Add fidelity-aware verification pilot

docs/collectivespec-p2-gate-20260713.md

@@ -0,0 +1,154 @@
+# CollectiveSpec P2：logical-plan 对照的审计与停止门槛
+
+## 决策
+
+**不启动正式的 P1/P2 SLO-goodput sweep，也不把 `compact-vs-padded` 作为
+CollectiveSpec 的研究主线。**
+
+原因不是这条机制一定没有工程收益，而是它的核心研究主张已经无法排除公开工作的
+覆盖：
+
+- [DSpark](https://arxiv.org/html/2607.05147) 已明确采用每请求的动态 verification
+  length，并把逻辑 sequence tracking 与物理 execution 解耦、flatten variable-length
+  token；
+- SGLang 的 [DSpark 集成说明](https://www.lmsys.org/blog/2026-07-06-dspark-sglang/)
+  已公开 `static`、`compact` 与 `cap-accept` 三种 verify mode。其中 `cap-accept`
+  执行完整 block、但只提交 compact window，且说明其输出与 `compact` 相同。这正是
+  “同一语义下 full/padded 与 compact”的 counterfactual；
+- 该实现还公开了 DP attention 下各 rank 使用最大 graph tier 的处理。因此，仅在
+  vLLM/H20 上再复现 compact 比 padded 快，只是环境复现，不是新的系统贡献。
+
+P0 还独立否定了原来的 liveness 动机：目标 runtime 已经以 scalar DP metadata 协调
+不同 DP replica 的物理 shape；异构 verifier candidate 没有引起运行期 collective
+错误。故不能再把“必须新增 canonical header 才能避免死锁”作为论文 premise。
+
+## 术语：什么必须相同
+
+### Logical plan（也称 semantic plan）
+
+logical plan 是一次 speculative verification **应当计算和提交什么**的不可变记录；它
+不包含 padding、CUDA graph tier、物理 rank 行数、worker PID 或耗时。每一个 verifier
+epoch 的最小条目为：
+
+```text
+(global_epoch, dp_rank, ordered request id,
+ logical_output_offset_before, scheduled_seq_len,
+ available_candidate_token_ids, requested_k, effective_k,
+ visible_candidate_token_ids_hash)
+```
+
+请求层还必须固定 `client_request_id`、server request id、prompt/body hash、arrival、DP
+assignment、提交顺序、temperature/seed 与预期 completion length。最终还要逐请求验证
+output token-id hash、completion length、finish reason、usage，以及 endpoint semantic
+transcript hash（content/reasoning/tool-call 的 canonical JSON）。
+
+这里的关键是 `k_i` 的 key 必须是
+`(server_request_id, logical_output_offset_before)`，而不是只有 request id：同一请求会
+经历多个 verification epoch。只有两个 cell 的这些事实都相同，才称为 *same logical
+plan*。
+
+### Compact vs. padded：只是同一 plan 的两个 lowering
+
+给定同一组 non-dummy logical entries：
+
+- **PaddedSync-semantic**：保留这些 entries，但为同步域插入 masked dummy rows，使各
+  DP peer 的 physical shape 对齐；
+- **CompactSync**：保留完全相同的 entries、candidate 和 commit semantics，用 ragged
+  packing / split vector 执行真实 rows，不计算 dummy rows。
+
+因此 `static K=3` 不能当作 padded 对照：它改变了每个请求可见 candidate prefix，改变了
+logical algorithm，而不是只改变 physical lowering。真实 physical-row 公式也不能简单
+写成 `N * (1 + max k_i)`；普通 decode、TP alignment 和 CUDA-graph alignment 都要从
+runner 的 row map 分开计数。
+
+## 当前 P0 对 P2 的限制
+
+P0 heterogeneous policy 按 vLLM **随机生成的 server request id** 哈希，而 client 没有
+发送 `X-Request-Id`。所以即使 trace 和 seed 相同，两个 cell 的每个 request/epoch 的
+`k_i` 也不可保证相同；日志只有 aggregate digest/histogram，也没有 per-row candidate
+token、assignment 或最终 token-id hash。P0 因而不能充当 same-logical-plan 的 P2 A/B。
+
+P0 的 padding 上界也已校正。66 个 target epoch 中 62 个 raw DP counts 不等：
+
+```text
+raw logical rows                 6,276
+local non-DP-aligned rows        6,536
+PaddedSync physical rows         7,024
+DP-global-max attributable rows    488  (= 6.95% of physical rows)
+```
+
+此前的 748 / 10.65% 将 260 行本地 TP/CUDA-graph alignment 混入 DP max padding。即使
+488 行都可回收，它仍只是 **target verifier row-count 的上界**：EAGLE3 仍按 Kmax=3
+完成 drafter 工作，也没有测得 EP bytes、collective critical path 或 E2E SLO-goodput。
+
+## 若未来重新打开，先补齐的测量契约
+
+不应先实现 compact lowering。先添加只用于 audit 的 telemetry：
+
+1. client 对每个请求发送固定 `X-Request-Id`、`X-Data-Parallel-Rank` 和
+   `return_token_ids=true`；记录 response id、prompt/output token-id hash、semantic
+   transcript hash 和 finish reason；
+2. scheduler 作为 semantic ledger 的唯一 writer，记录 per-epoch ordered entries、候选
+   token、requested/effective K、logical cursor 与 sampled token hash；
+3. worker 只记录物理事实：每 rank physical rows、DP/TP/graph padding 的原因、packed row
+   map；若要主张 EP 收益，额外记录 DeepEP all-to-all split vector、bytes、duration 和
+   rank wait；
+4. 汇总器先给出第一个 semantic/output diff，任一 mismatch 即标为 invalid，禁止读取
+   性能数字。
+
+最小 reproducibility smoke（仅在发现 topology gap 后执行）是 fresh engine 上的 16 个
+decode-only requests、每个 64 output tokens、temperature=0、DP0/DP1 各 8 个、显式
+`{0,3}` alternating manifest。先连续跑两次 **同一个** padded cell；只有 ledger 和逐请求
+token hash 全等，才允许运行 padded/compact mechanism probe。`return_token_ids` 会改变 SSE
+负担，故最终 latency cell 必须关掉该字段、改以 scheduler-side hash 审计。
+
+## 唯一尚可证伪的拓扑假设
+
+公开材料没有证明、也没有否定下列特殊情形：**独立 standard-DP scheduler 共享同一个 EP
+all-to-all domain** 时，DP global-max graph tier 之外仍有 EP split-vector / collective
+ordinal 的关键路径浪费。这不能从“论文没有写”推断为新颖性。
+
+只有一次短的 topology reconnaissance 观测到该额外瓶颈，才重新进行文献审计并考虑下列
+顺序严格的 gates：
+
+1. 与 SGLang-style DP global-max tier / current runtime 相比，compact plan 降低实际 EP
+   bytes、split imbalance 或 collective critical-path time；仅少几个 rows 不够；
+2. 在相同 semantic plan、token-exact 输出和无 tail-latency 退化下，至少三次 fresh-engine
+   paired runs 显示 E2E SLO-goodput 增益 >=10%；
+3. topology ablation 支持因果归因：DP=1 或 EP 不跨 DP 时收益消失或显著缩小，而
+   DP×shared-EP 时出现；
+4. 重新完成与 DSpark/SGLang 的逐项差异审计，证明贡献是 topology-aware collective
+   scheduling，而不是已有 ragged packing。
+
+任一 gate 不成立即结束 CollectiveSpec；不以 controller/K/queue knob 调优替代证据。
+
+## 如果 gate 重开时的固定环境与 setup
+
+下列是 P0 实际使用、后续必须 provenance-pin 的环境，而不是当前已启动的实验：
+
+| 项目 | 固定值 |
+|---|---|
+| host / accelerator | `dash0`，8 × NVIDIA H20 |
+| target / draft | Qwen3-235B-A22B FP8；EAGLE3，Kmax=3 |
+| parallelism | TP=4，DP=2，EP=8；`VLLM_MOE_USE_DEEPEP=1` |
+| engine | dash0 live installed vLLM wheel；记录 wheel metadata、import path、launch command 与 commit；不能以本地 checkout API 代替 |
+| execution | `FULL_DECODE_ONLY` CUDA graphs、FP8 KV cache、block size 64、`max-num-batched-tokens=1024`、`max-num-seqs=192`、max model len 262144 |
+| workload | immutable materialized `thinking_w20260327_1000` 的 decode-only window；机制 smoke 使用固定 burst，E2E 使用完整、session-closure 状态明确的 trace |
+| reproducibility | fresh engine per cell、temperature=0、固定 seed、固定 request ids/DP assignment、prefix-cache state 从空开始、ABBA cell order |
+| SLO（若进入 E2E） | 预注册 TPOT <= 40 ms、pass rate >= 0.95；同时报告 completion success、p50/p95/p99、deadline failures 与 output equivalence |
+
+remote source 必须从 Git 同步到
+`/home/admin/cpfs/wjh/collectivespec-pilot/20260713T054328Z/source`，并记录运行时实际
+source revision；任何远端 job 启动前在 artifact 中写明 resolved command、模型/trace path、
+预计 GPU 时间和结果目录。
+
+## 审计数据健全性
+
+- 新增实验数：n=0；本文件不报告任何新的性能数字。
+- 已复核的 P0 target epochs：n=66；两个 DP rank 的 raw row values 共 n=132，
+  min=1、max=77、distinct=35；physical rows 则 n=66，min=4、max=80、distinct=14。
+  原始 JSONL 可复核，不以 aggregate 值伪造每 epoch 分布。
+- 已用 aggregate row totals：n=4，min=488，max=7,024，distinct=4；均为非负。校正后的
+  关系 `6,276 <= 6,536 <= 7,024` 成立，且 `7,024 - 6,536 = 488`。
+- 外部材料覆盖判断区分为“论文明确描述”“官方公开实现明确描述”和“未公开拓扑细节”；
+  未从缺失的 EP 细节推导新颖性或性能收益。

docs/collectivespec-pilot-design-20260713.md

@@ -213,3 +213,107 @@ components；414 条 non-root edge 中 30 条（7.25%）父节点落在窗口外
 这只能称为 **window-session-closed**，不等于 full-session coherent：任何结果都必须报告
 这 30 条 boundary-parent residual，且不能据此声称跨窗口 KV reuse。若原始 span 恢复，必须
 重新从完整 source resolve root 与重新采样，不能沿用 fallback 的 score/threshold。
+
+## 2026-07-13 P0 v2：header/liveness premise 的实际结果
+
+### 先报异常
+
+两个 cell 都在 probe/result 已落盘、64 个请求都已完成后出现 teardown 异常。heterogeneous
+cell 有 `free(): corrupted unsorted chunks` 与共享资源泄漏；control 还出现 SIGTERM/SIG11
+和 TCPStore broken pipe。这些不是运行期 request/collective failure，但也意味着本实验**不
+证明干净退出或部署鲁棒性**。本节只使用完成前的 request、worker phase 与 DP metadata
+作为 P0 evidence；不报告任何 TPOT/QPS 比较。
+
+### 设计、判定与修正后的观测范围
+
+在 dash0 的 Qwen3-235B-A22B FP8 + EAGLE3、TP=4/DP=2/EP=8、DeepEP 配置上，P0 将 EAGLE
+已生成的 Kmax=3 candidates 按预先给定的 request-static 表截断为 `k_i ∈ {0,1,2,3}`。它只
+改变 verifier 可见 candidate，不消除 EAGLE 的 Kmax drafter 工作。
+
+原始 worker hook 还会记录 vLLM 的 profile/DP dummy run；空 `SchedulerOutput` 仍可能有
+physical rows。因此真实 target batch 的判据固定为：
+
+```text
+event == batch_execution_plan
+AND request_count > 0
+AND total_scheduled_rows > 0
+```
+
+最初 summary 将 676/640 条 dummy/profile record 混入 target phase，错误地把 control 的
+145/146 internal-call 差异解释为 rank mismatch。修正后的汇总只比较 target event，并将
+真实 DP pair 识别为 `[0,4]`、`[1,5]`、`[2,6]`、`[3,7]`；同一 logical DP replica 内的 TP
+group 则为 `[0,1,2,3]` 与 `[4,5,6,7]`。
+
+### 结果
+
+远端可复核 artifact：
+`/home/admin/cpfs/wjh/collectivespec-pilot/20260713T054328Z/p0_phase_v2_20260713T0944Z`
+（run source `bb698b5`；`summary.json`、`summary.md`、`driver_result.json` 和原始
+`p0_logs/*.jsonl` 均在该目录）。
+
+| cell | completion | 实际 candidate K | target worker records | target plan / DP coordination |
+|---|---:|---|---:|---|
+| control K=3 | 64/64，usage 均为 64 | `{3}` | 488；DP0 每 TP peer 65，DP1 每 peer 57 | 每个 logical DP replica 内序列完全一致；四个真实 DP pair 的 57 个 shared target epoch 的 scalar coordination signature 一致 |
+| heterogeneous | 64/64，usage 均为 64 | `{0,1,2,3}` | 528；8 个 peer 各 66 | 两个 logical DP replica 内序列完全一致；四个真实 DP pair 的 66/66 shared target epoch signature 一致 |
+
+heterogeneous 的 `candidate_truncate` 直方图为 `{0: 1408, 1: 670, 2: 672, 3: 398}`，而截断前
+全部为 K=3（共 3,148 个 candidate）。所以它不是只改变 log 的“伪异构”实验。两 cell 的
+target record 都满足：
+
+```text
+num_tokens_per_rank[dp_rank] == total_scheduled_rows
+physical_batch_rows == rows_across_dp[dp_rank]
+rows_across_dp[i] >= num_tokens_per_rank[i]
+len(rows_across_dp) == len(num_tokens_per_rank) == 2
+```
+
+这说明两个独立 scheduler 的 logical plan 可以不同，但 live runtime 已用 scalar DP metadata
+协调共同 physical shape，并让共享 EP domain 的真实请求完成；没有观察到运行期 deadlock 或
+collective error。它反驳的是“异构 verifier-side K 必须新增 canonical header 才能先保证
+liveness”的必要性，而不是一般性的形式化证明。
+
+### 留下的物理现象，以及为什么它仍不足以继续造系统
+
+heterogeneous 的 66 个 shared target epoch 中有 62 个的 raw DP token counts 不相等；现有
+runtime 将 `rows_across_dp` 同步为共同 shape。按每个 DP replica 的一个 TP anchor 计，
+target-only raw logical rows 为 6,276；逐 epoch 保留当前 TP/CUDA-graph local alignment 后为
+6,536；最终 physical rows 为 7,024。因此可单独归因给跨 DP global-max padding 的只有
+488 rows（6.95% physical rows）。此前用 7,024-6,276 得到的 748（10.65%）还混入了 260
+行本地 alignment，不能当作 compact 对照可回收的 DP work。control 的 row totals 也不与
+heterogeneous 直接比较，因为 logical plan 和 scheduler trajectory 不同。
+
+这只能看作 **P2 的 row-count upper bound**，绝不能把 control 与 heterogeneous 相减当作
+速度收益：两者 logical plan、scheduler trajectory 都不同。更关键的是 EAGLE drafter 仍完成
+Kmax 工作；即使理想 compact verifier 回收全部 6.95% 的 DP-only target rows，端到端
+SLO-goodput 增益也只会更小。
+
+因此决策为：
+
+1. **停止**把 canonical plan header / deadlock avoidance 当作 CollectiveSpec 的研究主线；
+   P0 已在目标部署上否定其必要前提。
+2. **停止**把“dynamic verification length + flattened ragged execution”本身当贡献；DSpark
+   已覆盖该组合，且它也指出固定长度 drafter 的前置工作不会因 verifier 截断自动消失。
+3. 仅保留一个很窄的、默认 no-go 的机会：同一 logical plan 下的 compact-vs-padded physical
+   execution。只有先以 P1 证明相对 best-static/global-sync 至少 10% E2E SLO-goodput，再以
+   P2 的因果对照证明关键路径 rows/bytes 真正下降，并完成 DSpark topology gap 审计，才值得
+   再投入实现。当前 P0 不满足这些条件。
+
+P0 未验证 greedy token-exact 输出、真实 DeepEP dispatch ordinal/split digest、取消/empty-rank
+stress，或其他模型/后端/更大 K 的泛化；这些都不能从本结果外推。
+
+### P0 data sanity
+
+- **teardown red flag 已单列**：control/heterogeneous 都在 completion 后发生 allocator/资源
+  清理异常；因此没有使用延迟、吞吐或 clean-shutdown 指标作结论。
+- n=2 cells，8 worker/cell，64 usage-verified completions/cell；completion count 的
+  min=max=64，distinct=1。
+- target worker record count：control=488、heterogeneous=528（min=488，max=528，distinct=2）；
+  dummy/profile records 分别为 676/640，已排除。
+- heterogeneous K：min=0，max=3，distinct=4；control K distinct=1。所有计数、rows 和
+  padding 非负；JSON parse errors=0。
+- heterogeneous 的 66 target epochs：两个 DP rank 的 raw row values 共 n=132，min=1、
+  max=77、distinct=35；physical rows n=66，min=4、max=80、distinct=14；分解
+  `6,276 <= 6,536 <= 7,024` 与 `7,024 - 6,536 = 488` 均成立。
+- 修正后的不变量均为 true：probe integrity、8 workers observed、每个 DP replica 内 target
+  phase/sequence 一致、target DP metadata 合法、DP coordination record 存在、四个真实 DP pair
+  的 shared scalar coordination signature 一致。

docs/fidelity-aware-harness-headroom-20260714.md

@@ -0,0 +1,95 @@
+# Fidelity-aware harness headroom audit
+
+Status: **PROMISING PREMISE, NO CONTRIBUTION CLAIM**.
+
+The audit answers whether engine instrumentation has enough incremental signal
+to justify a prospective experiment.  It does not establish generalization.
+
+## Simulator shortlist lower bound
+
+On the frozen 12-cell SimFid task, the strongest calibrated SLO simulator
+reading places TP2/MNS32 and TP2/MNS64 in the same first tie bucket.  Real-final
+evaluation of that two-cell bucket selects TP2/MNS32 and has zero real regret.
+A method requiring a real calibration probe plus final verification cannot beat
+two real cell evaluations on this task.  Therefore “better initial selection”
+is not a viable claim here; the remaining headroom is shorter real verification
+inside the same shortlist.
+
+## Five-second prefix result
+
+The retrospective Phase-6 dataset contains 37 primary anchors across 12 cells.
+Stable labels use the frozen same-placement 2-of-3 adjudication: 28 feasible and
+9 infeasible.  Three TP4 primary measurements disagree with their repeated
+labels, so single-run feasibility is not treated as ground truth.
+
+Using leave-one-cell-out folds, identical L2 logistic models, and a 5-second
+prefix:
+
+| Metric | Outcome-only | Instrumentation-aware | Delta |
+|---|---:|---:|---:|
+| Accuracy | 78.38% | 89.19% | +10.81 pp |
+| Balanced accuracy | 70.63% | 81.55% | +10.92 pp |
+| Brier score | 0.1297 | 0.0901 | -0.0396 |
+| Correct only in this model | 0 | 4 | +4 |
+| McNemar exact two-sided p | — | 0.125 | not significant |
+
+At the frozen conservative threshold 0.95, both policies make zero false
+accepts and zero false rejects on this retrospective set.  Outcome-only safely
+cuts 36.35% of measured primary-trial cost; instrumentation-aware safely cuts
+61.10%, an additional 24.75 percentage points.  Regularization sensitivity for
+accuracy delta is `[0.00, +10.81]` percentage points, so the sign is
+non-negative but the magnitude is not stable.
+
+Longer prefixes do not strengthen the case monotonically.  At 10 seconds,
+headline accuracy is 91.89% outcome-only versus 89.19% instrumentation-aware;
+at 15 seconds it is 88.89% versus 91.67%; at 20 seconds it is 86.11% versus
+91.67%, but both 0.95 policies make one false reject.  Five seconds is therefore
+a training-selected operating point, not a test result.
+
+## Interpretation
+
+There is enough headroom to run a held-out pilot, but not enough evidence to
+claim the harness contribution:
+
+- the 5-second cost gap is operationally large;
+- only four paired classifications differ, so significance is absent;
+- all examples share one workload/SLO/engine task;
+- completion timestamps are reconstructed from arrival + TTFT + TPOT rather
+  than recorded directly;
+- three adjudication disagreements are concentrated in transient TP4 runs;
+- outcome-only already recovers the simulator shortlist oracle with very few
+  real cells.
+
+The next experiment must therefore freeze the 5-second model and threshold,
+record exact monotonic completions, use a held-out trace, and label each anchor
+with three full repetitions.  The registered protocol is
+`docs/fidelity-aware-harness-protocol-20260714.md`.
+
+## Artifacts
+
+- `runs/fidelity-headroom/analyze_existing.py`
+- `runs/fidelity-headroom/metrics.json`
+- `runs/fidelity-headroom/analyze_prefixes.py`
+- `runs/fidelity-headroom/prefix-metrics.json`
+- `runs/fidelity-headroom/test_analysis.py`
+- `runs/fidelity-headroom/test_prefix_analysis.py`
+
+## Sanity block
+
+| Family | n | Min | Max | Distinct | Invariant/result |
+|---|---:|---:|---:|---:|---|
+| Real SimFid cell scores | 12 | 1.2833 | 3.2833 | 7 | Non-negative; not identical |
+| Prefix examples at 5 s | 37 | 5 s | 5 s | 1 expected | All 12 cells represented |
+| Adjudicated labels | 37 | 0 | 1 | 2 | 28 positive / 9 negative |
+| Primary/adjudicated disagreement | 37 | 0 | 1 | 2 | 3 TP4 disagreements retained |
+| Full primary elapsed time | 37 | 14.566 s | 62.064 s | 37 | Every 5 s prefix is in range |
+| Outcome probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
+| Instrumentation probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
+| Layer-1 streams | 12 | 14,174 records | 58,725 records | 12 | Contiguous, zero drops |
+
+Checked invariants: same folds/model family and cutoff; no full verdict in a
+feature; prefix-only Layer-1 slicing; non-negative costs/counters; bounded
+ratios/probabilities; both labels present; per-config results not identical;
+tie expansion before top-k; no imputation of non-monotonic frontiers.  The main
+limitation is reconstructed request completion time, explicitly marked on all
+37 five-second examples.

docs/fidelity-aware-harness-protocol-20260714.md

@@ -0,0 +1,193 @@
+# Fidelity-aware real-verification harness protocol
+
+Status: **PRE-REGISTERED STAGED EVALUATION; CONTRIBUTION NOT YET ESTABLISHED**.
+
+Date frozen: 2026-07-14 (Asia/Singapore).
+
+## Research question and contribution bar
+
+The harness has an independent systems contribution only if engine-internal
+instrumentation improves a tuning decision beyond what is already achievable
+with a simulator shortlist and external benchmark outcomes.  The intended
+claim is therefore deliberately stronger than “telemetry explains a run”:
+
+> Given the same simulator ranking, the same candidate order, and the same
+> short real-GPU probe, a learned instrumentation-aware verifier reaches a
+> configuration with at most 5% real SLO-goodput regret using materially fewer
+> H20-hours than both (a) simulator top-k followed by full real evaluation and
+> (b) an outcome-only verifier given exactly the same probe.
+
+The paper-facing gate is:
+
+- at least 20% lower real-verification H20-hours than outcome-only calibration;
+- at least 30% lower real-verification H20-hours than simulator top-k plus full
+  real final evaluation;
+- paired 95% task-bootstrap confidence interval for the outcome-only cost
+  reduction strictly above zero;
+- selected-configuration SLO-goodput regret at most 5% on every headline task;
+- no false-safe early accept in the pilot and at most 1% in the expanded suite;
+- profiling, warm-up, confirmation, instrumentation, and failed-run costs are
+  included rather than amortized away.  An amortized profile-cost view may be
+  reported only as a secondary result.
+
+If these conditions fail, instrumentation remains a debugging facility.  It is
+not an independent tuning-harness contribution.
+
+## What is learned, and what is not a rule
+
+The decision target is a stable, repeated real verdict, not a hand-authored
+diagnosis such as “queue length above N means reject.”  Each anchor receives
+three full real repetitions and a frozen 2-of-3 feasibility label.  A nested
+pair of regularized models predicts that label from a fixed prefix:
+
+- **Outcome-only input X:** configuration, offered rate, admitted/completed
+  progress, observed TTFT/TPOT margins, failures, and known workload lengths.
+- **Instrumentation input Z:** the same X plus generic engine state: running and
+  waiting queues, decode-batch shape, KV usage, graph mode and padding, prefill
+  share, preemptions, and model-step rate.
+
+Both models use the same L2 logistic family, train split, standardization,
+regularization, cutoff, and probability threshold.  The only experimental
+difference is Z.  The initial family is intentionally simple: a positive result
+then demonstrates value in the engine signal rather than capacity in a larger
+learner.  A sequence model is admissible only as a later, paired ablation.
+
+The frozen first policy uses a 5-second prefix, L2 regularization 1.0, and a
+two-sided abstaining threshold of 0.95: accept at `p(feasible)>=0.95`, reject at
+`p(feasible)<=0.05`, otherwise continue the exact same trial to completion.
+Threshold and cutoff were selected on the historical training task and are
+therefore not evidence; all claims come from subsequent held-out tasks.
+
+## Fair baselines
+
+| Method | Simulator | 5-second real prefix | External outcomes | Engine state | Full real continuation |
+|---|---:|---:|---:|---:|---:|
+| Real-only oracle | no | no | full | optional diagnostic | every candidate/anchor |
+| Sim top-k + real final | yes | included in full run | full | no decision use | every shortlisted candidate/anchor |
+| Outcome-only calibration | yes | yes | yes | no | only on abstention |
+| Instrumentation-aware | yes | yes | yes | yes | only on abstention |
+
+Tie buckets are expanded before top-k.  `k` is selected on training tasks and
+is fixed on held-out tasks; an oracle per-task k is forbidden.  Outcome-only
+receives all information available outside the engine, including config and
+workload features.  Instrumentation cannot use any record submitted after the
+cutoff.  The full label, confirmation votes, simulator error, and later
+requests are never model features.
+
+## Staged experiment
+
+### R0: historical premise and headroom audit
+
+The frozen SimFid surface has 12 cells.  The strongest calibrated SLO simulator
+reading has a top tie bucket `{TP2/MNS32, TP2/MNS64}`; full real evaluation of
+those two cells already finds the oracle with zero regret.  Consequently this
+single task cannot demonstrate a selection-count advantage: any method needing
+one real calibration probe and one real final verification has a lower bound of
+two real cells.
+
+The viable estimand is instead the duration and number of full real frontier
+evaluations inside a fixed shortlist.  Historical Phase-6 prefixes are analyzed
+only as training/premise data.  Their request completion times are reconstructed
+from arrival, TTFT, TPOT, and token count, so they cannot support a final claim.
+
+### P1: exact-timestamp prospective pilot
+
+- Engine/model/hardware: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B, one solo
+  server/client on dash0, NVIDIA H20, `TP in {1,2,4}`.
+- Held-out workload: `chat_w20260312_1000`, 60-second replay after the frozen
+  0.1 time scale, raw input `[0,8192]`, exactly 128 output tokens.
+- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, 95% request pass rate.
+- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
+- Per cell: one attainable low offered rate near 0.85x the historical v0.24
+  frontier and one high rate near 1.25x.  The exact threshold and selected
+  request hashes are frozen by a CPU preflight before launch.
+- Each cell uses a fresh server, the accepted long-request warm-up, one
+  unmeasured full-window burn-in, then three repetitions per rate.  Rate order
+  alternates and reverses across cells to prevent a fixed warm-state/order
+  confound.
+- The first repetition supplies the exact prefix.  All three repetitions supply
+  the 2-of-3 label.  Every request records a monotonic completion timestamp;
+  Layer-1 records are cut at the same monotonic boundary.
+- Placement is serialized.  Co-location is forbidden because Phase 6 observed
+  up to 92.86 percentage-point pass-rate shifts under co-location.
+- Hard cap: 3.5 H20-hours, including startup, warm-up, burn-in, all repetitions,
+  failures, and cleanup.  Projected cap violation stops before the next cell.
+
+P1 opens P2 only if all data invariants pass and instrumentation-aware has zero
+false accept/reject, is no worse than outcome-only, and either makes at least
+three additional correct early decisions or improves total valid trial-cost
+reduction by at least 15 absolute percentage points.  The pilot is a gate, not
+paper evidence.
+
+### P2: held-out task replication
+
+If P1 passes, freeze the model and run at least six independent task groups:
+three trace windows spanning distinct date/slot combinations and two SLO
+regimes.  No task used for threshold/model selection enters the headline test.
+The candidate surface is the full 12-cell `TP={1,2,4} x MNS={8,16,32,64}`
+surface.  Splits are by complete task, never by anchor or request.  A task-level
+paired bootstrap (10,000 repetitions, fixed seed) estimates cost and regret
+intervals.  Non-monotonic or split 2-of-3 anchors remain explicit; no frontier
+is imputed.
+
+### P3: end-to-end shortlist and search replay
+
+For each P2 task, run the same frozen simulator and tie-expanded top-k policy.
+Replay the real binary/frontier search under all three verification policies:
+full real, outcome-only, and instrumentation-aware.  The policy consumes only
+prefixes that would have been available at that decision point.  Report:
+
+- selected cell and real SLO-goodput regret;
+- number of real cells, anchors, and confirmations;
+- measured H20-hours and wall time;
+- false accept, false reject, and abstention counts;
+- profile, startup/warm-up, probe, full-continuation, confirmation, logging, and
+  failure cost breakdowns.
+
+### P4: simulator-rank-error attribution
+
+This phase distinguishes an outdated implementation/profile from a structural
+simulator limitation.  For each held-out task compare:
+
+1. the original simulator/profile;
+2. a version-matched re-profiled simulator;
+3. a trajectory-conditioned run supplied with the realized arrival and request
+   length sequence;
+4. outcome-only residual calibration;
+5. instrumentation-aware residual calibration.
+
+The engine trace is extended only as needed with a worker-level step UID and
+CUDA-event duration, because current async submit-to-complete spans overlap and
+are not GPU step time.  Residuals are decomposed into operator-profile error,
+scheduler/state error, and run-to-run noise.  If re-profiling alone restores the
+ranking, the old 30% loss was an implementation/profile defect.  If exact
+profiles and realized trajectories still mis-rank cells, and the residual is
+systematically explained by queue/KV/graph/batch state unavailable to the
+simulator, that is evidence of a structural state-abstraction gap.  Correlation
+alone is not called causal.
+
+## Failure modes that reject the route
+
+- Outcome-only matches or beats instrumentation-aware under the same cutoff.
+- Instrumentation gains average accuracy but introduces false-safe decisions.
+- Gains disappear under task-level rather than request/anchor-level splitting.
+- Savings come only from excluding startup, warm-up, profiling, confirmations,
+  or failed trials.
+- A different cutoff/threshold must be selected after seeing each test task.
+- The simulator top-k baseline already reaches the target with equal or lower
+  total H20-hours.
+- Exact instrumentation overhead exceeds 1% throughput or materially changes
+  p95/p99 latency.
+- Results depend on TP4 transient/non-monotonic trials and do not replicate on
+  held-out tasks.
+
+## Data sanity contract
+
+Every analysis ends with n, min/max, distinct count, label balance, and these
+invariants: non-negative counters/costs; probabilities and ratios in `[0,1]`;
+per-config results not all identical; timestamps monotonic; every prefix record
+at or before its cutoff; selected request ID/arrival/length hashes stable across
+repetitions; exact 128-token completion or counted failure; no dropped Layer-1
+records; 2-of-3 labels reproducible; no co-resident GPU process; total H20-hours
+below the hard cap; final GPUs idle.  A red flag is reported first and blocks
+the contribution claim.

docs/opprof/oracle-gap-protocol.md

@@ -0,0 +1,240 @@
+# Static-policy oracle-gap protocol
+
+Status: **FROZEN WITH A-OG-1 THROUGH A-OG-4 AMENDMENTS**.
+
+Date frozen: 2026-07-13 (Asia/Singapore). Existing Phase-3 measurements were
+inspected only to choose the workload pair and rate brackets. They are
+exploratory calibration data, not primary observations in this protocol.
+
+### A-OG-4 — close a majority-shifted boundary (before confirmation scores)
+
+After all four primary frontiers and 70 scores were complete, the controller
+was interrupted before the first confirmation produced a score. The partial
+`P01-C10-r26-rep1` attempt contained no result, sanity file, or score and is
+archived separately. This amendment is therefore blind to confirmation
+outcomes.
+
+The original confirmation schedule still runs first. Afterward, each cell's
+majority-vote frontier is recomputed. If either side of the *actual* final
+boundary has fewer than three trials because the provisional boundary moved,
+the controller runs only enough repetitions at that existing rate to reach
+three, high-to-low, on a fresh server for that config. It then recomputes the
+boundary and repeats for at most three closure rounds. No new rate anchor may
+be added. Failure to obtain a monotone, bracketed boundary with three trials
+per side within three rounds or the 6 H20-hour cap makes the experiment
+inconclusive.
+
+This fills a stopping-rule omission: it does not change any existing score,
+majority rule, phase, config, rate grid, SLO, or oracle calculation.
+
+### A-OG-3 — freeze a per-logical-trial transport retry rule (after 62 scores)
+
+The first attempt at `P06-C10-r1.9-rep0` reproduced the same isolated local
+transport signature seen in A-OG-2: one clean request raised
+`ClientOSError` 3.56 ms after admission, with HTTP status 0, no first token,
+and no output. The other 343 requests produced exactly 512 tokens, the server
+stayed healthy, drain took 4.00 s, and every other client invariant passed.
+The controller again rejected the attempt before creating a `score.json`.
+
+For the remainder of this experiment, an attempt may be quarantined and the
+identical logical trial retried once on a fresh server only if all of the
+following hold: the sole failed client invariant is `clean_failures_zero`;
+exactly one clean request failed; its error is `ClientOSError`, HTTP status is
+0, it produced neither a first token nor output, and it completed within 10
+ms of admission; every successful request has exact output; and the server
+has no crash or error. The attempt is never scored. A second invalid attempt
+for the same logical key, more than one failed request, or any other error
+signature is a stop condition and makes the experiment inconclusive.
+
+This rule is independent of config, rate, and observed performance, and
+supersedes the one-key wording in A-OG-2 without changing how that retry was
+executed. Existing scores remain immutable. No client, grid, policy,
+threshold, order, SLO, or oracle calculation changes.
+
+### A-OG-2 — retry one transport-invalid attempt (after 49 scored trials)
+
+The first attempt at `P06-C01-r2.2-rep0` produced one local
+`ClientOSError` 3.27 ms after admission, with HTTP status 0, no first token,
+and no output. The other 399 requests completed successfully, every successful
+request produced exactly 512 tokens, the server stayed healthy, and all other
+client invariants passed. The controller rejected the attempt before creating
+a `score.json`; no TTFT/TPOT result from this attempt was used to choose the
+recovery rule.
+
+This is a measurement-transport failure, not an observed server SLO outcome.
+The entire attempt is content-hashed and moved under `invalid-attempts/`, then
+the identical logical trial (phase, config, rate, repetition, derived manifest,
+seeds, timeline, and SLO) is run once on a fresh C01 server. All 49 existing
+scores remain immutable. A second client transport failure in the retry is a
+stop condition and makes the experiment inconclusive; it must not be retried
+again. No grid, policy, threshold, order, or oracle calculation changes.
+
+### A-OG-1 — extend the P06 upper bracket (after trial 33)
+
+The controller stopped as registered after C00/P06 remained SLO-feasible at
+every original and upward-extension anchor through 2.3 requests/s. At that
+point 33 trials and 1.519475 H20-hours were complete, all GPU memory had been
+released, and no C00/P06 infeasible upper bound existed. No oracle inference
+was performed.
+
+This amendment changes only the P06 upward-extension list from
+`2.1,2.2,2.3` to `2.1,2.2,2.3,2.4,2.5,2.6,2.8,3.0`. The controller stops at
+the first bracket exactly as before. Existing trials are immutable and are
+reused; config order, P01 rates, SLO, timelines, repetitions, placement,
+metrics, and decision threshold do not change. The resumable controller may
+accept the new code/protocol fingerprint only when all immutable runtime,
+client, model, manifest, config, and base-grid fields match the pre-amendment
+state, and it records both fingerprints under `A-OG-1`.
+
+## Question and decision gate
+
+The candidate motivation is:
+
+> A single global static batching policy leaves at least 10% end-to-end
+> SLO-goodput on the table when serving temporally heterogeneous phases; a
+> phase-aware runtime policy can recover that gap without changing hardware,
+> model, precision, or tensor-parallel topology.
+
+This experiment tests a necessary condition in the existing TP1 policy space
+`{C00,C10,C01,C11}`. The optimistic oracle knows the phase and switches with
+zero delay, zero state-transfer cost, and no prediction error. If even this
+oracle cannot beat the best one-config-for-all-phases policy by 10%, an online
+controller over these MNS/MBT choices cannot do so either.
+
+The primary gate uses a conservative capacity bracket:
+
+- `L[p,c]`: highest offered rate accepted as SLO-feasible for phase `p` and
+  config `c`;
+- `U[p,c]`: lowest higher offered rate accepted as SLO-infeasible;
+- oracle upper bound at phase-time weights `w`:
+  `sum_p w[p] * max_c U[p,c]`;
+- best-static lower bound:
+  `max_c sum_p w[p] * L[p,c]`.
+
+We scan every P01/P06 time mixture, including pure endpoints. The current
+motivation is **REFUTED** if the maximum conservative ratio
+`oracle_upper / static_lower - 1` is below 10%. It is **NOT ESTABLISHED** if the
+bound crosses 10% but the observed point estimate does not. A positive result
+requires a point-estimate gap of at least 10% and then a separately
+pre-registered interleaved-trace validation; this frontier experiment alone
+cannot establish a positive E2E contribution.
+
+The conclusion is scoped to the measured MNS/MBT policy family and the chosen
+strongest-conflict phase pair. It does not rule out new scheduling mechanisms,
+KV-state policies, topology changes, or other workload phases.
+
+## Fixed system boundary
+
+| Item | Frozen value |
+|---|---|
+| Host | `dash0`, one run at a time on physical GPU0 |
+| GPU | NVIDIA H20; no other GPU process anywhere on the host |
+| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, BF16 |
+| Runtime | `/tmp/wjh-opprof-phase2-dash0-20260711/.venv`, vLLM `0.24.1.dev3+g668cfb7e2` |
+| vLLM source | `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0` |
+| Topology | TP1, one server, no data/pipeline parallelism |
+| Fixed mechanisms | chunked prefill on; prefix caching on |
+| Client | Phase-5 timestamp/fixed-rate wrapper over the Phase-3 exact-token client |
+| Seeds | workload `20260712`; trial token-domain seed derived only from phase/rate/repetition, never config |
+
+SLO co-location results in Phase 6 showed pass-rate flips despite small
+throughput deltas. Therefore unused H20s remain idle: parallel placement is not
+authoritative for this experiment.
+
+## Workloads and policies
+
+The pair is chosen before new measurements because Phase 3 showed the strongest
+opposing static preference:
+
+- **P01:** input `U[128,512]`, output exactly 64 tokens, deterministic steady
+  arrivals. C10 lost 24.27% saturation throughput relative to C00.
+- **P06:** 50/50 input mixture `U[128,512]`/`U[4096,8192]`, output exactly 512
+  tokens, deterministic bursts of eight. C10 gained 3.37% over C00.
+
+Both reuse the immutable 32,768-row Phase-3 manifests. For every trial a
+derived manifest preserves request order, lengths, outputs, and arrival class,
+but applies a trial-specific token-seed offset. The same derived manifest is
+used for all four configs. This prevents prefix-cache carry-over when a hot
+server executes several anchors without changing the logical workload.
+
+| Config | Effective MNS | Effective MBT | Extra flags |
+|---|---:|---:|---|
+| C00 | 1024 | 8192 | none |
+| C10 | 64 | 8192 | `--max-num-seqs 64` |
+| C01 | 1024 | 2048 | `--max-num-batched-tokens 2048` |
+| C11 | 64 | 2048 | both flags |
+
+Startup logs must confirm these values. A default drift is a stop condition.
+
+## Load grid, order, and repetitions
+
+Primary grids:
+
+- P01: `{26,28,30,32,34,36}` requests/s; execution order
+  `32,26,36,28,34,30`.
+- P06: `{1.4,1.5,1.6,1.7,1.8,1.9,2.0}` requests/s; execution order
+  `1.7,1.4,2.0,1.5,1.9,1.6,1.8`.
+
+Every primary anchor runs once. For each phase/config, the highest primary
+feasible anchor and its next higher primary anchor are then run two more times,
+giving three trials at both sides of the boundary. If all primary anchors are
+feasible, extend upward in the fixed order P01 `38,40,42` or P06
+`2.1,2.2,2.3,2.4,2.5,2.6,2.8,3.0`.
+If all are infeasible, extend downward in the fixed order P01 `24,22,20` or P06
+`1.3,1.2,1.1`. Stop extending at the first bracket.
+
+One primary server is launched per config in order `C11,C00,C01,C10`.
+Confirmation servers are fresh and launch in reverse order
+`C10,C01,C00,C11`; their boundary anchors run high-to-low. This balances
+machine-time drift and makes confirmation independent of the primary server's
+cache/compiler state.
+
+Timelines:
+
+- P01: 60 s warm-up + 60 s clean measurement; drain cap 120 s.
+- P06: 60 s warm-up + 120 s clean measurement; drain cap 240 s.
+- no Kineto profiling; exact greedy output with `ignore_eos`; maximum client
+  concurrency 256.
+
+A trial is SLO-feasible when at least 95% of requests admitted during the clean
+interval eventually finish successfully and individually satisfy both:
+
+- TTFT <= 2 s for input <= 4,096 tokens; <= 4 s for input <= 32,768; <= 6 s
+  otherwise;
+- TPOT <= 50 ms, computed as `(completion - first_token)/(output_tokens - 1)`.
+
+SLO-goodput is the number of those passing clean-admission requests divided by
+clean seconds. Client schedule lag must stay <=1 s and achieved clean offered
+rate must be within 5% of target. Failure of either condition makes the anchor
+infeasible; its admitted-only latency is not used to rescue it.
+
+At a repeated boundary, feasibility is the majority of three trial verdicts.
+All accepted anchor verdicts must be monotone in offered rate. A persistent
+non-monotone result after the registered repeats is a red flag and stops the
+oracle-gap inference.
+
+## Validity and stopping rules
+
+Before every server launch record host, GPU, driver, clocks, runtime package
+versions, git/source hashes, manifest hashes, exact commands, and process
+contamination. Stop on another GPU process, request/output mismatch, manifest
+drift, server crash, non-finite latency, ratio outside `[0,1]`, negative
+counter, or discontinuous/non-monotone accepted frontier.
+
+The controller is detached and resumable. It kills only process groups it
+created, checks zero GPU memory after every server, never overwrites a complete
+trial, and writes state atomically. The hard budget is 6 H20-hours; expected
+cost is 3.0--4.0 H20-hours and approximately the same wall time because runs
+are serialized.
+
+## Required report
+
+The report includes every trial's target/achieved rate, clean cohort size,
+pass count/rate, SLO-goodput, TTFT/TPOT percentiles, schedule lag, failure
+reasons, accepted frontier brackets, per-phase oracle choices, best static
+choice, equal-time gap, worst-mixture conservative gap, and GPU-hours.
+
+The final statistics section ends with a data-sanity block containing `n`,
+min/max, distinct-value counts, and checks for non-negative counters, ratios in
+`[0,1]`, non-identical per-config results, exact output work, monotone
+frontiers, and continuous rate brackets.

docs/opprof/oracle-gap-results.md

@@ -0,0 +1,199 @@
+# Static-policy oracle-gap results
+
+Status: **FINAL — REFUTED WITHIN THE FROZEN TP1 MNS/MBT POLICY SPACE**.
+
+Date: 2026-07-13. The registered experiment completed 104 valid trials on
+`dash0` and returned `REFUTED`. Even a phase-perfect oracle with zero detection,
+switching, and state-transfer cost is bounded below the registered 10%
+SLO-goodput contribution gate.
+
+The machine result is
+`runs/opprof-oracle-gap/metrics.json` (SHA-256
+`250ba4c1657a8830795ee06392eea4e21c62d958fea11701ba60581ef0266543`).
+All 104 trial-level measurements are in
+`runs/opprof-oracle-gap/trials.csv` (SHA-256
+`60cd901f18bbf107eb130f0095e867c3b6d47a78a42e83bcbe57fecf23cc5f9c`).
+The final resumable controller state is
+`runs/opprof-oracle-gap/controller-state.json` (SHA-256
+`27ec5e9a3cd32a871d583ba8eb6d7d3fe2a338a8fd42369233ada57cd4da6436`).
+
+## Decision
+
+The tested motivation was:
+
+> One static batching policy leaves at least 10% end-to-end SLO-goodput on the
+> table across temporally heterogeneous phases, and a phase-aware runtime can
+> recover it by switching MNS/MBT policies.
+
+The registered 10,001-point mixture scan finds a worst conservative gap of
+**8.333219%**, at P01 time weight `0.0244`. Equal phase time gives
+**7.260726%**. An independent implementation that evaluates every exact
+static-policy crossover gives a slightly more conservative exact maximum of
+**8.333333% = 1/12**, at P01 weight `1/41`. This leaves 1.666667 percentage
+points below the 10% gate.
+
+The negative result is stronger than a failed online prototype. The oracle
+already knows the current phase and pays no switching cost. A realizable
+controller over the same four policies cannot exceed this oracle bound.
+
+This is a scoped refutation, not a universal claim about adaptive serving. It
+rules out the current contribution based on phase-aware selection among the
+four MNS/MBT configurations for the frozen P01/P06 pair, Qwen3-30B-A3B, TP1,
+H20, vLLM 0.24, and the registered SLO. It does **not** rule out a new scheduler,
+KV/cache policy, routing-aware mechanism, topology change, or a different
+workload family.
+
+## Fixed setup
+
+| Item | Value |
+|---|---|
+| Placement | `dash0`, serialized on physical GPU0; GPUs 1-7 idle |
+| GPU | NVIDIA H20, driver 580.95.05 |
+| Model | Qwen3-30B-A3B, BF16 |
+| Runtime | vLLM `0.24.1.dev3+g668cfb7e2`, source `4b253fd8619764b6971a7f2e3a3aa7545f6ace05` |
+| Topology | TP1; one server and one client |
+| Fixed mechanisms | chunked prefill on; prefix caching on |
+| P01 | input `U[128,512]`, exactly 64 output tokens, deterministic steady arrivals |
+| P06 | 50/50 input `U[128,512]` / `U[4096,8192]`, exactly 512 output tokens, deterministic bursts of eight |
+| P01 timeline | 60 s warm-up + 60 s clean measurement |
+| P06 timeline | 60 s warm-up + 120 s clean measurement |
+| SLO feasibility | at least 95% of clean-admitted requests pass both TTFT and TPOT |
+| TTFT SLO | <=2 s for input <=4096; <=4 s for input <=32768; otherwise <=6 s |
+| TPOT SLO | <=50 ms/token |
+
+The four policies were:
+
+| Config | MNS | MBT |
+|---|---:|---:|
+| C00 | 1024 | 8192 |
+| C10 | 64 | 8192 |
+| C01 | 1024 | 2048 |
+| C11 | 64 | 2048 |
+
+## Final SLO frontiers
+
+`L` is the highest majority-feasible offered rate. `U` is the next higher
+majority-infeasible rate. Both sides below have at least three trials, and all
+accepted rate sequences are monotone.
+
+| Config | P01 `L` | P01 `U` | P06 `L` | P06 `U` |
+|---|---:|---:|---:|---:|
+| C00 | 28.0 | 30.0 | 2.3 | 2.4 |
+| C10 | 24.0 | 26.0 | 2.4 | 2.5 |
+| C01 | 28.0 | 30.0 | 2.2 | 2.3 |
+| C11 | 24.0 | 26.0 | 2.2 | 2.3 |
+
+The optimistic oracle upper bound chooses C00/C01 at `U=30` for P01 and C10
+at `U=2.5` for P06. The static lower bound chooses C10 below P01 weight `1/41`
+and C00 above it; they tie at the exact worst point.
+
+| Mixture | Oracle upper | Best-static lower | Conservative gap |
+|---|---:|---:|---:|
+| Pure P06 | 2.500000 | 2.400000 | 4.166667% |
+| Exact worst, P01 weight `1/41` | 3.170732 | 2.926829 | **8.333333%** |
+| Equal phase time | 16.250000 | 15.150000 | **7.260726%** |
+| Pure P01 | 30.000000 | 28.000000 | 7.142857% |
+
+The specialization exists but is too small. Reducing MNS from 1024 to 64
+raises the conservative P06 lower bound from 2.3 to 2.4 req/s, while lowering
+P01 from 28 to 24 req/s. Even after using infeasible `U` values for the oracle
+and feasible `L` values for the static baseline, the best possible phase-aware
+selection cannot reach 10%.
+
+## Robustness finding
+
+The strongest system finding is not config specialization but a repeat-level
+mode flip at P01/26 rps. Both C10 and C11 show the identical verdict sequence
+`rep0=0% pass`, `rep1=100% pass`, `rep2=0% pass`. For C10, TTFT p50 is
+3864.62, 252.71, and 3994.88 ms; for C11 it is 6533.62, 593.44, and
+5812.71 ms. TPOT remains below the 50 ms SLO in these trials.
+
+These trials are measurement-valid: exact outputs, offered-rate tolerance,
+schedule lag, timestamps, and client invariants all pass. The majority rule
+therefore classifies 26 rps as infeasible for both configs. The result does not
+change the oracle-gap decision because neither config supplies the P01 oracle
+maximum or the relevant best-static P01 lower bound.
+
+Token-domain seed and server execution history change together across
+repetitions, so this experiment cannot attribute the flip to MoE routing,
+cache/compiler state, or another source. It does motivate a narrower factorial
+study that crosses token seed with fresh/reused server state and randomizes
+order, while recording routed-expert and per-step scheduler telemetry. That is
+a mechanism question; it should not be presented as evidence for a phase-aware
+MNS/MBT controller.
+
+## Execution and audit history
+
+The 104 scored trials comprise 52 base-grid primaries, 18 registered upward or
+downward extensions, 32 boundary confirmations, and two boundary-closure
+trials. The final closure moved C00/P06 from the provisional `[2.4,2.5)` to
+`[2.3,2.4)` and repeated 2.3 rps to 3/3 feasible.
+
+Four frozen amendments are recorded in the protocol:
+
+- A-OG-1 extended only the P06 upward anchors after C00 remained feasible
+  through the original 2.3-rps limit.
+- A-OG-2 quarantined one P06/C01/2.2 local `ClientOSError` attempt and retried
+  the identical logical trial on a fresh server.
+- A-OG-3 generalized the same pre-score transport rule after one
+  P06/C10/1.9 attempt showed the identical signature.
+- A-OG-4, frozen before any confirmation score, added closure for a
+  majority-shifted boundary without adding new anchors.
+
+The two transport-invalid attempts and one pre-A-OG-4 interrupted attempt
+created no score and do not enter any metric. Their retained tree hashes are,
+respectively,
+`434863ba90513cbc54534ffbc1a13c980b3ef7d567190a0aa3f97b55650acbb2`,
+`a57b1ac5f090680bb70c16b5d709eb2b8ac47dce57c7a03b8077cd9b6d80d831`,
+and `bc3feb53514601e641ef1db204c74cffe3d282b24ad797e5b161468f5d15de5c`.
+
+Ten deliberately overloaded primary anchors exceed both the 1 s schedule-lag
+gate and the 5% offered-rate tolerance. They are correctly classified as
+infeasible and are not used as final boundary points. All 48 final boundary
+trials pass both client-side gates; their maximum schedule lag is 573.59 ms.
+
+The final experiment fingerprint uses repository commit `16177b0`, analyzer
+SHA-256
+`d86ecb1f077472906cbb729bd2c9d4b3a82ac6dfdc90838a17ae300d0767110d`,
+controller SHA-256
+`f7c5c2f74f2002f1e4b097e608d165a3a2e9374fbf07935a4d6d6a7c5d45a83a`,
+and protocol SHA-256
+`173f969a4428643cf6c4b950413aa82bef25cafd163e20d7944370a5d87af435`.
+The archived launch log SHA-256 is
+`54e2f4804a5efee072b670043c4092cf41f1a27664faa05ea71bfc1412c3e9db`.
+
+## GPU accounting and cleanup
+
+The campaign used **4.8877033845 H20-hours**, below the 6.0-hour cap, across
+5 h 41 min 51 s wall time including amendment, audit, restart, and server
+startup intervals. At completion, all eight H20s report 0 MiB, 0% utilization,
+and zero compute processes.
+
+## Sanity block
+
+There are no data-sanity red flags. The P01/26 all-or-none mode flip is a
+scientific robustness finding, not an invalid trial signature; it repeats in
+two configs and all measurement invariants pass.
+
+| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---|
+| Score-row indicator | 104 | 1 | 1 | 1 expected | 104 score files; no overwrite |
+| Target rate (req/s) | 104 | 1.4 | 36.0 | 19 | Non-negative; fixed grid/extensions only |
+| Clean cohort per trial | 104 | 168 | 2160 | 30 | Non-empty and non-negative |
+| Pass rate | 104 | 0.0 | 1.0 | 57 | All ratios in `[0,1]` |
+| SLO-goodput (req/s) | 104 | 0.0 | 27.816667 | 59 | Non-negative; per-cell results not all identical |
+| Boundary pass rate | 48 | 0.0 | 1.0 | 33 | Both final sides have three trials |
+| Boundary max schedule lag (ms) | 48 | 2.145861 | 573.593768 | 48 | All below the 1000 ms gate |
+| Exact-output indicator | 85,402 | 1 | 1 | 1 expected | Every clean request produced the requested token count |
+| Frontier cells | 8 | 8 | 8 | 1 expected | 8/8 bracketed and monotone |
+| Registered weight scan | 10,001 | 0.0 | 1.0 | 10,001 | Continuous 0.0001 increments, endpoints included |
+| Campaign H20-hours | 1 | 4.887703 | 4.887703 | 1 | Non-negative and below 6.0 |
+| Final GPU memory/utilization | 8 | 0 MiB / 0% | 0 MiB / 0% | 1 expected | Zero compute processes |
+
+Checked invariants: fixed model, runtime, manifests, SLOs, config values, seeds,
+and serialized placement; exact output work; nondecreasing timestamps;
+non-negative counters and latencies; pass ratios in range; offered-rate and
+schedule gates; majority-of-three final boundaries; monotone accepted
+frontiers; complete 10,001-point scan; independent exact-crossover
+recomputation; transport-attempt quarantine; GPU hard-cap compliance; and
+complete GPU cleanup.

runs/fidelity-headroom/analyze_existing.py

@@ -0,0 +1,508 @@
+#!/usr/bin/env python3
+"""Retrospective headroom audit for a fidelity-aware tuning harness.
+
+This analysis intentionally separates two questions:
+
+1. How many real cell evaluations does a simulator top-k shortlist already
+   need to recover the real optimum on the frozen SimFid surface?
+2. On the P6 anchor ladder, do Layer-1 engine features predict the next
+   anchor's feasibility better than outcome-only features from the same
+   current anchor?
+
+The second question is diagnostic rather than decision-bearing: it uses a
+small, already-observed single-workload surface and full current-anchor
+summaries.  It is a premise check for a future prospective early-probe study.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable
+
+import numpy as np
+
+
+SCHEMA = "fidelity-headroom-v1"
+DEFAULT_REGULARIZATION = 1.0
+REGULARIZATION_SENSITIVITY = (0.1, 1.0, 10.0)
+BOOTSTRAP_SEED = 20260714
+BOOTSTRAP_REPLICATES = 10_000
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def numeric(values: Iterable[float | int]) -> dict[str, Any]:
+    array = [float(value) for value in values]
+    return {
+        "n": len(array),
+        "min": min(array) if array else None,
+        "max": max(array) if array else None,
+        "distinct_n": len(set(array)),
+    }
+
+
+def score_buckets(scores: dict[str, float], tolerance: float) -> dict[str, int]:
+    if tolerance <= 0:
+        raise ValueError("score tolerance must be positive")
+    return {cell: math.floor(float(score) / tolerance) for cell, score in scores.items()}
+
+
+def topk_curve(
+    real_scores: dict[str, float],
+    simulated_scores: dict[str, float],
+    tolerance: float,
+) -> dict[str, Any]:
+    if set(real_scores) != set(simulated_scores):
+        raise ValueError("real and simulator score cells differ")
+    buckets = score_buckets(simulated_scores, tolerance)
+    ordered = sorted(
+        simulated_scores,
+        key=lambda cell: (-buckets[cell], -float(simulated_scores[cell]), cell),
+    )
+    real_best = max(float(value) for value in real_scores.values())
+    points = []
+    for nominal_k in range(1, len(ordered) + 1):
+        cutoff_bucket = buckets[ordered[nominal_k - 1]]
+        candidates = [cell for cell in ordered if buckets[cell] >= cutoff_bucket]
+        selected = max(candidates, key=lambda cell: (float(real_scores[cell]), cell))
+        selected_score = float(real_scores[selected])
+        points.append(
+            {
+                "nominal_k": nominal_k,
+                "expanded_k": len(candidates),
+                "candidates": candidates,
+                "selected_cell_after_real_final": selected,
+                "selected_real_score": selected_score,
+                "real_regret": 1.0 - selected_score / real_best,
+            }
+        )
+
+    minimum_k = {}
+    for name, threshold in (("zero", 1e-15), ("one_percent", 0.01), ("five_percent", 0.05)):
+        eligible = [point for point in points if point["real_regret"] <= threshold]
+        minimum_k[name] = (
+            {
+                "nominal_k": eligible[0]["nominal_k"],
+                "expanded_k": eligible[0]["expanded_k"],
+            }
+            if eligible
+            else None
+        )
+    return {
+        "real_best": real_best,
+        "minimum_k": minimum_k,
+        "points": points,
+    }
+
+
+@dataclass(frozen=True)
+class Transition:
+    cell: str
+    current_anchor: float
+    next_anchor: float
+    external: tuple[float, ...]
+    instrumentation: tuple[float, ...]
+    next_feasible: int
+
+
+EXTERNAL_FEATURES = (
+    "log_current_rate_per_gpu",
+    "log_next_over_current_rate",
+    "log2_tp",
+    "log2_mns",
+    "current_pass_rate",
+    "ttft_max_over_6s",
+    "tpot_max_over_50ms",
+    "exact_output_fraction",
+    "early_stopped",
+)
+
+INSTRUMENTATION_FEATURES = (
+    "waiting_mean",
+    "waiting_max",
+    "decode_batch_mean",
+    "decode_batch_cv",
+    "kv_usage_mean",
+    "kv_usage_max",
+    "graph_none_share",
+    "graph_full_share",
+    "padding_fraction",
+    "prefill_token_fraction",
+    "model_steps_per_second",
+)
+
+
+def _finite(value: float | int | None) -> float:
+    if value is None:
+        return 0.0
+    result = float(value)
+    if not math.isfinite(result):
+        raise ValueError(f"non-finite feature: {value}")
+    return result
+
+
+def build_transitions(phase6: dict[str, Any]) -> list[Transition]:
+    transitions = []
+    for cell, cell_result in sorted(phase6["cells"].items()):
+        anchors = sorted(cell_result["anchors"], key=lambda item: float(item["anchor"]))
+        for current, following in zip(anchors, anchors[1:]):
+            if following["accepted_feasible"] is None:
+                continue
+            primary = current["primary"]
+            next_primary = following["primary"]
+            layer = current["layer1"]
+            rate = float(primary["selection"]["offered_req_s_per_gpu"])
+            next_rate = float(next_primary["selection"]["offered_req_s_per_gpu"])
+            selected_count = int(primary["selection"]["count"])
+            if rate <= 0 or next_rate <= 0 or selected_count <= 0:
+                raise ValueError("rates and selected counts must be positive")
+            external = (
+                math.log(rate),
+                math.log(next_rate / rate),
+                math.log2(float(cell_result["tp"])),
+                math.log2(float(cell_result["mns"])),
+                float(primary["pass_rate"]),
+                _finite(primary["ttft_ms"]["max"]) / 6000.0,
+                _finite(primary["tpot_ms"]["max"]) / 50.0,
+                float(primary["exact_output_count"]) / selected_count,
+                float(bool(primary["early_stopped"])),
+            )
+            graph_shares = layer.get("graph_mode_shares", {})
+            prefill_tokens = _finite(layer["prefill_tokens"])
+            decode_tokens = _finite(layer["decode_tokens"])
+            instrumentation = (
+                _finite(layer["waiting_mean"]),
+                _finite(layer["waiting_max"]),
+                _finite(layer["decode_B_mean"]),
+                _finite(layer["decode_B_cv"]),
+                _finite(layer["kv_usage_mean"]),
+                _finite(layer["kv_usage_max"]),
+                float(graph_shares.get("NONE", 0.0)),
+                float(graph_shares.get("FULL", 0.0)),
+                _finite(layer["padding_fraction"]),
+                prefill_tokens / max(1.0, prefill_tokens + decode_tokens),
+                _finite(layer["model_steps"]) / float(primary["interval"]["elapsed_s"]),
+            )
+            transitions.append(
+                Transition(
+                    cell=cell,
+                    current_anchor=float(current["anchor"]),
+                    next_anchor=float(following["anchor"]),
+                    external=external,
+                    instrumentation=instrumentation,
+                    next_feasible=int(bool(following["accepted_feasible"])),
+                )
+            )
+    return transitions
+
+
+def _sigmoid(values: np.ndarray) -> np.ndarray:
+    clipped = np.clip(values, -30.0, 30.0)
+    return 1.0 / (1.0 + np.exp(-clipped))
+
+
+def _fit_logistic(x: np.ndarray, y: np.ndarray, regularization: float) -> np.ndarray:
+    weights = np.zeros(x.shape[1], dtype=np.float64)
+    penalty = np.eye(x.shape[1], dtype=np.float64)
+    penalty[0, 0] = 0.0
+    for _ in range(100):
+        probability = _sigmoid(x @ weights)
+        gradient = x.T @ (probability - y) / len(y)
+        gradient += regularization * penalty @ weights / len(y)
+        curvature = probability * (1.0 - probability)
+        hessian = (x.T * curvature) @ x / len(y)
+        hessian += regularization * penalty / len(y)
+        step = np.linalg.lstsq(hessian, gradient, rcond=None)[0]
+        weights -= step
+        if float(np.max(np.abs(step))) < 1e-9:
+            break
+    return weights
+
+
+def _classification_metrics(y: np.ndarray, probability: np.ndarray) -> dict[str, Any]:
+    if np.any(probability < 0.0) or np.any(probability > 1.0):
+        raise ValueError("classification probabilities must be in [0, 1]")
+    prediction = probability >= 0.5
+    true_positive = int(np.sum(prediction & (y == 1)))
+    true_negative = int(np.sum(~prediction & (y == 0)))
+    false_positive = int(np.sum(prediction & (y == 0)))
+    false_negative = int(np.sum(~prediction & (y == 1)))
+    positive_total = true_positive + false_negative
+    negative_total = true_negative + false_positive
+    balanced = 0.5 * (
+        true_positive / positive_total + true_negative / negative_total
+    )
+    clipped = np.clip(probability, 1e-12, 1.0 - 1e-12)
+    return {
+        "accuracy": float(np.mean(prediction == y)),
+        "balanced_accuracy": float(balanced),
+        "brier": float(np.mean((probability - y) ** 2)),
+        "log_loss": float(np.mean(-(y * np.log(clipped) + (1 - y) * np.log(1 - clipped)))),
+        "confusion": {
+            "true_positive": true_positive,
+            "true_negative": true_negative,
+            "false_positive": false_positive,
+            "false_negative": false_negative,
+        },
+    }
+
+
+def _mcnemar_exact_p(outcome_only_correct: int, instrumentation_only_correct: int) -> float:
+    discordant = outcome_only_correct + instrumentation_only_correct
+    if discordant == 0:
+        return 1.0
+    tail = sum(
+        math.comb(discordant, value)
+        for value in range(min(outcome_only_correct, instrumentation_only_correct) + 1)
+    ) / (2**discordant)
+    return min(1.0, 2.0 * tail)
+
+
+def grouped_predictions(
+    transitions: list[Transition],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> tuple[np.ndarray, np.ndarray, list[str]]:
+    probabilities = []
+    labels = []
+    test_cells = []
+    for held_out in sorted({transition.cell for transition in transitions}):
+        train = [transition for transition in transitions if transition.cell != held_out]
+        test = [transition for transition in transitions if transition.cell == held_out]
+
+        def row(transition: Transition) -> np.ndarray:
+            values = transition.external
+            if instrumentation_aware:
+                values += transition.instrumentation
+            return np.asarray((1.0, *values), dtype=np.float64)
+
+        x_train = np.stack([row(transition) for transition in train])
+        x_test = np.stack([row(transition) for transition in test])
+        y_train = np.asarray([transition.next_feasible for transition in train], dtype=np.float64)
+        mean = x_train[:, 1:].mean(axis=0)
+        standard_deviation = x_train[:, 1:].std(axis=0)
+        standard_deviation[standard_deviation < 1e-8] = 1.0
+        x_train[:, 1:] = (x_train[:, 1:] - mean) / standard_deviation
+        x_test[:, 1:] = (x_test[:, 1:] - mean) / standard_deviation
+        weights = _fit_logistic(x_train, y_train, regularization)
+        probabilities.extend(_sigmoid(x_test @ weights).tolist())
+        labels.extend(transition.next_feasible for transition in test)
+        test_cells.extend(held_out for _ in test)
+    return (
+        np.asarray(labels, dtype=np.int64),
+        np.asarray(probabilities, dtype=np.float64),
+        test_cells,
+    )
+
+
+def _group_bootstrap_delta(
+    y: np.ndarray,
+    outcome_probability: np.ndarray,
+    instrumentation_probability: np.ndarray,
+    cells: list[str],
+) -> dict[str, Any]:
+    groups = sorted(set(cells))
+    indices = {group: np.asarray([i for i, cell in enumerate(cells) if cell == group]) for group in groups}
+    random = np.random.default_rng(BOOTSTRAP_SEED)
+    accuracy_deltas = []
+    brier_deltas = []
+    for _ in range(BOOTSTRAP_REPLICATES):
+        sampled = random.choice(groups, size=len(groups), replace=True)
+        selected = np.concatenate([indices[group] for group in sampled])
+        selected_y = y[selected]
+        outcome = outcome_probability[selected]
+        instrumentation = instrumentation_probability[selected]
+        accuracy_deltas.append(
+            float(np.mean((instrumentation >= 0.5) == selected_y))
+            - float(np.mean((outcome >= 0.5) == selected_y))
+        )
+        brier_deltas.append(
+            float(np.mean((instrumentation - selected_y) ** 2))
+            - float(np.mean((outcome - selected_y) ** 2))
+        )
+    return {
+        "semantics": "group bootstrap over cells; diagnostic confidence interval",
+        "replicates": BOOTSTRAP_REPLICATES,
+        "seed": BOOTSTRAP_SEED,
+        "accuracy_delta_instrumentation_minus_outcome": {
+            "point": float(np.mean((instrumentation_probability >= 0.5) == y))
+            - float(np.mean((outcome_probability >= 0.5) == y)),
+            "ci95": [float(x) for x in np.percentile(accuracy_deltas, [2.5, 97.5])],
+        },
+        "brier_delta_instrumentation_minus_outcome": {
+            "point": float(np.mean((instrumentation_probability - y) ** 2))
+            - float(np.mean((outcome_probability - y) ** 2)),
+            "ci95": [float(x) for x in np.percentile(brier_deltas, [2.5, 97.5])],
+        },
+    }
+
+
+def transition_analysis(transitions: list[Transition]) -> dict[str, Any]:
+    sensitivity = {}
+    headline_payload = None
+    for regularization in REGULARIZATION_SENSITIVITY:
+        y, outcome_probability, cells = grouped_predictions(
+            transitions,
+            instrumentation_aware=False,
+            regularization=regularization,
+        )
+        instrumentation_y, instrumentation_probability, instrumentation_cells = grouped_predictions(
+            transitions,
+            instrumentation_aware=True,
+            regularization=regularization,
+        )
+        if not np.array_equal(y, instrumentation_y) or cells != instrumentation_cells:
+            raise AssertionError("model folds or labels differ")
+        outcome_correct = (outcome_probability >= 0.5) == y
+        instrumentation_correct = (instrumentation_probability >= 0.5) == y
+        payload = {
+            "outcome_only": _classification_metrics(y, outcome_probability),
+            "instrumentation_aware": _classification_metrics(y, instrumentation_probability),
+            "paired_correctness": {
+                "both_correct": int(np.sum(outcome_correct & instrumentation_correct)),
+                "outcome_only_correct": int(np.sum(outcome_correct & ~instrumentation_correct)),
+                "instrumentation_only_correct": int(np.sum(~outcome_correct & instrumentation_correct)),
+                "both_wrong": int(np.sum(~outcome_correct & ~instrumentation_correct)),
+            },
+            "bootstrap": _group_bootstrap_delta(
+                y,
+                outcome_probability,
+                instrumentation_probability,
+                cells,
+            ),
+        }
+        payload["paired_correctness"]["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+            payload["paired_correctness"]["outcome_only_correct"],
+            payload["paired_correctness"]["instrumentation_only_correct"],
+        )
+        sensitivity[str(regularization)] = payload
+        if regularization == DEFAULT_REGULARIZATION:
+            headline_payload = payload
+    assert headline_payload is not None
+    labels = [transition.next_feasible for transition in transitions]
+    accuracy_deltas = [
+        value["instrumentation_aware"]["accuracy"] - value["outcome_only"]["accuracy"]
+        for value in sensitivity.values()
+    ]
+    brier_deltas = [
+        value["instrumentation_aware"]["brier"] - value["outcome_only"]["brier"]
+        for value in sensitivity.values()
+    ]
+    return {
+        "status": "RETROSPECTIVE_DIAGNOSTIC_ONLY",
+        "estimand": "next-anchor feasibility from the full current-anchor summary",
+        "split": "leave-one-cell-out",
+        "model": "L2 logistic regression with train-fold standardization",
+        "external_features": list(EXTERNAL_FEATURES),
+        "instrumentation_features": list(INSTRUMENTATION_FEATURES),
+        "headline_regularization": DEFAULT_REGULARIZATION,
+        "headline": headline_payload,
+        "regularization_sensitivity": sensitivity,
+        "sensitivity_summary": {
+            "accuracy_delta_min_max": [min(accuracy_deltas), max(accuracy_deltas)],
+            "brier_delta_min_max": [min(brier_deltas), max(brier_deltas)],
+            "incremental_signal_verdict": "NEEDS_PROSPECTIVE_EVIDENCE",
+        },
+        "label_sanity": {
+            **numeric(labels),
+            "positive": sum(labels),
+            "negative": len(labels) - sum(labels),
+        },
+    }
+
+
+def analyze(simfid_path: Path, phase6_path: Path) -> dict[str, Any]:
+    simfid = json.loads(simfid_path.read_text())
+    phase6 = json.loads(phase6_path.read_text())
+    real_scores = {cell: float(score) for cell, score in simfid["real_scores"].items()}
+    topk = {}
+    for reading, payload in sorted(simfid["analyses"].items()):
+        tie = payload["metrics"]["tie_buckets"]["simulator"]
+        topk[reading] = topk_curve(
+            real_scores,
+            {cell: float(score) for cell, score in payload["simulated_scores"].items()},
+            float(tie["tolerance"]),
+        )
+    transitions = build_transitions(phase6)
+    transition_result = transition_analysis(transitions)
+    red_flags = []
+    if len(real_scores) != 12:
+        red_flags.append("unexpected_simfid_cell_count")
+    if len(transitions) == 0 or len(set(x.next_feasible for x in transitions)) != 2:
+        red_flags.append("transition_labels_missing_or_single_class")
+    if any(not math.isfinite(value) or value < 0 for value in real_scores.values()):
+        red_flags.append("invalid_real_score")
+    return {
+        "schema": SCHEMA,
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": "retrospective single-workload premise audit; not prospective contribution evidence",
+        "provenance": {
+            "simfid_metrics": str(simfid_path.resolve()),
+            "simfid_sha256": sha256_file(simfid_path),
+            "phase6_metrics": str(phase6_path.resolve()),
+            "phase6_sha256": sha256_file(phase6_path),
+        },
+        "topk_headroom": topk,
+        "next_anchor_prediction": transition_result,
+        "decision": {
+            "current_surface_can_show_selection_contribution": False,
+            "reason": (
+                "The strongest frozen-calibrated SLO reading reaches zero real regret "
+                "after real evaluation of its first two-cell tie bucket. A method that "
+                "requires one calibration probe and one final verification cannot use "
+                "this single task to demonstrate fewer real cell evaluations."
+            ),
+            "prospective_target": (
+                "Test whether internal features from a short, shared real probe reduce "
+                "the number or duration of full frontier evaluations relative to an "
+                "outcome-only model given the same probe."
+            ),
+        },
+        "sanity": {
+            "real_scores": numeric(real_scores.values()),
+            "simulator_readings": len(topk),
+            "transitions": len(transitions),
+            "transition_cells": len({transition.cell for transition in transitions}),
+            "red_flags": red_flags,
+            "invariants": {
+                "same_cells_all_readings": all(
+                    set(payload["simulated_scores"]) == set(real_scores)
+                    for payload in simfid["analyses"].values()
+                ),
+                "scores_nonnegative": all(value >= 0 for value in real_scores.values()),
+                "transition_features_finite": all(
+                    all(math.isfinite(value) for value in (*item.external, *item.instrumentation))
+                    for item in transitions
+                ),
+                "probabilities_bounded": True,
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--simfid-metrics", type=Path, required=True)
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(args.simfid_metrics, args.phase6_metrics)
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(json.dumps({"status": result["status"], "output": str(args.output)}, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_pilot.py

@@ -0,0 +1,293 @@
+#!/usr/bin/env python3
+"""Evaluate frozen outcome-only and instrumentation-aware policies on P1."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from analyze_existing import _classification_metrics, _mcnemar_exact_p
+from analyze_prefixes import (
+    PrefixExample,
+    _load_jsonl,
+    _prefix_features,
+    numeric,
+    policy_metrics,
+    predict_frozen_model,
+    sha256_file,
+)
+
+
+def result_path(run_root: Path, cell: str, level: str, replicate: int) -> Path:
+    return run_root / "cells" / cell / f"{level}-rep{replicate}" / "result.json"
+
+
+def requests_path(run_root: Path, cell: str, level: str, replicate: int) -> Path:
+    return run_root / "cells" / cell / f"{level}-rep{replicate}" / "requests.jsonl"
+
+
+def selection_for(
+    manifest: dict[str, Any], cell: str, level: str, replicate: int
+) -> dict[str, Any]:
+    role = f"{level}{replicate}"
+    return manifest["cells"][cell]["targets"][level]["selections"][role]
+
+
+def build_pilot_examples(
+    manifest: dict[str, Any], run_root: Path, cutoff_s: float
+) -> tuple[list[PrefixExample], list[dict[str, Any]], list[str]]:
+    examples = []
+    details = []
+    red_flags = []
+    for cell, config in sorted(manifest["cells"].items()):
+        stream_path = next((run_root / "cells" / cell / "opprof").glob("*.jsonl"))
+        stream = _load_jsonl(stream_path, require_key="submit_mono_ns")
+        for level in ("low", "high"):
+            results = [
+                json.loads(result_path(run_root, cell, level, replicate).read_text())
+                for replicate in (1, 2, 3)
+            ]
+            votes = [bool(result["feasible"]) for result in results]
+            adjudicated = sum(votes) >= 2
+            primary = results[0]
+            requests = _load_jsonl(requests_path(run_root, cell, level, 1))
+            exact_timestamps = sum(
+                request.get("completed_elapsed_s") is not None for request in requests
+            )
+            actual_outcomes = sum(
+                request.get("completed_mono_ns") is not None for request in requests
+            )
+            if exact_timestamps != actual_outcomes:
+                red_flags.append(f"timestamp_count_mismatch_{cell}_{level}")
+            expected = selection_for(manifest, cell, level, 1)
+            if int(primary["selection"]["count"]) != int(expected["selected_count"]):
+                red_flags.append(f"selection_count_mismatch_{cell}_{level}")
+            for result_key, manifest_key in (
+                ("request_id_order_sha256", "request_id_order_sha256"),
+                ("arrival_order_sha256", "arrival_order_sha256"),
+                ("raw_length_order_sha256", "input_length_order_sha256"),
+            ):
+                if primary["selection"][result_key] != expected[manifest_key]:
+                    red_flags.append(f"selection_hash_mismatch_{cell}_{level}_{result_key}")
+            start_ns = int(primary["interval"]["start_mono_ns"])
+            end_ns = start_ns + int(cutoff_s * 1e9)
+            records = [
+                record
+                for record in stream
+                if record.get("model_executed")
+                and start_ns <= int(record["submit_mono_ns"]) <= end_ns
+            ]
+            outcome, instrumentation, completion_source = _prefix_features(
+                primary=primary,
+                tp=int(config["tp"]),
+                max_num_seqs=int(config["mns"]),
+                requests=requests,
+                records=records,
+                cutoff_s=cutoff_s,
+            )
+            example = PrefixExample(
+                cell=cell,
+                anchor=float(primary["anchor"]),
+                cutoff_s=cutoff_s,
+                tp=int(config["tp"]),
+                full_elapsed_s=float(primary["interval"]["elapsed_s"]),
+                feasible=int(adjudicated),
+                primary_feasible=int(bool(primary["feasible"])),
+                outcome=outcome,
+                instrumentation=instrumentation,
+                completion_time_source=completion_source,
+            )
+            examples.append(example)
+            details.append(
+                {
+                    "cell": cell,
+                    "level": level,
+                    "anchor_rep1": primary["anchor"],
+                    "selected_count_rep1": primary["selection"]["count"],
+                    "votes": votes,
+                    "pass_rates": [result["pass_rate"] for result in results],
+                    "adjudicated_feasible": adjudicated,
+                    "primary_feasible": bool(primary["feasible"]),
+                    "actual_timestamped_outcomes": actual_outcomes,
+                    "selected_outcomes": len(requests),
+                    "prefix_layer1_records": len(records),
+                    "completion_time_source": completion_source,
+                }
+            )
+    return examples, details, red_flags
+
+
+def analyze(
+    manifest_path: Path,
+    model_path: Path,
+    run_root: Path,
+) -> dict[str, Any]:
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    models = json.loads(model_path.read_text(encoding="utf-8"))
+    state_path = run_root / "controller-state.json"
+    state = json.loads(state_path.read_text(encoding="utf-8"))
+    cutoff_s = float(models["cutoff_s"])
+    threshold = float(models["accept_probability"])
+    examples, details, red_flags = build_pilot_examples(manifest, run_root, cutoff_s)
+    labels = np.asarray([example.feasible for example in examples], dtype=np.int64)
+    outcome_probability = predict_frozen_model(models["models"]["outcome_only"], examples)
+    instrumentation_probability = predict_frozen_model(
+        models["models"]["instrumentation_aware"], examples
+    )
+    outcome_policy = policy_metrics(
+        examples, labels, outcome_probability, threshold
+    )
+    instrumentation_policy = policy_metrics(
+        examples, labels, instrumentation_probability, threshold
+    )
+    outcome_correct = (outcome_probability >= 0.5) == labels
+    instrumentation_correct = (instrumentation_probability >= 0.5) == labels
+    paired = {
+        "both_correct": int(np.sum(outcome_correct & instrumentation_correct)),
+        "outcome_only_correct": int(np.sum(outcome_correct & ~instrumentation_correct)),
+        "instrumentation_only_correct": int(np.sum(~outcome_correct & instrumentation_correct)),
+        "both_wrong": int(np.sum(~outcome_correct & ~instrumentation_correct)),
+    }
+    paired["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+        paired["outcome_only_correct"], paired["instrumentation_only_correct"]
+    )
+    for detail, outcome_p, instrumentation_p in zip(
+        details, outcome_probability, instrumentation_probability
+    ):
+        detail["outcome_probability_feasible"] = float(outcome_p)
+        detail["instrumentation_probability_feasible"] = float(instrumentation_p)
+
+    positive = int(np.sum(labels))
+    negative = len(labels) - positive
+    if state["status"] != "complete" or int(state["completed_cells"]) != 6:
+        red_flags.append("campaign_incomplete")
+    if positive < 3 or negative < 3:
+        red_flags.append("insufficient_label_balance")
+    if any(
+        detail["actual_timestamped_outcomes"] == 0 for detail in details
+    ):
+        red_flags.append("no_exact_request_timestamps")
+    if float(state["gpu_hours_total"]) >= float(state["hard_cap_h20_hours"]):
+        red_flags.append("hard_cap_exceeded")
+
+    outcome_errors = outcome_policy["false_accept"] + outcome_policy["false_reject"]
+    instrumentation_errors = (
+        instrumentation_policy["false_accept"]
+        + instrumentation_policy["false_reject"]
+    )
+    outcome_decisions = outcome_policy["early_accept"] + outcome_policy["early_reject"]
+    instrumentation_decisions = (
+        instrumentation_policy["early_accept"]
+        + instrumentation_policy["early_reject"]
+    )
+    outcome_reduction = outcome_policy["valid_cost_reduction_fraction"]
+    instrumentation_reduction = instrumentation_policy["valid_cost_reduction_fraction"]
+    cost_delta = (
+        instrumentation_reduction - outcome_reduction
+        if outcome_reduction is not None and instrumentation_reduction is not None
+        else None
+    )
+    data_valid = not red_flags
+    safety_gate = instrumentation_errors == 0 and instrumentation_errors <= outcome_errors
+    incremental_gate = (
+        instrumentation_decisions - outcome_decisions >= 3
+        or (cost_delta is not None and cost_delta >= 0.15)
+    )
+    pilot_pass = data_valid and safety_gate and incremental_gate
+
+    return {
+        "schema": "fidelity-prefix-pilot-result-v1",
+        "status": "PILOT_PASS" if pilot_pass else "PILOT_FAIL",
+        "scope": "held-out single-task gate; not paper-facing contribution evidence",
+        "provenance": {
+            "manifest": str(manifest_path.resolve()),
+            "manifest_sha256": sha256_file(manifest_path),
+            "frozen_models": str(model_path.resolve()),
+            "frozen_models_sha256": sha256_file(model_path),
+            "controller_state": str(state_path.resolve()),
+            "controller_state_sha256": sha256_file(state_path),
+        },
+        "cutoff_s": cutoff_s,
+        "threshold": threshold,
+        "examples": details,
+        "outcome_only": {
+            "classification": _classification_metrics(labels, outcome_probability),
+            "policy": outcome_policy,
+        },
+        "instrumentation_aware": {
+            "classification": _classification_metrics(labels, instrumentation_probability),
+            "policy": instrumentation_policy,
+        },
+        "paired_correctness": paired,
+        "gate": {
+            "data_valid": data_valid,
+            "safety_gate": safety_gate,
+            "incremental_gate": incremental_gate,
+            "additional_early_decisions": instrumentation_decisions - outcome_decisions,
+            "valid_cost_reduction_fraction_delta": cost_delta,
+            "opens_expanded_p2": pilot_pass,
+        },
+        "gpu": {
+            "actual_h20_hours": state["gpu_hours_total"],
+            "hard_cap_h20_hours": state["hard_cap_h20_hours"],
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "labels": {
+                **numeric(labels.tolist()),
+                "positive": positive,
+                "negative": negative,
+            },
+            "full_elapsed_s": numeric(example.full_elapsed_s for example in examples),
+            "remaining_h20_hours": numeric(
+                example.remaining_h20_hours for example in examples
+            ),
+            "outcome_probability": numeric(outcome_probability.tolist()),
+            "instrumentation_probability": numeric(
+                instrumentation_probability.tolist()
+            ),
+            "invariants": {
+                "examples_12": len(examples) == 12,
+                "cells_6": len({example.cell for example in examples}) == 6,
+                "ratios_bounded": bool(
+                    np.all((outcome_probability >= 0) & (outcome_probability <= 1))
+                    and np.all(
+                        (instrumentation_probability >= 0)
+                        & (instrumentation_probability <= 1)
+                    )
+                ),
+                "costs_nonnegative": all(
+                    example.remaining_h20_hours >= 0 for example in examples
+                ),
+                "all_cell_validations": all(
+                    all(cell["validation"]["invariants"].values())
+                    for cell in state["cells"].values()
+                ),
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--frozen-models", type=Path, required=True)
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(args.manifest, args.frozen_models, args.run_root)
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(json.dumps({
+        "status": result["status"],
+        "gate": result["gate"],
+        "sanity_red_flags": result["sanity"]["red_flags"],
+    }, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_prefixes.py

@@ -0,0 +1,629 @@
+#!/usr/bin/env python3
+"""Retrospective, leakage-bounded audit of short real-probe prefixes.
+
+The outcome-only and instrumentation-aware models receive the same trial
+prefix.  The latter differs only by Layer-1 engine state.  Existing Phase-6
+request artifacts predate exact completion timestamps, so their completion
+time is reconstructed from arrival + TTFT + token intervals and is explicitly
+marked approximate.  New artifacts use ``completed_elapsed_s`` directly.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable
+
+import numpy as np
+
+from analyze_existing import (
+    DEFAULT_REGULARIZATION,
+    REGULARIZATION_SENSITIVITY,
+    _classification_metrics,
+    _fit_logistic,
+    _group_bootstrap_delta,
+    _mcnemar_exact_p,
+    _sigmoid,
+)
+
+
+SCHEMA = "fidelity-prefix-v1"
+DEFAULT_CUTOFFS = (5.0, 10.0, 15.0, 20.0)
+POLICY_THRESHOLDS = (0.8, 0.9, 0.95)
+
+OUTCOME_FEATURES = (
+    "log_offered_rate_per_gpu",
+    "log2_tp",
+    "log2_max_num_seqs",
+    "admitted_fraction",
+    "completed_over_admitted",
+    "completed_pass_rate",
+    "completed_fail_fraction_of_total",
+    "outstanding_over_admitted",
+    "ttft_max_over_slo_max",
+    "ttft_mean_over_slo_max",
+    "tpot_max_over_slo",
+    "tpot_mean_over_slo",
+    "admitted_input_tokens_mean_over_limit",
+)
+
+INSTRUMENTATION_FEATURES = (
+    "model_steps_per_second",
+    "waiting_mean",
+    "waiting_max",
+    "waiting_nonzero_share",
+    "running_mean",
+    "running_max",
+    "decode_batch_mean",
+    "decode_batch_max",
+    "decode_batch_cv",
+    "kv_usage_mean",
+    "kv_usage_max",
+    "kv_usage_end_minus_start",
+    "graph_none_share",
+    "graph_full_share",
+    "padding_fraction",
+    "prefill_token_fraction",
+    "preemptions",
+)
+
+
+@dataclass(frozen=True)
+class PrefixExample:
+    cell: str
+    anchor: float
+    cutoff_s: float
+    tp: int
+    full_elapsed_s: float
+    feasible: int
+    primary_feasible: int
+    outcome: tuple[float, ...]
+    instrumentation: tuple[float, ...]
+    completion_time_source: str
+
+    @property
+    def remaining_h20_hours(self) -> float:
+        return self.tp * max(0.0, self.full_elapsed_s - self.cutoff_s) / 3600.0
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def numeric(values: Iterable[float | int]) -> dict[str, Any]:
+    array = [float(value) for value in values]
+    return {
+        "n": len(array),
+        "min": min(array) if array else None,
+        "max": max(array) if array else None,
+        "distinct_n": len(set(array)),
+    }
+
+
+def _cv(values: list[float]) -> float:
+    if not values:
+        return 0.0
+    array = np.asarray(values, dtype=np.float64)
+    mean = float(array.mean())
+    return float(array.std(ddof=0) / mean) if mean else 0.0
+
+
+def completion_elapsed_s(request: dict[str, Any]) -> tuple[float | None, str]:
+    exact = request.get("completed_elapsed_s")
+    if exact is not None:
+        value = float(exact)
+        if value < 0 or not math.isfinite(value):
+            raise ValueError(f"invalid completed_elapsed_s={exact}")
+        return value, "exact_monotonic"
+    if not request.get("success"):
+        return None, "unobserved_failure"
+    required = (
+        request.get("arrival_s"),
+        request.get("ttft_ms"),
+        request.get("tpot_ms"),
+        request.get("completion_tokens"),
+    )
+    if any(value is None for value in required):
+        return None, "unobserved_failure"
+    arrival_s, ttft_ms, tpot_ms, completion_tokens = required
+    value = float(arrival_s) + (
+        float(ttft_ms) + max(int(completion_tokens) - 1, 0) * float(tpot_ms)
+    ) / 1000.0
+    if value < 0 or not math.isfinite(value):
+        raise ValueError(f"invalid reconstructed completion time={value}")
+    return value, "reconstructed_from_latency"
+
+
+def _load_jsonl(path: Path, *, require_key: str | None = None) -> list[dict[str, Any]]:
+    records = []
+    with path.open(encoding="utf-8") as source:
+        for line in source:
+            item = json.loads(line)
+            if require_key is None or require_key in item:
+                records.append(item)
+    return records
+
+
+def _anchor_directory(cell_root: Path, anchor: float) -> Path:
+    matches = []
+    for result_path in cell_root.glob("anchor-*/result.json"):
+        payload = json.loads(result_path.read_text(encoding="utf-8"))
+        if math.isclose(float(payload["anchor"]), anchor, rel_tol=0.0, abs_tol=1e-15):
+            matches.append(result_path.parent)
+    if len(matches) != 1:
+        raise ValueError(f"expected one primary directory for anchor {anchor}: {matches}")
+    return matches[0]
+
+
+def _prefix_features(
+    *,
+    primary: dict[str, Any],
+    tp: int,
+    max_num_seqs: int,
+    requests: list[dict[str, Any]],
+    records: list[dict[str, Any]],
+    cutoff_s: float,
+) -> tuple[tuple[float, ...], tuple[float, ...], str]:
+    admitted = [request for request in requests if float(request["arrival_s"]) <= cutoff_s]
+    completed = []
+    sources = set()
+    for request in requests:
+        completed_s, source = completion_elapsed_s(request)
+        if completed_s is None or completed_s > cutoff_s:
+            continue
+        completed.append(request)
+        sources.add(source)
+    if not admitted or not records:
+        raise ValueError("prefix has no admitted requests or Layer-1 records")
+    if any(request not in admitted for request in completed):
+        raise ValueError("completed request was not admitted inside prefix")
+
+    total = len(requests)
+    passed = sum(bool(request["slo_pass"]) for request in completed)
+    ttft = [float(request["ttft_ms"]) for request in completed if request["ttft_ms"] is not None]
+    tpot = [float(request["tpot_ms"]) for request in completed if request["tpot_ms"] is not None]
+    offered_rate = float(primary["selection"]["offered_req_s_per_gpu"])
+    if offered_rate <= 0 or total <= 0:
+        raise ValueError("offered rate and selected request count must be positive")
+
+    outcome = (
+        math.log(offered_rate),
+        math.log2(float(tp)),
+        math.log2(float(max_num_seqs)),
+        len(admitted) / total,
+        len(completed) / len(admitted),
+        passed / max(1, len(completed)),
+        (len(completed) - passed) / total,
+        (len(admitted) - len(completed)) / len(admitted),
+        max(ttft, default=0.0) / 6000.0,
+        float(np.mean(ttft)) / 6000.0 if ttft else 0.0,
+        max(tpot, default=0.0) / 50.0,
+        float(np.mean(tpot)) / 50.0 if tpot else 0.0,
+        float(np.mean([float(request["raw_input_tokens"]) for request in admitted])) / 8192.0,
+    )
+
+    waiting = [float(record["queues"]["waiting"]) for record in records]
+    running = [float(record["queues"]["running"]) for record in records]
+    decode_batch = [float(record["decode_batch_size"]) for record in records]
+    kv_usage = [float(record["kv"]["usage"]) for record in records]
+    graph_modes = [str(record["cudagraph"]["runtime_mode"]) for record in records]
+    bucket_tokens = sum(int(record["cudagraph"]["bucket_tokens"]) for record in records)
+    padding_tokens = sum(int(record["cudagraph"]["padding_tokens"]) for record in records)
+    prefill_tokens = sum(int(record["prefill_tokens"]) for record in records)
+    decode_tokens = sum(int(record["decode_tokens"]) for record in records)
+    instrumentation = (
+        len(records) / cutoff_s,
+        float(np.mean(waiting)),
+        max(waiting),
+        sum(value > 0 for value in waiting) / len(waiting),
+        float(np.mean(running)),
+        max(running),
+        float(np.mean(decode_batch)),
+        max(decode_batch),
+        _cv(decode_batch),
+        float(np.mean(kv_usage)),
+        max(kv_usage),
+        kv_usage[-1] - kv_usage[0],
+        graph_modes.count("NONE") / len(graph_modes),
+        graph_modes.count("FULL") / len(graph_modes),
+        padding_tokens / max(1, bucket_tokens),
+        prefill_tokens / max(1, prefill_tokens + decode_tokens),
+        float(sum(int(record["preemptions"]) for record in records)),
+    )
+    completion_source = "+".join(sorted(sources)) if sources else "none_completed"
+    return outcome, instrumentation, completion_source
+
+
+def build_examples(
+    phase6: dict[str, Any],
+    raw_root: Path,
+    cutoff_s: float,
+) -> list[PrefixExample]:
+    examples = []
+    for cell, cell_result in sorted(phase6["cells"].items()):
+        cell_root = raw_root / cell
+        stream_path = next((cell_root / "opprof").glob("*.jsonl"))
+        stream = _load_jsonl(stream_path, require_key="submit_mono_ns")
+        for anchor in cell_result["anchors"]:
+            primary = anchor["primary"]
+            full_elapsed_s = float(primary["interval"]["elapsed_s"])
+            if full_elapsed_s + 1e-9 < cutoff_s:
+                continue
+            anchor_value = float(primary["anchor"])
+            anchor_root = _anchor_directory(cell_root, anchor_value)
+            requests = _load_jsonl(anchor_root / "requests.jsonl")
+            start_ns = int(primary["interval"]["start_mono_ns"])
+            end_ns = start_ns + int(cutoff_s * 1e9)
+            records = [
+                record
+                for record in stream
+                if record.get("model_executed")
+                and start_ns <= int(record["submit_mono_ns"]) <= end_ns
+            ]
+            outcome, instrumentation, source = _prefix_features(
+                primary=primary,
+                tp=int(cell_result["tp"]),
+                max_num_seqs=int(cell_result["mns"]),
+                requests=requests,
+                records=records,
+                cutoff_s=cutoff_s,
+            )
+            examples.append(
+                PrefixExample(
+                    cell=cell,
+                    anchor=anchor_value,
+                    cutoff_s=cutoff_s,
+                    tp=int(cell_result["tp"]),
+                    full_elapsed_s=full_elapsed_s,
+                    feasible=int(bool(anchor["accepted_feasible"])),
+                    primary_feasible=int(bool(primary["feasible"])),
+                    outcome=outcome,
+                    instrumentation=instrumentation,
+                    completion_time_source=source,
+                )
+            )
+    return examples
+
+
+def grouped_predictions(
+    examples: list[PrefixExample],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> tuple[np.ndarray, np.ndarray, list[str]]:
+    probabilities = []
+    labels = []
+    groups = []
+    for held_out in sorted({example.cell for example in examples}):
+        train = [example for example in examples if example.cell != held_out]
+        test = [example for example in examples if example.cell == held_out]
+
+        def row(example: PrefixExample) -> np.ndarray:
+            values = example.outcome
+            if instrumentation_aware:
+                values += example.instrumentation
+            return np.asarray((1.0, *values), dtype=np.float64)
+
+        x_train = np.stack([row(example) for example in train])
+        x_test = np.stack([row(example) for example in test])
+        y_train = np.asarray([example.feasible for example in train], dtype=np.float64)
+        if len(set(y_train.tolist())) != 2:
+            raise ValueError(f"training fold for {held_out} has a single label")
+        mean = x_train[:, 1:].mean(axis=0)
+        standard_deviation = x_train[:, 1:].std(axis=0)
+        standard_deviation[standard_deviation < 1e-8] = 1.0
+        x_train[:, 1:] = (x_train[:, 1:] - mean) / standard_deviation
+        x_test[:, 1:] = (x_test[:, 1:] - mean) / standard_deviation
+        weights = _fit_logistic(x_train, y_train, regularization)
+        probabilities.extend(_sigmoid(x_test @ weights).tolist())
+        labels.extend(example.feasible for example in test)
+        groups.extend(held_out for _ in test)
+    return (
+        np.asarray(labels, dtype=np.int64),
+        np.asarray(probabilities, dtype=np.float64),
+        groups,
+    )
+
+
+def fit_frozen_model(
+    examples: list[PrefixExample],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> dict[str, Any]:
+    def row(example: PrefixExample) -> np.ndarray:
+        values = example.outcome
+        if instrumentation_aware:
+            values += example.instrumentation
+        return np.asarray((1.0, *values), dtype=np.float64)
+
+    matrix = np.stack([row(example) for example in examples])
+    labels = np.asarray([example.feasible for example in examples], dtype=np.float64)
+    if len(set(labels.tolist())) != 2:
+        raise ValueError("frozen model requires both feasibility labels")
+    mean = matrix[:, 1:].mean(axis=0)
+    standard_deviation = matrix[:, 1:].std(axis=0)
+    standard_deviation[standard_deviation < 1e-8] = 1.0
+    standardized = matrix.copy()
+    standardized[:, 1:] = (standardized[:, 1:] - mean) / standard_deviation
+    weights = _fit_logistic(standardized, labels, regularization)
+    probabilities = _sigmoid(standardized @ weights)
+    names = list(OUTCOME_FEATURES)
+    if instrumentation_aware:
+        names.extend(INSTRUMENTATION_FEATURES)
+    return {
+        "instrumentation_aware": instrumentation_aware,
+        "regularization": regularization,
+        "feature_names": names,
+        "feature_mean": mean.tolist(),
+        "feature_standard_deviation": standard_deviation.tolist(),
+        "weights_with_intercept_first": weights.tolist(),
+        "training_classification": _classification_metrics(labels, probabilities),
+    }
+
+
+def predict_frozen_model(
+    model: dict[str, Any],
+    examples: list[PrefixExample],
+) -> np.ndarray:
+    instrumentation_aware = bool(model["instrumentation_aware"])
+    rows = []
+    for example in examples:
+        values = example.outcome
+        if instrumentation_aware:
+            values += example.instrumentation
+        rows.append((1.0, *values))
+    matrix = np.asarray(rows, dtype=np.float64)
+    mean = np.asarray(model["feature_mean"], dtype=np.float64)
+    standard_deviation = np.asarray(
+        model["feature_standard_deviation"], dtype=np.float64
+    )
+    weights = np.asarray(model["weights_with_intercept_first"], dtype=np.float64)
+    if matrix.shape[1] != len(weights) or matrix.shape[1] - 1 != len(mean):
+        raise ValueError("frozen model feature dimensions do not match examples")
+    matrix[:, 1:] = (matrix[:, 1:] - mean) / standard_deviation
+    return _sigmoid(matrix @ weights)
+
+
+def policy_metrics(
+    examples: list[PrefixExample],
+    labels: np.ndarray,
+    probabilities: np.ndarray,
+    threshold: float,
+) -> dict[str, Any]:
+    accept = probabilities >= threshold
+    reject = probabilities <= 1.0 - threshold
+    decide = accept | reject
+    prediction = accept.astype(np.int64)
+    correct = prediction == labels
+    remaining = np.asarray(
+        [example.remaining_h20_hours for example in examples], dtype=np.float64
+    )
+    full_cost = sum(example.tp * example.full_elapsed_s / 3600.0 for example in examples)
+    saved = float(np.sum(remaining[decide]))
+    correct_saved = float(np.sum(remaining[decide & correct]))
+    invalid_saved = float(np.sum(remaining[decide & ~correct]))
+
+    def describe(mask: np.ndarray) -> list[dict[str, Any]]:
+        return [
+            {
+                "cell": example.cell,
+                "anchor": example.anchor,
+                "label_feasible": bool(label),
+                "probability_feasible": float(probability),
+                "remaining_h20_hours": example.remaining_h20_hours,
+            }
+            for example, label, probability, selected in zip(
+                examples, labels, probabilities, mask
+            )
+            if selected
+        ]
+
+    return {
+        "threshold": threshold,
+        "early_accept": int(np.sum(accept)),
+        "early_reject": int(np.sum(reject)),
+        "abstain_continue_full": int(np.sum(~decide)),
+        "false_accept": int(np.sum(accept & (labels == 0))),
+        "false_reject": int(np.sum(reject & (labels == 1))),
+        "false_accept_examples": describe(accept & (labels == 0)),
+        "false_reject_examples": describe(reject & (labels == 1)),
+        "decision_coverage": float(np.mean(decide)),
+        "full_trial_h20_hours": float(full_cost),
+        "remaining_h20_hours_at_cutoff": float(np.sum(remaining)),
+        "saved_h20_hours_if_decisions_used": saved,
+        "correctly_saved_h20_hours": correct_saved,
+        "invalidly_saved_h20_hours": invalid_saved,
+        "valid_zero_error_policy": bool(np.all(correct[decide])),
+        "valid_cost_reduction_fraction": (
+            correct_saved / full_cost if invalid_saved == 0.0 and full_cost else None
+        ),
+    }
+
+
+def analyze_cutoff(examples: list[PrefixExample]) -> dict[str, Any]:
+    sensitivity = {}
+    headline = None
+    for regularization in REGULARIZATION_SENSITIVITY:
+        labels, outcome_probability, groups = grouped_predictions(
+            examples,
+            instrumentation_aware=False,
+            regularization=regularization,
+        )
+        instrument_labels, instrument_probability, instrument_groups = grouped_predictions(
+            examples,
+            instrumentation_aware=True,
+            regularization=regularization,
+        )
+        if not np.array_equal(labels, instrument_labels) or groups != instrument_groups:
+            raise AssertionError("paired folds or labels differ")
+        if groups != [example.cell for example in examples]:
+            raise AssertionError("prediction order differs from example order")
+        outcome_correct = (outcome_probability >= 0.5) == labels
+        instrument_correct = (instrument_probability >= 0.5) == labels
+        result = {
+            "outcome_only": {
+                "classification": _classification_metrics(labels, outcome_probability),
+                "policies": [
+                    policy_metrics(examples, labels, outcome_probability, threshold)
+                    for threshold in POLICY_THRESHOLDS
+                ],
+            },
+            "instrumentation_aware": {
+                "classification": _classification_metrics(labels, instrument_probability),
+                "policies": [
+                    policy_metrics(examples, labels, instrument_probability, threshold)
+                    for threshold in POLICY_THRESHOLDS
+                ],
+            },
+            "paired_correctness": {
+                "both_correct": int(np.sum(outcome_correct & instrument_correct)),
+                "outcome_only_correct": int(np.sum(outcome_correct & ~instrument_correct)),
+                "instrumentation_only_correct": int(np.sum(~outcome_correct & instrument_correct)),
+                "both_wrong": int(np.sum(~outcome_correct & ~instrument_correct)),
+            },
+            "bootstrap": _group_bootstrap_delta(
+                labels,
+                outcome_probability,
+                instrument_probability,
+                groups,
+            ),
+        }
+        paired = result["paired_correctness"]
+        paired["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+            paired["outcome_only_correct"], paired["instrumentation_only_correct"]
+        )
+        sensitivity[str(regularization)] = result
+        if regularization == DEFAULT_REGULARIZATION:
+            headline = result
+    assert headline is not None
+    labels = [example.feasible for example in examples]
+    return {
+        "examples": len(examples),
+        "cells": len({example.cell for example in examples}),
+        "label_sanity": {
+            **numeric(labels),
+            "positive": sum(labels),
+            "negative": len(labels) - sum(labels),
+            "primary_adjudicated_disagreements": sum(
+                example.feasible != example.primary_feasible for example in examples
+            ),
+        },
+        "completion_time_sources": {
+            source: sum(example.completion_time_source == source for example in examples)
+            for source in sorted({example.completion_time_source for example in examples})
+        },
+        "headline_regularization": DEFAULT_REGULARIZATION,
+        "headline": headline,
+        "regularization_sensitivity": sensitivity,
+        "remaining_h20_hours": numeric(
+            example.remaining_h20_hours for example in examples
+        ),
+    }
+
+
+def analyze(
+    phase6_path: Path,
+    raw_root: Path,
+    cutoffs: tuple[float, ...],
+) -> dict[str, Any]:
+    phase6 = json.loads(phase6_path.read_text(encoding="utf-8"))
+    by_cutoff = {}
+    red_flags = []
+    for cutoff in cutoffs:
+        examples = build_examples(phase6, raw_root, cutoff)
+        if len({example.feasible for example in examples}) != 2:
+            red_flags.append(f"single_label_at_{cutoff:g}s")
+            continue
+        by_cutoff[f"{cutoff:g}"] = analyze_cutoff(examples)
+        if len({example.cell for example in examples}) != 12:
+            red_flags.append(f"incomplete_cells_at_{cutoff:g}s")
+        if not all(
+            math.isfinite(value)
+            for example in examples
+            for value in (*example.outcome, *example.instrumentation)
+        ):
+            red_flags.append(f"nonfinite_features_at_{cutoff:g}s")
+
+    headline_deltas = {
+        cutoff: {
+            "accuracy": (
+                result["headline"]["instrumentation_aware"]["classification"]["accuracy"]
+                - result["headline"]["outcome_only"]["classification"]["accuracy"]
+            ),
+            "brier": (
+                result["headline"]["instrumentation_aware"]["classification"]["brier"]
+                - result["headline"]["outcome_only"]["classification"]["brier"]
+            ),
+        }
+        for cutoff, result in by_cutoff.items()
+    }
+    return {
+        "schema": SCHEMA,
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": (
+            "retrospective single-workload prefix diagnostic; model selection, "
+            "threshold choice, and contribution claims require held-out prospective tasks"
+        ),
+        "estimand": (
+            "2-of-3 adjudicated anchor feasibility from the first primary trial's "
+            "identical short real prefix"
+        ),
+        "split": "leave-one-configuration-cell-out",
+        "model": "same L2 logistic model and folds; instrumentation model appends Layer-1 features",
+        "outcome_features": list(OUTCOME_FEATURES),
+        "instrumentation_features": list(INSTRUMENTATION_FEATURES),
+        "provenance": {
+            "phase6_metrics": str(phase6_path.resolve()),
+            "phase6_metrics_sha256": sha256_file(phase6_path),
+            "raw_root": str(raw_root.resolve()),
+        },
+        "cutoffs_s": list(cutoffs),
+        "cutoffs": by_cutoff,
+        "headline_incremental_deltas": headline_deltas,
+        "decision": {
+            "contribution_established": False,
+            "reason": (
+                "This dataset contains one workload and reconstructed rather than exact request "
+                "completion times. Three TP4 primary trials also disagree with their 2-of-3 "
+                "labels. It can reject a missing-signal premise but cannot establish "
+                "generalization or a paper-facing cost reduction."
+            ),
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "cutoff_count": len(by_cutoff),
+            "invariants": {
+                "cutoffs_positive": all(cutoff > 0 for cutoff in cutoffs),
+                "paired_same_model_family": True,
+                "probabilities_checked_in_unit_interval": True,
+                "full_trial_label_not_used_as_feature": True,
+                "records_strictly_prefix_sliced": True,
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--raw-root", type=Path, required=True)
+    parser.add_argument("--cutoffs", type=float, nargs="+", default=DEFAULT_CUTOFFS)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(args.phase6_metrics, args.raw_root, tuple(args.cutoffs))
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    print(json.dumps({"status": result["status"], "output": str(args.output)}, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/freeze_models.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+"""Freeze the training-task prefix models before prospective GPU work."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from analyze_prefixes import (
+    DEFAULT_REGULARIZATION,
+    POLICY_THRESHOLDS,
+    build_examples,
+    fit_frozen_model,
+    sha256_file,
+)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--prefix-metrics", type=Path, required=True)
+    parser.add_argument("--raw-root", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+
+    cutoff_s = 5.0
+    threshold = 0.95
+    if threshold not in POLICY_THRESHOLDS:
+        raise AssertionError("frozen threshold is outside audited policy thresholds")
+    phase6 = json.loads(args.phase6_metrics.read_text(encoding="utf-8"))
+    examples = build_examples(phase6, args.raw_root, cutoff_s)
+    payload = {
+        "schema": "fidelity-prefix-model-v1",
+        "status": "FROZEN_BEFORE_PROSPECTIVE_RUN",
+        "cutoff_s": cutoff_s,
+        "accept_probability": threshold,
+        "reject_probability": 1.0 - threshold,
+        "regularization": DEFAULT_REGULARIZATION,
+        "label": "same-placement 2-of-3 adjudicated anchor feasibility",
+        "training_split_role": "historical training only; never headline test",
+        "training_examples": [
+            {
+                "cell": example.cell,
+                "anchor": example.anchor,
+                "label_feasible": bool(example.feasible),
+                "primary_feasible": bool(example.primary_feasible),
+                "completion_time_source": example.completion_time_source,
+            }
+            for example in examples
+        ],
+        "models": {
+            "outcome_only": fit_frozen_model(
+                examples,
+                instrumentation_aware=False,
+                regularization=DEFAULT_REGULARIZATION,
+            ),
+            "instrumentation_aware": fit_frozen_model(
+                examples,
+                instrumentation_aware=True,
+                regularization=DEFAULT_REGULARIZATION,
+            ),
+        },
+        "provenance": {
+            "phase6_metrics": str(args.phase6_metrics.resolve()),
+            "phase6_metrics_sha256": sha256_file(args.phase6_metrics),
+            "prefix_metrics": str(args.prefix_metrics.resolve()),
+            "prefix_metrics_sha256": sha256_file(args.prefix_metrics),
+            "raw_root": str(args.raw_root.resolve()),
+        },
+        "sanity": {
+            "n": len(examples),
+            "positive": sum(example.feasible for example in examples),
+            "negative": sum(not example.feasible for example in examples),
+            "cells": len({example.cell for example in examples}),
+            "invariants": {
+                "n_37": len(examples) == 37,
+                "cells_12": len({example.cell for example in examples}) == 12,
+                "both_labels": len({example.feasible for example in examples}) == 2,
+                "cutoff_5s": cutoff_s == 5.0,
+                "threshold_0.95": threshold == 0.95,
+            },
+        },
+    }
+    if not all(payload["sanity"]["invariants"].values()):
+        raise RuntimeError(f"model freeze invariants failed: {payload['sanity']}")
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    print(json.dumps({"status": payload["status"], "output": str(args.output)}))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/frozen-models.json

@@ -0,0 +1,515 @@
+{
+  "accept_probability": 0.95,
+  "cutoff_s": 5.0,
+  "label": "same-placement 2-of-3 adjudicated anchor feasibility",
+  "models": {
+    "instrumentation_aware": {
+      "feature_mean": [
+        0.8984976998643891,
+        0.8378378378378378,
+        4.324324324324325,
+        0.07552086023066117,
+        0.6758807403968693,
+        0.9459459459459459,
+        0.0,
+        0.3241192596031305,
+        0.04468545442770093,
+        0.025590516908533558,
+        0.23873649352596996,
+        0.1943716628122394,
+        0.4321792125178198,
+        112.70270270270272,
+        0.21087752102856197,
+        0.918918918918919,
+        0.055470351361483664,
+        4.904239530899751,
+        10.162162162162161,
+        4.822982150502539,
+        10.135135135135135,
+        0.43557131397798415,
+        0.031387890158936414,
+        0.05804311436894179,
+        0.03298678556030958,
+        0.030437119300455177,
+        0.9503065396705037,
+        0.07127076398319926,
+        0.6234198543231205,
+        0.0
+      ],
+      "feature_names": [
+        "log_offered_rate_per_gpu",
+        "log2_tp",
+        "log2_max_num_seqs",
+        "admitted_fraction",
+        "completed_over_admitted",
+        "completed_pass_rate",
+        "completed_fail_fraction_of_total",
+        "outstanding_over_admitted",
+        "ttft_max_over_slo_max",
+        "ttft_mean_over_slo_max",
+        "tpot_max_over_slo",
+        "tpot_mean_over_slo",
+        "admitted_input_tokens_mean_over_limit",
+        "model_steps_per_second",
+        "waiting_mean",
+        "waiting_max",
+        "waiting_nonzero_share",
+        "running_mean",
+        "running_max",
+        "decode_batch_mean",
+        "decode_batch_max",
+        "decode_batch_cv",
+        "kv_usage_mean",
+        "kv_usage_max",
+        "kv_usage_end_minus_start",
+        "graph_none_share",
+        "graph_full_share",
+        "padding_fraction",
+        "prefill_token_fraction",
+        "preemptions"
+      ],
+      "feature_standard_deviation": [
+        0.2953332526155246,
+        0.8546696378833459,
+        1.1402715194448103,
+        0.006588255148989237,
+        0.2751217635728275,
+        0.22612433149569594,
+        1.0,
+        0.27512176357282747,
+        0.048292427420964075,
+        0.02574874589991541,
+        0.1635381690436309,
+        0.14098674719611365,
+        0.02516276437103069,
+        61.39272994412853,
+        0.7234949448561444,
+        2.198013579605131,
+        0.18326586413988316,
+        2.4542471212960844,
+        6.08726412391018,
+        2.4074006634033043,
+        6.067913672185017,
+        0.12556414020947543,
+        0.03256962310836033,
+        0.054049610008010444,
+        0.048321850100969746,
+        0.04298231458641556,
+        0.041906068064246155,
+        0.08212268757576466,
+        0.4089385238422411,
+        1.0
+      ],
+      "instrumentation_aware": true,
+      "regularization": 1.0,
+      "training_classification": {
+        "accuracy": 0.972972972972973,
+        "balanced_accuracy": 0.9444444444444444,
+        "brier": 0.02820726479488704,
+        "confusion": {
+          "false_negative": 0,
+          "false_positive": 1,
+          "true_negative": 8,
+          "true_positive": 28
+        },
+        "log_loss": 0.11247563885308659
+      },
+      "weights_with_intercept_first": [
+        2.109507425802979,
+        -0.8372240489271802,
+        -0.2476229678897366,
+        0.18172257646801393,
+        -0.07076358054975332,
+        0.3035586906752765,
+        0.08500005412355496,
+        -7.754818242684634e-26,
+        -0.3035586906752766,
+        0.4014234393196892,
+        0.513218716194957,
+        -0.35161457106287,
+        0.10558147889556725,
+        0.5674345291616134,
+        0.15895995157114373,
+        -0.4274260624362057,
+        -0.048791959001756195,
+        -0.37221380985270663,
+        -0.35527537277290255,
+        0.20582736797173468,
+        -0.35837576944545413,
+        0.2342062515631318,
+        0.45071068059490843,
+        0.3326948315186803,
+        0.2698892549960913,
+        0.017868065865726347,
+        -0.1540209080477302,
+        0.3412427440368233,
+        0.5831011876762794,
+        -0.583920360300169,
+        0.0
+      ]
+    },
+    "outcome_only": {
+      "feature_mean": [
+        0.8984976998643891,
+        0.8378378378378378,
+        4.324324324324325,
+        0.07552086023066117,
+        0.6758807403968693,
+        0.9459459459459459,
+        0.0,
+        0.3241192596031305,
+        0.04468545442770093,
+        0.025590516908533558,
+        0.23873649352596996,
+        0.1943716628122394,
+        0.4321792125178198
+      ],
+      "feature_names": [
+        "log_offered_rate_per_gpu",
+        "log2_tp",
+        "log2_max_num_seqs",
+        "admitted_fraction",
+        "completed_over_admitted",
+        "completed_pass_rate",
+        "completed_fail_fraction_of_total",
+        "outstanding_over_admitted",
+        "ttft_max_over_slo_max",
+        "ttft_mean_over_slo_max",
+        "tpot_max_over_slo",
+        "tpot_mean_over_slo",
+        "admitted_input_tokens_mean_over_limit"
+      ],
+      "feature_standard_deviation": [
+        0.2953332526155246,
+        0.8546696378833459,
+        1.1402715194448103,
+        0.006588255148989237,
+        0.2751217635728275,
+        0.22612433149569594,
+        1.0,
+        0.27512176357282747,
+        0.048292427420964075,
+        0.02574874589991541,
+        0.1635381690436309,
+        0.14098674719611365,
+        0.02516276437103069
+      ],
+      "instrumentation_aware": false,
+      "regularization": 1.0,
+      "training_classification": {
+        "accuracy": 0.9459459459459459,
+        "balanced_accuracy": 0.8888888888888888,
+        "brier": 0.051887373873176545,
+        "confusion": {
+          "false_negative": 0,
+          "false_positive": 2,
+          "true_negative": 7,
+          "true_positive": 28
+        },
+        "log_loss": 0.184988719119571
+      },
+      "weights_with_intercept_first": [
+        1.8996338126233983,
+        -1.1536861934230125,
+        -0.3806404559018098,
+        0.5901136731733696,
+        0.022432085012851908,
+        0.5805554730881304,
+        0.25786307099613026,
+        -8.077935669463161e-27,
+        -0.5805554730881304,
+        -0.15413292402348447,
+        0.0986842306063204,
+        -0.5181573573074624,
+        0.06283513013708956,
+        0.911619634884147
+      ]
+    }
+  },
+  "provenance": {
+    "phase6_metrics": "/home/gahow/phd/aituner/runs/opprof-phase6/phase6/metrics.json",
+    "phase6_metrics_sha256": "290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16",
+    "prefix_metrics": "/home/gahow/phd/aituner/runs/fidelity-headroom/prefix-metrics.json",
+    "prefix_metrics_sha256": "cda821bcde1ae8427507aa4f03a1c116ccc7f7b8b717f73ca587bee3670a0340",
+    "raw_root": "/home/gahow/phd/aituner/runs/opprof-phase6/phase6/solo-authoritative/cells"
+  },
+  "regularization": 1.0,
+  "reject_probability": 0.050000000000000044,
+  "sanity": {
+    "cells": 12,
+    "invariants": {
+      "both_labels": true,
+      "cells_12": true,
+      "cutoff_5s": true,
+      "n_37": true,
+      "threshold_0.95": true
+    },
+    "n": 37,
+    "negative": 9,
+    "positive": 28
+  },
+  "schema": "fidelity-prefix-model-v1",
+  "status": "FROZEN_BEFORE_PROSPECTIVE_RUN",
+  "training_examples": [
+    {
+      "anchor": 0.24609375,
+      "cell": "tp1_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.2421875,
+      "cell": "tp1_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.24609375,
+      "cell": "tp1_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns32",
+      "completion_time_source": "none_completed",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.2421875,
+      "cell": "tp1_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.24609375,
+      "cell": "tp1_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns64",
+      "completion_time_source": "none_completed",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.21875,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.2265625,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.23046875,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.234375,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.4921875,
+      "cell": "tp2_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.49609375,
+      "cell": "tp2_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp2_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.75,
+      "cell": "tp2_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.75390625,
+      "cell": "tp2_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp2_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.75,
+      "cell": "tp2_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.4921875,
+      "cell": "tp2_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.49609375,
+      "cell": "tp2_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.033182214016,
+      "cell": "tp4_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.033717411016,
+      "cell": "tp4_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.033717411016,
+      "cell": "tp4_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.033717411016,
+      "cell": "tp4_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.016055910008,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.016591107009,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.017126304009,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    }
+  ],
+  "training_split_role": "historical training only; never headline test"
+}

runs/fidelity-headroom/pilot_controller.py

@@ -0,0 +1,436 @@
+#!/usr/bin/env python3
+"""Serialized dash0 controller for the exact-timestamp prefix pilot."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shlex
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+
+HERE = Path(__file__).resolve().parent
+PHASE6 = HERE.parent / "opprof-phase6"
+sys.path.insert(0, str(PHASE6))
+
+import opprof_phase6_controller as base  # noqa: E402
+
+
+ORDER = (
+    "tp1_mns8",
+    "tp1_mns64",
+    "tp2_mns8",
+    "tp2_mns64",
+    "tp4_mns16",
+    "tp4_mns64",
+)
+CELL_ESTIMATE_H20_HOURS = {1: 0.20, 2: 0.40, 4: 0.80}
+SAFETY_H20_HOURS = 0.20
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    base.atomic_json(path, payload)
+
+
+def wait_all_idle(timeout_s: float = 30.0) -> None:
+    deadline = time.monotonic() + timeout_s
+    last_error: Exception | None = None
+    while time.monotonic() < deadline:
+        try:
+            base.assert_all_idle()
+            return
+        except RuntimeError as error:
+            last_error = error
+            time.sleep(1.0)
+    raise last_error or RuntimeError("GPU idle timeout")
+
+
+def configure_base(args: argparse.Namespace, manifest: dict[str, Any]) -> None:
+    base.WORKDIR = args.run_root.parent
+    base.RUN_ROOT = args.run_root
+    base.STATE = args.run_root / "controller-state.json"
+    base.SOURCE = args.vllm_source
+    base.VENV = args.venv
+    base.AITUNER = args.aituner_root
+    base.MODEL = args.model
+    base.CLIENT = args.client
+    base.GPU_LIMIT = float(manifest["execution"]["hard_cap_h20_hours"])
+    base.MARKER = "fidelity-prefix-pilot-20260714"
+    base.CELLS = {
+        cell: {"tp": int(config["tp"]), "mns": int(config["mns"])}
+        for cell, config in manifest["cells"].items()
+    }
+
+
+def load_state(path: Path, hard_cap: float) -> dict[str, Any]:
+    if path.exists():
+        return json.loads(path.read_text(encoding="utf-8"))
+    return {
+        "schema": "fidelity-prefix-pilot-state-v1",
+        "status": "initialized",
+        "hard_cap_h20_hours": hard_cap,
+        "gpu_hours_total": 0.0,
+        "completed_cells": 0,
+        "cells": {},
+        "failures": [],
+        "started_at": time.time(),
+    }
+
+
+def save_state(path: Path, state: dict[str, Any]) -> None:
+    atomic_json(path, state)
+
+
+def append_echo(run_root: Path, line: str) -> None:
+    run_root.mkdir(parents=True, exist_ok=True)
+    with (run_root / "launch-echo.log").open("a", encoding="utf-8") as target:
+        target.write(line + "\n")
+    print(line, flush=True)
+
+
+def remaining_projection(manifest: dict[str, Any], index: int) -> float:
+    return sum(
+        CELL_ESTIMATE_H20_HOURS[int(manifest["cells"][cell]["tp"])]
+        for cell in ORDER[index:]
+    ) + SAFETY_H20_HOURS
+
+
+def start_server(
+    *,
+    cell: str,
+    index: int,
+    run_root: Path,
+) -> dict[str, Any]:
+    config = base.CELLS[cell]
+    gpus = tuple(range(int(config["tp"])))
+    cell_root = run_root / "cells" / cell
+    cell_root.mkdir(parents=True, exist_ok=True)
+    port = 8900 + index
+    command = base.server_command(cell, gpus, port)
+    with (cell_root / "commands.log").open("a", encoding="utf-8") as log:
+        log.write(f"SERVER {shlex.join(command)}\n")
+    server_log = (cell_root / "server.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update(
+        {
+            "CUDA_VISIBLE_DEVICES": ",".join(map(str, gpus)),
+            "VLLM_OPPROF_DIR": str(cell_root / "opprof"),
+            "OPPROF_PHASE6_MARKER": base.MARKER,
+            "AITUNER_ROOT": str(base.AITUNER),
+            "HF_HUB_OFFLINE": "1",
+            "TRANSFORMERS_OFFLINE": "1",
+            "PYTHONUNBUFFERED": "1",
+        }
+    )
+    server = subprocess.Popen(
+        command,
+        cwd=base.SOURCE,
+        env=environment,
+        stdout=server_log,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    base.OWNED_PGIDS.add(server.pid)
+    return {
+        "cell": cell,
+        "gpus": gpus,
+        "port": port,
+        "dir": cell_root,
+        "server": server,
+        "server_handle": server_log,
+        "spawned_at": time.time(),
+        "results": [],
+    }
+
+
+def selection_for(
+    manifest: dict[str, Any], cell: str, role: str
+) -> tuple[str, dict[str, Any]]:
+    level = "low" if role == "burnin" or role.startswith("low") else "high"
+    return level, manifest["cells"][cell]["targets"][level]["selections"][role]
+
+
+def client_command(
+    entry: dict[str, Any],
+    *,
+    role: str,
+    selection: dict[str, Any],
+    output: Path,
+    warmup: bool,
+) -> list[str]:
+    config = base.CELLS[entry["cell"]]
+    return [
+        "taskset",
+        "-c",
+        base.cpu_mask(entry["gpus"]),
+        str(base.VENV / "bin/python"),
+        str(base.CLIENT),
+        "warmup" if warmup else "run-anchor",
+        "--study",
+        str(selection["study"]),
+        "--cell",
+        entry["cell"],
+        "--anchor",
+        str(selection["anchor"]),
+        "--tp",
+        str(config["tp"]),
+        "--mns",
+        str(config["mns"]),
+        "--base-url",
+        f"http://127.0.0.1:{entry['port']}",
+        "--result-dir",
+        str(output),
+    ]
+
+
+def run_client(
+    *,
+    entry: dict[str, Any],
+    role: str,
+    selection: dict[str, Any],
+    output: Path,
+    state: dict[str, Any],
+    warmup: bool = False,
+) -> dict[str, Any]:
+    command = client_command(
+        entry, role=role, selection=selection, output=output, warmup=warmup
+    )
+    with (entry["dir"] / "commands.log").open("a", encoding="utf-8") as log:
+        log.write(f"CLIENT role={role} {shlex.join(command)}\n")
+    handle = (output.parent / f"{output.name}.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update({"AITUNER_ROOT": str(base.AITUNER), "PYTHONUNBUFFERED": "1"})
+    process = subprocess.Popen(
+        command,
+        cwd=base.WORKDIR,
+        env=environment,
+        stdout=handle,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    deadline = time.monotonic() + 180.0
+    try:
+        while process.poll() is None:
+            if time.monotonic() > deadline:
+                process.terminate()
+                raise TimeoutError(f"client timeout: {entry['cell']} {role}")
+            if entry["server"].poll() is not None:
+                raise RuntimeError(f"server exited during {entry['cell']} {role}")
+            base.assert_no_other_compute()
+            if state["gpu_hours_total"] + base.live_gpu_hours([entry]) >= base.GPU_LIMIT:
+                process.terminate()
+                raise RuntimeError("pilot H20-hour hard cap reached")
+            time.sleep(1.0)
+    finally:
+        handle.close()
+    if process.returncode:
+        raise RuntimeError(
+            f"client failed: cell={entry['cell']} role={role} rc={process.returncode}"
+        )
+    result = json.loads((output / "result.json").read_text(encoding="utf-8"))
+    if int(result["selection"]["count"]) != int(selection["selected_count"]):
+        raise RuntimeError(f"selection count mismatch: {entry['cell']} {role}")
+    for key in (
+        "request_id_order_sha256",
+        "arrival_order_sha256",
+        "raw_length_order_sha256",
+    ):
+        manifest_key = (
+            "input_length_order_sha256" if key == "raw_length_order_sha256" else key
+        )
+        if result["selection"][key] != selection[manifest_key]:
+            raise RuntimeError(f"selection hash mismatch {key}: {entry['cell']} {role}")
+    entry["results"].append(
+        {"anchor": float(selection["anchor"]), "dir": str(output), "kind": result["kind"]}
+    )
+    return result
+
+
+def execute_cell(
+    *,
+    index: int,
+    cell: str,
+    manifest: dict[str, Any],
+    run_root: Path,
+    state_path: Path,
+    state: dict[str, Any],
+) -> None:
+    if state["cells"].get(cell, {}).get("status") == "complete":
+        return
+    projection = remaining_projection(manifest, index)
+    if state["gpu_hours_total"] + projection > base.GPU_LIMIT:
+        state["status"] = "budget_projection_stop"
+        state["budget_stop"] = {
+            "before_cell": cell,
+            "spent_h20_hours": state["gpu_hours_total"],
+            "remaining_projection_h20_hours": projection,
+            "hard_cap_h20_hours": base.GPU_LIMIT,
+        }
+        save_state(state_path, state)
+        raise RuntimeError(f"projected pilot cost exceeds hard cap before {cell}")
+
+    config = manifest["cells"][cell]
+    echo = (
+        f"PILOT_CELL_ECHO cell={cell} tp={config['tp']} mns={config['mns']} "
+        f"gpus=0-{int(config['tp']) - 1} workload={manifest['source']['window_id']} "
+        f"roles=burnin+low1/high1/low2/high2/low3/high3 "
+        f"spent_h20h={state['gpu_hours_total']:.6f} "
+        f"remaining_projection_h20h={projection:.3f} cap_h20h={base.GPU_LIMIT:.1f} "
+        f"manifest={run_root / 'pilot-manifest.json'}"
+    )
+    append_echo(run_root, echo)
+    wait_all_idle()
+    cell_state = {
+        "status": "starting",
+        "tp": int(config["tp"]),
+        "mns": int(config["mns"]),
+        "started_at": time.time(),
+        "runs": [],
+    }
+    state["status"] = "running"
+    state["cells"][cell] = cell_state
+    save_state(state_path, state)
+    entry = start_server(cell=cell, index=index, run_root=run_root)
+    failure: Exception | None = None
+    try:
+        base.wait_ready(entry)
+        _level, burnin = selection_for(manifest, cell, "burnin")
+        cell_state["status"] = "warmup"
+        save_state(state_path, state)
+        warmup = run_client(
+            entry=entry,
+            role="burnin",
+            selection=burnin,
+            output=entry["dir"] / "warmup",
+            state=state,
+            warmup=True,
+        )
+        cell_state["warmup"] = {
+            "exact_output_count": warmup["exact_output_count"],
+            "long_gt4096": warmup["selection"]["long_gt4096"],
+        }
+        cell_state["status"] = "burnin"
+        save_state(state_path, state)
+        burnin_result = run_client(
+            entry=entry,
+            role="burnin",
+            selection=burnin,
+            output=entry["dir"] / "burnin",
+            state=state,
+        )
+        cell_state["burnin"] = {
+            "pass_rate": burnin_result["pass_rate"],
+            "feasible": burnin_result["feasible"],
+        }
+        role_order = manifest["execution"][
+            "even_cell_order" if index % 2 == 0 else "odd_cell_order"
+        ]
+        cell_state["status"] = "measured"
+        cell_state["role_order"] = role_order
+        save_state(state_path, state)
+        for role in role_order:
+            level, selection = selection_for(manifest, cell, role)
+            result = run_client(
+                entry=entry,
+                role=role,
+                selection=selection,
+                output=entry["dir"] / f"{level}-rep{role[-1]}",
+                state=state,
+            )
+            cell_state["runs"].append(
+                {
+                    "role": role,
+                    "level": level,
+                    "anchor": selection["anchor"],
+                    "selected_count": selection["selected_count"],
+                    "pass_rate": result["pass_rate"],
+                    "feasible": result["feasible"],
+                    "elapsed_s": result["interval"]["elapsed_s"],
+                }
+            )
+            save_state(state_path, state)
+        cell_state["status"] = "stopping"
+        save_state(state_path, state)
+    except Exception as error:  # noqa: BLE001
+        failure = error
+    finally:
+        try:
+            base.stop_entry(entry)
+        except Exception as error:  # noqa: BLE001
+            failure = failure or error
+        time.sleep(2.0)
+        try:
+            wait_all_idle()
+        except Exception as error:  # noqa: BLE001
+            failure = failure or error
+
+    cell_hours = base.live_gpu_hours([entry])
+    state["gpu_hours_total"] += cell_hours
+    cell_state["gpu_hours"] = cell_hours
+    if failure is not None:
+        cell_state["status"] = "failed"
+        cell_state["failure"] = repr(failure)
+        state["status"] = "failed"
+        state["failures"].append({"cell": cell, "failure": repr(failure)})
+        save_state(state_path, state)
+        raise failure
+    validation = base.validate_cell(entry)
+    cell_state["validation"] = validation
+    cell_state["status"] = "complete"
+    cell_state["completed_at"] = time.time()
+    state["completed_cells"] += 1
+    save_state(state_path, state)
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("--manifest", type=Path, required=True)
+    result.add_argument("--run-root", type=Path, required=True)
+    result.add_argument("--aituner-root", type=Path, required=True)
+    result.add_argument("--vllm-source", type=Path, required=True)
+    result.add_argument("--venv", type=Path, required=True)
+    result.add_argument("--model", type=Path, required=True)
+    result.add_argument("--client", type=Path, required=True)
+    return result
+
+
+def main() -> None:
+    args = parser().parse_args()
+    manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
+    if manifest["status"] != "PASS":
+        raise RuntimeError("pilot manifest did not pass preflight")
+    args.run_root.mkdir(parents=True, exist_ok=True)
+    copied_manifest = args.run_root / "pilot-manifest.json"
+    if not copied_manifest.exists():
+        atomic_json(copied_manifest, manifest)
+    configure_base(args, manifest)
+    state_path = args.run_root / "controller-state.json"
+    state = load_state(state_path, base.GPU_LIMIT)
+    state["status"] = "running"
+    save_state(state_path, state)
+    for index, cell in enumerate(ORDER):
+        execute_cell(
+            index=index,
+            cell=cell,
+            manifest=manifest,
+            run_root=args.run_root,
+            state_path=state_path,
+            state=state,
+        )
+    state["status"] = "complete"
+    state["completed_at"] = time.time()
+    save_state(state_path, state)
+    print(json.dumps({
+        "status": state["status"],
+        "completed_cells": state["completed_cells"],
+        "gpu_hours_total": state["gpu_hours_total"],
+    }, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/prepare_pilot.py

@@ -0,0 +1,351 @@
+#!/usr/bin/env python3
+"""Materialize session-disjoint pilot repeats and freeze attainable anchors.
+
+The private outputs retain prompt text and stay on the experiment host.  The
+public manifest contains only aggregate counts, hashes, paths, and parameters.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+
+AITUNER_ROOT = Path(os.environ.get("AITUNER_ROOT", Path(__file__).resolve().parents[2]))
+sys.path.insert(0, str(AITUNER_ROOT / "src"))
+
+from aituner.spec import load_study_spec  # noqa: E402
+from aituner.trace import load_trace_requests, select_requests_for_threshold  # noqa: E402
+
+
+ROLES = ("burnin", "low1", "high1", "low2", "high2", "low3", "high3")
+CELLS = {
+    "tp1_mns8": {"tp": 1, "mns": 8, "frontier_req_s_gpu": 2.3833333333333333},
+    "tp1_mns64": {"tp": 1, "mns": 64, "frontier_req_s_gpu": 2.3833333333333333},
+    "tp2_mns8": {"tp": 2, "mns": 8, "frontier_req_s_gpu": 2.2416666666666667},
+    "tp2_mns64": {"tp": 2, "mns": 64, "frontier_req_s_gpu": 2.3},
+    "tp4_mns16": {"tp": 4, "mns": 16, "frontier_req_s_gpu": 2.5},
+    "tp4_mns64": {"tp": 4, "mns": 64, "frontier_req_s_gpu": 2.5},
+}
+TARGET_MULTIPLIERS = {"low": 0.85, "high": 1.25}
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_suffix(path.suffix + ".tmp")
+    tmp.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    os.replace(tmp, path)
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def order_hash(values: list[str]) -> str:
+    return hashlib.sha256("\n".join(values).encode()).hexdigest()
+
+
+def resolve_source_trace(windows_path: Path, window_id: str) -> tuple[dict[str, Any], Path]:
+    payload = json.loads(windows_path.read_text(encoding="utf-8"))
+    for window in payload["windows"]:
+        if window["window_id"] != window_id:
+            continue
+        trace = Path(window["trace_file"])
+        if not trace.is_absolute():
+            trace = (windows_path.parent / trace).resolve()
+        return window, trace
+    raise ValueError(f"window not found: {window_id}")
+
+
+def materialize_bands(
+    source_trace: Path,
+    source_window: dict[str, Any],
+    private_root: Path,
+) -> tuple[Path, dict[str, Any]]:
+    traces_root = private_root / "traces"
+    traces_root.mkdir(parents=True, exist_ok=True)
+    temporary = {role: traces_root / f".{role}.jsonl.tmp" for role in ROLES}
+    final = {role: traces_root / f"{role}.jsonl" for role in ROLES}
+    handles = {role: temporary[role].open("w", encoding="utf-8") for role in ROLES}
+    stats = {
+        role: {
+            "rows": 0,
+            "sum_input_tokens": 0,
+            "min_timestamp": None,
+            "max_timestamp": None,
+        }
+        for role in ROLES
+    }
+    try:
+        with source_trace.open(encoding="utf-8") as source:
+            for line_number, line in enumerate(source):
+                row = json.loads(line)
+                value = float(row["sampling_u"])
+                if not 0.0 <= value <= 1.0:
+                    raise ValueError(f"sampling_u outside [0,1] at line {line_number}")
+                band = min(len(ROLES) - 1, int(value * len(ROLES)))
+                role = ROLES[band]
+                remapped = value * len(ROLES) - band
+                row["sampling_u"] = min(remapped, math.nextafter(1.0, 0.0))
+                row["fidelity_pilot_band"] = role
+                handles[role].write(json.dumps(row, ensure_ascii=False) + "\n")
+                timestamp = float(row["timestamp"])
+                item = stats[role]
+                item["rows"] += 1
+                item["sum_input_tokens"] += int(row.get("input_length") or 0)
+                item["min_timestamp"] = (
+                    timestamp if item["min_timestamp"] is None
+                    else min(float(item["min_timestamp"]), timestamp)
+                )
+                item["max_timestamp"] = (
+                    timestamp if item["max_timestamp"] is None
+                    else max(float(item["max_timestamp"]), timestamp)
+                )
+    finally:
+        for handle in handles.values():
+            handle.close()
+    for role in ROLES:
+        os.replace(temporary[role], final[role])
+        stats[role]["sha256"] = sha256_file(final[role])
+        stats[role]["bytes"] = final[role].stat().st_size
+
+    windows = []
+    for role in ROLES:
+        window = dict(source_window)
+        window["window_id"] = f"fidelity_pilot_{role}"
+        window["trace_file"] = f"traces/{role}.jsonl"
+        window["num_requests"] = stats[role]["rows"]
+        window["sum_input_length"] = stats[role]["sum_input_tokens"]
+        window["sampling_strategy"] = "session_uniform_seven_disjoint_bands_remapped"
+        window["fidelity_pilot_role"] = role
+        windows.append(window)
+    private_windows = private_root / "windows.json"
+    atomic_json(
+        private_windows,
+        {
+            "schema": "fidelity-pilot-private-windows-v1",
+            "roles": list(ROLES),
+            "windows": windows,
+        },
+    )
+    return private_windows, stats
+
+
+def write_studies(
+    *,
+    base_primary: Path,
+    base_tp4: Path,
+    private_windows: Path,
+    private_root: Path,
+) -> dict[str, dict[str, Path]]:
+    bases = {
+        "primary": json.loads(base_primary.read_text(encoding="utf-8")),
+        "tp4": json.loads(base_tp4.read_text(encoding="utf-8")),
+    }
+    result: dict[str, dict[str, Path]] = {}
+    for role in ROLES:
+        result[role] = {}
+        for tier, base in bases.items():
+            payload = json.loads(json.dumps(base))
+            payload["study_id"] = f"fidelity-prefix-pilot-{role}-{tier}"
+            payload["hardware"]["host_candidates"] = ["dash0"]
+            payload["engine"]["engine_version"] = "0.24.1.dev3+opprof"
+            payload["trace"]["windows_path"] = str(private_windows)
+            payload["trace"]["window_id"] = f"fidelity_pilot_{role}"
+            path = private_root / "studies" / f"{role}-{tier}.json"
+            atomic_json(path, payload)
+            result[role][tier] = path
+    return result
+
+
+def attainable_anchor(requests: list[Any], target_count: int) -> tuple[float, list[Any]]:
+    ordered = sorted(float(request.sampling_u) for request in requests)
+    if not ordered:
+        raise ValueError("no requests after study filtering")
+    candidate_indices = sorted({
+        max(0, min(len(ordered) - 1, target_count - 1)),
+        max(0, min(len(ordered) - 1, target_count)),
+    })
+    candidates = []
+    for index in candidate_indices:
+        anchor = ordered[index]
+        selected = select_requests_for_threshold(requests, threshold=anchor)
+        candidates.append((abs(len(selected) - target_count), len(selected), anchor, selected))
+    _error, _count, anchor, selected = min(candidates, key=lambda item: (item[0], item[1]))
+    return anchor, selected
+
+
+def selected_record(selected: list[Any], *, tp: int, duration_s: float) -> dict[str, Any]:
+    return {
+        "anchor": max(float(request.sampling_u) for request in selected),
+        "selected_count": len(selected),
+        "offered_req_s": len(selected) / duration_s,
+        "offered_req_s_per_gpu": len(selected) / duration_s / tp,
+        "request_id_order_sha256": order_hash([request.row_id for request in selected]),
+        "arrival_order_sha256": order_hash([f"{request.arrival_s:.12f}" for request in selected]),
+        "input_length_order_sha256": order_hash(
+            [str(request.prompt_tokens_hint) for request in selected]
+        ),
+    }
+
+
+def build_manifest(
+    *,
+    studies: dict[str, dict[str, Path]],
+    private_windows: Path,
+    band_stats: dict[str, Any],
+    source_trace: Path,
+    source_windows: Path,
+    source_window_id: str,
+) -> dict[str, Any]:
+    loaded = {}
+    durations = {}
+    for role, tiers in studies.items():
+        loaded[role] = {}
+        for tier, path in tiers.items():
+            study = load_study_spec(path)
+            window, requests = load_trace_requests(study, study_spec_path=path)
+            loaded[role][tier] = requests
+            durations[role] = float(window.window_end - window.window_start)
+
+    cells = {}
+    all_hashes = []
+    for cell, config in CELLS.items():
+        tp = int(config["tp"])
+        tier = "tp4" if tp == 4 else "primary"
+        targets = {}
+        for level, multiplier in TARGET_MULTIPLIERS.items():
+            target_rate = float(config["frontier_req_s_gpu"]) * multiplier
+            target_count = round(target_rate * durations["low1"] * tp)
+            roles = [role for role in ROLES if role == "burnin" or role.startswith(level)]
+            selections = {}
+            for role in roles:
+                anchor, selected = attainable_anchor(loaded[role][tier], target_count)
+                record = selected_record(selected, tp=tp, duration_s=durations[role])
+                record["anchor"] = anchor
+                record["study"] = str(studies[role][tier])
+                selections[role] = record
+                all_hashes.append(record["request_id_order_sha256"])
+            targets[level] = {
+                "multiplier": multiplier,
+                "target_req_s_per_gpu": target_rate,
+                "target_count": target_count,
+                "selections": selections,
+            }
+        cells[cell] = {**config, "targets": targets}
+
+    red_flags = []
+    for cell, config in cells.items():
+        for level, target in config["targets"].items():
+            if not target["selections"]:
+                red_flags.append(f"missing_{cell}_{level}")
+            for selection in target["selections"].values():
+                if selection["selected_count"] <= 0:
+                    red_flags.append(f"empty_{cell}_{level}")
+    per_cell_distinct = {}
+    for cell, config in cells.items():
+        hashes = [
+            selection["request_id_order_sha256"]
+            for target in config["targets"].values()
+            for selection in target["selections"].values()
+        ]
+        per_cell_distinct[cell] = len(hashes) == len(set(hashes))
+        if not per_cell_distinct[cell]:
+            red_flags.append(f"session_bands_overlap_{cell}")
+    return {
+        "schema": "fidelity-prefix-pilot-manifest-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "source": {
+            "windows": str(source_windows),
+            "window_id": source_window_id,
+            "trace": str(source_trace),
+            "trace_sha256": sha256_file(source_trace),
+        },
+        "private": {
+            "windows": str(private_windows),
+            "windows_sha256": sha256_file(private_windows),
+            "band_stats": band_stats,
+            "studies": {
+                role: {tier: str(path) for tier, path in tiers.items()}
+                for role, tiers in studies.items()
+            },
+        },
+        "roles": list(ROLES),
+        "cells": cells,
+        "execution": {
+            "cutoff_s": 5.0,
+            "replicates_per_level": 3,
+            "label": "2-of-3 session-disjoint repetitions",
+            "even_cell_order": ["low1", "high1", "high2", "low2", "low3", "high3"],
+            "odd_cell_order": ["high1", "low1", "low2", "high2", "high3", "low3"],
+            "hard_cap_h20_hours": 3.5,
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "n_cells": len(cells),
+            "n_roles": len(ROLES),
+            "selected_sets": len(all_hashes),
+            "distinct_selected_sets": len(set(all_hashes)),
+            "per_cell_selected_sets_distinct": per_cell_distinct,
+            "invariants": {
+                "cells_6": len(cells) == 6,
+                "roles_7": len(ROLES) == 7,
+                "band_rows_nonzero": all(stats["rows"] > 0 for stats in band_stats.values()),
+                "session_bands_disjoint_per_cell": all(per_cell_distinct.values()),
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--source-windows", type=Path, required=True)
+    parser.add_argument("--source-window-id", default="chat_w20260312_1000")
+    parser.add_argument("--base-primary-study", type=Path, required=True)
+    parser.add_argument("--base-tp4-study", type=Path, required=True)
+    parser.add_argument("--private-root", type=Path, required=True)
+    parser.add_argument("--public-manifest", type=Path, required=True)
+    args = parser.parse_args()
+
+    source_window, source_trace = resolve_source_trace(
+        args.source_windows, args.source_window_id
+    )
+    private_windows, band_stats = materialize_bands(
+        source_trace, source_window, args.private_root
+    )
+    studies = write_studies(
+        base_primary=args.base_primary_study,
+        base_tp4=args.base_tp4_study,
+        private_windows=private_windows,
+        private_root=args.private_root,
+    )
+    manifest = build_manifest(
+        studies=studies,
+        private_windows=private_windows,
+        band_stats=band_stats,
+        source_trace=source_trace,
+        source_windows=args.source_windows,
+        source_window_id=args.source_window_id,
+    )
+    atomic_json(args.public_manifest, manifest)
+    print(json.dumps({
+        "status": manifest["status"],
+        "manifest": str(args.public_manifest),
+        "sanity": manifest["sanity"],
+    }, sort_keys=True))
+    if manifest["status"] != "PASS":
+        raise RuntimeError(f"pilot preflight failed: {manifest['sanity']['red_flags']}")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_analysis.py

@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import importlib.util
+import math
+import sys
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def load_analysis():
+    spec = importlib.util.spec_from_file_location("fidelity_headroom", HERE / "analyze_existing.py")
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def main() -> None:
+    analysis = load_analysis()
+    curve = analysis.topk_curve(
+        {"a": 3.0, "b": 2.0, "c": 1.0},
+        {"a": 1.0, "b": 2.0, "c": 2.0},
+        2e-6,
+    )
+    assert curve["points"][0]["expanded_k"] == 2
+    assert curve["points"][0]["candidates"] == ["b", "c"]
+    assert math.isclose(curve["points"][0]["real_regret"], 1.0 / 3.0)
+    assert curve["points"][2]["real_regret"] == 0.0
+    assert curve["minimum_k"]["five_percent"] == {"nominal_k": 3, "expanded_k": 3}
+    assert analysis._mcnemar_exact_p(0, 1) == 1.0
+    assert analysis._mcnemar_exact_p(0, 5) == 0.0625
+    print("fidelity headroom analysis: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_pilot_tools.py

@@ -0,0 +1,85 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import json
+import math
+import sys
+import tempfile
+from dataclasses import dataclass
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+
+import pilot_controller as controller  # noqa: E402
+import prepare_pilot as prepare  # noqa: E402
+
+
+@dataclass
+class Request:
+    row_id: str
+    sampling_u: float
+    arrival_s: float = 0.0
+    prompt_tokens_hint: int = 1
+
+
+def main() -> None:
+    requests = [
+        Request("a", 0.1),
+        Request("b", 0.2),
+        Request("c", 0.2),
+        Request("d", 0.9),
+    ]
+    anchor, selected = prepare.attainable_anchor(requests, target_count=2)
+    assert anchor == 0.2
+    assert [request.row_id for request in selected] == ["a", "b", "c"]
+
+    with tempfile.TemporaryDirectory() as temporary:
+        root = Path(temporary)
+        source = root / "source.jsonl"
+        rows = []
+        for index, role in enumerate(prepare.ROLES):
+            rows.append(
+                {
+                    "request_id": role,
+                    "timestamp": float(index),
+                    "sampling_u": (index + 0.5) / len(prepare.ROLES),
+                    "input_length": 16 + index,
+                    "messages": [{"role": "user", "content": role}],
+                }
+            )
+        source.write_text(
+            "".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8"
+        )
+        windows, stats = prepare.materialize_bands(
+            source,
+            {
+                "window_id": "source",
+                "trace_type": "chat",
+                "window_start": 0.0,
+                "window_end": 600.0,
+            },
+            root / "private",
+        )
+        assert windows.is_file()
+        assert all(stats[role]["rows"] == 1 for role in prepare.ROLES)
+        for role in prepare.ROLES:
+            row = json.loads((root / "private" / "traces" / f"{role}.jsonl").read_text())
+            assert row["fidelity_pilot_band"] == role
+            assert abs(float(row["sampling_u"]) - 0.5) < 1e-12
+
+    assert len(controller.ORDER) == 6
+    assert set(controller.ORDER) == set(prepare.CELLS)
+    assert math.isclose(
+        sum(
+            controller.CELL_ESTIMATE_H20_HOURS[int(config["tp"])]
+            for config in prepare.CELLS.values()
+        ) + controller.SAFETY_H20_HOURS,
+        3.0,
+    )
+    print("fidelity pilot tools: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_prefix_analysis.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import math
+import sys
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+
+import analyze_prefixes as analysis  # noqa: E402
+
+
+def main() -> None:
+    exact, exact_source = analysis.completion_elapsed_s(
+        {"completed_elapsed_s": 7.25}
+    )
+    assert exact == 7.25 and exact_source == "exact_monotonic"
+
+    reconstructed, reconstructed_source = analysis.completion_elapsed_s(
+        {
+            "success": True,
+            "arrival_s": 2.0,
+            "ttft_ms": 100.0,
+            "tpot_ms": 10.0,
+            "completion_tokens": 11,
+        }
+    )
+    assert math.isclose(reconstructed or 0.0, 2.2)
+    assert reconstructed_source == "reconstructed_from_latency"
+    missing, missing_source = analysis.completion_elapsed_s({"success": False})
+    assert missing is None and missing_source == "unobserved_failure"
+
+    examples = [
+        analysis.PrefixExample(
+            cell=f"c{index}",
+            anchor=float(index),
+            cutoff_s=5.0,
+            tp=1,
+            full_elapsed_s=65.0,
+            feasible=label,
+            primary_feasible=label,
+            outcome=(float(index),),
+            instrumentation=(float(index % 2),),
+            completion_time_source="exact_monotonic",
+        )
+        for index, label in enumerate((0, 1, 1))
+    ]
+    labels = analysis.np.asarray([0, 1, 1])
+    probabilities = analysis.np.asarray([0.01, 0.99, 0.60])
+    policy = analysis.policy_metrics(examples, labels, probabilities, 0.95)
+    assert policy["early_accept"] == 1
+    assert policy["early_reject"] == 1
+    assert policy["abstain_continue_full"] == 1
+    assert policy["false_accept"] == 0 and policy["false_reject"] == 0
+    assert policy["valid_zero_error_policy"]
+    assert policy["valid_cost_reduction_fraction"] is not None
+    model = analysis.fit_frozen_model(
+        examples,
+        instrumentation_aware=True,
+        regularization=1.0,
+    )
+    frozen_probability = analysis.predict_frozen_model(model, examples)
+    assert len(frozen_probability) == len(examples)
+    assert analysis.np.all(frozen_probability >= 0.0)
+    assert analysis.np.all(frozen_probability <= 1.0)
+    print("fidelity prefix analysis: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/opprof-oracle-gap/launch-echo.log

@@ -0,0 +1,13 @@
+RUN_ECHO stage=primary-C11 host=dash0 gpu=0 cpus=0-19 config=C11 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C00 host=dash0 gpu=0 cpus=0-19 config=C00 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C00 host=dash0 gpu=0 cpus=0-19 config=C00 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C01 host=dash0 gpu=0 cpus=0-19 config=C01 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C01 host=dash0 gpu=0 cpus=0-19 config=C01 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C10 host=dash0 gpu=0 cpus=0-19 config=C10 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C10 host=dash0 gpu=0 cpus=0-19 config=C10 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=confirm-C10 host=dash0 gpu=0 cpus=0-19 config=C10 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=confirm-C10 host=dash0 gpu=0 cpus=0-19 config=C10 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=confirm-C01 host=dash0 gpu=0 cpus=0-19 config=C01 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=confirm-C00 host=dash0 gpu=0 cpus=0-19 config=C00 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=confirm-C11 host=dash0 gpu=0 cpus=0-19 config=C11 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=closure-r1-C00 host=dash0 gpu=0 cpus=0-19 config=C00 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h

runs/opprof-oracle-gap/trials.csv

@@ -0,0 +1,105 @@
+trial_path,phase,config,target_rate_rps,repetition,role,clean_seconds,achieved_offered_rps,cohort_n,pass_n,pass_rate,slo_goodput_rps,offered_rate_valid,schedule_valid,raw_slo_feasible,feasible,exact_output_n,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,schedule_lag_max_ms,schedule_lag_p95_ms,schedule_lag_p99_ms,failure_reasons
+trials/P01-C00/rate-26/rep-0/score.json,P01,C00,26.0,0,primary,60.0,26.0,1560,1560,1.0,26.0,True,True,True,True,1560,110.87747401325032,143.8727189670317,159.6376050147228,41.272228730598556,43.98393507913819,44.77031942870882,2.3002910893410444,1.6921606729738414,2.0629908540286124,{}
+trials/P01-C00/rate-28/rep-0/score.json,P01,C00,28.0,0,primary,60.0,28.0,1680,1601,0.9529761904761904,26.683333333333334,True,True,True,True,1680,124.47624997003004,159.8520129919052,169.08409597817808,46.3585068253682,49.967933936776326,50.40311877752492,5.177478247787803,1.6857225564308465,2.054154989309609,"{""tpot_slo"":79}"
+trials/P01-C00/rate-28/rep-1/score.json,P01,C00,28.0,1,boundary-confirmation,60.0,28.0,1680,1659,0.9875,27.65,True,True,True,True,1680,123.72062000213191,159.21212401008233,168.65225398214534,46.01303692112514,49.41609907906414,50.04231119066447,2.9356907471083105,1.665140618570149,2.045694272965193,"{""tpot_slo"":21}"
+trials/P01-C00/rate-28/rep-2/score.json,P01,C00,28.0,2,boundary-confirmation,60.0,28.0,1680,1645,0.9791666666666666,27.416666666666668,True,True,True,True,1680,123.91230202047154,158.21082500042394,168.29883796162903,46.164931619488115,49.66844036521035,50.222261873487795,2.195626962929964,1.626886718440801,2.0519818062894046,"{""tpot_slo"":35}"
+trials/P01-C00/rate-30/rep-0/score.json,P01,C00,30.0,0,primary,60.0,30.0,1800,307,0.17055555555555554,5.116666666666666,True,True,False,False,1800,138.99163604946807,175.20886095007882,186.90624000737444,51.83528874638594,54.82435365122492,55.49848122182228,3.6000446416437626,1.7311419360339642,2.0488349837251008,"{""tpot_slo"":1493}"
+trials/P01-C00/rate-30/rep-1/score.json,P01,C00,30.0,1,boundary-confirmation,60.0,30.0,1800,479,0.26611111111111113,7.983333333333333,True,True,False,False,1800,137.74755800841376,174.40702999010682,187.4343649833463,51.48036580898666,54.92729398362812,55.44106577733709,4.635824996512383,1.635887660086155,2.0532096386887133,"{""tpot_slo"":1321}"
+trials/P01-C00/rate-30/rep-2/score.json,P01,C00,30.0,2,boundary-confirmation,60.0,30.0,1800,414,0.23,6.9,True,True,False,False,1800,138.20564799243584,174.3248249986209,184.11366897635162,51.62530752371938,55.15781777729798,55.70552152367161,2.614296681713313,1.703035377431661,2.073209674563259,"{""tpot_slo"":1386}"
+trials/P01-C00/rate-32/rep-0/score.json,P01,C00,32.0,0,primary,60.0,32.0,1920,87,0.0453125,1.45,True,True,False,False,1920,155.5941189872101,193.93265299731866,211.04579704115167,57.9167679839191,62.15287795260785,62.6541800161321,3.307300037704408,1.723894034512341,2.1944420295767486,"{""tpot_slo"":1833}"
+trials/P01-C00/rate-34/rep-0/score.json,P01,C00,34.0,0,primary,60.0,34.0,2040,88,0.043137254901960784,1.4666666666666666,True,True,False,False,2040,176.93204001989216,221.90839203540236,237.5196199864149,64.94585336518607,69.83023355548669,71.09916014301162,3.92269337316975,1.7322570784017444,2.3150334018282592,"{""tpot_slo"":1952}"
+trials/P01-C00/rate-36/rep-0/score.json,P01,C00,36.0,0,primary,60.0,36.0,2160,81,0.0375,1.35,True,True,False,False,2160,201.2797609786503,249.29071601945907,267.4241259810515,74.56418415919568,79.37573587303864,80.72255477715757,4.510630737058818,2.05585197545588,2.8801406733691692,"{""tpot_slo"":2079}"
+trials/P01-C01/rate-26/rep-0/score.json,P01,C01,26.0,0,primary,60.0,26.0,1560,1560,1.0,26.0,True,True,True,True,1560,109.35463797068223,141.4266600040719,156.78152500186116,40.75073330138352,43.68351201530516,44.807886191096806,2.4665576056577265,1.7485098796896636,2.0897933281958103,{}
+trials/P01-C01/rate-28/rep-0/score.json,P01,C01,28.0,0,primary,60.0,28.0,1680,1632,0.9714285714285714,27.2,True,True,True,True,1680,124.94011997478083,159.36688397778198,166.59722500480711,46.1645503168083,49.64453539726073,50.473119238472826,2.1891174255870283,1.6197741497308016,2.002480032388121,"{""tpot_slo"":48}"
+trials/P01-C01/rate-28/rep-1/score.json,P01,C01,28.0,1,boundary-confirmation,60.0,28.0,1680,1588,0.9452380952380952,26.466666666666665,True,True,False,False,1680,123.78435401478782,159.4017290044576,168.0457249749452,46.30296269804978,50.03878576237531,50.66540757090681,2.3081725812517107,1.547503750771284,2.0317512680776417,"{""tpot_slo"":92}"
+trials/P01-C01/rate-28/rep-2/score.json,P01,C01,28.0,2,boundary-confirmation,60.0,28.0,1680,1669,0.993452380952381,27.816666666666666,True,True,True,True,1680,123.87456797296181,159.58094800589606,168.1017209775746,46.167882237886445,49.62505357125626,49.97083171370572,2.81829503364861,1.630675047636032,2.110089873895049,"{""tpot_slo"":11}"
+trials/P01-C01/rate-30/rep-0/score.json,P01,C01,30.0,0,primary,60.0,30.0,1800,406,0.22555555555555556,6.766666666666667,True,True,False,False,1800,138.7023749994114,174.91436598356813,186.39138899743557,51.529372444502,55.032949429005384,55.51141852305995,3.2389176194556057,1.7149029881693423,2.075109339784831,"{""tpot_slo"":1394}"
+trials/P01-C01/rate-30/rep-1/score.json,P01,C01,30.0,1,boundary-confirmation,60.0,30.0,1800,383,0.2127777777777778,6.383333333333334,True,True,False,False,1800,139.0131320222281,174.85803802264854,185.4456909932196,51.60883849205833,55.771121270178504,56.22731407989733,2.5585776893422008,1.7057106597349048,2.1022360306233168,"{""tpot_slo"":1417}"
+trials/P01-C01/rate-30/rep-2/score.json,P01,C01,30.0,2,boundary-confirmation,60.0,30.0,1800,364,0.20222222222222222,6.066666666666666,True,True,False,False,1800,139.08331200946122,174.3557599838823,184.92947099730372,51.89378812746514,55.43139969767441,56.029993317289545,3.436596365645528,1.6893130377866328,2.101277350448072,"{""tpot_slo"":1436}"
+trials/P01-C01/rate-32/rep-0/score.json,P01,C01,32.0,0,primary,60.0,32.0,1920,89,0.04635416666666667,1.4833333333333334,True,True,False,False,1920,154.02632602490485,190.3878870070912,210.70248901378363,57.73119722208422,62.29816603156487,62.87742125347168,2.908719005063176,1.639701018575579,2.0793279982171953,"{""tpot_slo"":1831}"
+trials/P01-C01/rate-34/rep-0/score.json,P01,C01,34.0,0,primary,60.0,34.0,2040,90,0.04411764705882353,1.5,True,True,False,False,2040,176.2790570501238,220.11423198273405,235.4159569949843,64.42572111134521,69.2640547769972,70.48677477712137,4.586904076859355,1.7256339197047055,2.46569694718346,"{""tpot_slo"":1950}"
+trials/P01-C01/rate-36/rep-0/score.json,P01,C01,36.0,0,primary,60.0,36.0,2160,82,0.03796296296296296,1.3666666666666667,True,True,False,False,2160,200.13104798272252,246.79395399289206,266.32684899959713,74.13756723852,78.38002976218593,79.73950822183508,6.9154552184045315,2.069939684588462,2.8538682381622493,"{""tpot_slo"":2078}"
+trials/P01-C10/rate-24/rep-0/score.json,P01,C10,24.0,0,primary-extension,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,97.52241603564471,116.05227499967441,122.57484695874155,37.891577190113445,40.75966766639982,41.13237590469893,3.6939186975359917,1.7255899729207158,2.0753800054080784,{}
+trials/P01-C10/rate-24/rep-1/score.json,P01,C10,24.0,1,boundary-confirmation,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,98.17885496886447,118.82886500097811,125.91037503443658,38.32998042852278,40.7927984765376,41.027600936118574,2.9292047256603837,1.768625690601766,2.1016597165726125,{}
+trials/P01-C10/rate-24/rep-2/score.json,P01,C10,24.0,2,boundary-confirmation,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,97.96303004259244,126.96650595171377,137.19069201033562,38.67084376277432,41.26724865036233,41.506455126514155,2.5533746811561286,1.722060958854854,2.077668672427535,{}
+trials/P01-C10/rate-26/rep-0/score.json,P01,C10,26.0,0,primary,60.0,26.0,1560,0,0.0,0.0,True,True,False,False,1560,3864.615537051577,4926.1638119933195,5062.709584017284,39.198912619169626,40.58124177760282,41.15692004820125,2.5109824491664767,1.7559383413754404,2.064570668153465,"{""ttft_slo"":1560}"
+trials/P01-C10/rate-26/rep-1/score.json,P01,C10,26.0,1,boundary-confirmation,60.0,26.0,1560,1560,1.0,26.0,True,True,True,True,1560,252.71200499264523,497.90483101969585,632.6556759886444,37.49335380936308,39.14127271393284,39.90859952330264,3.0049763154238462,1.7557168612256646,2.0546196028590202,{}
+trials/P01-C10/rate-26/rep-2/score.json,P01,C10,26.0,2,boundary-confirmation,60.0,26.0,1560,0,0.0,0.0,True,True,False,False,1560,3994.878589990549,4745.065895025618,4836.570041952655,38.89002274659033,40.17117711168433,40.39965068278391,2.420338918454945,1.6692215576767921,2.05352931516245,"{""ttft_slo"":1560}"
+trials/P01-C10/rate-28/rep-0/score.json,P01,C10,28.0,0,primary,60.0,26.083333333333332,1565,0,0.0,0.0,False,False,False,False,1565,7570.850218005944,7743.26098500751,7767.141760967206,38.62161282542354,39.79506171382372,40.20180085688711,4137.868300836999,3841.5666787186638,4090.4739925754257,"{""ttft_slo"":1565}"
+trials/P01-C10/rate-30/rep-0/score.json,P01,C10,30.0,0,primary,60.0,26.066666666666666,1564,0,0.0,0.0,False,False,False,False,1564,7471.911213011481,7596.271771006286,7660.170332994312,37.75816839761175,38.96897019068193,39.542113761565396,9581.972879357636,9184.90837601712,9528.559824393597,"{""ttft_slo"":1564}"
+trials/P01-C10/rate-32/rep-0/score.json,P01,C10,32.0,0,primary,60.0,27.733333333333334,1664,0,0.0,0.0,False,False,False,False,1664,6994.684477976989,7170.971911051311,7199.6434789616615,34.7310509680519,36.133673142767435,36.555938174142426,9920.33471504692,9432.675201038364,9748.585979046766,"{""ttft_slo"":1664}"
+trials/P01-C10/rate-34/rep-0/score.json,P01,C10,34.0,0,primary,60.0,26.666666666666668,1600,0,0.0,0.0,False,False,False,False,1600,7292.200801020954,7465.953706996515,7510.689996997826,36.594293714087044,37.8891855555897,38.275920475522675,19949.719168245792,19317.845753917936,19914.817562676035,"{""ttft_slo"":1600}"
+trials/P01-C10/rate-36/rep-0/score.json,P01,C10,36.0,0,primary,60.0,26.966666666666665,1618,0,0.0,0.0,False,False,False,False,1618,7226.758364005946,7377.549752010964,7406.953382014763,36.146196222583214,37.36288985699445,37.69975279398736,24686.17197702406,24000.297317747027,24624.147864291444,"{""ttft_slo"":1618}"
+trials/P01-C11/rate-24/rep-0/score.json,P01,C11,24.0,0,primary-extension,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,102.60815400397405,147.0231090206653,167.74030699161813,39.74406866704128,41.34076804805931,42.04290312629873,2.937526674941182,1.6711382777430117,2.0573336514644325,{}
+trials/P01-C11/rate-24/rep-1/score.json,P01,C11,24.0,1,boundary-confirmation,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,97.34877798473462,117.04645899590105,125.5875930073671,37.771026253707646,40.67583511110454,41.213303270365394,5.921298987232149,1.7458907095715404,2.079935686197132,{}
+trials/P01-C11/rate-24/rep-2/score.json,P01,C11,24.0,2,boundary-confirmation,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,96.28819499630481,115.87436601985246,122.07344203488901,37.68616379423451,40.112987238292895,40.710791253433044,3.3418433158658445,1.6138353385031223,2.0559499971568584,{}
+trials/P01-C11/rate-26/rep-0/score.json,P01,C11,26.0,0,primary,60.0,25.783333333333335,1547,0,0.0,0.0,True,True,False,False,1547,6533.619961992372,7947.257625986822,7982.082434988115,40.18144979370788,41.07262126986854,42.167470412557975,573.5937684075907,386.3291516317986,548.7673947936855,"{""ttft_slo"":1547}"
+trials/P01-C11/rate-26/rep-1/score.json,P01,C11,26.0,1,boundary-confirmation,60.0,26.0,1560,1560,1.0,26.0,True,True,True,True,1560,593.4362710104324,827.8850860078819,934.9126719753258,37.31267711162449,38.505050603167284,38.729826237873304,2.241449663415551,1.7251850222237408,2.0647295750677586,{}
+trials/P01-C11/rate-26/rep-2/score.json,P01,C11,26.0,2,boundary-confirmation,60.0,26.0,1560,0,0.0,0.0,True,True,False,False,1560,5812.711999984458,6669.004526047502,6752.215686021373,39.62677446035077,41.18544706291268,41.53003749190017,2.1458613919094205,1.6790343215689063,2.039063081610948,"{""ttft_slo"":1560}"
+trials/P01-C11/rate-28/rep-0/score.json,P01,C11,28.0,0,primary,60.0,26.233333333333334,1574,0,0.0,0.0,False,False,False,False,1574,7565.811808046419,7749.4256109930575,7759.166635980364,38.48338009519798,39.82913404765968,40.06808598355819,3845.5762091325596,3568.3553874259815,3815.4344389913604,"{""ttft_slo"":1574}"
+trials/P01-C11/rate-30/rep-0/score.json,P01,C11,30.0,0,primary,60.0,26.1,1566,0,0.0,0.0,False,False,False,False,1566,7462.299443024676,7605.860833020415,7636.226687987801,37.638444031241335,38.706122920663645,39.0103649207583,9453.580788627733,9022.533191659022,9404.877326625865,"{""ttft_slo"":1566}"
+trials/P01-C11/rate-32/rep-0/score.json,P01,C11,32.0,0,primary,60.0,28.25,1695,0,0.0,0.0,False,False,False,False,1695,6933.729738055263,7114.611251046881,7164.560960023664,34.169790237986795,35.679708412360576,36.14399266717512,9283.821924007498,8650.737509015016,9040.9968290478,"{""ttft_slo"":1695}"
+trials/P01-C11/rate-34/rep-0/score.json,P01,C11,34.0,0,primary,60.0,26.6,1596,0,0.0,0.0,False,False,False,False,1596,7322.663314000238,7463.087519980036,7493.740423000418,36.76072112654173,38.06862412648837,39.04326098401927,20215.17994365422,19248.93292755587,19825.970116246026,"{""ttft_slo"":1596}"
+trials/P01-C11/rate-36/rep-0/score.json,P01,C11,36.0,0,primary,60.0,27.15,1629,0,0.0,0.0,False,False,False,False,1629,7185.944458993617,7345.619179017376,7373.497121967375,35.93770284131761,37.092669237782026,37.386540825190465,23960.0648356718,23284.954287286382,23910.954526276328,"{""ttft_slo"":1629}"
+trials/P06-C00/rate-1.4/rep-0/score.json,P06,C00,1.4,0,primary,120.0,1.4,168,168,1.0,1.4,True,True,True,True,168,999.2175319930539,1855.332705017645,2123.494433995802,14.184352559645326,17.348588929546235,18.162435246563156,7.325362705159932,6.399379402864724,7.314503251109272,{}
+trials/P06-C00/rate-1.5/rep-0/score.json,P06,C00,1.5,0,primary,120.0,1.4666666666666666,176,176,1.0,1.4666666666666666,True,True,True,True,176,993.2098449789919,1804.5693339663558,2139.430449984502,14.179927917742289,18.19816446378317,20.127085575295354,7.586089021060616,6.320645683445036,7.3406120063737035,{}
+trials/P06-C00/rate-1.6/rep-0/score.json,P06,C00,1.6,0,primary,120.0,1.6,192,192,1.0,1.6,True,True,True,True,192,997.0329970237799,1733.372846036218,2134.9421920021996,16.06630710572817,21.62564592766293,22.13207139725179,10.16304298536852,7.14888097718358,9.413381980266422,{}
+trials/P06-C00/rate-1.7/rep-0/score.json,P06,C00,1.7,0,primary,120.0,1.7333333333333334,208,208,1.0,1.7333333333333334,True,True,True,True,208,999.191242037341,1732.8481450094841,2074.4936399860308,18.194802745597148,22.597885336599326,24.458052634116655,7.205226575024426,6.399967067409307,6.824508309364319,{}
+trials/P06-C00/rate-1.8/rep-0/score.json,P06,C00,1.8,0,primary,120.0,1.8,216,215,0.9953703703703703,1.7916666666666667,True,True,True,True,216,1004.904159985017,1762.3687379527837,2076.484896009788,20.300264271963098,23.445108162404136,23.846686213302796,7.712147431448102,6.571699457708746,7.433776569087058,"{""ttft_slo"":1}"
+trials/P06-C00/rate-1.9/rep-0/score.json,P06,C00,1.9,0,primary,120.0,1.8666666666666667,224,223,0.9955357142857143,1.8583333333333334,True,True,True,True,224,1008.6866070050746,1725.8416549884714,2094.005164981354,22.55910544618855,27.32792746370436,28.425333408958693,7.935012050438672,6.445528415497392,7.354569446761161,"{""ttft_slo"":1}"
+trials/P06-C00/rate-2/rep-0/score.json,P06,C00,2.0,0,primary,120.0,2.0,240,237,0.9875,1.975,True,True,True,True,240,1008.1276059499942,1844.6353260078467,2217.431389959529,26.691506409000628,33.19407374552067,34.29328832297391,8.592377009335905,6.834403029642999,8.295246050693095,"{""ttft_slo"":3}"
+trials/P06-C00/rate-2.1/rep-0/score.json,P06,C00,2.1,0,primary-extension,120.0,2.1333333333333333,256,253,0.98828125,2.1083333333333334,True,True,True,True,256,1021.1376319639385,1874.3867019657046,2236.3475980237126,30.77564142275795,34.25728149906643,35.95762665954431,9.336929477285594,7.481765991542488,8.548939367756248,"{""ttft_slo"":3}"
+trials/P06-C00/rate-2.2/rep-0/score.json,P06,C00,2.2,0,primary-extension,120.0,2.2,264,260,0.9848484848484849,2.1666666666666665,True,True,True,True,264,1016.4398739580065,1892.5629690056667,2254.457817005459,33.431356234900626,38.75392900199817,40.532795230971026,8.67525755893439,6.830454338341951,8.405035536270589,"{""ttft_slo"":4}"
+trials/P06-C00/rate-2.3/rep-0/score.json,P06,C00,2.3,0,primary-extension,120.0,2.2666666666666666,272,268,0.9852941176470589,2.2333333333333334,True,True,True,True,272,1028.8451670203358,1908.7893930263817,2253.68108501425,37.45989758911344,42.8498864755738,44.0359943385287,8.370409661438316,6.645695248153061,8.179216703865677,"{""ttft_slo"":4}"
+trials/P06-C00/rate-2.3/rep-1/score.json,P06,C00,2.3,1,boundary-closure,120.0,2.2666666666666666,272,268,0.9852941176470589,2.2333333333333334,True,True,True,True,272,1056.8546410067938,1936.134911957197,2271.4714580215514,38.52701387477496,43.82504283363062,44.65749040118431,9.594396688044071,6.700532685499638,7.922522665467113,"{""ttft_slo"":4}"
+trials/P06-C00/rate-2.3/rep-2/score.json,P06,C00,2.3,2,boundary-closure,120.0,2.2666666666666666,272,268,0.9852941176470589,2.2333333333333334,True,True,True,True,272,1023.681657970883,2005.0558719667606,2281.9403469911776,37.73080680230924,44.07464489231429,44.84600109586295,9.688543679658324,6.723627564497292,9.51991870533675,"{""ttft_slo"":4}"
+trials/P06-C00/rate-2.4/rep-0/score.json,P06,C00,2.4,0,primary-extension,120.0,2.4,288,274,0.9513888888888888,2.283333333333333,True,True,True,True,288,1047.7718930342235,1939.0789800090715,2299.159614020027,44.111866635974835,49.369580082211705,51.03415431313074,8.399292710237205,6.617546721827239,8.075028716120869,"{""tpot_slo"":10,""ttft_slo"":4}"
+trials/P06-C00/rate-2.4/rep-1/score.json,P06,C00,2.4,1,boundary-confirmation,120.0,2.4,288,271,0.9409722222222222,2.2583333333333333,True,True,False,False,288,1038.3890920202248,1913.3997529861517,2282.3066530399956,44.10149966145149,49.90086417213272,52.977919430584045,8.468122687190771,6.662778672762215,7.901694276370108,"{""tpot_slo"":13,""ttft_slo"":4}"
+trials/P06-C00/rate-2.4/rep-2/score.json,P06,C00,2.4,2,boundary-confirmation,120.0,2.4,288,273,0.9479166666666666,2.275,True,True,False,False,288,1038.4550529997796,1935.180893051438,2284.888348018285,44.084652043025166,49.2430650137015,53.145144203447224,10.83466998534277,7.184286660049111,9.758414002135396,"{""tpot_slo"":11,""ttft_slo"":4}"
+trials/P06-C00/rate-2.5/rep-0/score.json,P06,C00,2.5,0,primary-extension,120.0,2.533333333333333,304,171,0.5625,1.425,True,True,False,False,304,1042.2218869789504,2067.6334840245545,2326.744472957216,48.563286011679544,54.620607958950735,57.85391967308046,8.668092952575535,6.844838964752853,7.631328015122563,"{""tpot_slo"":132,""ttft_slo"":5}"
+trials/P06-C00/rate-2.5/rep-1/score.json,P06,C00,2.5,1,boundary-confirmation,120.0,2.533333333333333,304,163,0.5361842105263158,1.3583333333333334,True,True,False,False,304,1058.9527579722926,2087.2758819605224,2304.234509996604,48.46152181804559,55.69985224658062,58.651440686896194,9.696525987237692,6.925722060259432,8.803103992249817,"{""tpot_slo"":139,""ttft_slo"":5}"
+trials/P06-C00/rate-2.5/rep-2/score.json,P06,C00,2.5,2,boundary-confirmation,120.0,2.533333333333333,304,155,0.5098684210526315,1.2916666666666667,True,True,False,False,304,1049.7184470295906,2060.312128975056,2316.979594004806,49.96833974953901,55.53648186496271,58.368578595004486,8.793669985607266,7.0653450093232095,8.415801043156534,"{""tpot_slo"":148,""ttft_slo"":5}"
+trials/P06-C01/rate-1.4/rep-0/score.json,P06,C01,1.4,0,primary,120.0,1.4,168,166,0.9880952380952381,1.3833333333333333,True,True,True,True,168,870.9071970079094,2094.233661016915,2450.609703955706,15.16279330330859,18.223098260277624,18.65344091000513,7.756370992865413,6.575752224307507,7.233251410070807,"{""ttft_slo"":2}"
+trials/P06-C01/rate-1.5/rep-0/score.json,P06,C01,1.5,0,primary,120.0,1.4666666666666666,176,174,0.9886363636363636,1.45,True,True,True,True,176,856.920883001294,2077.852713991888,2404.4516870053485,15.114829923712453,21.891871407088583,23.431140612450896,7.3857519892044365,6.386720982845873,7.349602004978806,"{""ttft_slo"":2}"
+trials/P06-C01/rate-1.6/rep-0/score.json,P06,C01,1.6,0,primary,120.0,1.6,192,190,0.9895833333333334,1.5833333333333333,True,True,True,True,192,877.8399180155247,1984.3015700462274,2398.247136035934,17.95897510373389,23.491408933458008,24.43991283751266,7.940459006931633,6.813998974394053,7.770671974867582,"{""ttft_slo"":2}"
+trials/P06-C01/rate-1.7/rep-0/score.json,P06,C01,1.7,0,primary,120.0,1.7333333333333334,208,206,0.9903846153846154,1.7166666666666666,True,True,True,True,208,865.2636900078505,1968.6543480493128,2176.717725000344,20.55894938951865,24.509686565572494,25.151764009755702,7.413975603412837,6.295370752923191,7.269357622135431,"{""ttft_slo"":2}"
+trials/P06-C01/rate-1.8/rep-0/score.json,P06,C01,1.8,0,primary,120.0,1.8,216,212,0.9814814814814815,1.7666666666666666,True,True,True,True,216,884.8230779985897,2010.2625639992766,2270.7025869749486,22.31720594712943,29.147994661486823,30.879378019546845,10.790552652906626,6.586181407328695,7.714397390373051,"{""ttft_slo"":4}"
+trials/P06-C01/rate-1.9/rep-0/score.json,P06,C01,1.9,0,primary,120.0,1.8666666666666667,224,220,0.9821428571428571,1.8333333333333333,True,True,True,True,224,871.5889130253345,2014.9384770193137,2264.04358696891,26.832327434502417,30.454946645747857,31.631232761352372,7.387189427390695,6.136250507552177,7.152303704060614,"{""ttft_slo"":4}"
+trials/P06-C01/rate-2/rep-0/score.json,P06,C01,2.0,0,primary,120.0,2.0,240,233,0.9708333333333333,1.9416666666666667,True,True,True,True,240,872.5066980114207,2134.159804030787,2552.078590961173,28.788431911978055,35.65545248732503,36.88229199800979,7.818170997779816,6.43993797712028,7.529255992267281,"{""ttft_slo"":7}"
+trials/P06-C01/rate-2.1/rep-0/score.json,P06,C01,2.1,0,primary-extension,120.0,2.1333333333333333,256,245,0.95703125,2.0416666666666665,True,True,True,True,256,917.0198910287581,2213.4800929925404,2593.613261007704,34.30701432673993,40.38027477887625,41.43307809392173,10.985759610775858,7.624904450494796,9.49472957290709,"{""ttft_slo"":11}"
+trials/P06-C01/rate-2.2/rep-0/score.json,P06,C01,2.2,0,primary-extension,120.0,2.2,264,252,0.9545454545454546,2.1,True,True,True,True,264,945.8785869646817,2318.5488900053315,2714.1117680002935,40.473415896204465,47.044261179954624,48.30041582584053,12.640100321732461,7.256975513882935,8.479340351186693,"{""ttft_slo"":12}"
+trials/P06-C01/rate-2.2/rep-1/score.json,P06,C01,2.2,1,boundary-confirmation,120.0,2.2,264,254,0.9621212121212122,2.1166666666666667,True,True,True,True,264,937.7531060017645,2288.793044979684,2689.1637159860693,39.88727447547205,44.97671687867084,46.09535447171016,8.372949319891632,6.535524502396584,7.7698222594335675,"{""ttft_slo"":10}"
+trials/P06-C01/rate-2.2/rep-2/score.json,P06,C01,2.2,2,boundary-confirmation,120.0,2.2,264,251,0.9507575757575758,2.091666666666667,True,True,True,True,264,943.6753669870086,2305.475205008406,2666.3335349876434,40.217721972609056,46.534055287730126,47.48751569861444,9.360338270198554,6.640697305556387,8.254165295511484,"{""ttft_slo"":13}"
+trials/P06-C01/rate-2.3/rep-0/score.json,P06,C01,2.3,0,primary-extension,120.0,2.2666666666666666,272,222,0.8161764705882353,1.85,True,True,False,False,272,971.3688230258413,2346.123350027483,2757.595370989293,45.034927745675944,51.62063973977225,55.295872516609336,8.836901630274951,7.390526297967881,8.478775678668171,"{""tpot_slo"":35,""ttft_slo"":16}"
+trials/P06-C01/rate-2.3/rep-1/score.json,P06,C01,2.3,1,boundary-confirmation,120.0,2.2666666666666666,272,232,0.8529411764705882,1.9333333333333333,True,True,False,False,272,963.1062899716198,2335.012650990393,2770.2474140096456,45.059407203469185,50.939032240677804,53.42882929345659,10.456104762852192,7.382042706012726,8.998437726404518,"{""tpot_slo"":25,""ttft_slo"":16}"
+trials/P06-C01/rate-2.3/rep-2/score.json,P06,C01,2.3,2,boundary-confirmation,120.0,2.2666666666666666,272,219,0.8051470588235294,1.825,True,True,False,False,272,968.3994479710236,2338.2130070240237,2772.5177410175093,45.12062682583381,51.60208501754631,55.46992149902508,9.490319702308625,6.954058830160648,7.915747701190412,"{""tpot_slo"":39,""ttft_slo"":15}"
+trials/P06-C10/rate-1.4/rep-0/score.json,P06,C10,1.4,0,primary,120.0,1.4,168,167,0.9940476190476191,1.3916666666666666,True,True,True,True,168,989.1352539998479,1854.9544009729289,2126.261135970708,14.310991064619834,17.33015237377005,18.133633745558114,8.03026300854981,6.68573915027082,7.717043161392212,"{""ttft_slo"":1}"
+trials/P06-C10/rate-1.5/rep-0/score.json,P06,C10,1.5,0,primary,120.0,1.4666666666666666,176,176,1.0,1.4666666666666666,True,True,True,True,176,989.0364960301667,1836.014078988228,2159.4515729811974,14.38420469275313,18.27264551281539,20.49906990411573,7.232828997075558,6.385942979250103,7.173349673394114,{}
+trials/P06-C10/rate-1.6/rep-0/score.json,P06,C10,1.6,0,primary,120.0,1.6,192,192,1.0,1.6,True,True,True,True,192,1005.6778370053507,1733.1810360192321,2129.3352029751986,16.17475541492144,21.547753428548223,22.38600313688087,11.067131999880075,6.535599008202553,10.750612011179328,{}
+trials/P06-C10/rate-1.7/rep-0/score.json,P06,C10,1.7,0,primary,120.0,1.7333333333333334,208,208,1.0,1.7333333333333334,True,True,True,True,208,1005.7410600129515,1793.3206979651004,2149.9399559688754,18.121018011802754,22.653281632191163,23.578543808204603,8.620336942840368,6.635099765844643,7.54233862971887,{}
+trials/P06-C10/rate-1.8/rep-0/score.json,P06,C10,1.8,0,primary,120.0,1.8,216,215,0.9953703703703703,1.7916666666666667,True,True,True,True,216,1003.2750739483163,1837.6042150193825,2153.3647059695795,20.33729544431135,23.54741711743584,24.420280583155453,7.7063715434633195,6.210717547219247,7.363768992945552,"{""ttft_slo"":1}"
+trials/P06-C10/rate-1.9/rep-0/score.json,P06,C10,1.9,0,primary,120.0,1.8666666666666667,224,223,0.9955357142857143,1.8583333333333334,True,True,True,True,224,1028.2902300241403,1734.957567998208,2139.568875019904,23.068141569465457,28.3249747396516,29.331570328873948,10.697210382204503,6.612500932533294,8.539305708836764,"{""ttft_slo"":1}"
+trials/P06-C10/rate-2/rep-0/score.json,P06,C10,2.0,0,primary,120.0,2.0,240,237,0.9875,1.975,True,True,True,True,240,1015.3626359533519,1876.5030350186862,2216.4827240048908,26.61050810754744,32.98075636598207,34.85392479450622,14.75413900334388,7.143530005123466,12.002518982626498,"{""ttft_slo"":3}"
+trials/P06-C10/rate-2.1/rep-0/score.json,P06,C10,2.1,0,primary-extension,120.0,2.1333333333333333,256,252,0.984375,2.1,True,True,True,True,256,1017.7665610099211,1897.7975359885022,2233.356000040658,29.193307035247233,34.04904124857821,35.716844309251684,8.122143452055752,7.0329742738977075,8.025550458114594,"{""ttft_slo"":4}"
+trials/P06-C10/rate-2.2/rep-0/score.json,P06,C10,2.2,0,primary-extension,120.0,2.2,264,260,0.9848484848484849,2.1666666666666665,True,True,True,True,264,1028.838190017268,1924.7021020273678,2267.4863139982335,34.656678156589145,39.05688540715225,40.56380550300068,8.019849366974086,6.283827999141067,7.548786816187203,"{""ttft_slo"":4}"
+trials/P06-C10/rate-2.3/rep-0/score.json,P06,C10,2.3,0,primary-extension,120.0,2.2666666666666666,272,268,0.9852941176470589,2.2333333333333334,True,True,True,True,272,1030.823885987047,2007.8507559956051,2252.5435159914196,37.69898847546957,43.23842918587786,44.374794215250965,8.239199698437005,6.477339018601924,7.5856699841097,"{""ttft_slo"":4}"
+trials/P06-C10/rate-2.4/rep-0/score.json,P06,C10,2.4,0,primary-extension,120.0,2.4,288,274,0.9513888888888888,2.283333333333333,True,True,True,True,288,1034.7871870035306,1914.081139024347,2285.277316987049,43.703152659488566,49.364211770989286,50.49469348336178,8.994451723992825,6.490265019237995,8.65559169324115,"{""tpot_slo"":10,""ttft_slo"":4}"
+trials/P06-C10/rate-2.4/rep-1/score.json,P06,C10,2.4,1,boundary-confirmation,120.0,2.4,288,278,0.9652777777777778,2.316666666666667,True,True,True,True,288,1042.35012200661,1931.6510139615275,2304.684264003299,45.058667197629426,49.32953689043244,50.9193064501407,8.361130661796778,6.718640972394496,8.037615334615111,"{""tpot_slo"":6,""ttft_slo"":4}"
+trials/P06-C10/rate-2.4/rep-2/score.json,P06,C10,2.4,2,boundary-confirmation,120.0,2.4,288,279,0.96875,2.325,True,True,True,True,288,1022.7319020195864,1937.5082700280473,2259.4223510241136,43.80097978273607,48.74295069858406,50.16730903327071,9.690148697700351,7.029823667835444,7.890908687841147,"{""tpot_slo"":5,""ttft_slo"":4}"
+trials/P06-C10/rate-2.5/rep-0/score.json,P06,C10,2.5,0,primary-extension,120.0,2.533333333333333,304,95,0.3125,0.7916666666666666,True,True,False,False,304,2603.841262985952,4117.584208026528,4780.076910043135,48.75482211150083,52.61768415659564,53.111906254408,9.685411059763283,6.941412982996553,8.32790503045544,"{""tpot_slo"":93,""ttft_slo"":129}"
+trials/P06-C10/rate-2.5/rep-1/score.json,P06,C10,2.5,1,boundary-confirmation,120.0,2.533333333333333,304,103,0.33881578947368424,0.8583333333333333,True,True,False,False,304,2862.704548984766,4183.153211022727,4453.049428993836,48.63343907633173,52.460244035260814,52.98061303326445,8.852682018186897,6.924911984242499,8.437061042059213,"{""tpot_slo"":78,""ttft_slo"":138}"
+trials/P06-C10/rate-2.5/rep-2/score.json,P06,C10,2.5,2,boundary-confirmation,120.0,2.533333333333333,304,65,0.2138157894736842,0.5416666666666666,True,True,False,False,304,3408.5888129775412,4655.079743999522,4986.388145014644,48.67060376908047,52.2114794285341,52.96427048145886,10.025584022514522,7.519927050452679,9.58072004141286,"{""tpot_slo"":82,""ttft_slo"":197}"
+trials/P06-C11/rate-1.4/rep-0/score.json,P06,C11,1.4,0,primary,120.0,1.4,168,166,0.9880952380952381,1.3833333333333333,True,True,True,True,168,878.8207969628274,2104.5628290157765,2456.5626359544694,15.149171608582197,18.309408667248558,18.975726387453484,7.393979234620929,6.200018804520369,7.383851974736899,"{""ttft_slo"":2}"
+trials/P06-C11/rate-1.5/rep-0/score.json,P06,C11,1.5,0,primary,120.0,1.4666666666666666,176,174,0.9886363636363636,1.45,True,True,True,True,176,868.387623981107,2078.5744480090216,2430.6161800050177,15.37368986883188,21.409829477497073,22.96634120153584,9.10428969655186,6.963004008866847,8.968600712250918,"{""ttft_slo"":2}"
+trials/P06-C11/rate-1.6/rep-0/score.json,P06,C11,1.6,0,primary,120.0,1.6,192,190,0.9895833333333334,1.5833333333333333,True,True,True,True,192,880.2933589904569,1996.0739740054123,2426.3907080166973,19.148470602775586,23.556329107637392,24.3911435792301,12.246161000803113,6.944428023416549,11.784875998273492,"{""ttft_slo"":2}"
+trials/P06-C11/rate-1.7/rep-0/score.json,P06,C11,1.7,0,primary,120.0,1.7333333333333334,208,206,0.9903846153846154,1.7166666666666666,True,True,True,True,208,868.3885979698971,1974.1372629650868,2170.4040309996344,20.554707737693313,24.361233739774683,25.208697178052034,8.172314323019236,6.486376631073654,7.166995259467512,"{""ttft_slo"":2}"
+trials/P06-C11/rate-1.8/rep-0/score.json,P06,C11,1.8,0,primary,120.0,1.8,216,212,0.9814814814814815,1.7666666666666666,True,True,True,True,216,860.0498370360583,2015.8445590059273,2296.1058749933727,22.033081184015582,28.35712986695261,29.818611892387672,8.214946719817817,6.435671588405967,7.979147601872683,"{""ttft_slo"":4}"
+trials/P06-C11/rate-1.9/rep-0/score.json,P06,C11,1.9,0,primary,120.0,1.8666666666666667,224,220,0.9821428571428571,1.8333333333333333,True,True,True,True,224,873.2819089782424,2015.0140480254777,2233.055104035884,26.420655446181733,30.60916253037973,31.97331095106149,7.478082727175206,6.193810375407338,7.1653242921456695,"{""ttft_slo"":4}"
+trials/P06-C11/rate-2/rep-0/score.json,P06,C11,2.0,0,primary,120.0,2.0,240,233,0.9708333333333333,1.9416666666666667,True,True,True,True,240,885.916200990323,2135.891862970311,2545.2551289927214,29.176486246565762,36.21475781799784,37.257001457900834,8.14571394585073,6.473987945355475,7.991001999471337,"{""ttft_slo"":7}"
+trials/P06-C11/rate-2.1/rep-0/score.json,P06,C11,2.1,0,primary-extension,120.0,2.1333333333333333,256,244,0.953125,2.033333333333333,True,True,True,True,256,910.2991609834135,2218.1496039847843,2635.4756229557097,34.070244894377566,40.17389689240907,40.98456636594732,9.201513486914337,6.535229331348091,8.932627330068499,"{""ttft_slo"":12}"
+trials/P06-C11/rate-2.2/rep-0/score.json,P06,C11,2.2,0,primary-extension,120.0,2.2,264,252,0.9545454545454546,2.1,True,True,True,True,264,939.8728320375085,2325.3240960184485,2723.1857150327414,40.232674399146596,46.644165585115516,47.97150479452393,8.847062883432955,7.093118503689766,8.64546129014343,"{""ttft_slo"":12}"
+trials/P06-C11/rate-2.2/rep-1/score.json,P06,C11,2.2,1,boundary-confirmation,120.0,2.2,264,255,0.9659090909090909,2.125,True,True,True,True,264,937.060812022537,2310.2555879740976,2691.6481269872747,39.889533819955155,45.028635876735315,46.05559236791492,8.61785898450762,6.791460502427071,8.322948298882693,"{""ttft_slo"":9}"
+trials/P06-C11/rate-2.2/rep-2/score.json,P06,C11,2.2,2,boundary-confirmation,120.0,2.2,264,252,0.9545454545454546,2.1,True,True,True,True,264,938.2475850288756,2306.774304015562,2683.795469987672,40.23246330137541,46.400342516622956,47.64727340899991,8.344491885509342,6.439735821913928,7.930240361019969,"{""ttft_slo"":12}"
+trials/P06-C11/rate-2.3/rep-0/score.json,P06,C11,2.3,0,primary-extension,120.0,2.2666666666666666,272,203,0.7463235294117647,1.6916666666666667,True,True,False,False,272,1074.6694429544732,2435.8343050116673,2784.457146015484,44.76343049704682,52.034396264262845,54.29683911359874,8.639712177682668,6.902627006638795,8.276157022919506,"{""tpot_slo"":51,""ttft_slo"":19}"
+trials/P06-C11/rate-2.3/rep-1/score.json,P06,C11,2.3,1,boundary-confirmation,120.0,2.2666666666666666,272,180,0.6617647058823529,1.5,True,True,False,False,272,1291.9953179662116,2771.1534849950112,3167.952421004884,47.08591616236825,52.35070534633225,54.310699465772075,13.530849944800138,7.158174004871398,9.898048709146678,"{""tpot_slo"":74,""ttft_slo"":23}"
+trials/P06-C11/rate-2.3/rep-2/score.json,P06,C11,2.3,2,boundary-confirmation,120.0,2.2666666666666666,272,177,0.6507352941176471,1.475,True,True,False,False,272,1149.452324025333,2552.789599983953,2787.1654250193387,45.428387461906325,52.92104801173102,55.13124520753268,11.129230260848999,7.671434723306447,10.892333288211375,"{""tpot_slo"":75,""ttft_slo"":25}"

runs/opprof-phase6/opprof_phase6_client.py

@@ -122,6 +122,11 @@ def run_replay(args: argparse.Namespace, *, warmup: bool) -> dict[str, Any]:
             "tpot_ms": outcome.tpot_ms,
             "completion_tokens": outcome.completion_tokens,
             "completion_tokens_source": outcome.completion_tokens_source,
+            "completed_mono_ns": outcome.completed_mono_ns,
+            "completed_elapsed_s": (
+                (outcome.completed_mono_ns - interval_start_mono_ns) / 1e9
+                if outcome.completed_mono_ns is not None else None
+            ),
             "slo_pass": evaluation.passed,
             "reasons": evaluation.reasons,
             "error": outcome.error,

scripts/collectivespec/summarize_p0_phase_trace.py

@@ -328,11 +328,29 @@ def target_execution_agreement(worker_events: list[dict[str, Any]]) -> dict[str,
         }
 
     metadata_valid_count = 0
+    anchor_local_ranks: dict[int, int] = {}
+    for event in target_events:
+        dp_rank = integer(event.get("data_parallel_rank"))
+        local_rank = integer(event.get("local_rank"))
+        if dp_rank is None or local_rank is None:
+            continue
+        previous = anchor_local_ranks.get(dp_rank)
+        if previous is None or local_rank < previous:
+            anchor_local_ranks[dp_rank] = local_rank
+    anchor_events = [
+        event
+        for event in target_events
+        if integer(event.get("data_parallel_rank")) in anchor_local_ranks
+        and integer(event.get("local_rank"))
+        == anchor_local_ranks[integer(event.get("data_parallel_rank"))]
+    ]
     metadata_padding: list[int] = []
     raw_unequal_count = 0
     for event in target_events:
-        valid, padding, raw_unequal = _target_metadata_check(event)
+        valid, _, _ = _target_metadata_check(event)
         metadata_valid_count += valid
+    for event in anchor_events:
+        _, padding, raw_unequal = _target_metadata_check(event)
         if padding is not None:
             metadata_padding.append(padding)
         raw_unequal_count += bool(raw_unequal)
@@ -352,8 +370,9 @@ def target_execution_agreement(worker_events: list[dict[str, Any]]) -> dict[str,
             "valid_record_count": metadata_valid_count,
             "all_records_valid": len(target_events) > 0
             and metadata_valid_count == len(target_events),
-            "raw_dp_counts_unequal_record_count": raw_unequal_count,
-            "post_coordinate_minus_local_raw_rows": {
+            "anchor_record_count": len(anchor_events),
+            "raw_dp_counts_unequal_anchor_record_count": raw_unequal_count,
+            "per_dp_replica_anchor_post_coordinate_minus_local_raw_rows": {
                 "n": len(metadata_padding),
                 "min": min(metadata_padding) if metadata_padding else None,
                 "max": max(metadata_padding) if metadata_padding else None,

scripts/oracle_gap/__init__.py

@@ -0,0 +1 @@
+"""Oracle-gap experiment helpers."""

scripts/oracle_gap/analyze.py

@@ -0,0 +1,366 @@
+#!/usr/bin/env python3
+"""Score fixed-rate request logs and summarize static-vs-oracle frontiers."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+from collections import Counter, defaultdict
+from pathlib import Path
+from typing import Any, Iterable
+
+
+TARGET_PASS_RATE = 0.95
+TPOT_LIMIT_MS = 50.0
+PHASES = ("P01", "P06")
+CONFIGS = ("C00", "C10", "C01", "C11")
+
+
+def atomic_json(path: Path, value: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(f"{path.name}.tmp.{os.getpid()}")
+    temporary.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n")
+    os.replace(temporary, path)
+
+
+def numeric(values: Iterable[float | int | None]) -> dict[str, Any]:
+    materialized = list(values)
+    finite = [
+        float(value)
+        for value in materialized
+        if value is not None and math.isfinite(float(value))
+    ]
+    return {
+        "n": len(materialized),
+        "finite_n": len(finite),
+        "missing_n": len(materialized) - len(finite),
+        "min": min(finite) if finite else None,
+        "max": max(finite) if finite else None,
+        "distinct_n": len(set(finite)),
+    }
+
+
+def percentile(values: Iterable[float], quantile: float) -> float | None:
+    ordered = sorted(float(value) for value in values)
+    if not ordered:
+        return None
+    index = max(0, min(len(ordered) - 1, math.ceil(quantile * len(ordered)) - 1))
+    return ordered[index]
+
+
+def ttft_limit_ms(input_tokens: int) -> float:
+    if input_tokens <= 4096:
+        return 2000.0
+    if input_tokens <= 32768:
+        return 4000.0
+    return 6000.0
+
+
+def score_trial(
+    request_path: Path,
+    result_path: Path,
+    *,
+    phase: str,
+    config: str,
+    target_rate: float,
+    repetition: int,
+    role: str,
+) -> dict[str, Any]:
+    result = json.loads(result_path.read_text())
+    rows = [json.loads(line) for line in request_path.read_text().splitlines() if line]
+    clean_start = float(result["warmup_seconds"])
+    clean_seconds = float(result["clean_segment_seconds"]) * int(
+        result["num_clean_segments"]
+    )
+    clean_end = clean_start + clean_seconds
+    cohort = [row for row in rows if clean_start <= float(row["admitted_s"]) < clean_end]
+
+    ttft_values: list[float] = []
+    tpot_values: list[float] = []
+    lag_values: list[float] = []
+    reasons: Counter[str] = Counter()
+    passes = 0
+    exact_outputs = 0
+    for row in cohort:
+        lag_ms = (float(row["admitted_s"]) - float(row["scheduled_s"])) * 1000.0
+        lag_values.append(lag_ms)
+        request_reasons: list[str] = []
+        if not bool(row["success"]):
+            request_reasons.append(str(row.get("error_kind") or "request_failed"))
+        first = row.get("first_token_s")
+        if first is None:
+            request_reasons.append("ttft_missing")
+            ttft = None
+        else:
+            ttft = (float(first) - float(row["admitted_s"])) * 1000.0
+            ttft_values.append(ttft)
+            if ttft > ttft_limit_ms(int(row["input_tokens"])):
+                request_reasons.append("ttft_slo")
+        actual = row.get("actual_output_tokens")
+        requested = int(row["requested_output_tokens"])
+        if actual == requested:
+            exact_outputs += 1
+        if first is None or actual is None or int(actual) <= 1:
+            request_reasons.append("tpot_missing")
+            tpot = None
+        else:
+            tpot = (
+                (float(row["completed_s"]) - float(first))
+                * 1000.0
+                / (int(actual) - 1)
+            )
+            tpot_values.append(tpot)
+            if tpot > TPOT_LIMIT_MS:
+                request_reasons.append("tpot_slo")
+        if request_reasons:
+            reasons.update(set(request_reasons))
+        else:
+            passes += 1
+
+    achieved_rate = len(cohort) / clean_seconds if clean_seconds else 0.0
+    pass_rate = passes / len(cohort) if cohort else 0.0
+    max_lag_ms = max(lag_values, default=math.inf)
+    offered_rate_valid = (
+        target_rate > 0 and abs(achieved_rate / target_rate - 1.0) <= 0.05
+    )
+    schedule_valid = bool(cohort) and max_lag_ms <= 1000.0
+    raw_feasible = pass_rate >= TARGET_PASS_RATE
+    feasible = raw_feasible and offered_rate_valid and schedule_valid
+    invariants = {
+        "cohort_nonempty": bool(cohort),
+        "clean_duration_positive": clean_seconds > 0,
+        "timestamps_nondecreasing": all(
+            float(row["scheduled_s"]) <= float(row["admitted_s"])
+            <= float(row["completed_s"])
+            for row in cohort
+        ),
+        "exact_output_or_failed": all(
+            (not bool(row["success"]))
+            or row.get("actual_output_tokens") == row.get("requested_output_tokens")
+            for row in cohort
+        ),
+        "latencies_nonnegative": all(value >= 0 for value in ttft_values + tpot_values),
+        "pass_rate_in_0_1": 0.0 <= pass_rate <= 1.0,
+        "goodput_nonnegative": passes >= 0,
+    }
+    if not all(invariants.values()):
+        raise RuntimeError(f"trial data invariant failed: {invariants}")
+    return {
+        "schema": 1,
+        "phase": phase,
+        "config": config,
+        "target_rate_rps": target_rate,
+        "repetition": repetition,
+        "role": role,
+        "clean_start_s": clean_start,
+        "clean_end_s": clean_end,
+        "clean_seconds": clean_seconds,
+        "cohort_n": len(cohort),
+        "pass_n": passes,
+        "pass_rate": pass_rate,
+        "slo_goodput_rps": passes / clean_seconds,
+        "achieved_offered_rps": achieved_rate,
+        "offered_rate_valid": offered_rate_valid,
+        "schedule_valid": schedule_valid,
+        "raw_slo_feasible": raw_feasible,
+        "feasible": feasible,
+        "exact_output_n": exact_outputs,
+        "failure_reasons": dict(sorted(reasons.items())),
+        "ttft_ms": {
+            **numeric(ttft_values),
+            "p50": percentile(ttft_values, 0.50),
+            "p95": percentile(ttft_values, 0.95),
+            "p99": percentile(ttft_values, 0.99),
+        },
+        "tpot_ms": {
+            **numeric(tpot_values),
+            "p50": percentile(tpot_values, 0.50),
+            "p95": percentile(tpot_values, 0.95),
+            "p99": percentile(tpot_values, 0.99),
+        },
+        "schedule_lag_ms": {
+            **numeric(lag_values),
+            "p95": percentile(lag_values, 0.95),
+            "p99": percentile(lag_values, 0.99),
+        },
+        "invariants": invariants,
+    }
+
+
+def accepted_rate(rows: list[dict[str, Any]]) -> dict[str, Any]:
+    verdicts = [bool(row["feasible"]) for row in rows]
+    pass_n = sum(int(row["pass_n"]) for row in rows)
+    cohort_n = sum(int(row["cohort_n"]) for row in rows)
+    return {
+        "trials": len(rows),
+        "trial_feasible": verdicts,
+        "accepted_feasible": sum(verdicts) > len(verdicts) / 2,
+        "pooled_pass_n": pass_n,
+        "pooled_cohort_n": cohort_n,
+        "pooled_pass_rate": pass_n / cohort_n if cohort_n else 0.0,
+        "median_goodput_rps": sorted(float(row["slo_goodput_rps"]) for row in rows)[
+            len(rows) // 2
+        ],
+    }
+
+
+def frontier_for_cell(rows: list[dict[str, Any]]) -> dict[str, Any]:
+    by_rate: dict[float, list[dict[str, Any]]] = defaultdict(list)
+    for row in rows:
+        by_rate[float(row["target_rate_rps"])].append(row)
+    rates = []
+    for rate in sorted(by_rate):
+        rates.append({"rate_rps": rate, **accepted_rate(by_rate[rate])})
+    verdicts = [bool(row["accepted_feasible"]) for row in rates]
+    # Once a failure appears, no higher anchor may pass.
+    monotone = not any(
+        (not verdicts[i]) and any(verdicts[i + 1 :]) for i in range(len(verdicts))
+    )
+    feasible_rates = [row["rate_rps"] for row in rates if row["accepted_feasible"]]
+    infeasible_rates = [row["rate_rps"] for row in rates if not row["accepted_feasible"]]
+    lower = max(feasible_rates, default=None)
+    upper_candidates = [rate for rate in infeasible_rates if lower is None or rate > lower]
+    upper = min(upper_candidates, default=None)
+    boundary_repeated = bool(
+        lower is not None
+        and upper is not None
+        and next(row for row in rates if row["rate_rps"] == lower)["trials"] >= 3
+        and next(row for row in rates if row["rate_rps"] == upper)["trials"] >= 3
+    )
+    bracketed = lower is not None and upper is not None and monotone and boundary_repeated
+    return {
+        "rates": rates,
+        "lower_feasible_rps": lower,
+        "upper_infeasible_rps": upper,
+        "bracketed": bracketed,
+        "boundary_repeated": boundary_repeated,
+        "monotone": monotone,
+    }
+
+
+def gap_at_weight(
+    lower: dict[str, dict[str, float]],
+    upper: dict[str, dict[str, float]],
+    p01_weight: float,
+) -> dict[str, Any]:
+    weights = {"P01": p01_weight, "P06": 1.0 - p01_weight}
+    oracle = sum(weights[p] * max(upper[p].values()) for p in PHASES)
+    static_values = {
+        config: sum(weights[p] * lower[p][config] for p in PHASES)
+        for config in CONFIGS
+    }
+    best_config = max(static_values, key=static_values.get)
+    static = static_values[best_config]
+    return {
+        "p01_weight": p01_weight,
+        "oracle_upper_rps": oracle,
+        "static_lower_rps": static,
+        "best_static_config": best_config,
+        "gap": oracle / static - 1.0,
+    }
+
+
+def summarize_trials(rows: list[dict[str, Any]]) -> dict[str, Any]:
+    grouped: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
+    for row in rows:
+        grouped[(str(row["phase"]), str(row["config"]))].append(row)
+    expected = {(phase, config) for phase in PHASES for config in CONFIGS}
+    if set(grouped) != expected:
+        raise RuntimeError(f"cell coverage mismatch: {sorted(set(grouped) ^ expected)}")
+    frontiers = {
+        phase: {
+            config: frontier_for_cell(grouped[(phase, config)])
+            for config in CONFIGS
+        }
+        for phase in PHASES
+    }
+    all_bracketed = all(
+        frontiers[p][c]["bracketed"] for p in PHASES for c in CONFIGS
+    )
+    all_monotone = all(
+        frontiers[p][c]["monotone"] for p in PHASES for c in CONFIGS
+    )
+    lower = {
+        p: {c: float(frontiers[p][c]["lower_feasible_rps"]) for c in CONFIGS}
+        for p in PHASES
+    } if all_bracketed else {}
+    upper = {
+        p: {c: float(frontiers[p][c]["upper_infeasible_rps"]) for c in CONFIGS}
+        for p in PHASES
+    } if all_bracketed else {}
+    scan = [gap_at_weight(lower, upper, step / 10000) for step in range(10001)] if all_bracketed else []
+    worst = max(scan, key=lambda row: row["gap"]) if scan else None
+    equal = gap_at_weight(lower, upper, 0.5) if all_bracketed else None
+    distinct_by_cell = {
+        f"{p}-{c}": len(
+            {round(float(row["slo_goodput_rps"]), 12) for row in grouped[(p, c)]}
+        )
+        for p in PHASES for c in CONFIGS
+    }
+    sanity = {
+        "trial_count": numeric([row["cohort_n"] for row in rows]),
+        "target_rates": numeric([row["target_rate_rps"] for row in rows]),
+        "pass_rates": numeric([row["pass_rate"] for row in rows]),
+        "goodput_rps": numeric([row["slo_goodput_rps"] for row in rows]),
+        "distinct_goodput_by_cell": distinct_by_cell,
+        "invariants": {
+            "all_counters_nonnegative": all(
+                int(row["cohort_n"]) >= 0 and int(row["pass_n"]) >= 0 for row in rows
+            ),
+            "all_ratios_in_0_1": all(0 <= float(row["pass_rate"]) <= 1 for row in rows),
+            "all_trial_invariants": all(all(row["invariants"].values()) for row in rows),
+            "all_cells_bracketed": all_bracketed,
+            "all_frontiers_monotone": all_monotone,
+            "per_cell_results_not_all_identical": all(value > 1 for value in distinct_by_cell.values()),
+            "weight_scan_continuous": len(scan) in (0, 10001),
+        },
+    }
+    verdict = "INCONCLUSIVE"
+    if all(sanity["invariants"].values()) and worst is not None:
+        verdict = "REFUTED" if float(worst["gap"]) < 0.10 else "NOT_ESTABLISHED"
+    return {
+        "schema": 1,
+        "verdict": verdict,
+        "threshold": 0.10,
+        "frontiers": frontiers,
+        "equal_time_conservative": equal,
+        "worst_mixture_conservative": worst,
+        "sanity": sanity,
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    sub = parser.add_subparsers(dest="command", required=True)
+    score = sub.add_parser("score")
+    score.add_argument("--requests", required=True)
+    score.add_argument("--result", required=True)
+    score.add_argument("--phase", choices=PHASES, required=True)
+    score.add_argument("--config", choices=CONFIGS, required=True)
+    score.add_argument("--target-rate", type=float, required=True)
+    score.add_argument("--repetition", type=int, required=True)
+    score.add_argument("--role", required=True)
+    score.add_argument("--out", required=True)
+    summary = sub.add_parser("summarize")
+    summary.add_argument("--trial-glob", required=True)
+    summary.add_argument("--out", required=True)
+    args = parser.parse_args()
+    if args.command == "score":
+        value = score_trial(
+            Path(args.requests), Path(args.result), phase=args.phase,
+            config=args.config, target_rate=args.target_rate,
+            repetition=args.repetition, role=args.role,
+        )
+    else:
+        import glob
+
+        paths = [Path(path) for path in sorted(glob.glob(args.trial_glob, recursive=True))]
+        value = summarize_trials([json.loads(path.read_text()) for path in paths])
+    atomic_json(Path(args.out), value)
+    print(json.dumps(value, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

scripts/oracle_gap/phase3_upper_bound.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""Retrospective point-estimate oracle bound from accepted Phase-3 cells."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+from analyze import CONFIGS, PHASES, atomic_json, numeric
+
+
+def build(metrics: dict[str, Any]) -> dict[str, Any]:
+    tables: dict[str, dict[str, dict[str, float]]] = {}
+    for load in ("saturation", "moderate"):
+        table: dict[str, dict[str, float]] = {}
+        for phase in PHASES:
+            row = {}
+            for config in CONFIGS:
+                run = metrics.get("runs", {}).get(f"{phase}-{config}-{load}")
+                if run is not None:
+                    row[config] = float(run["clean"]["completed_throughput_rps"])
+            table[phase] = row
+        tables[load] = table
+
+    analyses = {}
+    all_values = []
+    for load, table in tables.items():
+        complete = all(set(row) == set(CONFIGS) for row in table.values())
+        if not complete:
+            raise RuntimeError(f"missing sentinel throughput cell for {load}: {table}")
+        per_phase = {}
+        c00_regrets = []
+        for phase, row in table.items():
+            oracle_config = max(row, key=row.get)
+            oracle = row[oracle_config]
+            regret = oracle / row["C00"] - 1.0
+            c00_regrets.append(regret)
+            per_phase[phase] = {
+                "throughput_rps": row,
+                "oracle_config": oracle_config,
+                "oracle_rps": oracle,
+                "c00_regret": regret,
+            }
+            all_values.extend(row.values())
+        equal_oracle = sum(item["oracle_rps"] for item in per_phase.values()) / len(PHASES)
+        static = {
+            config: sum(table[p][config] for p in PHASES) / len(PHASES)
+            for config in CONFIGS
+        }
+        analyses[load] = {
+            "per_phase": per_phase,
+            "universal_c00_point_bound": max(c00_regrets),
+            "equal_time_oracle_rps": equal_oracle,
+            "equal_time_best_static_config": max(static, key=static.get),
+            "equal_time_best_static_rps": max(static.values()),
+            "equal_time_gap": equal_oracle / max(static.values()) - 1.0,
+            "interpretation": (
+                "valid additive point-estimate precheck"
+                if load == "saturation"
+                else "diagnostic only: each config used its own offered rate"
+            ),
+        }
+    return {
+        "schema": 1,
+        "scope": list(PHASES),
+        "analyses": analyses,
+        "sanity": {
+            "throughput_rps": numeric(all_values),
+            "invariants": {
+                "all_nonnegative": all(value >= 0 for value in all_values),
+                "all_cells_present": len(all_values) == 2 * len(PHASES) * len(CONFIGS),
+                "per_config_not_identical": len(set(all_values)) > len(CONFIGS),
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--metrics", required=True)
+    parser.add_argument("--out", required=True)
+    args = parser.parse_args()
+    result = build(json.loads(Path(args.metrics).read_text()))
+    atomic_json(Path(args.out), result)
+    print(json.dumps(result, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

scripts/oracle_gap/run_frontier.py

@@ -0,0 +1,881 @@
+#!/usr/bin/env python3
+"""Detached, resumable solo-H20 controller for the oracle-gap frontier."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import shlex
+import shutil
+import signal
+import subprocess
+import time
+import urllib.request
+from pathlib import Path
+from typing import Any
+
+from analyze import (
+    CONFIGS,
+    PHASES,
+    atomic_json,
+    frontier_for_cell,
+    score_trial,
+    summarize_trials,
+)
+
+
+SCHEMA = 1
+AMENDMENT = "A-OG-4"
+AMENDMENT_REASON = (
+    "close any majority-shifted final boundary to three trials per side "
+    "without adding rate anchors"
+)
+REMOTE_ROOT = Path("/home/admin/cpfs/wjh/oracle-gap-20260713")
+RUN_ROOT = REMOTE_ROOT / "runs"
+STATE = RUN_ROOT / "controller-state.json"
+PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests")
+MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B")
+SOURCE = Path("/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0")
+VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv")
+REPO = Path(os.environ.get("AITUNER_ORACLE_REPO", Path(__file__).resolve().parents[2]))
+P5_CLIENT = REPO / "runs/opprof-phase5/opprof_phase5_client.py"
+P3_CLIENT_DIR = REPO / "runs/opprof-phase3/provenance"
+GPU = 0
+CPU_MASK = "0-19"
+PORT = 8820
+GPU_HOUR_LIMIT = 6.0
+MAX_CLOSURE_ROUNDS = 3
+PRIMARY_ORDER = ("C11", "C00", "C01", "C10")
+CONFIRM_ORDER = tuple(reversed(PRIMARY_ORDER))
+CONFIG_DETAILS = {
+    "C00": {"mns": 1024, "mbt": 8192, "flags": []},
+    "C10": {"mns": 64, "mbt": 8192, "flags": ["--max-num-seqs", "64"]},
+    "C01": {
+        "mns": 1024,
+        "mbt": 2048,
+        "flags": ["--max-num-batched-tokens", "2048"],
+    },
+    "C11": {
+        "mns": 64,
+        "mbt": 2048,
+        "flags": [
+            "--max-num-seqs", "64", "--max-num-batched-tokens", "2048"
+        ],
+    },
+}
+BASE_RATES = {
+    "P01": (32.0, 26.0, 36.0, 28.0, 34.0, 30.0),
+    "P06": (1.7, 1.4, 2.0, 1.5, 1.9, 1.6, 1.8),
+}
+UP_EXTENSIONS = {
+    "P01": (38.0, 40.0, 42.0),
+    "P06": (2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.8, 3.0),
+}
+DOWN_EXTENSIONS = {"P01": (24.0, 22.0, 20.0), "P06": (1.3, 1.2, 1.1)}
+TIMELINE = {
+    "P01": {"warmup": 60, "clean": 60, "drain": 120},
+    "P06": {"warmup": 60, "clean": 120, "drain": 240},
+}
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def run_text(command: list[str], *, check: bool = True) -> str:
+    result = subprocess.run(
+        command, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
+    )
+    if check and result.returncode:
+        raise RuntimeError(
+            f"command failed ({result.returncode}): {shlex.join(command)}\n{result.stdout}"
+        )
+    return result.stdout
+
+
+def load_state(resume: bool) -> dict[str, Any]:
+    if STATE.exists():
+        if not resume:
+            raise RuntimeError(f"state exists; use --resume: {STATE}")
+        return json.loads(STATE.read_text())
+    return {
+        "schema": SCHEMA,
+        "status": "created",
+        "created_at": time.time(),
+        "gpu_hours": 0.0,
+        "completed_trials": [],
+        "stages": {},
+        "fingerprint": {},
+        "owned_pgids": [],
+    }
+
+
+def save_state(state: dict[str, Any]) -> None:
+    state["updated_at"] = time.time()
+    state["controller_pid"] = os.getpid()
+    atomic_json(STATE, state)
+
+
+def compute_apps() -> list[dict[str, Any]]:
+    output = run_text(
+        [
+            "nvidia-smi",
+            "--query-compute-apps=gpu_uuid,pid,process_name,used_memory",
+            "--format=csv,noheader,nounits",
+        ],
+        check=False,
+    )
+    rows = []
+    for line in output.splitlines():
+        parts = [part.strip() for part in line.split(",", 3)]
+        if len(parts) == 4 and parts[1].isdigit():
+            rows.append(
+                {
+                    "gpu_uuid": parts[0],
+                    "pid": int(parts[1]),
+                    "process_name": parts[2],
+                    "used_memory_mib": int(parts[3].split()[0]),
+                }
+            )
+    return rows
+
+
+def gpu_snapshot() -> dict[str, Any]:
+    query = run_text(
+        [
+            "nvidia-smi",
+            "--query-gpu=index,name,uuid,memory.used,utilization.gpu,clocks.sm,clocks.mem,power.draw",
+            "--format=csv,noheader,nounits",
+        ]
+    )
+    return {
+        "time": time.time(),
+        "gpus": query.splitlines(),
+        "compute_apps": compute_apps(),
+        "loadavg": list(os.getloadavg()),
+    }
+
+
+def assert_idle() -> None:
+    deadline = time.monotonic() + 30
+    while time.monotonic() < deadline:
+        if not compute_apps():
+            return
+        time.sleep(1)
+    raise RuntimeError(f"GPU host is not idle: {compute_apps()}")
+
+
+def descendants(root_pid: int) -> set[int]:
+    output = run_text(["ps", "-e", "-o", "pid=,ppid="], check=False)
+    children: dict[int, list[int]] = {}
+    for line in output.splitlines():
+        parts = line.split()
+        if len(parts) == 2:
+            pid, parent = map(int, parts)
+            children.setdefault(parent, []).append(pid)
+    result = {root_pid}
+    pending = [root_pid]
+    while pending:
+        for child in children.get(pending.pop(), []):
+            if child not in result:
+                result.add(child)
+                pending.append(child)
+    return result
+
+
+def assert_only_server_apps(server_pid: int) -> None:
+    allowed = descendants(server_pid)
+    unexpected = [row for row in compute_apps() if int(row["pid"]) not in allowed]
+    if unexpected:
+        raise RuntimeError(f"unexpected GPU process during run: {unexpected}")
+
+
+def wait_idle_after_stop() -> None:
+    deadline = time.monotonic() + 60
+    while time.monotonic() < deadline:
+        output = run_text(
+            [
+                "nvidia-smi", "--query-gpu=index,memory.used",
+                "--format=csv,noheader,nounits",
+            ]
+        )
+        memory = [int(line.split(",")[1].strip()) for line in output.splitlines()]
+        if not compute_apps() and all(value == 0 for value in memory):
+            return
+        time.sleep(1)
+    raise RuntimeError("GPU memory/processes did not return to zero")
+
+
+def fingerprint() -> dict[str, Any]:
+    manifests = {}
+    for phase in PHASES:
+        path = PRIVATE / f"{phase}.jsonl"
+        summary = json.loads(path.with_suffix(path.suffix + ".summary.json").read_text())
+        if int(summary["rows"]) != 32768 or summary["sha256"] != sha256_file(path):
+            raise RuntimeError(f"manifest mismatch: {phase}")
+        manifests[phase] = {"path": str(path), "sha256": summary["sha256"]}
+    return {
+        "controller_sha256": sha256_file(Path(__file__).resolve()),
+        "analyzer_sha256": sha256_file(Path(__file__).with_name("analyze.py")),
+        "protocol_sha256": sha256_file(
+            REPO / "docs/opprof/oracle-gap-protocol.md"
+        ),
+        "p5_client_sha256": sha256_file(P5_CLIENT),
+        "p3_client_sha256": sha256_file(P3_CLIENT_DIR / "opprof_phase3_client.py"),
+        "repo_commit": run_text(["git", "-C", str(REPO), "rev-parse", "HEAD"]).strip(),
+        "repo_tree": run_text(["git", "-C", str(REPO), "rev-parse", "HEAD^{tree}"]).strip(),
+        "vllm_commit": run_text(["git", "-C", str(SOURCE), "rev-parse", "HEAD"]).strip(),
+        "model": str(MODEL),
+        "manifests": manifests,
+        "runtime": run_text(
+            [str(VENV / "bin/python"), "-c", "import torch,vllm; print(torch.__version__,torch.version.cuda,vllm.__version__)"]
+        ).strip(),
+        "driver": run_text(["nvidia-smi", "--query-gpu=driver_version", "--format=csv,noheader"]).splitlines()[0],
+        "config_details": CONFIG_DETAILS,
+        "base_rates": {key: list(value) for key, value in BASE_RATES.items()},
+        "up_extensions": {
+            key: list(value) for key, value in UP_EXTENSIONS.items()
+        },
+    }
+
+
+def resume_compatible(old: dict[str, Any], current: dict[str, Any]) -> bool:
+    immutable = (
+        "analyzer_sha256",
+        "p5_client_sha256",
+        "p3_client_sha256",
+        "vllm_commit",
+        "model",
+        "manifests",
+        "runtime",
+        "driver",
+        "config_details",
+        "base_rates",
+    )
+    if not all(old.get(key) == current.get(key) for key in immutable):
+        return False
+    old_extensions = old.get("up_extensions")
+    current_extensions = current.get("up_extensions")
+    if old_extensions == current_extensions:
+        return True
+    # The only earlier compatible transition was A-OG-1, which appended the
+    # five registered P06 anchors after the original 2.3-rps list.
+    if not isinstance(old_extensions, dict) or not isinstance(current_extensions, dict):
+        return False
+    return (
+        old_extensions.get("P01") == current_extensions.get("P01")
+        and list(old_extensions.get("P06", [])) + [2.4, 2.5, 2.6, 2.8, 3.0]
+        == current_extensions.get("P06")
+    )
+
+
+def ensure_provenance(current: dict[str, Any]) -> None:
+    destination = RUN_ROOT / "provenance"
+    destination.mkdir(parents=True, exist_ok=True)
+    for source in (
+        Path(__file__).resolve(),
+        Path(__file__).with_name("analyze.py"),
+        P5_CLIENT,
+        P3_CLIENT_DIR / "opprof_phase3_client.py",
+        REPO / "docs/opprof/oracle-gap-protocol.md",
+    ):
+        target = destination / source.name
+        if target.exists() and sha256_file(target) != sha256_file(source):
+            digest = sha256_file(source)
+            target = destination / f"{source.stem}.{digest[:12]}{source.suffix}"
+        if target.exists() and sha256_file(target) != sha256_file(source):
+            raise RuntimeError(f"content-addressed provenance mismatch: {target}")
+        if not target.exists():
+            shutil.copy2(source, target)
+    fingerprint_path = (
+        destination / f"fingerprint.{current['repo_commit'][:8]}.json"
+    )
+    atomic_json(fingerprint_path, current)
+    if not (destination / "fingerprint.json").exists():
+        atomic_json(destination / "fingerprint.json", current)
+    atomic_json(destination / "host-before.json", gpu_snapshot())
+    (destination / "nvidia-smi-q.txt").write_text(run_text(["nvidia-smi", "-q"]))
+
+
+def rate_label(rate: float) -> str:
+    return f"{rate:.3f}".rstrip("0").rstrip(".")
+
+
+def trial_key(phase: str, config: str, rate: float, repetition: int) -> str:
+    return f"{phase}-{config}-r{rate_label(rate)}-rep{repetition}"
+
+
+def derived_manifest(phase: str, rate: float, repetition: int) -> Path:
+    label = f"{phase}-r{rate_label(rate)}-rep{repetition}"
+    output = RUN_ROOT / "manifests" / f"{label}.jsonl"
+    summary_path = output.with_suffix(output.suffix + ".summary.json")
+    source = PRIVATE / f"{phase}.jsonl"
+    domain = int.from_bytes(
+        hashlib.sha256(f"oracle-gap:{label}".encode()).digest()[:4], "big"
+    )
+    if output.exists() and summary_path.exists():
+        summary = json.loads(summary_path.read_text())
+        if summary["sha256"] == sha256_file(output) and summary["source_sha256"] == sha256_file(source):
+            return output
+        raise RuntimeError(f"derived manifest mismatch: {output}")
+    output.parent.mkdir(parents=True, exist_ok=True)
+    temporary = output.with_name(f"{output.name}.tmp.{os.getpid()}")
+    rows = 0
+    input_sum = 0
+    output_sum = 0
+    with source.open() as src, temporary.open("w") as dst:
+        for line in src:
+            row = json.loads(line)
+            row["token_seed"] = (int(row.get("token_seed", 0)) + domain * 1000003) & ((1 << 63) - 1)
+            row["token_domain"] = domain
+            dst.write(json.dumps(row, sort_keys=True, separators=(",", ":")) + "\n")
+            rows += 1
+            input_sum += int(row["input_tokens"])
+            output_sum += int(row["output_tokens"])
+        dst.flush()
+        os.fsync(dst.fileno())
+    os.replace(temporary, output)
+    summary = {
+        "schema": 1,
+        "phase": phase,
+        "rate": rate,
+        "repetition": repetition,
+        "rows": rows,
+        "input_token_sum": input_sum,
+        "output_token_sum": output_sum,
+        "token_domain": domain,
+        "source_sha256": sha256_file(source),
+        "sha256": sha256_file(output),
+        "invariants": {"rows_32768": rows == 32768, "positive_work": input_sum > 0 and output_sum > 0},
+    }
+    if not all(summary["invariants"].values()):
+        raise RuntimeError(f"derived manifest invalid: {summary}")
+    atomic_json(summary_path, summary)
+    return output
+
+
+def server_command(config: str) -> list[str]:
+    return [
+        "taskset", "-c", CPU_MASK, str(VENV / "bin/vllm"), "serve", str(MODEL),
+        "--host", "127.0.0.1", "--port", str(PORT),
+        "--tensor-parallel-size", "1", "--enable-chunked-prefill",
+        "--enable-prefix-caching", "--shutdown-timeout", "120",
+        *CONFIG_DETAILS[config]["flags"],
+    ]
+
+
+def wait_ready(process: subprocess.Popen[Any], timeout: float = 300) -> None:
+    deadline = time.monotonic() + timeout
+    while time.monotonic() < deadline:
+        if process.poll() is not None:
+            raise RuntimeError(f"server exited before ready: {process.returncode}")
+        try:
+            with urllib.request.urlopen(f"http://127.0.0.1:{PORT}/health", timeout=1) as response:
+                if response.status == 200:
+                    return
+        except Exception:
+            pass
+        time.sleep(1)
+    raise TimeoutError("server readiness timeout")
+
+
+def validate_startup(log_path: Path, config: str) -> None:
+    log = log_path.read_text(errors="replace")
+    details = CONFIG_DETAILS[config]
+    invariants = {
+        "triton_moe": "Using TRITON Unquantized MoE backend" in log,
+        "tp1": "tensor_parallel_size=1" in log,
+        "mbt": (
+            "Chunked prefill is enabled with max_num_batched_tokens=8192" in log
+            if details["mbt"] == 8192
+            else "'max_num_batched_tokens': 2048" in log
+        ),
+        "mns": details["mns"] == 1024 or "'max_num_seqs': 64" in log,
+    }
+    if not all(invariants.values()):
+        raise RuntimeError(f"server startup invariants failed {config}: {invariants}")
+
+
+def start_server(config: str, stage: str, state: dict[str, Any]) -> dict[str, Any]:
+    assert_idle()
+    directory = RUN_ROOT / "servers" / stage
+    directory.mkdir(parents=True, exist_ok=True)
+    command = server_command(config)
+    echo = (
+        f"RUN_ECHO stage={stage} host=dash0 gpu=0 cpus={CPU_MASK} config={config} "
+        f"model={MODEL} source={SOURCE} manifests={PRIVATE}/P01,P06.jsonl "
+        f"output={RUN_ROOT} expected_server_plus_trials=20-45min budget_cap={GPU_HOUR_LIMIT}H20h"
+    )
+    with (RUN_ROOT / "launch-echo.log").open("a") as handle:
+        handle.write(echo + "\n")
+    print(echo, flush=True)
+    (directory / "command.txt").write_text(shlex.join(command) + "\n")
+    atomic_json(directory / "gpu-before.json", gpu_snapshot())
+    handle = (directory / "server.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update(
+        {
+            "CUDA_VISIBLE_DEVICES": str(GPU),
+            "VLLM_OPPROF_DIR": str(directory / "opprof"),
+            "HF_HUB_OFFLINE": "1",
+            "TRANSFORMERS_OFFLINE": "1",
+            "PYTHONUNBUFFERED": "1",
+            "AITUNER_ORACLE_GAP_MARKER": stage,
+        }
+    )
+    process = subprocess.Popen(
+        command, cwd=SOURCE, env=environment, stdout=handle,
+        stderr=subprocess.STDOUT, start_new_session=True,
+    )
+    state["owned_pgids"] = [process.pid]
+    save_state(state)
+    wait_ready(process)
+    validate_startup(directory / "server.log", config)
+    return {
+        "config": config, "stage": stage, "dir": directory, "process": process,
+        "handle": handle, "started_at": time.time(),
+    }
+
+
+def stop_server(entry: dict[str, Any], state: dict[str, Any]) -> None:
+    process = entry["process"]
+    if process.poll() is None:
+        try:
+            os.kill(process.pid, signal.SIGINT)
+        except ProcessLookupError:
+            pass
+        try:
+            process.wait(timeout=150)
+        except subprocess.TimeoutExpired:
+            for signum, timeout in ((signal.SIGTERM, 10), (signal.SIGKILL, 30)):
+                if process.poll() is not None:
+                    break
+                try:
+                    os.killpg(process.pid, signum)
+                except ProcessLookupError:
+                    pass
+                try:
+                    process.wait(timeout=timeout)
+                except subprocess.TimeoutExpired:
+                    continue
+            if process.poll() is None:
+                raise TimeoutError(f"server process group did not stop: {entry['stage']}")
+    entry["handle"].close()
+    elapsed = time.time() - float(entry["started_at"])
+    state["gpu_hours"] = float(state["gpu_hours"]) + elapsed / 3600.0
+    state["owned_pgids"] = []
+    atomic_json(entry["dir"] / "gpu-after.json", gpu_snapshot())
+    log = (entry["dir"] / "server.log").read_text(errors="replace")
+    if "mode=drain timeout=120s" not in log:
+        raise RuntimeError(f"server did not use drain shutdown: {entry['stage']}")
+    wait_idle_after_stop()
+    save_state(state)
+
+
+def client_command(
+    phase: str, rate: float, repetition: int, output: Path
+) -> list[str]:
+    timeline = TIMELINE[phase]
+    return [
+        "taskset", "-c", CPU_MASK, str(VENV / "bin/python"), str(P5_CLIENT), "run",
+        "--manifest", str(derived_manifest(phase, rate, repetition)),
+        "--base-url", f"http://127.0.0.1:{PORT}", "--model", str(MODEL),
+        "--load-point", "moderate", "--fixed-request-rate", str(rate),
+        "--max-concurrency", "256", "--ignore-eos", "--temperature", "0",
+        "--warmup-seconds", str(timeline["warmup"]),
+        "--clean-segment-seconds", str(timeline["clean"]),
+        "--num-clean-segments", "1", "--post-clean-seconds", "0",
+        "--drain-timeout-seconds", str(timeline["drain"]),
+        "--workload-seed", "20260712", "--server-seed", "20260712",
+        "--result-dir", str(output / "client"),
+    ]
+
+
+def run_trial(
+    state: dict[str, Any], server: dict[str, Any], phase: str, rate: float,
+    repetition: int, role: str,
+) -> dict[str, Any]:
+    config = str(server["config"])
+    key = trial_key(phase, config, rate, repetition)
+    output = RUN_ROOT / "trials" / f"{phase}-{config}" / f"rate-{rate_label(rate)}" / f"rep-{repetition}"
+    score_path = output / "score.json"
+    if key in state["completed_trials"]:
+        if not score_path.exists():
+            raise RuntimeError(f"completed state lacks score: {key}")
+        return json.loads(score_path.read_text())
+    output.mkdir(parents=True, exist_ok=True)
+    command = client_command(phase, rate, repetition, output)
+    (output / "command.txt").write_text(shlex.join(command) + "\n")
+    if server["process"].poll() is not None:
+        raise RuntimeError(f"server exited before trial: {key}")
+    assert_only_server_apps(int(server["process"].pid))
+    environment = os.environ.copy()
+    environment["PYTHONPATH"] = str(P3_CLIENT_DIR) + os.pathsep + environment.get("PYTHONPATH", "")
+    started = time.time()
+    with (output / "client.log").open("ab", buffering=0) as handle:
+        result = subprocess.run(
+            command, cwd=REPO, env=environment, stdout=handle,
+            stderr=subprocess.STDOUT, timeout=900,
+        )
+    assert_only_server_apps(int(server["process"].pid))
+    if server["process"].poll() is not None:
+        raise RuntimeError(f"server exited during trial: {key}")
+    client_result = output / "client/result.json"
+    client_sanity = output / "client/sanity.json"
+    if not client_result.exists() or not client_sanity.exists():
+        raise RuntimeError(f"client produced no result: {key} rc={result.returncode}")
+    sanity = json.loads(client_sanity.read_text())["invariants"]
+    failed = [name for name, passed in sanity.items() if not passed]
+    allowed = {"moderate_offered_within_5pct"}
+    if result.returncode != 0 and not failed:
+        raise RuntimeError(f"client failed without sanity marker: {key}")
+    if set(failed) - allowed:
+        raise RuntimeError(f"client validity failure {key}: {failed}")
+    score = score_trial(
+        output / "client/requests.jsonl", client_result, phase=phase,
+        config=config, target_rate=rate, repetition=repetition, role=role,
+    )
+    score["wall_seconds"] = time.time() - started
+    score["client_returncode"] = result.returncode
+    score["client_failed_invariants"] = failed
+    score["manifest_sha256"] = sha256_file(derived_manifest(phase, rate, repetition))
+    atomic_json(score_path, score)
+    state["completed_trials"].append(key)
+    state["last_trial"] = {
+        "key": key, "feasible": score["feasible"], "pass_rate": score["pass_rate"],
+        "goodput": score["slo_goodput_rps"], "completed_at": time.time(),
+    }
+    save_state(state)
+    print(
+        f"TRIAL {key} feasible={score['feasible']} pass={score['pass_rate']:.6f} "
+        f"goodput={score['slo_goodput_rps']:.6f} lagmax={score['schedule_lag_ms']['max']:.3f}",
+        flush=True,
+    )
+    return score
+
+
+def primary_bracket(rows: list[dict[str, Any]]) -> tuple[float, float] | None:
+    verdict = {float(row["target_rate_rps"]): bool(row["feasible"]) for row in rows}
+    ordered = sorted(verdict)
+    passes = [rate for rate in ordered if verdict[rate]]
+    failures = [rate for rate in ordered if not verdict[rate]]
+    if not passes or not failures:
+        return None
+    lower = max(passes)
+    upper = min((rate for rate in failures if rate > lower), default=None)
+    if upper is None or any(verdict[rate] for rate in ordered if rate > upper):
+        raise RuntimeError(f"non-monotone primary frontier: {verdict}")
+    return lower, upper
+
+
+def run_primary_config(state: dict[str, Any], config: str) -> None:
+    stage = f"primary-{config}"
+    if state["stages"].get(stage, {}).get("status") == "complete":
+        return
+    state["stages"][stage] = {"status": "starting", "started_at": time.time()}
+    save_state(state)
+    server = start_server(config, stage, state)
+    failure = None
+    try:
+        for phase in PHASES:
+            phase_rows = [run_trial(state, server, phase, rate, 0, "primary") for rate in BASE_RATES[phase]]
+            bracket = primary_bracket(phase_rows)
+            if bracket is None:
+                extension = UP_EXTENSIONS[phase] if all(row["feasible"] for row in phase_rows) else DOWN_EXTENSIONS[phase]
+                for rate in extension:
+                    phase_rows.append(run_trial(state, server, phase, rate, 0, "primary-extension"))
+                    bracket = primary_bracket(phase_rows)
+                    if bracket is not None:
+                        break
+            if bracket is None:
+                raise RuntimeError(f"failed to bracket {phase}-{config}")
+            state["stages"][stage].setdefault("brackets", {})[phase] = list(bracket)
+            save_state(state)
+    except Exception as error:
+        failure = error
+    finally:
+        try:
+            stop_server(server, state)
+        except Exception as error:
+            failure = failure or error
+    if failure is not None:
+        state["stages"][stage]["status"] = "failed"
+        state["stages"][stage]["failure"] = repr(failure)
+        state["status"] = "failed"
+        save_state(state)
+        raise failure
+    state["stages"][stage]["status"] = "complete"
+    state["stages"][stage]["completed_at"] = time.time()
+    save_state(state)
+
+
+def run_confirm_config(state: dict[str, Any], config: str) -> None:
+    stage = f"confirm-{config}"
+    if state["stages"].get(stage, {}).get("status") == "complete":
+        return
+    brackets = state["stages"][f"primary-{config}"]["brackets"]
+    state["stages"][stage] = {"status": "starting", "started_at": time.time(), "brackets": brackets}
+    save_state(state)
+    server = start_server(config, stage, state)
+    failure = None
+    try:
+        for phase in PHASES:
+            lower, upper = (float(value) for value in brackets[phase])
+            for rate in (upper, lower):
+                for repetition in (1, 2):
+                    run_trial(state, server, phase, rate, repetition, "boundary-confirmation")
+    except Exception as error:
+        failure = error
+    finally:
+        try:
+            stop_server(server, state)
+        except Exception as error:
+            failure = failure or error
+    if failure is not None:
+        state["stages"][stage]["status"] = "failed"
+        state["stages"][stage]["failure"] = repr(failure)
+        state["status"] = "failed"
+        save_state(state)
+        raise failure
+    state["stages"][stage]["status"] = "complete"
+    state["stages"][stage]["completed_at"] = time.time()
+    save_state(state)
+
+
+def load_cell_scores(phase: str, config: str) -> list[dict[str, Any]]:
+    paths = sorted((RUN_ROOT / "trials" / f"{phase}-{config}").glob("**/score.json"))
+    return [json.loads(path.read_text()) for path in paths]
+
+
+def boundary_closure_needs(rows: list[dict[str, Any]]) -> list[float]:
+    frontier = frontier_for_cell(rows)
+    if not frontier["monotone"]:
+        raise RuntimeError("cannot close a non-monotone frontier")
+    lower = frontier["lower_feasible_rps"]
+    upper = frontier["upper_infeasible_rps"]
+    if lower is None or upper is None:
+        raise RuntimeError("cannot close an unbracketed frontier")
+    trials_by_rate = {
+        float(row["rate_rps"]): int(row["trials"])
+        for row in frontier["rates"]
+    }
+    # Preserve the confirmation convention of running high-to-low.
+    return [
+        float(rate)
+        for rate in (upper, lower)
+        if trials_by_rate[float(rate)] < 3
+    ]
+
+
+def run_closure_config(
+    state: dict[str, Any],
+    round_index: int,
+    config: str,
+    needs: dict[str, list[float]],
+) -> None:
+    stage = f"closure-r{round_index}-{config}"
+    if state["stages"].get(stage, {}).get("status") == "complete":
+        return
+    state["stages"][stage] = {
+        "status": "starting",
+        "started_at": time.time(),
+        "needs": needs,
+    }
+    save_state(state)
+    server = start_server(config, stage, state)
+    failure = None
+    try:
+        for phase in PHASES:
+            for rate in needs.get(phase, []):
+                rows = load_cell_scores(phase, config)
+                repetitions = {
+                    int(row["repetition"])
+                    for row in rows
+                    if float(row["target_rate_rps"]) == rate
+                }
+                repetition = 0
+                while len(repetitions) < 3:
+                    while repetition in repetitions:
+                        repetition += 1
+                    run_trial(
+                        state,
+                        server,
+                        phase,
+                        rate,
+                        repetition,
+                        "boundary-closure",
+                    )
+                    repetitions.add(repetition)
+    except Exception as error:
+        failure = error
+    finally:
+        try:
+            stop_server(server, state)
+        except Exception as error:
+            failure = failure or error
+    if failure is not None:
+        state["stages"][stage]["status"] = "failed"
+        state["stages"][stage]["failure"] = repr(failure)
+        state["status"] = "failed"
+        save_state(state)
+        raise failure
+    state["stages"][stage]["status"] = "complete"
+    state["stages"][stage]["completed_at"] = time.time()
+    save_state(state)
+
+
+def run_boundary_closure(state: dict[str, Any]) -> None:
+    stage = "boundary-closure"
+    closure = state["stages"].setdefault(
+        stage, {"status": "starting", "started_at": time.time(), "rounds": []}
+    )
+    save_state(state)
+    for round_index in range(1, MAX_CLOSURE_ROUNDS + 1):
+        needs = {
+            config: {
+                phase: boundary_closure_needs(load_cell_scores(phase, config))
+                for phase in PHASES
+            }
+            for config in CONFIGS
+        }
+        closure["rounds"].append({"round": round_index, "needs": needs})
+        save_state(state)
+        if not any(rates for config in needs.values() for rates in config.values()):
+            closure["status"] = "complete"
+            closure["completed_at"] = time.time()
+            save_state(state)
+            return
+        for config in CONFIRM_ORDER:
+            config_needs = {
+                phase: rates for phase, rates in needs[config].items() if rates
+            }
+            if not config_needs:
+                continue
+            if float(state["gpu_hours"]) >= GPU_HOUR_LIMIT:
+                raise RuntimeError("GPU-hour hard cap reached during boundary closure")
+            run_closure_config(state, round_index, config, config_needs)
+    remaining = {
+        config: {
+            phase: boundary_closure_needs(load_cell_scores(phase, config))
+            for phase in PHASES
+        }
+        for config in CONFIGS
+    }
+    closure["final_needs"] = remaining
+    if not any(rates for config in remaining.values() for rates in config.values()):
+        closure["status"] = "complete"
+        closure["completed_at"] = time.time()
+        save_state(state)
+        return
+    raise RuntimeError(
+        f"failed to close final boundaries in {MAX_CLOSURE_ROUNDS} rounds"
+    )
+
+
+def cleanup_recorded(state: dict[str, Any]) -> None:
+    for pgid in state.get("owned_pgids", []):
+        try:
+            os.killpg(int(pgid), signal.SIGKILL)
+        except ProcessLookupError:
+            pass
+    state["owned_pgids"] = []
+    save_state(state)
+    wait_idle_after_stop()
+
+
+def execute(resume: bool) -> None:
+    RUN_ROOT.mkdir(parents=True, exist_ok=True)
+    state = load_state(resume)
+    if resume and state.get("owned_pgids"):
+        cleanup_recorded(state)
+    assert_idle()
+    current = fingerprint()
+    if state["fingerprint"] and state["fingerprint"] != current:
+        if not (
+            resume
+            and state.get("status") == "failed"
+            and resume_compatible(state["fingerprint"], current)
+        ):
+            raise RuntimeError("resume fingerprint changed incompatibly")
+        state.setdefault("amendments", []).append(
+            {
+                "id": AMENDMENT,
+                "accepted_at": time.time(),
+                "reason": AMENDMENT_REASON,
+                "completed_trials_before": len(
+                    state.get("completed_trials", [])
+                ),
+                "gpu_hours_before": state.get("gpu_hours"),
+                "old_fingerprint": state["fingerprint"],
+                "new_fingerprint": current,
+            }
+        )
+    state["fingerprint"] = current
+    state["status"] = "running"
+    save_state(state)
+    ensure_provenance(current)
+    for config in PRIMARY_ORDER:
+        if float(state["gpu_hours"]) >= GPU_HOUR_LIMIT:
+            raise RuntimeError("GPU-hour hard cap reached before primary completion")
+        run_primary_config(state, config)
+    for config in CONFIRM_ORDER:
+        if float(state["gpu_hours"]) >= GPU_HOUR_LIMIT:
+            raise RuntimeError("GPU-hour hard cap reached before confirmations")
+        run_confirm_config(state, config)
+    run_boundary_closure(state)
+    score_paths = sorted((RUN_ROOT / "trials").glob("**/score.json"))
+    summary = summarize_trials([json.loads(path.read_text()) for path in score_paths])
+    summary["gpu_hours"] = state["gpu_hours"]
+    summary["trial_files"] = len(score_paths)
+    atomic_json(RUN_ROOT / "metrics.json", summary)
+    state["status"] = "complete"
+    state["completed_at"] = time.time()
+    state["verdict"] = summary["verdict"]
+    save_state(state)
+    print(json.dumps({"status": "complete", "verdict": summary["verdict"], "gpu_hours": state["gpu_hours"]}, sort_keys=True))
+
+
+def plan() -> dict[str, Any]:
+    primary = sum(len(BASE_RATES[phase]) for phase in PHASES) * len(CONFIGS)
+    confirmations = 2 * 2 * len(PHASES) * len(CONFIGS)
+    return {
+        "schema": 1,
+        "placement": "serialized solo GPU0",
+        "primary_order": list(PRIMARY_ORDER),
+        "confirm_order": list(CONFIRM_ORDER),
+        "primary_trials_without_extensions": primary,
+        "confirmation_trials": confirmations,
+        "max_boundary_closure_rounds": MAX_CLOSURE_ROUNDS,
+        "expected_total_trials": primary + confirmations,
+        "expected_h20_hours": "3.0-4.0",
+        "hard_cap_h20_hours": GPU_HOUR_LIMIT,
+        "rates": {key: list(value) for key, value in BASE_RATES.items()},
+        "timelines": TIMELINE,
+        "output": str(RUN_ROOT),
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    sub = parser.add_subparsers(dest="command", required=True)
+    run = sub.add_parser("run")
+    run.add_argument("--resume", action="store_true")
+    sub.add_parser("plan")
+    sub.add_parser("status")
+    args = parser.parse_args()
+    if args.command == "run":
+        execute(args.resume)
+    elif args.command == "plan":
+        print(json.dumps(plan(), indent=2, sort_keys=True))
+    else:
+        print(STATE.read_text() if STATE.exists() else json.dumps({"status": "not-started"}))
+
+
+if __name__ == "__main__":
+    main()

src/aituner/slo.py

@@ -16,6 +16,7 @@ class RequestOutcome:
     completion_tokens: int | None
     error: str = ""
     completion_tokens_source: str = ""
+    completed_mono_ns: int | None = None
 
 
 @dataclass(frozen=True)

src/aituner/worker.py

@@ -127,6 +127,7 @@ def _run_one_request(
                         f"actual={actual_completion_tokens}"
                     ),
                     completion_tokens_source=completion_tokens_source,
+                    completed_mono_ns=time.monotonic_ns(),
                 )
             if actual_completion_tokens != expected_completion_tokens:
                 return RequestOutcome(
@@ -142,6 +143,7 @@ def _run_one_request(
                         f"actual={actual_completion_tokens}"
                     ),
                     completion_tokens_source=completion_tokens_source,
+                    completed_mono_ns=time.monotonic_ns(),
                 )
         return RequestOutcome(
             request_id=request.row_id,
@@ -151,6 +153,7 @@ def _run_one_request(
             prompt_tokens=request.prompt_tokens_hint,
             completion_tokens=actual_completion_tokens or request.completion_tokens_hint,
             completion_tokens_source=completion_tokens_source,
+            completed_mono_ns=time.monotonic_ns(),
         )
     except HttpClientError as exc:
         return RequestOutcome(
@@ -161,6 +164,7 @@ def _run_one_request(
             prompt_tokens=request.prompt_tokens_hint,
             completion_tokens=request.completion_tokens_hint,
             error=str(exc),
+            completed_mono_ns=time.monotonic_ns(),
         )
 
 

tests/test_core_flow.py

@@ -5604,6 +5604,7 @@ class CoreFlowTests(unittest.TestCase):
                 completion_tokens=1,
             ),
         ):
+            with mock.patch("aituner.worker.time.monotonic_ns", return_value=123456789):
                 outcome = _run_one_request(
                     request,
                     base_url="http://127.0.0.1:8000",
@@ -5613,6 +5614,7 @@ class CoreFlowTests(unittest.TestCase):
         self.assertFalse(outcome.success)
         self.assertEqual(outcome.error, "completion_tokens_mismatch expected=2 actual=1")
         self.assertEqual(outcome.completion_tokens, 1)
+        self.assertEqual(outcome.completed_mono_ns, 123456789)
 
     def test_build_prompt_mentions_completion_tokens_override(self) -> None:
         with tempfile.TemporaryDirectory() as tmp:

tests/test_oracle_gap.py

@@ -0,0 +1,215 @@
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+
+ORACLE_GAP = Path(__file__).resolve().parents[1] / "scripts/oracle_gap"
+sys.path.insert(0, str(ORACLE_GAP))
+
+from analyze import score_trial, summarize_trials  # noqa: E402
+from run_frontier import (  # noqa: E402
+    AMENDMENT,
+    UP_EXTENSIONS,
+    boundary_closure_needs,
+    resume_compatible,
+)
+
+
+def _request(
+    request_id: str,
+    admitted: float,
+    first: float,
+    completed: float,
+    *,
+    input_tokens: int = 512,
+    output_tokens: int = 64,
+) -> dict:
+    return {
+        "request_id": request_id,
+        "scheduled_s": admitted - 0.001,
+        "admitted_s": admitted,
+        "first_token_s": first,
+        "completed_s": completed,
+        "input_tokens": input_tokens,
+        "requested_output_tokens": output_tokens,
+        "actual_output_tokens": output_tokens,
+        "success": True,
+        "error_kind": None,
+    }
+
+
+def test_score_trial_uses_clean_admission_cohort_and_both_slos(tmp_path: Path) -> None:
+    result_path = tmp_path / "result.json"
+    result_path.write_text(
+        json.dumps(
+            {
+                "warmup_seconds": 10,
+                "clean_segment_seconds": 20,
+                "num_clean_segments": 1,
+            }
+        )
+    )
+    rows = [
+        _request("warmup", 9.0, 9.1, 10.0),
+        _request("pass", 11.0, 11.1, 13.0),
+        # (16.0 - 12.1) / 63 = 61.9 ms: TPOT miss.
+        _request("tpot-fail", 12.0, 12.1, 16.0),
+        _request("after-clean", 31.0, 31.1, 32.0),
+    ]
+    request_path = tmp_path / "requests.jsonl"
+    request_path.write_text("".join(json.dumps(row) + "\n" for row in rows))
+
+    scored = score_trial(
+        request_path,
+        result_path,
+        phase="P01",
+        config="C00",
+        target_rate=0.1,
+        repetition=0,
+        role="test",
+    )
+
+    assert scored["cohort_n"] == 2
+    assert scored["pass_n"] == 1
+    assert scored["pass_rate"] == 0.5
+    assert scored["slo_goodput_rps"] == 0.05
+    assert scored["failure_reasons"] == {"tpot_slo": 1}
+    assert scored["offered_rate_valid"]
+    assert not scored["feasible"]
+
+
+def _frontier_row(
+    phase: str,
+    config: str,
+    rate: float,
+    feasible: bool,
+) -> dict:
+    cohort = 100
+    passes = 99 if feasible else 80
+    return {
+        "phase": phase,
+        "config": config,
+        "target_rate_rps": rate,
+        "repetition": 0,
+        "role": "test",
+        "cohort_n": cohort,
+        "pass_n": passes,
+        "pass_rate": passes / cohort,
+        "slo_goodput_rps": rate * passes / cohort,
+        "feasible": feasible,
+        "invariants": {"test": True},
+    }
+
+
+def test_conservative_oracle_bound_can_refute_ten_percent_gate() -> None:
+    brackets = {
+        "P01": {
+            "C00": (30.0, 32.0),
+            "C10": (28.0, 30.0),
+            "C01": (29.0, 31.0),
+            "C11": (27.0, 29.0),
+        },
+        "P06": {
+            "C00": (1.6, 1.7),
+            "C10": (1.7, 1.8),
+            "C01": (1.55, 1.65),
+            "C11": (1.6, 1.7),
+        },
+    }
+    rows = []
+    for phase, configs in brackets.items():
+        for config, (lower, upper) in configs.items():
+            for _ in range(3):
+                rows.append(_frontier_row(phase, config, lower, True))
+                rows.append(_frontier_row(phase, config, upper, False))
+
+    summary = summarize_trials(rows)
+
+    assert summary["verdict"] == "REFUTED"
+    assert summary["worst_mixture_conservative"]["gap"] < 0.10
+    assert summary["sanity"]["invariants"]["all_cells_bracketed"]
+    assert summary["sanity"]["invariants"]["all_frontiers_monotone"]
+
+
+def test_nonmonotone_frontier_blocks_inference() -> None:
+    rows = []
+    for phase in ("P01", "P06"):
+        for config in ("C00", "C10", "C01", "C11"):
+            for _ in range(3):
+                rows.append(_frontier_row(phase, config, 1.0, True))
+                rows.append(_frontier_row(phase, config, 2.0, False))
+    for _ in range(3):
+        rows.append(_frontier_row("P01", "C00", 3.0, True))
+
+    summary = summarize_trials(rows)
+
+    assert summary["verdict"] == "INCONCLUSIVE"
+    assert not summary["sanity"]["invariants"]["all_frontiers_monotone"]
+
+
+def test_a_og_1_extends_only_mutable_resume_fields() -> None:
+    immutable = {
+        "p5_client_sha256": "p5",
+        "p3_client_sha256": "p3",
+        "vllm_commit": "vllm",
+        "model": "model",
+        "manifests": {"P01": "a", "P06": "b"},
+        "runtime": "runtime",
+        "driver": "driver",
+        "analyzer_sha256": "analyzer",
+        "config_details": {"C00": {}},
+        "base_rates": {"P01": [26], "P06": [1.4]},
+        "up_extensions": {"P01": [38.0], "P06": [2.1, 2.2, 2.3]},
+    }
+    amended = {
+        **immutable,
+        "controller_sha256": "new",
+        "repo_commit": "new",
+        "up_extensions": {
+            "P01": [38.0],
+            "P06": [2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.8, 3.0],
+        },
+    }
+
+    assert UP_EXTENSIONS["P06"][-5:] == (2.4, 2.5, 2.6, 2.8, 3.0)
+    assert resume_compatible(immutable, amended)
+    assert not resume_compatible(
+        immutable, {**amended, "manifests": {"P01": "changed"}}
+    )
+
+
+def test_a_og_4_requires_the_amended_grid_to_stay_fixed() -> None:
+    old = {
+        "analyzer_sha256": "analyzer",
+        "p5_client_sha256": "p5",
+        "p3_client_sha256": "p3",
+        "vllm_commit": "vllm",
+        "model": "model",
+        "manifests": {"P01": "a", "P06": "b"},
+        "runtime": "runtime",
+        "driver": "driver",
+        "config_details": {"C00": {}},
+        "base_rates": {"P01": [26], "P06": [1.4]},
+        "up_extensions": {"P01": [38.0], "P06": [2.1, 2.2, 2.3, 2.4]},
+    }
+
+    assert AMENDMENT == "A-OG-4"
+    assert resume_compatible(old, {**old, "controller_sha256": "new"})
+    assert not resume_compatible(
+        old,
+        {**old, "up_extensions": {"P01": [38.0], "P06": [2.1, 2.2, 2.3, 2.5]}},
+    )
+
+
+def test_boundary_closure_follows_a_majority_shift() -> None:
+    rows = []
+    rows.append(_frontier_row("P06", "C00", 2.3, True))
+    rows.extend(_frontier_row("P06", "C00", 2.4, value) for value in (True, False, False))
+    rows.extend(_frontier_row("P06", "C00", 2.5, False) for _ in range(3))
+
+    assert boundary_closure_needs(rows) == [2.3]
+
+    rows.extend(_frontier_row("P06", "C00", 2.3, True) for _ in range(2))
+    assert boundary_closure_needs(rows) == []