Add simulator-aware fidelity pilot audit

Entry-point summary of both closed campaigns: verdicts, corrected
readings (P5 supersedes P3's P10-vs-P04 magnitude), methodological
findings (co-location SLO validity, arrival uniformization, compile-
factor env poisoning), artifact map, and GPU accounting.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.gitignore

@@ -14,3 +14,8 @@ configs/examples/dash0_llm_10min_study_run[0-9]*.json
 configs/examples/dash0_smoke_study_run*.json
 infra/gpu_fleet/config/fleet.toml
 infra/gpu_fleet/config/jobs.toml
+# Raw Layer-1 telemetry streams (507 MB; decision-bearing JSON/log evidence stays tracked)
+runs/**/*.jsonl
+.ruff_cache/
+# Recovered dash1 interaction-run stores (100 MB raw tune logs, kept on disk only)
+recovered-stores/

docs/collectivespec-p2-gate-20260713.md

@@ -0,0 +1,154 @@
+# CollectiveSpec P2：logical-plan 对照的审计与停止门槛
+
+## 决策
+
+**不启动正式的 P1/P2 SLO-goodput sweep，也不把 `compact-vs-padded` 作为
+CollectiveSpec 的研究主线。**
+
+原因不是这条机制一定没有工程收益，而是它的核心研究主张已经无法排除公开工作的
+覆盖：
+
+- [DSpark](https://arxiv.org/html/2607.05147) 已明确采用每请求的动态 verification
+  length，并把逻辑 sequence tracking 与物理 execution 解耦、flatten variable-length
+  token；
+- SGLang 的 [DSpark 集成说明](https://www.lmsys.org/blog/2026-07-06-dspark-sglang/)
+  已公开 `static`、`compact` 与 `cap-accept` 三种 verify mode。其中 `cap-accept`
+  执行完整 block、但只提交 compact window，且说明其输出与 `compact` 相同。这正是
+  “同一语义下 full/padded 与 compact”的 counterfactual；
+- 该实现还公开了 DP attention 下各 rank 使用最大 graph tier 的处理。因此，仅在
+  vLLM/H20 上再复现 compact 比 padded 快，只是环境复现，不是新的系统贡献。
+
+P0 还独立否定了原来的 liveness 动机：目标 runtime 已经以 scalar DP metadata 协调
+不同 DP replica 的物理 shape；异构 verifier candidate 没有引起运行期 collective
+错误。故不能再把“必须新增 canonical header 才能避免死锁”作为论文 premise。
+
+## 术语：什么必须相同
+
+### Logical plan（也称 semantic plan）
+
+logical plan 是一次 speculative verification **应当计算和提交什么**的不可变记录；它
+不包含 padding、CUDA graph tier、物理 rank 行数、worker PID 或耗时。每一个 verifier
+epoch 的最小条目为：
+
+```text
+(global_epoch, dp_rank, ordered request id,
+ logical_output_offset_before, scheduled_seq_len,
+ available_candidate_token_ids, requested_k, effective_k,
+ visible_candidate_token_ids_hash)
+```
+
+请求层还必须固定 `client_request_id`、server request id、prompt/body hash、arrival、DP
+assignment、提交顺序、temperature/seed 与预期 completion length。最终还要逐请求验证
+output token-id hash、completion length、finish reason、usage，以及 endpoint semantic
+transcript hash（content/reasoning/tool-call 的 canonical JSON）。
+
+这里的关键是 `k_i` 的 key 必须是
+`(server_request_id, logical_output_offset_before)`，而不是只有 request id：同一请求会
+经历多个 verification epoch。只有两个 cell 的这些事实都相同，才称为 *same logical
+plan*。
+
+### Compact vs. padded：只是同一 plan 的两个 lowering
+
+给定同一组 non-dummy logical entries：
+
+- **PaddedSync-semantic**：保留这些 entries，但为同步域插入 masked dummy rows，使各
+  DP peer 的 physical shape 对齐；
+- **CompactSync**：保留完全相同的 entries、candidate 和 commit semantics，用 ragged
+  packing / split vector 执行真实 rows，不计算 dummy rows。
+
+因此 `static K=3` 不能当作 padded 对照：它改变了每个请求可见 candidate prefix，改变了
+logical algorithm，而不是只改变 physical lowering。真实 physical-row 公式也不能简单
+写成 `N * (1 + max k_i)`；普通 decode、TP alignment 和 CUDA-graph alignment 都要从
+runner 的 row map 分开计数。
+
+## 当前 P0 对 P2 的限制
+
+P0 heterogeneous policy 按 vLLM **随机生成的 server request id** 哈希，而 client 没有
+发送 `X-Request-Id`。所以即使 trace 和 seed 相同，两个 cell 的每个 request/epoch 的
+`k_i` 也不可保证相同；日志只有 aggregate digest/histogram，也没有 per-row candidate
+token、assignment 或最终 token-id hash。P0 因而不能充当 same-logical-plan 的 P2 A/B。
+
+P0 的 padding 上界也已校正。66 个 target epoch 中 62 个 raw DP counts 不等：
+
+```text
+raw logical rows                 6,276
+local non-DP-aligned rows        6,536
+PaddedSync physical rows         7,024
+DP-global-max attributable rows    488  (= 6.95% of physical rows)
+```
+
+此前的 748 / 10.65% 将 260 行本地 TP/CUDA-graph alignment 混入 DP max padding。即使
+488 行都可回收，它仍只是 **target verifier row-count 的上界**：EAGLE3 仍按 Kmax=3
+完成 drafter 工作，也没有测得 EP bytes、collective critical path 或 E2E SLO-goodput。
+
+## 若未来重新打开，先补齐的测量契约
+
+不应先实现 compact lowering。先添加只用于 audit 的 telemetry：
+
+1. client 对每个请求发送固定 `X-Request-Id`、`X-Data-Parallel-Rank` 和
+   `return_token_ids=true`；记录 response id、prompt/output token-id hash、semantic
+   transcript hash 和 finish reason；
+2. scheduler 作为 semantic ledger 的唯一 writer，记录 per-epoch ordered entries、候选
+   token、requested/effective K、logical cursor 与 sampled token hash；
+3. worker 只记录物理事实：每 rank physical rows、DP/TP/graph padding 的原因、packed row
+   map；若要主张 EP 收益，额外记录 DeepEP all-to-all split vector、bytes、duration 和
+   rank wait；
+4. 汇总器先给出第一个 semantic/output diff，任一 mismatch 即标为 invalid，禁止读取
+   性能数字。
+
+最小 reproducibility smoke（仅在发现 topology gap 后执行）是 fresh engine 上的 16 个
+decode-only requests、每个 64 output tokens、temperature=0、DP0/DP1 各 8 个、显式
+`{0,3}` alternating manifest。先连续跑两次 **同一个** padded cell；只有 ledger 和逐请求
+token hash 全等，才允许运行 padded/compact mechanism probe。`return_token_ids` 会改变 SSE
+负担，故最终 latency cell 必须关掉该字段、改以 scheduler-side hash 审计。
+
+## 唯一尚可证伪的拓扑假设
+
+公开材料没有证明、也没有否定下列特殊情形：**独立 standard-DP scheduler 共享同一个 EP
+all-to-all domain** 时，DP global-max graph tier 之外仍有 EP split-vector / collective
+ordinal 的关键路径浪费。这不能从“论文没有写”推断为新颖性。
+
+只有一次短的 topology reconnaissance 观测到该额外瓶颈，才重新进行文献审计并考虑下列
+顺序严格的 gates：
+
+1. 与 SGLang-style DP global-max tier / current runtime 相比，compact plan 降低实际 EP
+   bytes、split imbalance 或 collective critical-path time；仅少几个 rows 不够；
+2. 在相同 semantic plan、token-exact 输出和无 tail-latency 退化下，至少三次 fresh-engine
+   paired runs 显示 E2E SLO-goodput 增益 >=10%；
+3. topology ablation 支持因果归因：DP=1 或 EP 不跨 DP 时收益消失或显著缩小，而
+   DP×shared-EP 时出现；
+4. 重新完成与 DSpark/SGLang 的逐项差异审计，证明贡献是 topology-aware collective
+   scheduling，而不是已有 ragged packing。
+
+任一 gate 不成立即结束 CollectiveSpec；不以 controller/K/queue knob 调优替代证据。
+
+## 如果 gate 重开时的固定环境与 setup
+
+下列是 P0 实际使用、后续必须 provenance-pin 的环境，而不是当前已启动的实验：
+
+| 项目 | 固定值 |
+|---|---|
+| host / accelerator | `dash0`，8 × NVIDIA H20 |
+| target / draft | Qwen3-235B-A22B FP8；EAGLE3，Kmax=3 |
+| parallelism | TP=4，DP=2，EP=8；`VLLM_MOE_USE_DEEPEP=1` |
+| engine | dash0 live installed vLLM wheel；记录 wheel metadata、import path、launch command 与 commit；不能以本地 checkout API 代替 |
+| execution | `FULL_DECODE_ONLY` CUDA graphs、FP8 KV cache、block size 64、`max-num-batched-tokens=1024`、`max-num-seqs=192`、max model len 262144 |
+| workload | immutable materialized `thinking_w20260327_1000` 的 decode-only window；机制 smoke 使用固定 burst，E2E 使用完整、session-closure 状态明确的 trace |
+| reproducibility | fresh engine per cell、temperature=0、固定 seed、固定 request ids/DP assignment、prefix-cache state 从空开始、ABBA cell order |
+| SLO（若进入 E2E） | 预注册 TPOT <= 40 ms、pass rate >= 0.95；同时报告 completion success、p50/p95/p99、deadline failures 与 output equivalence |
+
+remote source 必须从 Git 同步到
+`/home/admin/cpfs/wjh/collectivespec-pilot/20260713T054328Z/source`，并记录运行时实际
+source revision；任何远端 job 启动前在 artifact 中写明 resolved command、模型/trace path、
+预计 GPU 时间和结果目录。
+
+## 审计数据健全性
+
+- 新增实验数：n=0；本文件不报告任何新的性能数字。
+- 已复核的 P0 target epochs：n=66；两个 DP rank 的 raw row values 共 n=132，
+  min=1、max=77、distinct=35；physical rows 则 n=66，min=4、max=80、distinct=14。
+  原始 JSONL 可复核，不以 aggregate 值伪造每 epoch 分布。
+- 已用 aggregate row totals：n=4，min=488，max=7,024，distinct=4；均为非负。校正后的
+  关系 `6,276 <= 6,536 <= 7,024` 成立，且 `7,024 - 6,536 = 488`。
+- 外部材料覆盖判断区分为“论文明确描述”“官方公开实现明确描述”和“未公开拓扑细节”；
+  未从缺失的 EP 细节推导新颖性或性能收益。

docs/collectivespec-pilot-design-20260713.md

@@ -0,0 +1,319 @@
+# CollectiveSpec：先做机会判定的实验设计
+
+## 结论先行
+
+当前不应先实现一个“按请求动态 K、再做一次 DP collective 同步”的原型。该路径的
+工程修复很薄，而且相邻公开工作已经覆盖了 request-level dynamic speculation 与
+ragged verification 的大量空间。CollectiveSpec 只有在
+一个更强、可证伪的事实成立时才值得继续：**在 wide-EP MoE、DP>1 的生产负载中，
+不同 DP rank / request 所需的投机深度确实不同，并且全局 static K 明显浪费了 SLO
+可行 goodput。**
+
+本文件把第一轮实验定义为一个机会判定（opportunity gate），不是最终性能主张。
+
+## 固定条件
+
+- Host: `dash0`, 8x NVIDIA H20。
+- Model: Qwen3-235B-A22B FP8；draft: EAGLE3。
+- Deployment: TP=4, DP=2, EP=8，`VLLM_MOE_USE_DEEPEP=1`。
+- Trace: `thinking_w20260327_1000`，600 秒 decode-only 窗口。
+- SLO: TPOT <= 40 ms，pass rate >= 0.95。
+- 同一 engine revision、同一模型/trace 路径、同一环境变量；实验串行执行，避免 GPU
+  互相干扰。
+
+这里的 resolved topology 来自远端实际 StudySpec，而不是仓库 README 中可能已过期的
+配置描述。
+
+## 假设与可证伪指标
+
+### G0：static-K 是否有足够可利用的空间？
+
+- H0：在该固定拓扑和负载下，NoSpec/K=1/2/3 的 SLO-goodput 差异很小；最佳固定 K 已经
+  足够好。此时停止 CollectiveSpec。
+- H1：不同 static K 的可行前沿存在实质差异，且最优 K 对负载区间敏感。只有 H1 才
+  说明 dynamic policy 可能有直接性能价值。
+
+主要指标：每个 K 在相同 SLO 下可达到的最大 `sampling_u`，以及对应请求率、TPOT
+pass rate、p50/p95 TPOT、成功/失败原因。`sampling_u` 是现有 replay 使用的一致 trace
+抽样旋钮，因此只能作为此 trace 的 SLO-goodput proxy，不能直接外推为线上 QPS。
+
+本 trace 的 output-length 分布很重尾；replayer 的 drain deadline 可能在长输出尚未完成
+时终止 probe。故每个结果必须同时报告 completion-success count 和 deadline failure；不能
+只因 TPOT pass rate 达标就把截断 request 当作“无成本”。本轮的原始 static screen 仍沿用
+现有 SLO 以便和项目 baseline 可比，但它不能替代完整 completion 的确认实验。
+
+判定门槛（预注册）：
+
+1. 在 K=1/2/3 之间，最优 K 相对次优 K 的最大可行 `sampling_u` 小于 5%，或
+   置信区间/重复实验重叠很大：
+   **停止**。
+2. 最优 speculative K 比次优 speculative K 至少高 10%，且在两个独立重复中方向一致：
+   进入 G1。NoSpec 仅作为“是否值得用 draft model”的部署对照，不能替代这条判据。
+3. 如果 K=3 不受当前 engine 支持、任一配置启动失败，记录为兼容性结果，不把它误作
+   性能差。
+
+## 第一阶段：static-K screening
+
+配置为 `{NoSpec, K=1, K=2, K=3}`。NoSpec 会删除 `--speculative-config`，而不是传
+非法的 `num_speculative_tokens=0`。它释放 draft model 相关资源，因此不等于
+same-stack 的 logical K=0；后者在 EAGLE 类实现中仍可能需要一次 draft forward 来保持
+KV 同步。当前 dash0 binary 的 MLA indexer 明确限制 `num_speculative_tokens <= 3`，
+故这已穷尽该 binary 的合法 static horizon。原计划每个配置：
+
+- 搜索范围 `sampling_u in [0.005, 0.020]`；
+- 最多 3 次 probe、tolerance=0.003；
+- 每个 probe 使用完整 600 秒 trace replay（不会使用 `max_requests_per_probe` 的
+  截断模式）；
+- 启动顺序 `2,1,3,0`，降低冷启动或时间漂移与 K 单调对应的风险；
+- 每个 K 都使用独立 Store、不可变派生 StudySpec、完整 stdout/stderr log。
+
+这是一轮筛选而非 final frontier。它一旦显示值得继续，才对 top-2 K 做交叉顺序的完整
+搜索与至少两次重复。
+
+### 2026-07-13 数据质量修正：controlled screen
+
+原始 trace 第一个 K=2 probe 暴露了一个不应隐藏的 measurement red flag：worker 的
+drain deadline 按 selected set 的 p99 output length 计算，而该 set 含一个 36,034-token
+completion。该 request 因 deadline 被裁掉；尽管 TPOT-only pass rate 仍可达标，censoring
+会随 `sampling_u` 改变 selected set，不能用于比较 static-K frontier。
+
+因此原始 run 只保留为诊断 artifact，停止后改跑一个明确标为 **controlled** 的 screen：
+保持相同 arrival、prompt、sampling seed 和 topology，但把每个 request 的
+`min_tokens=max_tokens=4096`。4096 接近原始 output mean 3,924.6，且使 p99 deadline 覆盖
+每个 request 的完整 completion。它回答的是“在相同输入/到达条件下，static K 是否留下
+可利用空间”，不是 production trace 的最终 goodput 结论。最终论文实验必须同时有：
+
+1. 该受控 curve（机制和可重复性）；
+2. 原始长度 trace 的完整-completion 版本（不能使用 p99 censoring）；
+3. 至少一个 held-out window。
+
+### 2026-07-13 数据质量修正 #2：fresh-engine fixed grid
+
+受控 screen 的第一条 K=2、`u=0.0125` probe 本身通过了完整性检查（152/152 success、
+usage 返回的 completion token 均精确为 4096、无 early stop）。但随后发现原二分搜索会
+在同一 engine 内连续执行多个 probe；第二、三 probe 继承前一 probe 的 prefix/KV cache，
+也继承全局 RNG 的已消耗状态。不同 K 的二分分支/中止路径不同，因此不能把该搜索输出的
+`best_sampling_u` 差异直接归因于 K。该 run 在第二 probe 中主动停止，**不作为 G0 结果**。
+
+替代协议是每个 `(offered-load, K)` 只运行一次、每次均启动一个 fresh engine。两个固定
+负载由原始 immutable trace 的统一 `sampling_u` 阈值物化：`u=0.0125`（152 requests,
+0.2533 req/s）和 `u=0.0200`（263 requests, 0.4383 req/s）。物化后的 request 仍保留原
+prompt、arrival 和 sampling provenance，但强制 `temperature=0`、显式 engine `seed=0`，
+并用统一的 4096 completion override。每个 K 只有一个 probe，所以 accelerator KV/prefix
+cache 为空且 RNG 从相同 seed 开始；每次都从 `probe_details.jsonl` 验证：
+
+1. `early_stopped=false`；
+2. outcome count = selected count，且每个 request success；
+3. TTFT/TPOT 均非空；
+4. completion token 的 source 为 usage，且实际/预期均为 4096；
+5. result 无 partial-probe failure，且只包含一个 primary probe。
+
+运行顺序在两个负载间反向（ABBA），避免 K 与时间漂移完全共线。这个 grid 仍然只回答
+static-K 是否存在足够大的机会；它不估计 production sampling goodput，也不证明 rank-local
+K 的上界。当前 vLLM deployment 的 `reasoning_parser=''`；其 SSE 实现将生成文本放在
+`delta.content`，所以本协议中的 token-time 定义覆盖当前 `<think>` 输出。若以后启用
+reasoning parser，客户端必须同时记录 `reasoning_content` 后才可复用此指标。
+
+## G1：只有在 G0 通过后才做的直接验证
+
+目标不是“不同请求有不同 K”这种已经很常见的说法，而是验证下面的系统命题：
+
+> 在 DP+EP MoE 下，局部独立的 K 决策会让 collective 序列分歧；把它们编译为
+> rank-agreement 的 ragged execution plan，可保留异质请求的计算节省，同时不改变
+> collective order。
+
+当前 dash0 vLLM 已经有一个很好的切入点：每个 DP step 会 all-reduce 一段 metadata，
+并把各 rank 的 total token count padding 到最大值；CUDA graph mode 也会取跨 rank 的
+共同模式。这说明论文的最小机制不应另造一个 scheduler，而应把现有 scalar
+`(num_tokens, num_reqs, graph_mode)` agreement 扩展为 canonical speculative-plan header。
+关键增量是让 header 描述真实 active frontiers，并保证后续 verifier/EP split vector 的
+collective ordinal 相同；若最后仍 padding 到 global max，就没有可主张的性能机制。
+
+需要实现/测量：
+
+1. **oracle trace replayer**：利用 G0 的 per-K service curve，为每个到达时刻选择
+   SLO-feasible K，比较 best-static K 与 oracle 的 upper bound。若 oracle gain <10%，
+   停止，避免把噪声当论文方向。
+2. **collective trace**：按 DP rank 记录每个 decode step 的 collective 序列、token
+   shape、active-sequence mask、MoE all-to-all bytes 和 rank idle time。验证“local K
+   不同”是否真的导致 sequence divergence，而不是仅是一个 API 限制。
+3. **CollectiveSpec prototype**：固定 collective order，用全局 agreement header 和
+   ragged/padded verification plan；对比 `best static K`、global-max-K、oracle 和当前
+   upstream dynamic-spec baseline（包括 DSpark/FASER 能实现的部分）。
+4. **ablation**：去掉 agreement、去掉 ragged packing、去掉 queue/SLO policy；报告
+   goodput、p50/p95/p99 TPOT、acceptance、MoE communication bytes、GPU SM/HBM util、
+   rank skew。
+
+## 主要风险
+
+- 最新 upstream 动态投机对 DP>1 的处理可能本身只需一个 global-K broadcast；那是
+  feature patch，不构成研究贡献。
+- 当前 dash0 runtime 已验证 DP=2 + static EAGLE 可以工作；尚未在这个 binary 上证明
+  “local dynamic K 会 deadlock”。因此研究动机必须写成固定 EAGLE horizon 的执行限制，
+  不能把未运行的 dynamic-K 路径当作既成故障。
+- FASER/DSpark 等相邻工作会把“dynamic K + ragged verify”作为强 baseline；必须在
+  做任何大实现前进行逐项复现/排除。
+- trace 的 `sampling_u` 是 proxy；最终结论必须在固定 arrival trace、真实请求长度和
+  至少一个不同 workload 上复现。
+
+## 2026-07-13 系统重审：收紧主张和后续 gate
+
+本轮查阅 [vLLM Dynamic SD 限制](https://docs.vllm.ai/en/latest/features/speculative_decoding/dynamic_speculative_decoding/)
+与 [DSpark](https://arxiv.org/abs/2607.05147) 后，原先“dynamic K 会使 collective
+diverge，因此做 ragged verifier”这一表述太宽，不能作为实现前提：
+
+- DSpark 已公开主张按请求动态 verification length、跨请求 token flatten 与 ragged
+  physical execution；这些本身不再构成新颖性。
+- dash0 当前实际 import 的 vLLM wheel 与本地 checkout 不同。实验中的运行日志称
+  `v0.11.1`，而 wheel metadata 为 `0.13.0rc2.dev2111+gb44b43f43.d20260309`；后续任何
+  hook 必须对这个 live source 做 provenance pin，不能把本地 v0.24 API 当作证据。
+- 这个 live runtime 已原生使用 per-request `list[list[int]]` draft token IDs，并将真实
+  长度送入 scheduler/metadata；EP all-to-all 也具有 variable-split data path。因此
+  **per-request horizon 不等于 collective divergence**。必须先观测到不同 DP replica
+  的 collective call count、phase 或 branch trace 的真实不一致。
+
+修订后的唯一可能研究命题是更窄的：
+
+> 对共享一个 EP collective domain 的独立 DP scheduler，如何将异构 verification plan
+> 编译为可证明 liveness 的 canonical execution trace，同时在同一逻辑 plan 下回收
+> global physical-max padding 的关键路径成本。
+
+它有四个顺序严格的停止门槛：
+
+1. **P0 / 真实 premise**：以预先给定的 `k_i ∈ {0,1,2,3}` replay 表截断已生成的
+   EAGLE candidates，在每个 DP/TP rank 记录 target、EAGLE 和 EP phase signature。若
+   没有 trace mismatch、强制 global padding 或 liveness 问题，就停止把 plan header 当作
+   研究贡献。
+2. **P1 / 机会量**：即使存在 mismatch，也必须证明 rank-local oracle 相对 best static
+   或 globally synchronized K 有至少 10% 的 SLO-goodput headroom；全局同步若恢复 90%
+   以上 gap，则只值得做 upstream patch。
+3. **P2 / 因果物理对照**：同一个 `k_i` replay plan 必须对比
+   `PaddedSync-semantic`（物理 `N × (1 + max k_i)` rows）与 `CompactSync`
+   （物理 `Σ_i(1+k_i)` rows）。static K=3 不是 PaddedSync，因为二者 logical algorithm
+   不同。若实际 target/EP work 与关键路径未减少，停止。
+4. **P3 / 正确性与部署**：所有 TP peers 的 plan digest 相同、所有 EP ranks 观察到同一
+   header vector、每 epoch 的 target/EAGLE/MoE signature 一致；greedy output token-exact，
+   并通过 empty-rank、`{0,3}` 交替、长时间 stress。只有随后在两个 session-coherent
+   workload 上重现 SLO-goodput 才能讨论论文。
+
+若 P0--P3 任一项失败，合理结论是停止 CollectiveSpec，而不是继续调 controller、K 或
+queue policy。若全部通过，最小原型的边界也只应是 verifier-side compaction：当前
+EAGLE 仍按 Kmax 产生 candidate，短 `k_i` 不会自动消除 draft-side work，不能把 verifier
+节省误报为完整 draft+verify speedup。
+
+### Trace 数据可用性与 window-closure fallback
+
+本轮发现 `prepare_trace_windows.py` 依赖的 2026-03-27 原始格式化 trace span 在 dash0
+已不存在，不能把它的 streaming full-session root 当作已经复验。现有的完整 600 秒
+materialized window 仍含 prompt、`chat_id` 与 `parent_chat_id`，因此新增的 fallback 仅将
+窗口内图的 connected component 作为选择原子：保留 384/384 条窗口内 parent edge，且把
+同一个窗口外 parent 的 sibling 归为同一 component。该窗口有 15,479 requests、15,095
+components；414 条 non-root edge 中 30 条（7.25%）父节点落在窗口外。
+
+这只能称为 **window-session-closed**，不等于 full-session coherent：任何结果都必须报告
+这 30 条 boundary-parent residual，且不能据此声称跨窗口 KV reuse。若原始 span 恢复，必须
+重新从完整 source resolve root 与重新采样，不能沿用 fallback 的 score/threshold。
+
+## 2026-07-13 P0 v2：header/liveness premise 的实际结果
+
+### 先报异常
+
+两个 cell 都在 probe/result 已落盘、64 个请求都已完成后出现 teardown 异常。heterogeneous
+cell 有 `free(): corrupted unsorted chunks` 与共享资源泄漏；control 还出现 SIGTERM/SIG11
+和 TCPStore broken pipe。这些不是运行期 request/collective failure，但也意味着本实验**不
+证明干净退出或部署鲁棒性**。本节只使用完成前的 request、worker phase 与 DP metadata
+作为 P0 evidence；不报告任何 TPOT/QPS 比较。
+
+### 设计、判定与修正后的观测范围
+
+在 dash0 的 Qwen3-235B-A22B FP8 + EAGLE3、TP=4/DP=2/EP=8、DeepEP 配置上，P0 将 EAGLE
+已生成的 Kmax=3 candidates 按预先给定的 request-static 表截断为 `k_i ∈ {0,1,2,3}`。它只
+改变 verifier 可见 candidate，不消除 EAGLE 的 Kmax drafter 工作。
+
+原始 worker hook 还会记录 vLLM 的 profile/DP dummy run；空 `SchedulerOutput` 仍可能有
+physical rows。因此真实 target batch 的判据固定为：
+
+```text
+event == batch_execution_plan
+AND request_count > 0
+AND total_scheduled_rows > 0
+```
+
+最初 summary 将 676/640 条 dummy/profile record 混入 target phase，错误地把 control 的
+145/146 internal-call 差异解释为 rank mismatch。修正后的汇总只比较 target event，并将
+真实 DP pair 识别为 `[0,4]`、`[1,5]`、`[2,6]`、`[3,7]`；同一 logical DP replica 内的 TP
+group 则为 `[0,1,2,3]` 与 `[4,5,6,7]`。
+
+### 结果
+
+远端可复核 artifact：
+`/home/admin/cpfs/wjh/collectivespec-pilot/20260713T054328Z/p0_phase_v2_20260713T0944Z`
+（run source `bb698b5`；`summary.json`、`summary.md`、`driver_result.json` 和原始
+`p0_logs/*.jsonl` 均在该目录）。
+
+| cell | completion | 实际 candidate K | target worker records | target plan / DP coordination |
+|---|---:|---|---:|---|
+| control K=3 | 64/64，usage 均为 64 | `{3}` | 488；DP0 每 TP peer 65，DP1 每 peer 57 | 每个 logical DP replica 内序列完全一致；四个真实 DP pair 的 57 个 shared target epoch 的 scalar coordination signature 一致 |
+| heterogeneous | 64/64，usage 均为 64 | `{0,1,2,3}` | 528；8 个 peer 各 66 | 两个 logical DP replica 内序列完全一致；四个真实 DP pair 的 66/66 shared target epoch signature 一致 |
+
+heterogeneous 的 `candidate_truncate` 直方图为 `{0: 1408, 1: 670, 2: 672, 3: 398}`，而截断前
+全部为 K=3（共 3,148 个 candidate）。所以它不是只改变 log 的“伪异构”实验。两 cell 的
+target record 都满足：
+
+```text
+num_tokens_per_rank[dp_rank] == total_scheduled_rows
+physical_batch_rows == rows_across_dp[dp_rank]
+rows_across_dp[i] >= num_tokens_per_rank[i]
+len(rows_across_dp) == len(num_tokens_per_rank) == 2
+```
+
+这说明两个独立 scheduler 的 logical plan 可以不同，但 live runtime 已用 scalar DP metadata
+协调共同 physical shape，并让共享 EP domain 的真实请求完成；没有观察到运行期 deadlock 或
+collective error。它反驳的是“异构 verifier-side K 必须新增 canonical header 才能先保证
+liveness”的必要性，而不是一般性的形式化证明。
+
+### 留下的物理现象，以及为什么它仍不足以继续造系统
+
+heterogeneous 的 66 个 shared target epoch 中有 62 个的 raw DP token counts 不相等；现有
+runtime 将 `rows_across_dp` 同步为共同 shape。按每个 DP replica 的一个 TP anchor 计，
+target-only raw logical rows 为 6,276；逐 epoch 保留当前 TP/CUDA-graph local alignment 后为
+6,536；最终 physical rows 为 7,024。因此可单独归因给跨 DP global-max padding 的只有
+488 rows（6.95% physical rows）。此前用 7,024-6,276 得到的 748（10.65%）还混入了 260
+行本地 alignment，不能当作 compact 对照可回收的 DP work。control 的 row totals 也不与
+heterogeneous 直接比较，因为 logical plan 和 scheduler trajectory 不同。
+
+这只能看作 **P2 的 row-count upper bound**，绝不能把 control 与 heterogeneous 相减当作
+速度收益：两者 logical plan、scheduler trajectory 都不同。更关键的是 EAGLE drafter 仍完成
+Kmax 工作；即使理想 compact verifier 回收全部 6.95% 的 DP-only target rows，端到端
+SLO-goodput 增益也只会更小。
+
+因此决策为：
+
+1. **停止**把 canonical plan header / deadlock avoidance 当作 CollectiveSpec 的研究主线；
+   P0 已在目标部署上否定其必要前提。
+2. **停止**把“dynamic verification length + flattened ragged execution”本身当贡献；DSpark
+   已覆盖该组合，且它也指出固定长度 drafter 的前置工作不会因 verifier 截断自动消失。
+3. 仅保留一个很窄的、默认 no-go 的机会：同一 logical plan 下的 compact-vs-padded physical
+   execution。只有先以 P1 证明相对 best-static/global-sync 至少 10% E2E SLO-goodput，再以
+   P2 的因果对照证明关键路径 rows/bytes 真正下降，并完成 DSpark topology gap 审计，才值得
+   再投入实现。当前 P0 不满足这些条件。
+
+P0 未验证 greedy token-exact 输出、真实 DeepEP dispatch ordinal/split digest、取消/empty-rank
+stress，或其他模型/后端/更大 K 的泛化；这些都不能从本结果外推。
+
+### P0 data sanity
+
+- **teardown red flag 已单列**：control/heterogeneous 都在 completion 后发生 allocator/资源
+  清理异常；因此没有使用延迟、吞吐或 clean-shutdown 指标作结论。
+- n=2 cells，8 worker/cell，64 usage-verified completions/cell；completion count 的
+  min=max=64，distinct=1。
+- target worker record count：control=488、heterogeneous=528（min=488，max=528，distinct=2）；
+  dummy/profile records 分别为 676/640，已排除。
+- heterogeneous K：min=0，max=3，distinct=4；control K distinct=1。所有计数、rows 和
+  padding 非负；JSON parse errors=0。
+- heterogeneous 的 66 target epochs：两个 DP rank 的 raw row values 共 n=132，min=1、
+  max=77、distinct=35；physical rows n=66，min=4、max=80、distinct=14；分解
+  `6,276 <= 6,536 <= 7,024` 与 `7,024 - 6,536 = 488` 均成立。
+- 修正后的不变量均为 true：probe integrity、8 workers observed、每个 DP replica 内 target
+  phase/sequence 一致、target DP metadata 合法、DP coordination record 存在、四个真实 DP pair
+  的 shared scalar coordination signature 一致。

docs/fidelity-aware-harness-headroom-20260714.md

@@ -0,0 +1,129 @@
+# Fidelity-aware harness headroom audit
+
+Status: **PROMISING PREMISE, NO CONTRIBUTION CLAIM**.
+
+The audit answers whether engine instrumentation has enough incremental signal
+to justify a prospective experiment.  It does not establish generalization.
+
+## Simulator shortlist lower bound
+
+On the frozen 12-cell SimFid task, the strongest calibrated SLO simulator
+reading places TP2/MNS32 and TP2/MNS64 in the same first tie bucket.  Real-final
+evaluation of that two-cell bucket selects TP2/MNS32 and has zero real regret.
+A method requiring a real calibration probe plus final verification cannot beat
+two real cell evaluations on this task.  Therefore “better initial selection”
+is not a viable claim here; the remaining headroom is shorter real verification
+inside the same shortlist.
+
+## Five-second prefix result
+
+The retrospective Phase-6 dataset contains 37 primary anchors across 12 cells.
+Stable labels use the frozen same-placement 2-of-3 adjudication: 28 feasible and
+9 infeasible.  Three TP4 primary measurements disagree with their repeated
+labels, so single-run feasibility is not treated as ground truth.
+
+Using leave-one-cell-out folds, identical L2 logistic models, and a 5-second
+prefix:
+
+| Metric | Outcome-only | Instrumentation-aware | Delta |
+|---|---:|---:|---:|
+| Accuracy | 78.38% | 89.19% | +10.81 pp |
+| Balanced accuracy | 70.63% | 81.55% | +10.92 pp |
+| Brier score | 0.1297 | 0.0901 | -0.0396 |
+| Correct only in this model | 0 | 4 | +4 |
+| McNemar exact two-sided p | — | 0.125 | not significant |
+
+At the frozen conservative threshold 0.95, both policies make zero false
+accepts and zero false rejects on this retrospective set.  Outcome-only safely
+cuts 36.35% of measured primary-trial cost; instrumentation-aware safely cuts
+61.10%, an additional 24.75 percentage points.  Regularization sensitivity for
+accuracy delta is `[0.00, +10.81]` percentage points, so the sign is
+non-negative but the magnitude is not stable.
+
+Longer prefixes do not strengthen the case monotonically.  At 10 seconds,
+headline accuracy is 91.89% outcome-only versus 89.19% instrumentation-aware;
+at 15 seconds it is 88.89% versus 91.67%; at 20 seconds it is 86.11% versus
+91.67%, but both 0.95 policies make one false reject.  Five seconds is therefore
+a training-selected operating point, not a test result.
+
+## Strong simulator-aware calibration baseline
+
+The original nested comparison used the same simulator shortlist but did not
+put Frontier's per-anchor prediction in either model.  A stronger retrospective
+audit now gives both models frozen-calibrated simulated throughput, simulated
+SLO pass rate, and simulated feasibility.  Under the same leave-one-cell-out
+folds, 5-second cutoff, L2 logistic family, regularization 1.0, and threshold
+0.95:
+
+| Metric | Sim + outcome | Sim + outcome + instrumentation | Delta |
+|---|---:|---:|---:|
+| Accuracy | 81.08% | 89.19% | +8.11 pp |
+| Balanced accuracy | 72.42% | 81.55% | +9.13 pp |
+| Brier score | 0.1058 | 0.0957 | -0.0101 |
+| Safe early decisions | 20/37 | 25/37 | +5 |
+| Valid full-trial cost reduction | 50.89% | 68.98% | +18.09 pp |
+| Residual verification H20-hours | 0.5240 | 0.3310 | -36.84% |
+
+Both 0.95 policies have zero false accept and zero false reject on this
+retrospective task.  Only three 0.5-threshold classifications differ in favor
+of instrumentation and none in favor of the strong baseline; McNemar's exact
+two-sided p-value is 0.25.  The cell-bootstrap accuracy-delta interval is
+`[0.00,+18.18]` percentage points.  The result is not robust to regularization:
+at 0.1 the strong baseline is more accurate and the instrumentation policy
+makes two unsafe decisions; at 10.0 the strong baseline is also more accurate.
+Thus the stronger comparison still has enough point-estimate headroom for a
+held-out test, but it materially weakens the evidence and makes a prospective
+task-level result mandatory.
+
+## Interpretation
+
+There is enough headroom to run a held-out pilot, but not enough evidence to
+claim the harness contribution:
+
+- the 5-second cost gap is operationally large;
+- only four paired classifications differ, so significance is absent;
+- all examples share one workload/SLO/engine task;
+- completion timestamps are reconstructed from arrival + TTFT + TPOT rather
+  than recorded directly;
+- three adjudication disagreements are concentrated in transient TP4 runs;
+- outcome-only already recovers the simulator shortlist oracle with very few
+  real cells.
+
+The next experiment must therefore freeze the 5-second model and threshold,
+record exact monotonic completions, use a held-out trace, and label each anchor
+with three full repetitions.  The registered protocol is
+`docs/fidelity-aware-harness-protocol-20260714.md`.
+
+## Artifacts
+
+- `runs/fidelity-headroom/analyze_existing.py`
+- `runs/fidelity-headroom/metrics.json`
+- `runs/fidelity-headroom/analyze_prefixes.py`
+- `runs/fidelity-headroom/prefix-metrics.json`
+- `runs/fidelity-headroom/test_analysis.py`
+- `runs/fidelity-headroom/test_prefix_analysis.py`
+- `runs/fidelity-headroom/analyze_strong_baseline.py`
+- `runs/fidelity-headroom/strong-baseline-metrics.json`
+- `runs/fidelity-headroom/test_strong_baseline.py`
+
+## Sanity block
+
+| Family | n | Min | Max | Distinct | Invariant/result |
+|---|---:|---:|---:|---:|---|
+| Real SimFid cell scores | 12 | 1.2833 | 3.2833 | 7 | Non-negative; not identical |
+| Prefix examples at 5 s | 37 | 5 s | 5 s | 1 expected | All 12 cells represented |
+| Adjudicated labels | 37 | 0 | 1 | 2 | 28 positive / 9 negative |
+| Primary/adjudicated disagreement | 37 | 0 | 1 | 2 | 3 TP4 disagreements retained |
+| Full primary elapsed time | 37 | 14.566 s | 62.064 s | 37 | Every 5 s prefix is in range |
+| Outcome probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
+| Instrumentation probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
+| Layer-1 streams | 12 | 14,174 records | 58,725 records | 12 | Contiguous, zero drops |
+| Matched frozen simulator anchors | 37 | pass rate 0.0688 | pass rate 1.0 | 12 pass-rate values | Every prefix matched exactly once |
+| Frozen simulator anchor corpus | 92 | positive throughput | positive throughput | >1 | No duplicate cell/anchor run |
+
+Checked invariants: same folds/model family and cutoff; no full verdict in a
+feature; prefix-only Layer-1 slicing; non-negative costs/counters; bounded
+ratios/probabilities; both labels present; per-config results not identical;
+tie expansion before top-k; no imputation of non-monotonic frontiers.  The main
+limitation is reconstructed request completion time, explicitly marked on all
+37 five-second examples.

docs/fidelity-aware-harness-protocol-20260714.md

@@ -0,0 +1,222 @@
+# Fidelity-aware real-verification harness protocol
+
+Status: **PRE-REGISTERED STAGED EVALUATION; CONTRIBUTION NOT YET ESTABLISHED**.
+
+Date frozen: 2026-07-14 (Asia/Singapore).
+
+## Research question and contribution bar
+
+The harness has an independent systems contribution only if engine-internal
+instrumentation improves a tuning decision beyond what is already achievable
+with a simulator shortlist and external benchmark outcomes.  The intended
+claim is therefore deliberately stronger than “telemetry explains a run”:
+
+> Given the same simulator ranking, the same candidate order, and the same
+> short real-GPU probe, a learned instrumentation-aware verifier reaches a
+> configuration with at most 5% real SLO-goodput regret using materially fewer
+> H20-hours than both (a) simulator top-k followed by full real evaluation and
+> (b) an outcome-only verifier given exactly the same probe.
+
+The paper-facing gate is:
+
+- at least 20% lower real-verification H20-hours than outcome-only calibration;
+- at least 30% lower real-verification H20-hours than simulator top-k plus full
+  real final evaluation;
+- paired 95% task-bootstrap confidence interval for the outcome-only cost
+  reduction strictly above zero;
+- selected-configuration SLO-goodput regret at most 5% on every headline task;
+- no false-safe early accept in the pilot and at most 1% in the expanded suite;
+- profiling, warm-up, confirmation, instrumentation, and failed-run costs are
+  included rather than amortized away.  An amortized profile-cost view may be
+  reported only as a secondary result.
+
+If these conditions fail, instrumentation remains a debugging facility.  It is
+not an independent tuning-harness contribution.
+
+## What is learned, and what is not a rule
+
+The decision target is a stable, repeated real verdict, not a hand-authored
+diagnosis such as “queue length above N means reject.”  Each anchor receives
+three full real repetitions and a frozen 2-of-3 feasibility label.  A nested
+pair of regularized models predicts that label from a fixed prefix:
+
+- **Outcome-only input X:** configuration, offered rate, admitted/completed
+  progress, observed TTFT/TPOT margins, failures, and known workload lengths.
+- **Instrumentation input Z:** the same X plus generic engine state: running and
+  waiting queues, decode-batch shape, KV usage, graph mode and padding, prefill
+  share, preemptions, and model-step rate.
+
+Both models use the same L2 logistic family, train split, standardization,
+regularization, cutoff, and probability threshold.  The only experimental
+difference is Z.  The initial family is intentionally simple: a positive result
+then demonstrates value in the engine signal rather than capacity in a larger
+learner.  A sequence model is admissible only as a later, paired ablation.
+
+### Amendment A1: strengthen the calibration baseline before P2
+
+Frozen 2026-07-14 13:08 Asia/Singapore, after P1 launch but before P1
+completion or analysis.  A baseline audit found that the first frozen P1
+models use the simulator only to define candidate order; their feature vectors
+do not contain the simulator's per-anchor prediction.  This is insufficient
+for the stronger term **outcome-only calibration**.  P1 therefore remains a
+prospective test of the originally frozen cross-workload predictor, but cannot
+by itself open a contribution claim.
+
+For P2/P3, both nested models must additionally receive the identical frozen
+simulator outputs available at that decision: predicted completed throughput
+per GPU, predicted SLO pass rate, and predicted feasibility.  The comparison
+is consequently `sim + config + workload + real outcome prefix` versus that
+exact vector plus real engine state.  Simulator features, regularization,
+cutoff, and thresholds are frozen before any P2 task.  If telemetry does not
+improve this stronger baseline, the harness has no independent contribution.
+
+The same audit also separates algorithm cost from benchmark-oracle cost.
+Headline method cost includes every action the method would execute online:
+simulator profiling/calibration, model onboarding, server startup, warm-up,
+real prefix, continuation after abstention, method-requested confirmation,
+logging overhead, failures, and cleanup.  Exhaustive real-oracle runs and the
+extra repetitions used only to construct 2-of-3 evaluation labels are common
+benchmark annotation cost; they are reported separately and charged to no
+method.  A second, deliberately conservative table adds that common cost to
+all methods.  This prevents both hiding real method cost and making the
+percentage gate mathematically depend on offline ground-truth annotation.
+
+The frozen first policy uses a 5-second prefix, L2 regularization 1.0, and a
+two-sided abstaining threshold of 0.95: accept at `p(feasible)>=0.95`, reject at
+`p(feasible)<=0.05`, otherwise continue the exact same trial to completion.
+Threshold and cutoff were selected on the historical training task and are
+therefore not evidence; all claims come from subsequent held-out tasks.
+
+## Fair baselines
+
+| Method | Simulator | 5-second real prefix | External outcomes | Engine state | Full real continuation |
+|---|---:|---:|---:|---:|---:|
+| Real-only oracle | no | no | full | optional diagnostic | every candidate/anchor |
+| Sim top-k + real final | yes | included in full run | full | no decision use | every shortlisted candidate/anchor |
+| Outcome-only calibration | yes, including its prediction features | yes | yes | no | only on abstention |
+| Instrumentation-aware | same prediction features | yes | yes | yes | only on abstention |
+
+Tie buckets are expanded before top-k.  `k` is selected on training tasks and
+is fixed on held-out tasks; an oracle per-task k is forbidden.  Outcome-only
+receives all information available outside the engine, including config,
+workload, and frozen simulator-prediction features.  Instrumentation cannot use
+any record submitted after the cutoff.  The full label, confirmation votes,
+realized simulator error, and later requests are never model features.
+
+## Staged experiment
+
+### R0: historical premise and headroom audit
+
+The frozen SimFid surface has 12 cells.  The strongest calibrated SLO simulator
+reading has a top tie bucket `{TP2/MNS32, TP2/MNS64}`; full real evaluation of
+those two cells already finds the oracle with zero regret.  Consequently this
+single task cannot demonstrate a selection-count advantage: any method needing
+one real calibration probe and one real final verification has a lower bound of
+two real cells.
+
+The viable estimand is instead the duration and number of full real frontier
+evaluations inside a fixed shortlist.  Historical Phase-6 prefixes are analyzed
+only as training/premise data.  Their request completion times are reconstructed
+from arrival, TTFT, TPOT, and token count, so they cannot support a final claim.
+
+### P1: exact-timestamp prospective pilot
+
+- Engine/model/hardware: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B, one solo
+  server/client on dash0, NVIDIA H20, `TP in {1,2,4}`.
+- Held-out workload: `chat_w20260312_1000`, 60-second replay after the frozen
+  0.1 time scale, raw input `[0,8192]`, exactly 128 output tokens.
+- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, 95% request pass rate.
+- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
+- Per cell: one attainable low offered rate near 0.85x the historical v0.24
+  frontier and one high rate near 1.25x.  The exact threshold and selected
+  request hashes are frozen by a CPU preflight before launch.
+- Each cell uses a fresh server, the accepted long-request warm-up, one
+  unmeasured full-window burn-in, then three repetitions per rate.  Rate order
+  alternates and reverses across cells to prevent a fixed warm-state/order
+  confound.
+- The first repetition supplies the exact prefix.  All three repetitions supply
+  the 2-of-3 label.  Every request records a monotonic completion timestamp;
+  Layer-1 records are cut at the same monotonic boundary.
+- Placement is serialized.  Co-location is forbidden because Phase 6 observed
+  up to 92.86 percentage-point pass-rate shifts under co-location.
+- Hard cap: 3.5 H20-hours, including startup, warm-up, burn-in, all repetitions,
+  failures, and cleanup.  Projected cap violation stops before the next cell.
+
+P1 opens P2 only if all data invariants pass and instrumentation-aware has zero
+false accept/reject, is no worse than outcome-only, and either makes at least
+three additional correct early decisions or improves total valid trial-cost
+reduction by at least 15 absolute percentage points.  The pilot is a gate, not
+paper evidence.
+
+### P2: held-out task replication
+
+If P1 passes, freeze the model and run at least six independent task groups:
+three trace windows spanning distinct date/slot combinations and two SLO
+regimes.  No task used for threshold/model selection enters the headline test.
+The candidate surface is the full 12-cell `TP={1,2,4} x MNS={8,16,32,64}`
+surface.  Splits are by complete task, never by anchor or request.  A task-level
+paired bootstrap (10,000 repetitions, fixed seed) estimates cost and regret
+intervals.  Non-monotonic or split 2-of-3 anchors remain explicit; no frontier
+is imputed.
+
+### P3: end-to-end shortlist and search replay
+
+For each P2 task, run the same frozen simulator and tie-expanded top-k policy.
+Replay the real binary/frontier search under all three verification policies:
+full real, outcome-only, and instrumentation-aware.  The policy consumes only
+prefixes that would have been available at that decision point.  Report:
+
+- selected cell and real SLO-goodput regret;
+- number of real cells, anchors, and confirmations;
+- measured H20-hours and wall time;
+- false accept, false reject, and abstention counts;
+- profile, startup/warm-up, probe, full-continuation, confirmation, logging, and
+  failure cost breakdowns.
+
+### P4: simulator-rank-error attribution
+
+This phase distinguishes an outdated implementation/profile from a structural
+simulator limitation.  For each held-out task compare:
+
+1. the original simulator/profile;
+2. a version-matched re-profiled simulator;
+3. a trajectory-conditioned run supplied with the realized arrival and request
+   length sequence;
+4. outcome-only residual calibration;
+5. instrumentation-aware residual calibration.
+
+The engine trace is extended only as needed with a worker-level step UID and
+CUDA-event duration, because current async submit-to-complete spans overlap and
+are not GPU step time.  Residuals are decomposed into operator-profile error,
+scheduler/state error, and run-to-run noise.  If re-profiling alone restores the
+ranking, the old 30% loss was an implementation/profile defect.  If exact
+profiles and realized trajectories still mis-rank cells, and the residual is
+systematically explained by queue/KV/graph/batch state unavailable to the
+simulator, that is evidence of a structural state-abstraction gap.  Correlation
+alone is not called causal.
+
+## Failure modes that reject the route
+
+- Outcome-only matches or beats instrumentation-aware under the same cutoff.
+- Instrumentation gains average accuracy but introduces false-safe decisions.
+- Gains disappear under task-level rather than request/anchor-level splitting.
+- Savings come only from excluding startup, warm-up, profiling, confirmations,
+  or failed trials.
+- A different cutoff/threshold must be selected after seeing each test task.
+- The simulator top-k baseline already reaches the target with equal or lower
+  total H20-hours.
+- Exact instrumentation overhead exceeds 1% throughput or materially changes
+  p95/p99 latency.
+- Results depend on TP4 transient/non-monotonic trials and do not replicate on
+  held-out tasks.
+
+## Data sanity contract
+
+Every analysis ends with n, min/max, distinct count, label balance, and these
+invariants: non-negative counters/costs; probabilities and ratios in `[0,1]`;
+per-config results not all identical; timestamps monotonic; every prefix record
+at or before its cutoff; selected request ID/arrival/length hashes stable across
+repetitions; exact 128-token completion or counted failure; no dropped Layer-1
+records; 2-of-3 labels reproducible; no co-resident GPU process; total H20-hours
+below the hard cap; final GPUs idle.  A red flag is reported first and blocks
+the contribution claim.

docs/opprof-simfid-overview-20260713.md

@@ -0,0 +1,131 @@
+# SimFid + OpProf Campaign Overview
+
+Date: 2026-07-13. Status: both campaigns **CLOSED**. This is the entry-point
+summary; every claim below links to a frozen protocol and a results document.
+Decision history lives in `docs/opprof_campaign_state.md` (this repo) and
+`replayserve/docs/simfid_campaign_state.md`.
+
+## Why these campaigns exist
+
+The paper's motivation requires three evidence-backed claims:
+
+1. **Simulators cannot replace real replay** for config tuning → SimFid.
+2. **The leverage point is engine-knob configuration, not operator
+   implementation**, at the measured regimes → OpProf P3–P5.
+3. **Tuning is genuinely hard**: the surface is workload-conditioned
+   (sign-flips defeat defaults), real evaluation is expensive (measured GPU
+   cost), and the surface churns across engine versions → OpProf P4/P6.
+
+All three now rest on pre-registered protocols with frozen decision rules,
+Holm-corrected contrasts, and no imputation over censored data.
+
+## Campaign 1 — SimFid (replayserve repo)
+
+**Question.** Can a Frontier-class simulator replace real replay for AITuner
+config tuning? **Verdict: NOT ADEQUATE** under the pre-declared decision rule.
+
+- S2-E (3-config TP family, same model/HW): ranking perfect after throughput
+  calibration, but latency is uncalibrated (TTFT p95 sim/real 0.30–0.38, TPOT
+  0.63–0.79) → false-feasible counterexample across a 50 ms TPOT SLO.
+- S2-R-b (12-cell TP×MNS surface, exact C1 workload replayed in Frontier,
+  184 runs, 0 failures): the decision-bearing frozen-calibrated
+  throughput-proxy reading picks TP1/MNS64 → **30.46% real top-1 regret**;
+  trap detection 3/6; LOAO 0/92. All three adequacy components FAIL.
+- Post-hoc diagnostic (NOT decision-bearing): an SLO-gated reading achieves
+  0–0.76% regret, tau-b 0.967, trap 6/6 — anchor-level SLO errors partially
+  cancel at per-cell peaks. This motivates hybrid sim-prune + real-final
+  designs but must always carry the post-hoc label.
+
+Sources: `replayserve/docs/simfid_s2e_report.md`,
+`replayserve/docs/simfid_s2rb_results.md`.
+
+## Campaign 2 — OpProf (this repo, `docs/opprof/`)
+
+**Question.** Do operators face materially different patterns online than in
+offline rectangular benchmarks ("workload-conditioned operator profiling"),
+and if so, where is the recoverable gain? Patched vLLM 0.24.0
+(`patches/vllm-0.24.0-opprof/`), Qwen3-30B-A3B BF16, dash0 H20.
+
+| Phase | Deliverable | Verdict / headline |
+|---|---|---|
+| P0–P2 | Dual-layer instrumentation (always-on Layer-1 telemetry + sampled Layer-2 Kineto) | Overhead **−0.04%** (CI [−0.17, +0.05]) after the compile-factor fix; Layer-2 perturbs 51.3% → sampled-only by design |
+| P3 | 10-pattern × config matrix (`phase3-{protocol,results}.md`) | **H1b PASS** (5/6 evaluable contrasts, Holm p≈0): irregular patterns carry R64 raggedness +23.0 to +44.8 pp over rectangular controls with 8.3–44.7% useful-token efficiency loss. **H1a INCONCLUSIVE** (Layer-2 window representativeness) |
+| P4 | Ranked optimization plan (`phase4-optimization-plan.md`) | #1 prefix-affine routing: **+82.14%** saturation req/s (P08 vs matched P07, 62.3% fewer prefill tokens). #4 MNS is pattern-conditioned: MNS64 gives +3.4/+3.7% on P06/P10 but **−24.27% on P01** — the sign flips by workload |
+| P5 | Mechanism decomposition (`phase5-{protocol,results}.md`) | All four intuitive mechanisms ≈0 at rho=0.60: raggedness 3.8% n.s., capture-size fix +1.6% n.s. (padding 13.74%→2.40% with no E2E gain), prefix ~0. **Real finding:** P3's P10-vs-P04 gap was largely an arrival-uniformization artifact — replaying recorded (bursty) arrival recovers +12.9%; burstiness helps batch formation at low rate |
+| P6 | Cross-version churn, paired 12-cell surface, vLLM 0.20→0.24 (`phase6-{protocol,results}.md`) | Old #2 config TP2/MNS64 **−29.41%** (solo-confirmed, bounded both sides); TP1 plateau **+13.5%** at MNS8; old argmax TP2/MNS32 held (−0.76%). Formal ARGMAX/RANKING/TRAP **INCONCLUSIVE** — 4 cells right-censored/non-monotonic, no imputation |
+
+**P6 mechanism note.** TP2/MNS64's old anchor now fails with decode-batch
+means 11.5–18.1 and 4.6–9.5% of steps executing outside cudagraph coverage;
+all 37 solo primaries had zero preemptions.
+
+**Combined churn claim (paper-usable).** Across one ~2-month engine upgrade
+the surface below the argmax reorganized (rank-2 config −29%, plateau +13%)
+even though the argmax survived → warm-start/transfer from stale tuning
+surfaces is unreliable; every engine upgrade is a retuning trigger.
+
+## Cross-cutting methodological findings (reusable beyond this paper)
+
+1. **Co-location validity is metric-dependent.** 21 exact same-request pairs
+   (co-located vs solo, P6): throughput and operator shares move <3%, but SLO
+   pass rates flip by up to **+92.86 pp** (0.071→1.000) at frontier anchors —
+   feasible/infeasible verdicts invert. Deltas are cell- and anchor-dependent
+   (0 to +92.9 pp), so no fixed correction exists. Rule adopted: SLO-frontier
+   measurements must be solo; mean-type metrics may be co-located behind the
+   pre-registered A-P3-1 validity gate (which rejected 8-way, passed 4-way).
+2. **Uniformizing arrival distorts efficiency** (−12.9% at low rate) in the
+   opposite direction from intuition — offline benchmarks that regularize
+   arrival misestimate real efficiency.
+3. **Env vars hashed into vLLM compile factors** silently cause cold
+   torch.compile caches and ~4% slower artifacts (root cause of our phantom
+   overhead; fixed with a one-line ignore-list entry). Upstream-report
+   candidate.
+4. **Long-context real traces break short-load-calibrated harness
+   assumptions** (drain deadlines, warm-up stabilization); P10/TP2 never
+   stabilizes (36.8% drift).
+
+## Corrections and honest limits — do NOT quote these stale readings
+
+- P3's "P10 is 14.3% worse than P04" is **superseded by P5**: largely a
+  materialization artifact of uniformized arrival. Quote pattern-vs-control
+  raggedness/padding effects (H1b) and the P5-corrected arrival result
+  instead. The remaining P10-vs-P03 gap (~36%) is workload physics, not
+  recoverable waste at this regime.
+- H1a (operator bottleneck-ranking inversions) is **not refuted** — it is
+  inconclusive at the measured regimes; saturation-regime decomposition
+  remains open.
+- P6 formal verdicts are INCONCLUSIVE by right-censoring, not by data
+  invalidity; the −29.41%/+13.5%/−0.76% numbers are bounded and quotable.
+- Only solo-tier SLO numbers are quotable; co-located W1–W3 artifacts are
+  preserved but superseded.
+- The <3% co-location bound is an empirical gate verified at moderate load on
+  specific patterns, not a theorem; re-run the gate before reusing 4-way
+  placement in new regimes.
+- SimFid's SLO-gated 0–0.76% regret reading is post-hoc, not decision-bearing.
+
+## Artifact map
+
+| Path | Content |
+|---|---|
+| `docs/opprof_campaign_state.md` | Full OpProf decision ledger (echoes, amendments, acceptances) |
+| `docs/opprof/phase{0,2}-*.md`, `docs/opprof/patch-design.md` | Recon, patch design, smoke + overhead evidence |
+| `docs/opprof/phase{3,5,6}-protocol.md` | Frozen pre-registered protocols incl. amendments |
+| `docs/opprof/phase{3,5,6}-results.md`, `phase4-optimization-plan.md` | Results; phase6 metrics pinned by SHA-256 `290ba7fc…` |
+| `patches/vllm-0.24.0-opprof/` | 7-patch series + apply.sh + tests (base `ee0da84a`) |
+| `runs/opprof-phase{3,5,6}/` | Decision-bearing evidence (metrics/manifests/validation); raw Layer-1 JSONL streams are git-ignored (507 MB, kept on disk and dash0) |
+| `docs/simulator-fidelity-frontier-20260711.md` | Standalone review: does the data show simulator mis-ranking rigorously? |
+| `replayserve/docs/simfid_*` | SimFid protocols, results, ledger |
+
+## GPU accounting
+
+OpProf total ≈ **22.2 H20-hours** (P0–P5 ≈ 16.5, P6 5.64 of a 6.0 cap).
+SimFid accounting lives in the replayserve ledger. No prompt or generated
+text appears in any committed artifact; the prompt-bearing trace copy stays
+in git-ignored `trace_windows/`.
+
+## Open items (not committed to)
+
+- Bound the 3 right-censored TP4 cells + TP2/MNS16 (~1.6 H20-h, exceeds the
+  P6 cap) to convert ARGMAX/TRAP into formal verdicts.
+- Saturation-regime mechanism decomposition (P5 analogue at high rho).
+- Upstream reports: compile-factor env poisoning; co-location SLO validity.
+- Synthesis of both campaigns into the paper's motivation section.

docs/opprof/oracle-gap-protocol.md

@@ -0,0 +1,240 @@
+# Static-policy oracle-gap protocol
+
+Status: **FROZEN WITH A-OG-1 THROUGH A-OG-4 AMENDMENTS**.
+
+Date frozen: 2026-07-13 (Asia/Singapore). Existing Phase-3 measurements were
+inspected only to choose the workload pair and rate brackets. They are
+exploratory calibration data, not primary observations in this protocol.
+
+### A-OG-4 — close a majority-shifted boundary (before confirmation scores)
+
+After all four primary frontiers and 70 scores were complete, the controller
+was interrupted before the first confirmation produced a score. The partial
+`P01-C10-r26-rep1` attempt contained no result, sanity file, or score and is
+archived separately. This amendment is therefore blind to confirmation
+outcomes.
+
+The original confirmation schedule still runs first. Afterward, each cell's
+majority-vote frontier is recomputed. If either side of the *actual* final
+boundary has fewer than three trials because the provisional boundary moved,
+the controller runs only enough repetitions at that existing rate to reach
+three, high-to-low, on a fresh server for that config. It then recomputes the
+boundary and repeats for at most three closure rounds. No new rate anchor may
+be added. Failure to obtain a monotone, bracketed boundary with three trials
+per side within three rounds or the 6 H20-hour cap makes the experiment
+inconclusive.
+
+This fills a stopping-rule omission: it does not change any existing score,
+majority rule, phase, config, rate grid, SLO, or oracle calculation.
+
+### A-OG-3 — freeze a per-logical-trial transport retry rule (after 62 scores)
+
+The first attempt at `P06-C10-r1.9-rep0` reproduced the same isolated local
+transport signature seen in A-OG-2: one clean request raised
+`ClientOSError` 3.56 ms after admission, with HTTP status 0, no first token,
+and no output. The other 343 requests produced exactly 512 tokens, the server
+stayed healthy, drain took 4.00 s, and every other client invariant passed.
+The controller again rejected the attempt before creating a `score.json`.
+
+For the remainder of this experiment, an attempt may be quarantined and the
+identical logical trial retried once on a fresh server only if all of the
+following hold: the sole failed client invariant is `clean_failures_zero`;
+exactly one clean request failed; its error is `ClientOSError`, HTTP status is
+0, it produced neither a first token nor output, and it completed within 10
+ms of admission; every successful request has exact output; and the server
+has no crash or error. The attempt is never scored. A second invalid attempt
+for the same logical key, more than one failed request, or any other error
+signature is a stop condition and makes the experiment inconclusive.
+
+This rule is independent of config, rate, and observed performance, and
+supersedes the one-key wording in A-OG-2 without changing how that retry was
+executed. Existing scores remain immutable. No client, grid, policy,
+threshold, order, SLO, or oracle calculation changes.
+
+### A-OG-2 — retry one transport-invalid attempt (after 49 scored trials)
+
+The first attempt at `P06-C01-r2.2-rep0` produced one local
+`ClientOSError` 3.27 ms after admission, with HTTP status 0, no first token,
+and no output. The other 399 requests completed successfully, every successful
+request produced exactly 512 tokens, the server stayed healthy, and all other
+client invariants passed. The controller rejected the attempt before creating
+a `score.json`; no TTFT/TPOT result from this attempt was used to choose the
+recovery rule.
+
+This is a measurement-transport failure, not an observed server SLO outcome.
+The entire attempt is content-hashed and moved under `invalid-attempts/`, then
+the identical logical trial (phase, config, rate, repetition, derived manifest,
+seeds, timeline, and SLO) is run once on a fresh C01 server. All 49 existing
+scores remain immutable. A second client transport failure in the retry is a
+stop condition and makes the experiment inconclusive; it must not be retried
+again. No grid, policy, threshold, order, or oracle calculation changes.
+
+### A-OG-1 — extend the P06 upper bracket (after trial 33)
+
+The controller stopped as registered after C00/P06 remained SLO-feasible at
+every original and upward-extension anchor through 2.3 requests/s. At that
+point 33 trials and 1.519475 H20-hours were complete, all GPU memory had been
+released, and no C00/P06 infeasible upper bound existed. No oracle inference
+was performed.
+
+This amendment changes only the P06 upward-extension list from
+`2.1,2.2,2.3` to `2.1,2.2,2.3,2.4,2.5,2.6,2.8,3.0`. The controller stops at
+the first bracket exactly as before. Existing trials are immutable and are
+reused; config order, P01 rates, SLO, timelines, repetitions, placement,
+metrics, and decision threshold do not change. The resumable controller may
+accept the new code/protocol fingerprint only when all immutable runtime,
+client, model, manifest, config, and base-grid fields match the pre-amendment
+state, and it records both fingerprints under `A-OG-1`.
+
+## Question and decision gate
+
+The candidate motivation is:
+
+> A single global static batching policy leaves at least 10% end-to-end
+> SLO-goodput on the table when serving temporally heterogeneous phases; a
+> phase-aware runtime policy can recover that gap without changing hardware,
+> model, precision, or tensor-parallel topology.
+
+This experiment tests a necessary condition in the existing TP1 policy space
+`{C00,C10,C01,C11}`. The optimistic oracle knows the phase and switches with
+zero delay, zero state-transfer cost, and no prediction error. If even this
+oracle cannot beat the best one-config-for-all-phases policy by 10%, an online
+controller over these MNS/MBT choices cannot do so either.
+
+The primary gate uses a conservative capacity bracket:
+
+- `L[p,c]`: highest offered rate accepted as SLO-feasible for phase `p` and
+  config `c`;
+- `U[p,c]`: lowest higher offered rate accepted as SLO-infeasible;
+- oracle upper bound at phase-time weights `w`:
+  `sum_p w[p] * max_c U[p,c]`;
+- best-static lower bound:
+  `max_c sum_p w[p] * L[p,c]`.
+
+We scan every P01/P06 time mixture, including pure endpoints. The current
+motivation is **REFUTED** if the maximum conservative ratio
+`oracle_upper / static_lower - 1` is below 10%. It is **NOT ESTABLISHED** if the
+bound crosses 10% but the observed point estimate does not. A positive result
+requires a point-estimate gap of at least 10% and then a separately
+pre-registered interleaved-trace validation; this frontier experiment alone
+cannot establish a positive E2E contribution.
+
+The conclusion is scoped to the measured MNS/MBT policy family and the chosen
+strongest-conflict phase pair. It does not rule out new scheduling mechanisms,
+KV-state policies, topology changes, or other workload phases.
+
+## Fixed system boundary
+
+| Item | Frozen value |
+|---|---|
+| Host | `dash0`, one run at a time on physical GPU0 |
+| GPU | NVIDIA H20; no other GPU process anywhere on the host |
+| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, BF16 |
+| Runtime | `/tmp/wjh-opprof-phase2-dash0-20260711/.venv`, vLLM `0.24.1.dev3+g668cfb7e2` |
+| vLLM source | `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0` |
+| Topology | TP1, one server, no data/pipeline parallelism |
+| Fixed mechanisms | chunked prefill on; prefix caching on |
+| Client | Phase-5 timestamp/fixed-rate wrapper over the Phase-3 exact-token client |
+| Seeds | workload `20260712`; trial token-domain seed derived only from phase/rate/repetition, never config |
+
+SLO co-location results in Phase 6 showed pass-rate flips despite small
+throughput deltas. Therefore unused H20s remain idle: parallel placement is not
+authoritative for this experiment.
+
+## Workloads and policies
+
+The pair is chosen before new measurements because Phase 3 showed the strongest
+opposing static preference:
+
+- **P01:** input `U[128,512]`, output exactly 64 tokens, deterministic steady
+  arrivals. C10 lost 24.27% saturation throughput relative to C00.
+- **P06:** 50/50 input mixture `U[128,512]`/`U[4096,8192]`, output exactly 512
+  tokens, deterministic bursts of eight. C10 gained 3.37% over C00.
+
+Both reuse the immutable 32,768-row Phase-3 manifests. For every trial a
+derived manifest preserves request order, lengths, outputs, and arrival class,
+but applies a trial-specific token-seed offset. The same derived manifest is
+used for all four configs. This prevents prefix-cache carry-over when a hot
+server executes several anchors without changing the logical workload.
+
+| Config | Effective MNS | Effective MBT | Extra flags |
+|---|---:|---:|---|
+| C00 | 1024 | 8192 | none |
+| C10 | 64 | 8192 | `--max-num-seqs 64` |
+| C01 | 1024 | 2048 | `--max-num-batched-tokens 2048` |
+| C11 | 64 | 2048 | both flags |
+
+Startup logs must confirm these values. A default drift is a stop condition.
+
+## Load grid, order, and repetitions
+
+Primary grids:
+
+- P01: `{26,28,30,32,34,36}` requests/s; execution order
+  `32,26,36,28,34,30`.
+- P06: `{1.4,1.5,1.6,1.7,1.8,1.9,2.0}` requests/s; execution order
+  `1.7,1.4,2.0,1.5,1.9,1.6,1.8`.
+
+Every primary anchor runs once. For each phase/config, the highest primary
+feasible anchor and its next higher primary anchor are then run two more times,
+giving three trials at both sides of the boundary. If all primary anchors are
+feasible, extend upward in the fixed order P01 `38,40,42` or P06
+`2.1,2.2,2.3,2.4,2.5,2.6,2.8,3.0`.
+If all are infeasible, extend downward in the fixed order P01 `24,22,20` or P06
+`1.3,1.2,1.1`. Stop extending at the first bracket.
+
+One primary server is launched per config in order `C11,C00,C01,C10`.
+Confirmation servers are fresh and launch in reverse order
+`C10,C01,C00,C11`; their boundary anchors run high-to-low. This balances
+machine-time drift and makes confirmation independent of the primary server's
+cache/compiler state.
+
+Timelines:
+
+- P01: 60 s warm-up + 60 s clean measurement; drain cap 120 s.
+- P06: 60 s warm-up + 120 s clean measurement; drain cap 240 s.
+- no Kineto profiling; exact greedy output with `ignore_eos`; maximum client
+  concurrency 256.
+
+A trial is SLO-feasible when at least 95% of requests admitted during the clean
+interval eventually finish successfully and individually satisfy both:
+
+- TTFT <= 2 s for input <= 4,096 tokens; <= 4 s for input <= 32,768; <= 6 s
+  otherwise;
+- TPOT <= 50 ms, computed as `(completion - first_token)/(output_tokens - 1)`.
+
+SLO-goodput is the number of those passing clean-admission requests divided by
+clean seconds. Client schedule lag must stay <=1 s and achieved clean offered
+rate must be within 5% of target. Failure of either condition makes the anchor
+infeasible; its admitted-only latency is not used to rescue it.
+
+At a repeated boundary, feasibility is the majority of three trial verdicts.
+All accepted anchor verdicts must be monotone in offered rate. A persistent
+non-monotone result after the registered repeats is a red flag and stops the
+oracle-gap inference.
+
+## Validity and stopping rules
+
+Before every server launch record host, GPU, driver, clocks, runtime package
+versions, git/source hashes, manifest hashes, exact commands, and process
+contamination. Stop on another GPU process, request/output mismatch, manifest
+drift, server crash, non-finite latency, ratio outside `[0,1]`, negative
+counter, or discontinuous/non-monotone accepted frontier.
+
+The controller is detached and resumable. It kills only process groups it
+created, checks zero GPU memory after every server, never overwrites a complete
+trial, and writes state atomically. The hard budget is 6 H20-hours; expected
+cost is 3.0--4.0 H20-hours and approximately the same wall time because runs
+are serialized.
+
+## Required report
+
+The report includes every trial's target/achieved rate, clean cohort size,
+pass count/rate, SLO-goodput, TTFT/TPOT percentiles, schedule lag, failure
+reasons, accepted frontier brackets, per-phase oracle choices, best static
+choice, equal-time gap, worst-mixture conservative gap, and GPU-hours.
+
+The final statistics section ends with a data-sanity block containing `n`,
+min/max, distinct-value counts, and checks for non-negative counters, ratios in
+`[0,1]`, non-identical per-config results, exact output work, monotone
+frontiers, and continuous rate brackets.

docs/opprof/oracle-gap-results.md

@@ -0,0 +1,199 @@
+# Static-policy oracle-gap results
+
+Status: **FINAL — REFUTED WITHIN THE FROZEN TP1 MNS/MBT POLICY SPACE**.
+
+Date: 2026-07-13. The registered experiment completed 104 valid trials on
+`dash0` and returned `REFUTED`. Even a phase-perfect oracle with zero detection,
+switching, and state-transfer cost is bounded below the registered 10%
+SLO-goodput contribution gate.
+
+The machine result is
+`runs/opprof-oracle-gap/metrics.json` (SHA-256
+`250ba4c1657a8830795ee06392eea4e21c62d958fea11701ba60581ef0266543`).
+All 104 trial-level measurements are in
+`runs/opprof-oracle-gap/trials.csv` (SHA-256
+`60cd901f18bbf107eb130f0095e867c3b6d47a78a42e83bcbe57fecf23cc5f9c`).
+The final resumable controller state is
+`runs/opprof-oracle-gap/controller-state.json` (SHA-256
+`27ec5e9a3cd32a871d583ba8eb6d7d3fe2a338a8fd42369233ada57cd4da6436`).
+
+## Decision
+
+The tested motivation was:
+
+> One static batching policy leaves at least 10% end-to-end SLO-goodput on the
+> table across temporally heterogeneous phases, and a phase-aware runtime can
+> recover it by switching MNS/MBT policies.
+
+The registered 10,001-point mixture scan finds a worst conservative gap of
+**8.333219%**, at P01 time weight `0.0244`. Equal phase time gives
+**7.260726%**. An independent implementation that evaluates every exact
+static-policy crossover gives a slightly more conservative exact maximum of
+**8.333333% = 1/12**, at P01 weight `1/41`. This leaves 1.666667 percentage
+points below the 10% gate.
+
+The negative result is stronger than a failed online prototype. The oracle
+already knows the current phase and pays no switching cost. A realizable
+controller over the same four policies cannot exceed this oracle bound.
+
+This is a scoped refutation, not a universal claim about adaptive serving. It
+rules out the current contribution based on phase-aware selection among the
+four MNS/MBT configurations for the frozen P01/P06 pair, Qwen3-30B-A3B, TP1,
+H20, vLLM 0.24, and the registered SLO. It does **not** rule out a new scheduler,
+KV/cache policy, routing-aware mechanism, topology change, or a different
+workload family.
+
+## Fixed setup
+
+| Item | Value |
+|---|---|
+| Placement | `dash0`, serialized on physical GPU0; GPUs 1-7 idle |
+| GPU | NVIDIA H20, driver 580.95.05 |
+| Model | Qwen3-30B-A3B, BF16 |
+| Runtime | vLLM `0.24.1.dev3+g668cfb7e2`, source `4b253fd8619764b6971a7f2e3a3aa7545f6ace05` |
+| Topology | TP1; one server and one client |
+| Fixed mechanisms | chunked prefill on; prefix caching on |
+| P01 | input `U[128,512]`, exactly 64 output tokens, deterministic steady arrivals |
+| P06 | 50/50 input `U[128,512]` / `U[4096,8192]`, exactly 512 output tokens, deterministic bursts of eight |
+| P01 timeline | 60 s warm-up + 60 s clean measurement |
+| P06 timeline | 60 s warm-up + 120 s clean measurement |
+| SLO feasibility | at least 95% of clean-admitted requests pass both TTFT and TPOT |
+| TTFT SLO | <=2 s for input <=4096; <=4 s for input <=32768; otherwise <=6 s |
+| TPOT SLO | <=50 ms/token |
+
+The four policies were:
+
+| Config | MNS | MBT |
+|---|---:|---:|
+| C00 | 1024 | 8192 |
+| C10 | 64 | 8192 |
+| C01 | 1024 | 2048 |
+| C11 | 64 | 2048 |
+
+## Final SLO frontiers
+
+`L` is the highest majority-feasible offered rate. `U` is the next higher
+majority-infeasible rate. Both sides below have at least three trials, and all
+accepted rate sequences are monotone.
+
+| Config | P01 `L` | P01 `U` | P06 `L` | P06 `U` |
+|---|---:|---:|---:|---:|
+| C00 | 28.0 | 30.0 | 2.3 | 2.4 |
+| C10 | 24.0 | 26.0 | 2.4 | 2.5 |
+| C01 | 28.0 | 30.0 | 2.2 | 2.3 |
+| C11 | 24.0 | 26.0 | 2.2 | 2.3 |
+
+The optimistic oracle upper bound chooses C00/C01 at `U=30` for P01 and C10
+at `U=2.5` for P06. The static lower bound chooses C10 below P01 weight `1/41`
+and C00 above it; they tie at the exact worst point.
+
+| Mixture | Oracle upper | Best-static lower | Conservative gap |
+|---|---:|---:|---:|
+| Pure P06 | 2.500000 | 2.400000 | 4.166667% |
+| Exact worst, P01 weight `1/41` | 3.170732 | 2.926829 | **8.333333%** |
+| Equal phase time | 16.250000 | 15.150000 | **7.260726%** |
+| Pure P01 | 30.000000 | 28.000000 | 7.142857% |
+
+The specialization exists but is too small. Reducing MNS from 1024 to 64
+raises the conservative P06 lower bound from 2.3 to 2.4 req/s, while lowering
+P01 from 28 to 24 req/s. Even after using infeasible `U` values for the oracle
+and feasible `L` values for the static baseline, the best possible phase-aware
+selection cannot reach 10%.
+
+## Robustness finding
+
+The strongest system finding is not config specialization but a repeat-level
+mode flip at P01/26 rps. Both C10 and C11 show the identical verdict sequence
+`rep0=0% pass`, `rep1=100% pass`, `rep2=0% pass`. For C10, TTFT p50 is
+3864.62, 252.71, and 3994.88 ms; for C11 it is 6533.62, 593.44, and
+5812.71 ms. TPOT remains below the 50 ms SLO in these trials.
+
+These trials are measurement-valid: exact outputs, offered-rate tolerance,
+schedule lag, timestamps, and client invariants all pass. The majority rule
+therefore classifies 26 rps as infeasible for both configs. The result does not
+change the oracle-gap decision because neither config supplies the P01 oracle
+maximum or the relevant best-static P01 lower bound.
+
+Token-domain seed and server execution history change together across
+repetitions, so this experiment cannot attribute the flip to MoE routing,
+cache/compiler state, or another source. It does motivate a narrower factorial
+study that crosses token seed with fresh/reused server state and randomizes
+order, while recording routed-expert and per-step scheduler telemetry. That is
+a mechanism question; it should not be presented as evidence for a phase-aware
+MNS/MBT controller.
+
+## Execution and audit history
+
+The 104 scored trials comprise 52 base-grid primaries, 18 registered upward or
+downward extensions, 32 boundary confirmations, and two boundary-closure
+trials. The final closure moved C00/P06 from the provisional `[2.4,2.5)` to
+`[2.3,2.4)` and repeated 2.3 rps to 3/3 feasible.
+
+Four frozen amendments are recorded in the protocol:
+
+- A-OG-1 extended only the P06 upward anchors after C00 remained feasible
+  through the original 2.3-rps limit.
+- A-OG-2 quarantined one P06/C01/2.2 local `ClientOSError` attempt and retried
+  the identical logical trial on a fresh server.
+- A-OG-3 generalized the same pre-score transport rule after one
+  P06/C10/1.9 attempt showed the identical signature.
+- A-OG-4, frozen before any confirmation score, added closure for a
+  majority-shifted boundary without adding new anchors.
+
+The two transport-invalid attempts and one pre-A-OG-4 interrupted attempt
+created no score and do not enter any metric. Their retained tree hashes are,
+respectively,
+`434863ba90513cbc54534ffbc1a13c980b3ef7d567190a0aa3f97b55650acbb2`,
+`a57b1ac5f090680bb70c16b5d709eb2b8ac47dce57c7a03b8077cd9b6d80d831`,
+and `bc3feb53514601e641ef1db204c74cffe3d282b24ad797e5b161468f5d15de5c`.
+
+Ten deliberately overloaded primary anchors exceed both the 1 s schedule-lag
+gate and the 5% offered-rate tolerance. They are correctly classified as
+infeasible and are not used as final boundary points. All 48 final boundary
+trials pass both client-side gates; their maximum schedule lag is 573.59 ms.
+
+The final experiment fingerprint uses repository commit `16177b0`, analyzer
+SHA-256
+`d86ecb1f077472906cbb729bd2c9d4b3a82ac6dfdc90838a17ae300d0767110d`,
+controller SHA-256
+`f7c5c2f74f2002f1e4b097e608d165a3a2e9374fbf07935a4d6d6a7c5d45a83a`,
+and protocol SHA-256
+`173f969a4428643cf6c4b950413aa82bef25cafd163e20d7944370a5d87af435`.
+The archived launch log SHA-256 is
+`54e2f4804a5efee072b670043c4092cf41f1a27664faa05ea71bfc1412c3e9db`.
+
+## GPU accounting and cleanup
+
+The campaign used **4.8877033845 H20-hours**, below the 6.0-hour cap, across
+5 h 41 min 51 s wall time including amendment, audit, restart, and server
+startup intervals. At completion, all eight H20s report 0 MiB, 0% utilization,
+and zero compute processes.
+
+## Sanity block
+
+There are no data-sanity red flags. The P01/26 all-or-none mode flip is a
+scientific robustness finding, not an invalid trial signature; it repeats in
+two configs and all measurement invariants pass.
+
+| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---|
+| Score-row indicator | 104 | 1 | 1 | 1 expected | 104 score files; no overwrite |
+| Target rate (req/s) | 104 | 1.4 | 36.0 | 19 | Non-negative; fixed grid/extensions only |
+| Clean cohort per trial | 104 | 168 | 2160 | 30 | Non-empty and non-negative |
+| Pass rate | 104 | 0.0 | 1.0 | 57 | All ratios in `[0,1]` |
+| SLO-goodput (req/s) | 104 | 0.0 | 27.816667 | 59 | Non-negative; per-cell results not all identical |
+| Boundary pass rate | 48 | 0.0 | 1.0 | 33 | Both final sides have three trials |
+| Boundary max schedule lag (ms) | 48 | 2.145861 | 573.593768 | 48 | All below the 1000 ms gate |
+| Exact-output indicator | 85,402 | 1 | 1 | 1 expected | Every clean request produced the requested token count |
+| Frontier cells | 8 | 8 | 8 | 1 expected | 8/8 bracketed and monotone |
+| Registered weight scan | 10,001 | 0.0 | 1.0 | 10,001 | Continuous 0.0001 increments, endpoints included |
+| Campaign H20-hours | 1 | 4.887703 | 4.887703 | 1 | Non-negative and below 6.0 |
+| Final GPU memory/utilization | 8 | 0 MiB / 0% | 0 MiB / 0% | 1 expected | Zero compute processes |
+
+Checked invariants: fixed model, runtime, manifests, SLOs, config values, seeds,
+and serialized placement; exact output work; nondecreasing timestamps;
+non-negative counters and latencies; pass ratios in range; offered-rate and
+schedule gates; majority-of-three final boundaries; monotone accepted
+frontiers; complete 10,001-point scan; independent exact-crossover
+recomputation; transport-attempt quarantine; GPU hard-cap compliance; and
+complete GPU cleanup.

docs/opprof/patch-design.md

@@ -0,0 +1,434 @@
+# OpProf dual-layer instrumentation patch design
+
+Status: **design for review; no vLLM patch has been implemented**.
+
+Target source: vLLM `v0.24.0`, commit `ee0da84ab9e04ac7610e28580af62c365e898389`.
+Target campaign: Qwen3-30B-A3B serving on NVIDIA H20 (SM90), using the V1 engine.
+All source paths and line numbers below are relative to the pinned clone.
+
+## Approved dispositions (2026-07-11)
+
+- Use JSONL with `msgspec`, the proposed context/chunk histogram edges, and an
+  8192-record bounded queue.
+- Keep exact expert loads Layer-2-only.
+- Use the community BF16 Qwen3-30B-A3B checkpoint. TP1 is primary, with TP2 and
+  TP4 counterpoints; record the selected MoE backend logs in every run.
+- Use two profiler warm-up iterations followed by eight active iterations.
+- Apply the 3% Layer-1 overhead gate to the **upper bound of the 95% confidence
+  interval** for every primary serving metric.
+- Reject `--disable-log-stats` when OpProf is enabled; retain the five-file
+  fail-fast design.
+
+## Goal and success criteria
+
+The patch should make every serving iteration conditionable on its request and
+execution pattern while keeping the always-on path below a 3% serving overhead
+budget. Heavy kernel tracing and exact MoE routes are sampled separately.
+
+The proposed split is:
+
+1. **Layer 1:** one compact composition record per scheduler step, emitted by
+   the scheduler process rather than every tensor-parallel worker.
+2. **Layer 2:** short, sampled windows using vLLM's existing torch-profiler
+   configuration and `/start_profile` and `/stop_profile` endpoints.
+
+The design deliberately does not add hooks to GPU kernels, attention layers,
+the Qwen model, or the OpenAI serving API. Those surfaces are not needed to
+answer the Phase 0 profiling questions.
+
+## Assumptions and tradeoffs
+
+- The campaign uses the V1 engine and the default vLLM scheduler. Both the
+  synchronous and `AsyncScheduler` paths inherit the base scheduler hooks.
+- The H20 backend statement assumes the usual unquantized BF16/FP16
+  Qwen3-30B-A3B checkpoint, `moe_backend=auto`, and no LoRA. Quantization,
+  batched expert format, or an explicit backend can change kernel selection.
+- Layer 1 requires normal stats collection, which is enabled by default. If
+  OpProf is enabled together with `--disable-log-stats`, initialization should
+  fail with an actionable error. Supporting that unusual combination would
+  require an extra stats path and a larger patch.
+- The context-length histogram records the sequence length at the end of the
+  scheduled model input (`num_computed_tokens` before scheduling plus tokens
+  scheduled in this step). This matches the worker-side sequence-length
+  construction at `vllm/v1/worker/gpu_model_runner.py:2010-2019` and avoids
+  retaining raw request lengths.
+- “Decode tokens” means tokens scheduled for requests classified by vLLM as
+  generation-phase requests, including scheduled speculative tokens. It is not
+  the number of subsequently accepted output tokens. The existing classifier
+  and its chunked-prefill semantics are at `vllm/v1/utils.py:780-813`.
+
+## Existing facilities to reuse
+
+These are zero-patch wins:
+
+| Need | Existing v0.24.0 facility | Design consequence |
+|---|---|---|
+| Scheduled request and token map | `SchedulerOutput.num_scheduled_tokens`, total tokens, preempted IDs, and new/cached request data at `vllm/v1/core/sched/output.py:180-219` | Derive composition in the scheduler; do not add fields to `SchedulerOutput`. |
+| Prefill/decode classification | `compute_iteration_details()` at `vllm/v1/utils.py:780-813` | Reuse the exact definition already used by the iteration log. |
+| Queue/KV/prefix stats | `SchedulerStats` at `vllm/v1/metrics/stats.py:170-198` and construction at `vllm/v1/core/sched/scheduler.py:2228-2264` | Reuse field semantics, but snapshot at schedule time to avoid async misattribution. |
+| Per-step CUDA-graph descriptor | `CUDAGraphStat` at `vllm/compilation/cuda_graph.py:32-37`, populated at `vllm/v1/worker/gpu_model_runner.py:3919-3934` | Reuse the object; only broaden the condition under which it is returned. |
+| Sampled CPU/CUDA tracing | profiler config and schedule at `vllm/config/profiler.py:33-105`, endpoints at `vllm/entrypoints/serve/profile/api_router.py:21-45` | Layer 2 needs configuration and orchestration, not a new profiler implementation or endpoint. |
+| Exact routed expert IDs | routed-expert capture callback and buffers at `vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58-84` and `vllm/v1/worker/gpu_model_runner.py:7382-7437` | Use only in a separate sampled Layer-2 run; it is too expensive for Layer 1. |
+
+## Layer 1: always-on composition records
+
+### Data flow and hook placement
+
+```text
+Scheduler.schedule()
+  -> snapshot request composition, queues, KV and prefix deltas
+  -> SchedulerOutput -> TP workers -> ModelRunnerOutput.cudagraph_stats
+  -> Scheduler.update_from_output()
+  -> finalize the same step record -> bounded queue -> background JSONL writer
+```
+
+The scheduler is the ownership boundary for Layer 1. It sees the full logical
+batch once, so recording in each TP worker would duplicate data and require an
+aggregation protocol.
+
+Exact proposed hooks against the pinned clone:
+
+1. **Initialize one recorder in the base scheduler constructor.** The scheduler
+   already owns stats state and passes the normal `log_stats` setting into the
+   KV-cache manager (`vllm/v1/core/sched/scheduler.py:78-87` and
+   `vllm/v1/core/sched/scheduler.py:250-260`). If `VLLM_OPPROF_DIR` is set,
+   construct the recorder and reject `log_stats=False`.
+2. **Begin a record inside `Scheduler.schedule()`.** Read prefix counters at
+   method entry, then finish the snapshot immediately after `SchedulerOutput`
+   and connector metadata are constructed and before
+   `_update_after_schedule()` mutates request state
+   (`vllm/v1/core/sched/scheduler.py:1012-1100`). Prefix lookup counters are
+   updated synchronously during this schedule call
+   (`vllm/v1/core/kv_cache_manager.py:202-242` and
+   `vllm/v1/core/sched/scheduler.py:675-712,897-915`), so before/after deltas
+   remain attributable even when several batches are in flight.
+3. **Finalize in `Scheduler.update_from_output()`.** Pair the returned
+   `ModelRunnerOutput` with the exact `SchedulerOutput` at
+   `vllm/v1/core/sched/scheduler.py:1464-1477`, and enqueue after normal output
+   processing near `vllm/v1/core/sched/scheduler.py:1791-1803`. Store pending
+   drafts by `id(scheduler_output)` and remove them on finalize; the identifier
+   never leaves the process. This is robust to the batch queue, which retains
+   and later returns the matching output object
+   (`vllm/v1/engine/core.py:519-632`).
+4. **Return the existing CUDA-graph stat when OpProf is enabled.** Change the
+   condition at `vllm/v1/worker/gpu_model_runner.py:3919-3926` from only
+   `observability_config.cudagraph_metrics` to that flag **or**
+   `VLLM_OPPROF_DIR`. The file already imports `vllm.envs` at line 23. No CUDA
+   synchronization or tensor transfer is introduced.
+5. **Close through the existing scheduler shutdown.** Drain and join the writer
+   before the scheduler reports shutdown complete at
+   `vllm/v1/core/sched/scheduler.py:2285-2295`; retain an `atexit` fallback for
+   abnormal embedding/test lifecycles.
+
+The scheduler assigns a monotonically increasing local `step_index` at begin.
+Every real scheduler call is represented, including zero-token steps. A DP-only
+dummy execution has no `SchedulerOutput` and is outside this composition
+stream; if DP is later in campaign scope, add a distinct `dummy_step` marker
+rather than pretending it has a request composition.
+
+### Record schema
+
+Use schema-versioned JSON Lines. An illustrative record is:
+
+```json
+{"schema":1,"engine_id":"dp0-pid1234","step_index":42,"submit_wall_ns":1783761000000000000,"submit_mono_ns":8920000000,"complete_mono_ns":8922371000,"model_executed":true,"scheduled_requests":37,"decode_batch_size":32,"prefill_requests":5,"prefill_tokens":896,"decode_tokens":32,"chunked_prefill":{"first":2,"middle":1,"final":1,"unsplit":1,"tokens":896,"chunk_size_hist":[0,0,1,1,1,1,1,0,0]},"context_length_hist":[0,0,3,6,12,10,5,1,0,0,0,0],"preemptions":0,"queues":{"running":37,"waiting":9,"deferred":0},"kv":{"total_blocks":120000,"free_blocks":42000,"used_blocks":78000,"usage":0.65},"prefix":{"local":{"requests":2,"queries":4096,"hits":3072,"preempted_requests":0,"preempted_queries":0,"preempted_hits":0},"external":null},"cudagraph":{"hit":true,"runtime_mode":"PIECEWISE","unpadded_tokens":928,"bucket_tokens":1024,"padding_tokens":96},"moe_expert_load":null,"dropped_records_before":0}
+```
+
+Required field semantics:
+
+- `submit_wall_ns` permits joining to external workload logs;
+  `submit_mono_ns` and `complete_mono_ns` provide stable within-process
+  ordering and elapsed time without wall-clock jumps.
+- `scheduled_requests` is the number of entries in
+  `num_scheduled_tokens`. `decode_batch_size`, `prefill_requests`,
+  `prefill_tokens`, and `decode_tokens` reuse vLLM's classifier.
+- `chunked_prefill` contains aggregate split information, never request IDs.
+  A context request is `first` when the scheduled range ends before its prompt
+  ends, `middle` when it was already a prefill chunk and remains incomplete,
+  `final` when a prior chunk completes, and `unsplit` when its prefill completes
+  in one step. `chunk_size_hist` uses token-count buckets
+  `(16, 32, 64, 128, 256, 512, 1024, 2048, +inf)`.
+- `context_length_hist` includes every scheduled request, with fixed upper
+  edges `(128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536,
+  131072, +inf)`. The final array therefore has 12 bins. The example above is
+  illustrative and will be checked against the final chosen edges in tests.
+- `preemptions` is the size of the step's `preempted_req_ids`, created at
+  `vllm/v1/core/sched/scheduler.py:1059-1076`. It is not an interval-derived
+  Prometheus value.
+- Queue lengths and KV blocks are captured immediately after scheduling. KV
+  usage follows the existing null-block-adjusted calculation at
+  `vllm/v1/core/block_pool.py:692-711`. `deferred` means skipped/deferred
+  waiting requests, matching `num_skipped_waiting_reqs` in
+  `vllm/v1/metrics/stats.py:174-183`.
+- Local and external prefix fields are per-schedule deltas copied from the
+  existing mutable counters. The existing stats drain resets those objects at
+  `vllm/v1/core/kv_cache_manager.py:190-200` and
+  `vllm/v1/core/sched/scheduler.py:2235-2242`; OpProf must read, not drain or
+  replace, them. These retain vLLM's lookup semantics: a local lookup can be
+  counted before allocation later rejects that waiting request, so prefix
+  `requests` is not required to be less than or equal to scheduled prefill
+  requests.
+- `cudagraph.hit` is the operational serving definition `runtime_mode != NONE`.
+  `FULL` selects the full graph path and `PIECEWISE` selects graph-wrapped
+  compiled regions; the latter must not be interpreted as full-step graph
+  coverage. Startup normally captures all dispatcher descriptors at
+  `vllm/v1/worker/gpu_model_runner.py:6595-6643`, but the stat itself has no
+  capture-versus-replay bit, so a rare lazy first capture cannot be
+  distinguished from replay. `bucket_tokens` is `num_padded_tokens` in the
+  existing stat.
+- `moe_expert_load` is explicitly null in Layer 1. This avoids accidentally
+  presenting unavailable data as zero.
+
+### Encoding, buffering, and flush
+
+JSONL is recommended over a compact binary ring buffer for Phase 1 because the
+schema will evolve during campaign bring-up, records need to be directly
+inspectable beside Kineto traces, and `msgspec` is already a vLLM dependency
+(`requirements/common.txt:35`).
+Encoding one small dictionary with a reused `msgspec.json.Encoder` avoids the
+standard `json` module's larger CPU cost. Once the schema stabilizes, the same
+record can be moved to a binary format only if Phase 2 measurements show JSONL
+is the bottleneck.
+
+The foreground path encodes once and performs non-blocking `put_nowait()` into
+a bounded queue of 8192 records. A single daemon writer thread drains into one
+file per EngineCore/DP rank and process. It flushes userspace buffers every
+1 MiB or one second, whichever comes first, and on clean process exit; it never
+calls `fsync()` per record. When the queue is full, the serving thread drops the
+new record, increments a counter, and the next successful record reports the
+gap in `dropped_records_before`. Shutdown emits a footer with encoded, written,
+and dropped counts. The file name includes schema, DP rank, PID, and start time,
+but not TP rank because there is one scheduler record stream.
+
+The sole on/off switch is:
+
+```text
+VLLM_OPPROF_DIR=/absolute/output/directory
+```
+
+Unset or empty means a true no-op: no recorder, queue, thread, record
+construction, or graph-stat broadening. Directory validation and a clear
+startup log happen before serving. Runtime toggling is intentionally omitted;
+it adds synchronization and ambiguous partial files without helping the
+campaign.
+
+### Expected cost and Phase 2 overhead gate
+
+These are estimates, not measurements:
+
+- composition and two fixed histograms: about 20-80 microseconds per step,
+  linear in scheduled requests;
+- `msgspec` encoding plus a non-blocking queue insertion: about 5-20
+  microseconds per step;
+- CUDA-graph stat construction: below 2 microseconds and no GPU sync;
+- disk I/O: off the serving critical path, subject to bounded-queue drops.
+
+The expected foreground total is roughly 25-100 microseconds per step. At a
+2 ms decode step, the high end would exceed 3%, so the budget is a measurement
+gate, not a claim.
+
+Phase 2 should use the same Qwen3 checkpoint, request trace, random seed,
+hardware, vLLM commit, TP/EP topology, CUDA-graph configuration, and cache
+state for off/on comparisons. Alternate off/on ordering, warm up before each
+measurement, and run at least five paired repeats. Report throughput and
+p50/p95 TTFT and TPOT, plus CPU utilization, log bytes per step, queue high
+watermark, and dropped-record count. The acceptance rule is less than 3%
+regression for every declared primary serving metric, with bootstrap confidence
+intervals reported. Whether the point estimate or upper 95% bound is the hard
+gate is an open decision.
+
+Correctness checks for every run are: contiguous step indices except explicitly
+reported drops, scheduled token sums matching the prefill/decode split,
+histogram counts matching scheduled request/chunk counts, KV ratio in `[0,1]`,
+non-negative counters, and identical generated outputs for the deterministic
+test trace.
+
+## Layer 2: sampled kernel windows
+
+### Trigger and window
+
+Use the existing profiler API with a run-specific output directory and
+`ignore_frontend=true`. The worker start/stop RPC already fans out to all
+workers (`vllm/v1/executor/abstract.py:256-257` and
+`vllm/v1/executor/multiproc_executor.py:340-402`), and each GPU worker creates a
+CPU+CUDA profiler with a rank-qualified trace name
+(`vllm/v1/worker/gpu_worker.py:929-980`). No Layer-2 vLLM code patch is needed.
+
+Recommended initial configuration:
+
+```text
+profiler=torch
+torch_profiler_dir=<run>/kineto
+ignore_frontend=true
+delay_iterations=0
+max_iterations=8
+wait_iterations=0
+warmup_iterations=2
+active_iterations=8
+torch_profiler_record_shapes=false
+torch_profiler_with_memory=false
+torch_profiler_with_stack=false
+torch_profiler_with_flops=false
+torch_profiler_use_gzip=true
+torch_profiler_dump_cuda_time_total=true
+```
+
+The primary trigger should be step-count based: an external campaign controller
+POSTs `/start_profile` immediately before a desired composition regime. The
+built-in schedule records two warm-up plus eight active model iterations and
+`max_iterations=8` then stops the underlying profiler on the following worker
+step. The “exceeds max” control semantics are explicit at
+`vllm/profiler/wrapper.py:83-114` and in
+`tests/v1/worker/test_gpu_profiler.py:77-98`; the controller should still POST
+`/stop_profile` afterward to clear the active control state.
+Worker iteration boundaries already call `profiler.step()` and annotate context
+and generation token counts at `vllm/v1/worker/gpu_worker.py:803-827`. On-demand
+manual POST remains useful for debugging. A time-based trigger is a fallback
+for live traffic but is less reproducible because it yields a variable number
+of steps. Phase 1 should confirm that the trace contains exactly eight active
+iteration annotations; profiler scheduling and trace callbacks live at
+`vllm/profiler/wrapper.py:159-226,290-307`.
+
+The profile directory should be unique per campaign server run. The HTTP start
+endpoint does not accept a `profile_prefix`, and a worker that is restarted for
+another window retains the trace name chosen on first initialization
+(`vllm/v1/worker/gpu_worker.py:939-975`). Multiple windows in one server run
+therefore share a directory and need a controller manifest containing start/
+stop wall times and produced filenames. Use a new directory only when the
+server is restarted; adding a new endpoint parameter is not justified for
+Phase 1. Every TP worker emits its own trace, and the rank suffix contains DP,
+PP, TP, DCP, EP, and global rank information
+(`vllm/distributed/utils.py:664-695`). Keep these traces separate and aggregate
+only offline.
+
+### CUDA graphs and Kineto interpretation
+
+Keep CUDA graphs enabled in the primary sampled windows so kernel time reflects
+the serving configuration. The pinned code profiles CUDA activity
+(`vllm/v1/worker/gpu_worker.py:953-960`) while a captured path invokes
+`CUDAGraph.replay()` instead of rerunning the Python callable
+(`vllm/compilation/cuda_graph.py:233-360`). Therefore Kineto/CUPTI can observe
+CUDA activity launched during replay, but the Python/PyTorch operator scopes
+that created the graph are not re-executed and cannot be assumed to retain
+per-layer attribution. vLLM explicitly documents the related limitation that
+layerwise NVTX tracing does not work with CUDA graphs
+(`vllm/config/observability.py:60-63`).
+
+The clone does not contain a stronger guarantee about whether a particular
+PyTorch/CUPTI build will present every graph node as an individually named
+kernel versus a coarser graph-launch view. Treat that as an empirical Phase 2
+check. Run one matched eager (`cudagraph_mode=NONE`) taxonomy window to identify
+kernel families, but do not use its timings as the production baseline.
+
+### Calibration and MoE routing
+
+Join each profiler iteration annotation to Layer-1 records by rank-independent
+step order, the profiler window marker, timestamps, and the prefill/decode token
+counts. Aggregate kernels into attention, router/top-k, expert GEMMs, dense
+GEMMs, normalization/activation, collectives, and sampling. For TP, report both
+sum-of-rank GPU work and the maximum per-rank critical-path time. Fit or tabulate
+kernel-time attribution conditioned on composition, context histogram, graph
+runtime mode, and capture bucket, then validate on held-out windows rather than
+the same samples used for calibration.
+
+For the Phase 2 “operator time approximately equals iteration time” gate, do
+not naively add overlapping kernels. Compute the union of CUDA kernel intervals
+per rank, use the maximum rank as the distributed GPU critical path, and retain
+an explicit CPU/queue/unattributed residual against Layer 1's submit-to-complete
+span. Operator-category interval unions plus the residual must reconstruct the
+span; summed GPU work is reported separately and may legitimately exceed wall
+time.
+
+Exact per-layer expert loads are Layer-2-only. A separate server start with
+`--enable-return-routed-experts` exposes per-token, per-layer top-k IDs through
+the existing router callback (`vllm/model_executor/layers/fused_moe/router/base_router.py:233-278`).
+Aggregate those arrays offline into per-layer expert counts, entropy, max/mean
+load, coefficient of variation, and top-k imbalance. Do not enable it in the
+normal Layer-1 or baseline Kineto run: its worker transit buffer costs a few MB,
+and the scheduler-side slot buffer can reach multiple GB with a CPU fancy-index
+copy per step (`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:223-305`).
+
+## Patch surface estimate
+
+Proposed implementation and tests:
+
+| File | Approximate changed/new lines | Purpose |
+|---|---:|---|
+| `vllm/envs.py` | 4 | Declare and parse `VLLM_OPPROF_DIR`. |
+| `vllm/v1/opprof.py` (new) | 220 | Schema, schedule snapshot/deltas, histograms, pending-step pairing, encoder, bounded writer, drop/footer handling. |
+| `vllm/v1/core/sched/scheduler.py` | 32 | Initialize recorder; begin before state mutation; finalize with the matching model output. |
+| `vllm/v1/worker/gpu_model_runner.py` | 4 | Return existing `CUDAGraphStat` when either built-in metrics or OpProf needs it. |
+| `tests/v1/core/test_opprof.py` (new) | 190 | Schema/invariants, chunk classification, async pairing, disabled no-op, bounded-queue drops, shutdown flush. |
+| **Total** | **5 files / about 450 lines** | About 260 production lines and 190 test lines. |
+
+No patch is proposed for `ProfilerConfig`, profile endpoints, profiler wrapper,
+`SchedulerOutput`, `SchedulerStats`, Prometheus/logging, CUDA kernels, fused MoE
+kernels, the Qwen model, or frontend APIs. If support for
+`--disable-log-stats` is required, add about 12 lines in
+`vllm/v1/core/kv_cache_manager.py`, making the estimate six files and about 462
+lines; the smaller fail-fast design is recommended for this campaign.
+
+## Risks and mitigations
+
+- **Foreground CPU overhead:** request scanning and JSON encoding are the likely
+  hot spots. Use fixed-size arrays, a reused encoder, no raw lists, no blocking
+  I/O, and enforce the Phase 2 gate.
+- **Async-scheduling races:** queue/KV/prefix state observed during
+  `update_from_output()` may include later in-flight schedules. Snapshot it
+  inside the synchronous `schedule()` call; only the immutable returned
+  CUDA-graph stat is added at completion. Pair by the exact `SchedulerOutput`
+  object and assert one begin/one finalize.
+- **TP and EP aggregation:** Layer 1 is scheduler-owned and emitted once. Layer
+  2 remains per rank. Interpret TP critical path as the slowest rank and, for an
+  expert-parallel routed-expert sample, aggregate expert ownership offline.
+- **Log volume:** at 500 steps/s, a 1.0 KiB record is about 44 GB/day per
+  EngineCore. Rotate by campaign run, compress completed files, and use the
+  bounded queue/drop counters. Production-long captures need a later sampling
+  or rotation policy; Phase 1 files should be bounded by run duration.
+- **Graph semantics:** `PIECEWISE` is partial graph coverage, not a binary full
+  hit. Preserve `runtime_mode`, unpadded tokens, bucket, and padding rather than
+  reducing the record to one boolean.
+- **Profiler perturbation:** Kineto has medium-to-high overhead and trace flushes
+  can stall. Use short windows, unique directories, disabled stacks/shapes and
+  memory by default, and never use Layer-2 latency as an unprofiled performance
+  result.
+- **Backend drift:** the unquantized SM90 oracle prioritizes Triton, but
+  shape-specific fallbacks remain possible. Record startup backend logs and the
+  full checkpoint/parallel/quantization configuration with every run.
+
+## Open decisions for review
+
+1. Approve JSONL with `msgspec`, the two proposed histogram edge sets, and a
+   bounded 8192-record writer queue.
+2. Approve exact expert-load collection as Layer-2-only, rather than adding a
+   new always-on GPU histogram kernel.
+3. Confirm the Qwen3 checkpoint precision/quantization and intended TP/EP/DP
+   topology; the Triton backend conclusion depends on these inputs.
+4. Choose the first profiler window: recommended two warm-up plus eight active
+   iterations, or a longer active window.
+5. Decide whether the 3% gate applies to metric point estimates or to the upper
+   bound of their 95% confidence intervals.
+6. Confirm that campaign runs may reject `--disable-log-stats`; supporting it
+   adds one file and about 12 lines.
+
+## Data sanity block
+
+- **Patch estimate:** n=5 files; per-file line-estimate min=4, max=220,
+  distinct values=4; total about 450 lines, of which about 260 are production
+  and 190 are tests.
+- **Histogram definitions:** n=2 arrays; bin-count min=9, max=12,
+  distinct=2. In the illustrative record, context-bin sum=37 equals scheduled
+  requests, chunk-bin sum=5 equals prefill requests, KV used=120000-42000, and
+  graph padding=1024-928.
+- **Estimated foreground components:** n=3 timed components; min is below 2
+  microseconds for graph-stat construction, max is 80 microseconds for
+  composition; distinct ranges=3. Their conservative combined foreground
+  estimate is 25-100 microseconds per step.
+- **Invariants checked:** counters and histogram bins are non-negative; KV
+  usage and prefix hit ratios are bounded in `[0,1]`; graph bucket is at least
+  unpadded tokens; `NONE` is the only graph miss mode; no raw request IDs or
+  context-length lists are serialized; one scheduler stream avoids TP
+  duplicates.
+- **Measurement status:** n=0 benchmark runs; min/max and cross-configuration
+  distinctness are not applicable. The cost figures are estimates and cannot
+  support an overhead conclusion until Phase 2.

docs/opprof/phase0-recon-vllm-0.24.0.md

@@ -0,0 +1,446 @@
+# Phase 0 recon: vLLM 0.24.0 observability
+
+Status: source recon and patch design complete; **no instrumentation patch was
+implemented**.
+
+Campaign target: operator-level, pattern-conditioned profiling of
+Qwen3-30B-A3B serving on NVIDIA H20. This report is based on the pinned source
+clone only. No GPU was used, no package or model was installed, and no dash
+host was contacted.
+
+Unless explicitly prefixed with `v0.20.0:`, every source path and line number
+below is relative to `/home/gahow/phd/vllm-v0.24.0` at the pinned commit.
+
+## Outcome
+
+vLLM 0.24.0 already contains most of the raw ingredients, but they are split
+across scheduler internals, aggregate metrics, worker traces, and an optional
+routed-expert return path. The smallest useful patch is therefore not a new
+profiler. It is one scheduler-owned per-step record that joins existing batch
+composition to the existing `CUDAGraphStat`; short torch-profiler windows and
+exact expert routes remain sampled Layer 2 facilities.
+
+The proposed patch is detailed in
+[`patch-design.md`](patch-design.md). Estimated surface: five files and about
+450 lines including tests, with about 260 production lines.
+
+## 1. Source pin and compatibility record
+
+The requested tag existed and the exact requested clone command completed
+successfully:
+
+```text
+git clone --depth 1 --branch v0.24.0 https://github.com/vllm-project/vllm /home/gahow/phd/vllm-v0.24.0
+```
+
+| Field | Pinned value |
+|---|---|
+| Repository | `https://github.com/vllm-project/vllm` |
+| Local clone | `/home/gahow/phd/vllm-v0.24.0` |
+| Tag | `v0.24.0` (`git describe --tags --exact-match`) |
+| Commit | `ee0da84ab9e04ac7610e28580af62c365e898389` |
+| Clone completion | `2026-07-11T08:26:16Z` (`2026-07-11 16:26:16 +08:00`) |
+| Checkout | Detached HEAD, clean |
+| Top-level entries | 45, including hidden entries and `.git` |
+| Listing SHA-256 | `75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58` |
+
+The listing digest is reproducible from entry names only, not file contents:
+
+```text
+LC_ALL=C find . -mindepth 1 -maxdepth 1 -printf '%f\n' |
+  LC_ALL=C sort | sha256sum
+```
+
+This definition includes `.git`, sorts bytewise under the C locale, and hashes
+the 45 LF-terminated root names. It is recorded to make the otherwise ambiguous
+phrase “sha256 of the top-level directory listing” precise.
+
+### Torch and CUDA requirements
+
+- Build and NVIDIA runtime requirements pin `torch==2.11.0` in
+  `pyproject.toml:1-13`, `requirements/build/cuda.txt:1-10`, and
+  `requirements/cuda.txt:1-10`; CMake repeats 2.11.0 as the supported CUDA and
+  ROCm torch version at `CMakeLists.txt:62-72`.
+- The NVIDIA requirements also pin `torchvision==0.26.0`,
+  `flashinfer-python==0.6.12`, `flashinfer-cubin==0.6.12`, and CUDA-13 extras
+  for CUTLASS DSL and Humming kernels (`requirements/cuda.txt:6-29`). Setup
+  strips/substitutes those extras for CUDA 12 builds
+  (`setup.py:1066-1083`).
+- The source default is CUDA 13.0 (`vllm/envs.py:85-87,555-563`) and the Docker
+  default is CUDA 13.0.2 (`docker/Dockerfile:14-42`). Wheel detection maps CUDA
+  major 12 to `cu129` and major 13 to `cu130` (`setup.py:528-568`).
+- The pinned installation document still describes the default precompiled
+  binaries as CUDA 12.9 and lists CUDA 12.8 and 13.0 variants
+  (`docs/getting_started/installation/gpu.cuda.inc.md:4-19,24-54`). This is a
+  packaging/default distinction, not evidence that 13.0 is unsupported.
+- NVIDIA compute capability 7.5+ is documented, and SM90 is in every relevant
+  CMake CUDA architecture set (`docs/getting_started/installation/gpu.cuda.inc.md:9-19`;
+  `CMakeLists.txt:105-118`). Optional FA3 and DeepGEMM builds need CUDA 12.3+,
+  FlashMLA needs 12.9+, and the Hopper CUTLASS MoE build needs 12.3+
+  (`setup.py:1111-1139`; `CMakeLists.txt:844-868`).
+
+Implication for dash0 later: H20/SM90 is a supported architecture, but the
+actual driver and installed CUDA were intentionally not queried in Phase 0.
+Phase 2 must choose a torch 2.11.0-compatible `cu129` or `cu130` artifact after
+checking the host driver. A wheel built against a different torch/CUDA build is
+not interchangeable; the source documentation explicitly warns about binary
+incompatibility (`docs/getting_started/installation/gpu.cuda.inc.md:14-19`).
+
+## 2. Built-in observability inventory
+
+### 2.1 Torch profiler integration
+
+**Configuration and control.** v0.24.0 does not contain the legacy
+`VLLM_TORCH_PROFILER_DIR` symbol: a repository-wide exact-name search at the
+pinned commit returned zero matches. Profiling is now configured through
+`ProfilerConfig`: `profiler` is `torch` or `cuda`, and
+`torch_profiler_dir` holds the absolute/URI output location
+(`vllm/config/profiler.py:33-46,125-145`). Optional controls include stacks,
+FLOPs, gzip, CUDA-time tables, shapes, memory, frontend exclusion, delayed
+start, maximum iterations, and wait/warmup/active iteration scheduling
+(`vllm/config/profiler.py:48-105`).
+
+The documented server form is a `--profiler-config` JSON object, for example
+`{"profiler":"torch","torch_profiler_dir":"/abs/run/kineto"}`, followed by
+POSTs to the two endpoints (`docs/contributing/profiling.md:46-85`).
+
+When profiler config is present, the serving router exposes POST
+`/start_profile` and `/stop_profile`; without profiler config the router is not
+attached (`vllm/entrypoints/serve/profile/api_router.py:21-45`). The endpoints
+are therefore not an unauthenticated always-on feature independent of server
+configuration; they exist within the configured serving application and should
+be protected like the rest of the control plane.
+
+**Captured activity.** Each GPU worker creates a torch profiler with both CPU
+and CUDA activities, using the configured shapes/memory/stack/FLOPs switches
+(`vllm/v1/worker/gpu_worker.py:929-980` and
+`vllm/profiler/wrapper.py:159-226`). TensorBoard-compatible traces are written
+with an optional gzip handler. The wrapper also writes CPU/CUDA aggregate
+tables; only rank 0 prints the table to stdout
+(`vllm/profiler/wrapper.py:251-287`). The source documentation calls the
+profiler medium-overhead and warns that trace size and final flushing can be
+large (`docs/contributing/profiling.md:1-38`).
+
+The V1 worker calls `profiler.step()` once per execution and wraps model
+execution in a record-function name containing context/generation request and
+token counts (`vllm/v1/worker/gpu_worker.py:803-827,895-898`). If frontend
+profiling is not ignored, `AsyncLLM` also creates a separate CPU-only trace
+(`vllm/v1/engine/async_llm.py:178-200`).
+
+**TP behavior.** `EngineCore.profile()` delegates to the executor
+(`vllm/v1/engine/core.py:662-663`), whose profile RPC is a collective worker
+call (`vllm/v1/executor/abstract.py:256-257`). Multiprocessing broadcasts calls
+that do not declare a unique output rank, so the profile call reaches every TP
+worker (`vllm/v1/executor/multiproc_executor.py:340-402`). Each worker emits its
+own rank-qualified trace; the suffix includes DP/PP/TP/DCP/EP/global rank
+components (`vllm/distributed/utils.py:664-695`). For TP, kernel analysis must
+therefore preserve per-rank files, use the slowest rank for critical-path time,
+and optionally sum ranks to quantify total GPU work.
+
+### 2.2 Iteration-level scheduler data and exported metrics
+
+The V1 scheduler interface states that each `schedule()` output corresponds to
+one model forward and maps request IDs to scheduled tokens
+(`vllm/v1/core/sched/interface.py:51-80`). `SchedulerOutput` already carries
+new/cached request data, per-request and total scheduled tokens, common-prefix
+blocks, and the request IDs preempted in that step
+(`vllm/v1/core/sched/output.py:180-219`). The base scheduler constructs this
+object at `vllm/v1/core/sched/scheduler.py:1012-1076`, then advances request
+computed-token and chunk state at `vllm/v1/core/sched/scheduler.py:1130-1155`.
+
+What is available today:
+
+| Requested datum | Internal per-step source | Externally exposed today |
+|---|---|---|
+| Batch size | `len(SchedulerOutput.num_scheduled_tokens)`; the map is exact | No raw per-step batch-size Prometheus series. Optional iteration log reports context and generation request counts. |
+| Scheduled prefill/decode tokens | `compute_iteration_details()` classifies new/cached context requests and sums scheduled tokens (`vllm/v1/utils.py:766-813`) | Optional log reports exact scheduled context/generation counts and elapsed time around the execution wait (`vllm/v1/engine/core.py:433-472`). Prometheus prompt/generation counters are cumulative output-side stats, not a raw scheduled-step stream. |
+| Chunked-prefill splits | `Request.is_prefill_chunk`, `num_computed_tokens`, and prompt/output state exist (`vllm/v1/request.py:130-175`); extended chunks are classified as context | No chunk ordinal, first/middle/final split, or chunk-size series. These must be derived before `_update_after_schedule()` mutates state. |
+| Preemptions | Exact step set in `SchedulerOutput.preempted_req_ids` | Stdout aggregates over its logging interval; `vllm:num_preemptions` is cumulative (`vllm/v1/metrics/loggers.py:219-281,624-631`). |
+| Running/waiting/deferred queues | `SchedulerStats.num_running_reqs`, `num_waiting_reqs`, and `num_skipped_waiting_reqs` (`vllm/v1/metrics/stats.py:170-183`) | Latest-value gauges `vllm:num_requests_running`, `vllm:num_requests_waiting`, and reason-labeled capacity/deferred gauges (`vllm/v1/metrics/loggers.py:453-496`). Running queue size is not the same as scheduled batch size. |
+| KV-cache usage | Scheduler reads `kv_cache_manager.usage`; the block pool computes a null-block-adjusted ratio in `[0,1]` (`vllm/v1/core/kv_cache_manager.py:181-188`; `vllm/v1/core/block_pool.py:692-711`) | Latest-value `vllm:kv_cache_usage_perc`; stdout prints the latest percent (`vllm/v1/metrics/loggers.py:250-260,524-532`). Raw total/free blocks are not exported per step. |
+| Prefix-cache counters | `PrefixCacheStats` contains request/query/hit and separately preempted request/query/hit counters (`vllm/v1/metrics/stats.py:114-143`) for local and optional connector caches | Prometheus exposes cumulative token queries/hits for local and external caches (`vllm/v1/metrics/loggers.py:547-565,1063-1101`); stdout reports interval-derived hit rates. It does not expose the preempted sub-counters separately. |
+| Total step tokens | Exact scheduled total is in `SchedulerOutput` | `vllm:iteration_tokens_total` is a histogram observed from output-side computed prompt plus generation tokens (`vllm/v1/metrics/loggers.py:693-724,1148-1171`), not a joinable raw step record. |
+
+`IterationStats` also has a wall timestamp, output-side prompt/generation data,
+preemptions, and finished-request latency samples
+(`vllm/v1/metrics/stats.py:325-347`). `Scheduler.make_stats()` produces latest
+queue/KV state, drains prefix stats, and passes through the step's CUDA-graph
+stat when an output is processed
+(`vllm/v1/core/sched/scheduler.py:2228-2264`). These are valuable existing
+plumbing, but async scheduling can schedule newer batches before an older
+output is processed. Consequently, those latest/drained values are not a safe
+per-step composition join without a schedule-time snapshot.
+
+Bottom line: no new scheduler instrumentation is needed to discover batch and
+token composition, preemptions, queues, KV use, or prefix events. A patch is
+needed only to serialize their **same-step association** and the missing
+histograms/chunk split.
+
+### 2.3 CUDA-graph observability
+
+Yes, v0.24.0 can determine the runtime graph mode and capture bucket for each
+model step internally:
+
+- `CUDAGraphStat` records unpadded tokens, padded tokens, padding, and runtime
+  mode (`vllm/compilation/cuda_graph.py:32-37`).
+- The GPU model runner asks the dispatcher for a concrete runtime mode and
+  `BatchDescriptor`, then conditionally attaches the stat to its output
+  (`vllm/v1/worker/gpu_model_runner.py:3822-3934`). It reaches the scheduler in
+  `ModelRunnerOutput` and `update_from_output()`
+  (`vllm/v1/outputs.py:231-281`;
+  `vllm/v1/core/sched/scheduler.py:1464-1477`).
+- The dispatcher is the source of truth: it pads to a captured descriptor,
+  tries `FULL`, then `PIECEWISE`, and returns `NONE` when no graph key matches
+  (`vllm/v1/cudagraph_dispatcher.py:235-324`; the design explanation is at
+  `docs/design/cuda_graphs.md:81-120`). At serving time, `FULL` and `PIECEWISE`
+  dispatch to a captured wrapper; `NONE` calls the runnable directly
+  (`vllm/compilation/cuda_graph.py:233-360`).
+
+Thus `runtime_mode != NONE` is a graph-path dispatch, and `num_padded_tokens` is
+the selected token bucket. `PIECEWISE` means graph-wrapped compiled pieces, not
+a full-step graph hit. Startup normally captures all dispatcher descriptors
+(`vllm/v1/worker/gpu_model_runner.py:6595-6643`), but `CUDAGraphStat` has no
+capture-versus-replay bit; a lazy first capture and a replay have the same stat.
+With built-in `cudagraph_metrics`, the stat is aggregated
+into a frequency table and printed at the logging interval
+(`vllm/config/observability.py:56-58`;
+`vllm/compilation/cuda_graph.py:40-124`); it is not exposed as a raw per-step
+Prometheus event. OpProf only needs to preserve the already-computed object on
+every enabled step.
+
+Capture modes and sizes live in `CompilationConfig`:
+`cudagraph_mode`, `cudagraph_capture_sizes`, and
+`max_cudagraph_capture_size` are defined at
+`vllm/config/compilation.py:587-688`. The V1 default is
+`FULL_AND_PIECEWISE`. Final sizes are generated or validated in
+`VllmConfig._set_cudagraph_sizes()`; the default is 1/2/4, then steps of 8 below
+256 and 16 above it, bounded by token/batch configuration, with user overrides
+written back to the final list (`vllm/config/vllm.py:1635-1800`).
+
+### 2.4 MoE routing and Qwen3-30B-A3B backend
+
+Qwen3 MoE creates a replicated gate and `FusedMoE` with the model's number of
+experts and top-k, and normally runs its router internally
+(`vllm/model_executor/models/qwen3_moe.py:137-249`). The causal-LM wrapper
+retains the list of MoE layers and their expert topology
+(`vllm/model_executor/models/qwen3_moe.py:662-729`). Router top-k IDs are
+available immediately after routing and before EPLB remapping
+(`vllm/model_executor/layers/fused_moe/router/base_router.py:233-278`).
+
+There is an exact, built-in access path: `--enable-return-routed-experts`
+(`vllm/config/model.py:209-215`; `vllm/engine/arg_utils.py:826-830`). It installs
+per-layer callbacks, writes top-k IDs to preallocated GPU buffers, copies the
+step to pinned CPU memory, and returns it through `ModelRunnerOutput`
+(`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58-84`;
+`vllm/v1/worker/gpu_model_runner.py:3652-3666,4637-4683,7382-7437`). This is
+reachable without modifying the fused kernels or saving router logits.
+Raw router-logit tensors are not plumbed to engine outputs; exposing those
+would be deeper and much higher-volume surgery than using the existing top-k
+ID callback.
+
+It is not cheap enough for Layer 1. The per-worker transit buffer is a few MB;
+the scheduler's whole-slot buffer can reach multiple GB and performs a CPU
+fancy-index write each step
+(`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:223-305`;
+the scheduler store is at `vllm/v1/core/sched/scheduler.py:1501-1520`). It is
+also incompatible with pipeline parallelism greater than one and with KV
+connectors (`vllm/config/vllm.py:871-893`). Exact per-layer expert load should
+therefore be collected in a separate sampled Layer-2 run and reduced offline.
+EPLB has load state, but enabling it adds communication and its logs are
+aggregate balancing summaries; it is not a free observability tap
+(`vllm/distributed/eplb/eplb_state.py:62-171,478-575`).
+
+For the normal unquantized BF16/FP16 Qwen3-30B-A3B configuration on H20/SM90,
+the automatic unquantized oracle explicitly moves both FlashInfer backends
+behind Triton on Hopper because they are expected to be slower
+(`vllm/model_executor/layers/fused_moe/oracle/unquantized.py:43-86`). It selects
+the first supported backend and documents possible shape-specific fallbacks
+(`vllm/model_executor/layers/fused_moe/oracle/unquantized.py:152-250`). The
+primary backend is therefore **Triton** under those assumptions. A quantized
+checkpoint, LoRA, expert activation format, or explicit `--moe-backend` can
+change that conclusion and must be recorded with the campaign run.
+
+### 2.5 V1 execution loop and natural hooks
+
+The control path is:
+
+```text
+EngineCore busy loop
+  -> Scheduler.schedule()
+  -> executor.execute_model(SchedulerOutput)
+  -> GPUWorker -> GPUModelRunner.execute_model()
+  -> Scheduler.update_from_output(SchedulerOutput, ModelRunnerOutput)
+```
+
+Evidence and hook implications:
+
+- `EngineCore.run_busy_loop()` and `_process_engine_step()` drive steps at
+  `vllm/v1/engine/core.py:1259-1317`.
+- The normal step retains the exact scheduler output across the future and
+  passes it back to `update_from_output()`
+  (`vllm/v1/engine/core.py:479-508`). The batch-queue path does the same for
+  multiple in-flight batches (`vllm/v1/engine/core.py:519-632`).
+- The scheduler's stable interface is schedule/update at
+  `vllm/v1/core/sched/interface.py:51-107`. The base implementation builds the
+  output at `vllm/v1/core/sched/scheduler.py:1012-1100`, mutates request state
+  immediately afterward at lines 1130-1155, and consumes model results at
+  lines 1464-1803.
+- Worker execution wraps `gpu_model_runner.execute_model()` at
+  `vllm/v1/worker/gpu_worker.py:895-898`; the model runner's entry is
+  `vllm/v1/worker/gpu_model_runner.py:4055-4069`.
+
+The natural Layer-1 hooks are therefore in the base scheduler: snapshot after
+the `SchedulerOutput` is complete but before request state advances, then
+finalize with the matching model output. This covers the async subclass and
+avoids frontend/output aggregation. The only worker change is broadening the
+existing CUDA-graph-stat condition. Layer 2 uses the existing profile control
+path unchanged.
+
+### 2.6 Relevant changes from 0.20.0
+
+For comparison only, tag `v0.20.0` was fetched into the allowed clone without
+changing the pinned working tree. It resolves to commit
+`88d34c6409e9fb3c7b8ca0c04756f061d2099eb1`.
+
+- Async scheduling and the batch-queue architecture already existed in 0.20.0;
+  `SchedulerConfig` selected `AsyncScheduler` at
+  `v0.20.0:vllm/config/scheduler.py:146-176`, and the core schedule/future/update
+  paths were at `v0.20.0:vllm/v1/engine/core.py:402-431,443-533`. It is
+  incorrect to treat async scheduling as new in 0.24.0.
+- 0.24.0 adds DP prefill cadence/throttling and passes a prefill-throttle flag
+  into `schedule()` (`vllm/config/scheduler.py:140-161` and
+  `vllm/v1/engine/core.py:474-490`). It also hardens zero-token/dummy-iteration
+  handling (`vllm/v1/engine/core.py:433-472`) and changes when the batch queue
+  fills.
+- The 0.24 scheduler contains explicit scheduled/processed sequence fences and
+  snapshots state that must survive later async schedules, including routed
+  experts (`vllm/v1/core/sched/scheduler.py:293-322,1093-1165`). That is direct
+  evidence that an OpProf hook must not read mutable request/queue state only
+  when an older output returns.
+- `vllm/config/profiler.py`, `vllm/profiler/wrapper.py`, and
+  `vllm/entrypoints/serve/profile/api_router.py` are byte-identical between the
+  two tags (`git diff --quiet v0.20.0..v0.24.0` returned success). Profiler
+  endpoint semantics did not move; the scheduler/core line numbers and safe
+  composition hook did.
+
+Practical conclusion: do not port a 0.20 line-number patch into EngineCore.
+Hook the 0.24 base scheduler's schedule/update pair so sync, async, and batch
+queue paths share one implementation.
+
+## 3. Proposed dual-layer design
+
+Layer 1 emits one schema-versioned JSONL record per scheduler step from the
+EngineCore/scheduler process. It contains step/timestamps; scheduled and decode
+batch counts; scheduled prefill/decode tokens; aggregate first/middle/final
+chunk information; fixed-bucket context and chunk-size histograms; exact step
+preemptions; schedule-time running/waiting/deferred queues; total/free/used KV
+blocks and ratio; local/external prefix deltas; and CUDA-graph mode/bucket. No
+request IDs or raw context-length lists are written. MoE expert load is null and
+explicitly marked Layer-2-only.
+
+Records are encoded with the existing `msgspec` dependency
+(`requirements/common.txt:35`), placed into a
+bounded non-blocking queue, and drained by one background JSONL writer. It
+flushes every 1 MiB or one second and at clean shutdown, with explicit drop/gap
+counters and no per-step `fsync`. `VLLM_OPPROF_DIR` is the startup-only switch;
+unset is a true no-op. The unmeasured foreground estimate is 25-100
+microseconds per step, so Phase 2 must enforce the less-than-3% overhead gate
+with paired off/on serving runs.
+
+Layer 2 configures the already-built torch profiler and samples short windows,
+initially two warm-up plus eight active worker iterations. Set
+`max_iterations=8`; the wrapper stops the underlying profiler on the subsequent
+step after the counter exceeds the limit, as its existing test documents
+(`vllm/profiler/wrapper.py:83-114`;
+`tests/v1/worker/test_gpu_profiler.py:77-98`). The controller then calls
+`/stop_profile` to clear control state. Step-count/on-demand endpoint control is
+preferred to wall time. Layer 2 writes one trace per TP worker and joins to
+Layer 1 through the existing iteration annotation, order, token counts, and
+timestamps.
+
+Under CUDA-graph replay, the profiler is configured for CUDA activity, but the
+Python callable is not re-executed: vLLM calls `CUDAGraph.replay()`
+(`vllm/compilation/cuda_graph.py:233-360`). CUDA activity may therefore be
+visible through Kineto/CUPTI, while Python/PyTorch scopes cannot be assumed to
+provide the original per-layer attribution. Layerwise NVTX is explicitly
+unsupported with graphs (`vllm/config/observability.py:60-63`). Phase 2 must
+check whether the pinned torch/CUPTI combination exposes individual replayed
+kernels or a coarser graph launch. A matched eager window can identify kernel
+families but must not replace graph-enabled production timing.
+
+Layer-2 kernel time is grouped into attention, router/top-k, expert/dense GEMMs,
+normalization/activation, collectives, and sampling. Per-rank critical-path
+times calibrate a composition/graph-bucket-conditioned Layer-1 attribution;
+held-out profiler windows validate that mapping. Exact routed experts are
+enabled only in a separate sampled server run.
+
+### Patch surface
+
+| File | Approximate lines |
+|---|---:|
+| `vllm/envs.py` | 4 |
+| `vllm/v1/opprof.py` (new) | 220 |
+| `vllm/v1/core/sched/scheduler.py` | 32 |
+| `vllm/v1/worker/gpu_model_runner.py` | 4 |
+| `tests/v1/core/test_opprof.py` (new) | 190 |
+| **Total** | **5 files / about 450 lines** |
+
+Zero-patch wins: profiler config/endpoints/wrapper, `SchedulerOutput`, existing
+metrics plumbing, CUDA kernels, fused MoE/router code, Qwen3 model code, and
+frontend APIs. Supporting OpProf together with `--disable-log-stats` would add
+about 12 lines in `vllm/v1/core/kv_cache_manager.py`; the design recommends
+failing fast instead.
+
+### Main risks
+
+- Histogram construction and JSON encoding are the likely foreground overhead;
+  the 3% budget must be measured, not inferred.
+- Async scheduling can misassociate latest queue/cache state with an older
+  output; snapshot inside `schedule()` and pair by the exact output object.
+- Layer 1 must be scheduler-owned to avoid TP duplicates; Layer 2 remains
+  per-rank and needs offline aggregation.
+- At 500 steps/s and about 1 KiB/record, an uncompressed stream is roughly
+  44 GB/day per EngineCore. Runs must be duration-bounded, with queue/drop
+  counters and post-run compression.
+- Kineto perturbs execution, and graph replay loses Python-scope attribution;
+  it is a sampled calibration instrument, not an always-on latency source.
+
+## 4. Open design decisions
+
+1. Approve JSONL/`msgspec`, an 8192-record queue, context edges
+   `(128, 256, 512, 1K, 2K, 4K, 8K, 16K, 32K, 64K, 128K, +inf)`, and chunk-size
+   edges `(16, 32, 64, 128, 256, 512, 1K, 2K, +inf)`.
+2. Approve exact MoE expert load as Layer-2-only. The alternative is a new GPU
+   histogram path, with a larger patch and an overhead/graph-capture risk.
+3. Confirm checkpoint precision/quantization and intended TP/EP/DP topology.
+   The H20 Triton conclusion assumes unquantized BF16/FP16, auto backend, and no
+   LoRA.
+4. Choose the initial profiler window: recommended two warm-up plus eight
+   active iterations, or a longer active window.
+5. Decide whether the less-than-3% gate applies to point estimates or the upper
+   95% confidence bound for every primary metric.
+6. Confirm that OpProf campaign runs may reject `--disable-log-stats`; support
+   for it increases the patch to six files and about 462 lines.
+
+## Data sanity block
+
+- **Pinned root listing:** n=45 names; lexicographic min=`.buildkite`,
+  max=`vllm`; distinct=45; SHA-256=`75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58`.
+- **Source versions compared:** n=2 tags; min=`v0.20.0`, max=`v0.24.0`;
+  distinct=2; pinned working-tree version remains exactly `v0.24.0`.
+- **Torch pin observations:** n=4 declarations
+  (`pyproject`, build requirements, CUDA runtime requirements, CMake);
+  min=max=`2.11.0`; distinct=1.
+- **Evidence-bearing source files inspected and cited:** n=44
+  distinct v0.24.0 paths; min/max not applicable to categorical paths;
+  v0.20.0 comparison touched n=5 distinct paths; min/max not applicable.
+- **Invariants checked:** exact tag equals `v0.24.0`; HEAD equals the recorded
+  commit; clone is clean; top-level names are unique; requested path was used;
+  torch pins agree; CUDA-graph modes are in `{NONE, PIECEWISE, FULL}`; KV usage
+  definition is bounded in `[0,1]`; scheduled-token total is defined as the sum
+  of per-request scheduled tokens; graph bucket is not smaller than unpadded
+  tokens; no remote/GPU/install/model-download action occurred.
+- **Benchmark/statistical conclusion check:** no performance benchmark was run,
+  so there are no per-configuration curves or measured overhead values on which
+  to perform identical-value, monotonicity, or continuity checks. All overhead
+  numbers in the design are labeled estimates.

docs/opprof/phase3-results.md

@@ -0,0 +1,262 @@
+# OpProf Phase 3 dash0 results
+
+Status: **FINAL A-P3-7 PARTIAL-MATRIX ANALYSIS — H1a INCONCLUSIVE; H1b PASS**.
+
+Date: 2026-07-12. The matrix is permanently frozen at 40/52 accepted measured
+runs and 20/24 complete pattern/config cells. A-P3-7 permits existential
+confirmation on complete comparisons but forbids refutation from incomplete
+coverage. The result is therefore **PARTIAL**: H1b is confirmed by five
+evaluable contrasts, while H1a cannot be evaluated on a valid pattern pair.
+
+The final machine result is
+`runs/opprof-phase3/phase3/metrics.json` (SHA-256
+`dcc44941bfe1eb56c7a069148e5565eddd1ccf966e6914aa16b64bd67d02f0f0`).
+The final analyzer SHA-256 is
+`483c170838a1e81a170d8425b4be22cc1be02ea4bc9492b474daf7ae86186d67`;
+7/7 no-GPU analysis tests passed, including ResourceWarning-as-error.
+
+## Data-sanity result first
+
+All **23/23** machine-checked invariants pass before inference: 40 unique
+accepted primary markers selected only through completed-stage records; exact
+240-second clean windows; zero clean failures; finite moderate rates within
+5%; balanced Layer-1 footer/sidecar accounting; ratios in range; 72 accepted
+trace files with the registered eight-file first-wave deviation; zero other
+GPU processes; exact 20-cell coverage; and the exact A-P3-7 missing-cell and
+missing-contrast sets.
+
+The accepted data contain 77,109 clean completions, 352,907 clean Layer-1
+steps, and 542,350 total Layer-1 records. Device-time classifiability is
+97.05–99.64%. No malformed trace, accounting mismatch, counter underflow,
+private-text leak, or unexpected identical-result family was found.
+
+Declared limitations are not hidden as sanity passes: eight accepted
+saturation trace files were lost during the registered first-wave controller
+cleanup; all four confirmation runs are absent; MoE per-layer CV is unavailable
+because layer scopes cover less than 80% of MoE GEMM time; and only one of nine
+completed C00-moderate patterns passes the Layer-2 inference gates.
+
+## Frozen coverage and A-P3-7 logic
+
+The four incomplete cells, each missing saturation and moderate, are exactly:
+
+- P03/C11;
+- P05/C00;
+- P10/C00-TP2; and
+- P11/C00.
+
+The four missing confirmations are P10, P06, P03, and P01 C00-moderate.
+Formal H1a coverage is therefore the nine completed C00 patterns P01, P02,
+P03, P04, P06, P07, P08, P09, and P10. Formal H1b has six evaluable and two
+non-evaluable frozen contrasts. A-P3-7's asymmetry is mandatory: one valid hit
+can confirm an existential hypothesis, but absence of a hit cannot support the
+original complete-matrix null.
+
+## H1a — operator composition
+
+**Verdict: INCONCLUSIVE; refutation is not allowed.**
+
+All moderate traces were parseable and classifiable, but only P04's two
+windows jointly passed classifiability, `abs(SMD)<=0.25` representativeness,
+and fixed recovery. P01/P02/P03/P09 failed representativeness in both windows;
+P06/P07/P08/P10 additionally failed at least one recovery. Thus only one of
+nine completed patterns is inferentially evaluable, leaving zero pattern pairs,
+zero ranking tests with data, and no Kendall tau-b pairs. No qualifying
+inversion can be accepted, and this is not evidence for a shared ranking.
+
+Descriptively, P04 is attention-led at both loads. P06 and P10 change from
+attention-led at saturation to MoE-GEMM-led at moderate load; the other
+available patterns remain MoE-GEMM-led. These are ranking changes in sampled
+windows, not H1a evidence because their window gates fail.
+
+### Per-pattern operator shares, both load points
+
+Values are the mean percentage across the two Layer-2 windows. `E` means the
+row is inferentially evaluable; `D` means data are available but descriptive
+only because a registered window/recovery gate failed; `NE-cell` means the cell
+is missing; `NE-trace` means the accepted run has no retained trace. `Other` is
+unclassified device activity. Collective, sampler, and dense-GEMM shares round
+to 0.00% in every available C00 row.
+
+| Pattern | Load | Use | Attention | MoE GEMM | Router | Collective | Sampler | Dense GEMM | Norm/elt | KV | Other | Top family |
+|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
+| P01 | saturation | D | 16.32 | 77.81 | 0.85 | 0.00 | 0.00 | 0.00 | 3.90 | 0.10 | 1.02 | MoE GEMM |
+| P01 | moderate | D | 12.09 | 82.43 | 1.03 | 0.00 | 0.00 | 0.00 | 3.38 | 0.11 | 0.96 | MoE GEMM |
+| P02 | saturation | D | 32.67 | 62.39 | 1.05 | 0.00 | 0.00 | 0.00 | 2.98 | 0.16 | 0.74 | MoE GEMM |
+| P02 | moderate | D | 20.91 | 72.53 | 1.64 | 0.00 | 0.00 | 0.00 | 3.68 | 0.25 | 1.00 | MoE GEMM |
+| P03 | saturation | D | 39.11 | 56.60 | 0.44 | 0.00 | 0.00 | 0.00 | 2.98 | 0.03 | 0.84 | MoE GEMM |
+| P03 | moderate | D | 23.32 | 57.45 | 3.54 | 0.00 | 0.00 | 0.00 | 12.29 | 0.83 | 2.57 | MoE GEMM |
+| P04 | saturation | D | 65.88 | 29.77 | 1.16 | 0.00 | 0.00 | 0.00 | 2.30 | 0.17 | 0.72 | Attention |
+| P04 | moderate | **E** | 47.88 | 40.64 | 3.28 | 0.00 | 0.00 | 0.00 | 5.85 | 0.45 | 1.90 | Attention |
+| P05 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — |
+| P05 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — |
+| P06 | saturation | D | 56.59 | 39.35 | 1.04 | 0.00 | 0.00 | 0.00 | 2.25 | 0.15 | 0.62 | Attention |
+| P06 | moderate | D | 31.60 | 57.52 | 2.97 | 0.00 | 0.00 | 0.00 | 5.69 | 0.43 | 1.80 | MoE GEMM |
+| P07 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — |
+| P07 | moderate | D | 28.67 | 62.18 | 2.42 | 0.00 | 0.00 | 0.00 | 4.86 | 0.37 | 1.49 | MoE GEMM |
+| P08 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — |
+| P08 | moderate | D | 34.59 | 57.06 | 2.22 | 0.00 | 0.00 | 0.00 | 4.44 | 0.34 | 1.34 | MoE GEMM |
+| P09 | saturation | D | 26.23 | 69.03 | 0.36 | 0.00 | 0.00 | 0.00 | 3.43 | 0.00 | 0.95 | MoE GEMM |
+| P09 | moderate | D | 16.19 | 75.00 | 2.14 | 0.00 | 0.00 | 0.00 | 4.83 | 0.35 | 1.49 | MoE GEMM |
+| P10 | saturation | D | 67.80 | 28.90 | 0.76 | 0.00 | 0.00 | 0.00 | 1.87 | 0.10 | 0.57 | Attention |
+| P10 | moderate | D | 30.61 | 54.04 | 2.81 | 0.00 | 0.00 | 0.00 | 9.80 | 0.68 | 2.06 | MoE GEMM |
+| P11 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — |
+| P11 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — |
+
+### Clean CUDA-graph modes at moderate load
+
+| Pattern | FULL | PIECEWISE | NONE/eager |
+|---|---:|---:|---:|
+| P01 | 1.98% | 73.19% | 24.83% |
+| P02 | 83.16% | 13.00% | 3.84% |
+| P03 | 96.37% | 0.00% | 3.63% |
+| P04 | 98.32% | 0.02% | 1.66% |
+| P05 | — | — | — |
+| P06 | 99.10% | 0.05% | 0.85% |
+| P07 | 98.76% | 0.00% | 1.24% |
+| P08 | 98.64% | 0.01% | 1.36% |
+| P09 | 91.79% | 3.69% | 4.52% |
+| P10 | 98.33% | 0.12% | 1.55% |
+| P11 | — | — | — |
+
+P01 is the clear mode outlier: PIECEWISE covers 73.19% of clean moderate
+steps. Full per-mode operator segments and active-step counts remain in the
+machine result; no mode with fewer than eight sampled steps is summarized.
+
+## H1b — rectangular-benchmark miss
+
+**Verdict: PASS.** Five of six evaluable frozen contrasts pass at least one
+waste threshold with a simultaneous positive bound and at least 5% useful-token
+efficiency loss. The two P05 contrasts are `NOT EVALUABLE`; no value is
+imputed. Holm correction retains the original eight planned contrasts per
+metric. Reported simultaneous intervals below are percentage-point effects;
+passing bootstrap p-values are zero at 100,000-resample resolution.
+
+| Frozen contrast | Status | Passing evidence | Useful-token efficiency loss |
+|---|---|---|---:|
+| P05 vs P01 | **NOT EVALUABLE** | P05/C00 missing | — |
+| P05 vs P03 | **NOT EVALUABLE** | P05/C00 missing | — |
+| P06 vs P02 | **PASS** | R64 +23.015 pp, simultaneous 95% [22.345, 23.683] | 11.609% |
+| P06 vs P04 | **PASS** | R64 +35.440 pp [34.826, 36.059] | 22.846% |
+| P09 vs P01 | **PASS** | padding +6.673 pp [5.887, 7.492]; R64 +39.616 pp [39.065, 40.166] | 8.325% |
+| P09 vs P03 | EVALUABLE, NO HIT | padding +8.542 pp and R64 +52.041 pp are material, but association gate fails | **3.840%**, below 5% |
+| P10 vs P03 | **PASS** | padding +5.565 pp [2.010, 10.341]; R64 +44.787 pp [43.635, 45.971] | 44.693% |
+| P10 vs P04 | **PASS** | padding +5.399 pp [1.868, 10.213]; R64 +44.787 pp [43.638, 45.949] | 14.260% |
+
+This confirms the existential H1b claim on completed contrasts. It cannot
+support a null claim about P05 or the missing matrix.
+
+## Waste accounting
+
+The preregistered contrast thresholds are 5 percentage points for padding,
+10 points for graph miss/overflow, 0.15 absolute for R64, 0.10 for mixed-batch
+interference, and 0.15 for MoE layer CV, plus the 5% efficiency/residual
+association gate. Values below are clean C00-moderate results; efficiency is
+scheduled useful tokens per model-step millisecond.
+
+| Pattern | Padding | Graph miss | Overflow | R64 | Mixed interference | Efficiency | Supported/total mixed steps |
+|---|---:|---:|---:|---:|---|---:|---:|
+| P01 | 1.869% | 24.826% | 24.826% | 0.3687 | N/A | 4.9673 | 0/5,646 |
+| P02 | 1.998% | 3.837% | 3.837% | 0.3687 | N/A | 2.6664 | 0/1,536 |
+| P03 | 0.000% | 1.736% | 1.736% | 0.2444 | N/A | 4.7356 | 0/138 |
+| P04 | 0.167% | 1.301% | 1.301% | 0.2444 | N/A | 3.0547 | 0/141 |
+| P05 | — | — | — | — | — | — | — |
+| P06 | 0.276% | 0.830% | 0.830% | 0.5988 | N/A | 2.3568 | 0/152 |
+| P07 | 0.140% | 1.240% | 1.240% | 0.0000 | N/A | 2.7513 | 0/186 |
+| P08 | 0.005% | 1.357% | 1.357% | 0.0000 | N/A | 2.1478 | 0/178 |
+| P09 | **8.542%** | 4.518% | 4.518% | **0.7648** | N/A | 4.5537 | 0/1,054 |
+| P10 | **5.565%** | 0.921% | 0.921% | **0.6923** | N/A | 2.6191 | 0/92 |
+| P11 | — | — | — | — | — | — | — |
+
+No irregular-versus-control graph-miss or overflow contrast reaches the
+positive 10-point H1b threshold. P01 has the largest absolute miss rate but is
+a rectangular control; P09 actually reduces miss versus P01 by 20.308 points.
+
+The registered leave-one-pattern-out robust fits were applied for mixed-batch
+interference, but no pattern retained 30 supported mixed steps inside both
+pure-fit convex supports; every result is N/A rather than extrapolated. No
+separate LOAO operator-ranking procedure was preregistered, so none was added
+post hoc. The required two-window rule is used unchanged, and confirmation-run
+robustness is `NOT EVALUABLE` because all four confirmations are missing.
+
+### Top three waste findings
+
+1. **P10 real trace:** R64 is 44.787 points above both long rectangular
+   controls; padding is 5.565/5.399 points higher; efficiency is 44.693% worse
+   than P03 and 14.260% worse than P04.
+2. **P09 production-shaped mix:** versus P01, R64 rises 39.616 points and
+   padding 6.673 points with 8.325% lower efficiency. Its still-larger P03
+   contrast does not pass because efficiency loss is only 3.840%.
+3. **P06 bimodal long-output burst:** R64 rises 23.015 points versus P02 and
+   35.440 points versus P04, paired with 11.609% and 22.846% efficiency loss.
+
+## Pattern-conditioned operational findings
+
+- **P10/TP2 non-stabilization:** the fresh run completed 17 warm-up requests,
+  but trailing scheduled-token throughput fell 18,022.4 → 16,384.0 → 14,062.6
+  tokens/s. Its 36.76% fitted drift exceeds A-P3-6's 10% limit; same-wave
+  synthetic P11 and P03 completed 512 and 102 warm-up requests and passed their
+  applicable registered gates. The orchestrator accepts this as a
+  pattern-conditioned finding, not an accepted throughput measurement.
+- **Long-context drain:** P10/C01 saturation naturally drained for 288.619
+  seconds. The clean window and accounting were valid; the original 120-second
+  short-pattern watchdog was miscalibrated, while the amended 600-second P10
+  budget correctly retained the run.
+- **Failure boundary:** P01/C01 moderate had 5,828 clean successes and zero
+  clean failures. Two `ServerDisconnectedError` records occurred only in
+  excluded post-profile time; separating clean and auxiliary windows prevented
+  valid data from being discarded.
+- **Layer-2 perturbation:** despite 97%+ kernel classifiability, eight of nine
+  completed moderate patterns failed representativeness or recovery. Together
+  with the Phase-2 51.3% active-window perturbation, this shows that sparse
+  Kineto windows are operationally fragile for these serving patterns.
+
+These incidents consistently show long-context and mixed serving patterns
+breaking assumptions calibrated on shorter rectangular loads: fixed drain
+watchdogs, completion-count warm-up, global failure scope, and profile-window
+representativeness.
+
+## Matrix and GPU accounting
+
+| Item | Final value |
+|---|---:|
+| Planned measured runs | 52 |
+| Accepted measured runs | **40** |
+| Complete cells | **20/24** |
+| Accepted confirmations | 0/4 |
+| Accepted clean failures | 0 |
+| Drain-quarantined accepted runs | 0 |
+| Accepted Layer-2 trace files | 72 |
+| Registered missing trace files | 8 |
+| Other-user GPU processes | 0 |
+| Cumulative GPU use | **14.025875 H20-hours** |
+| Reserved/unused headroom | **1.974125 H20-hours** |
+
+No GPU work was run for A-P3-7 analysis. The reserved headroom remains unused,
+and all eight GPUs were at 0 MiB/0% before and after the CPU-only analysis.
+
+## Data sanity block
+
+| Numeric family | n | finite | missing | min | max | distinct | Check |
+|---|---:|---:|---:|---:|---:|---:|---|
+| Accepted throughput (req/s) | 40 | 40 | 0 | 0.445833 | 43.300000 | 37 | finite, positive, not identical |
+| Token efficiency (tokens/ms) | 40 | 40 | 0 | 2.147827 | 8.295849 | 40 | finite, positive, distinct |
+| Drain duration (s) | 40 | 40 | 0 | 0.305419 | 288.619108 | 40 | non-negative; no quarantine |
+| Clean Layer-1 steps/run | 40 | 40 | 0 | 718 | 20,171 | 40 | sum 352,907; continuous per run |
+| Operator classifiable fraction | 72 | 72 | 0 | 0.970458 | 0.996378 | 72 | all >=0.70 and within [0,1] |
+| Operator-family shares | 576 | 576 | 0 | 0.000000 | 0.873417 | 348 | within [0,1] |
+| Waste ratios | 240 | 240 | 0 | 0.000000 | 1.000000 | 168 | within [0,1] |
+| Waste-contrast effects | 36 | 24 | 12 expected | -0.203082 | 0.520413 | 17 | signed; N/A preserved |
+| Kendall tau-b | 0 | 0 | 0 | — | — | 0 | expected: only one valid pattern |
+| Accepted clean failures | 40 | 40 | 0 | 0 | 0 | 1 expected | sum zero |
+| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | cleanup passed |
+
+Checked invariants: accepted IDs unique; exact 40-run/20-cell coverage; exact
+four missing cells and two missing contrasts; 240-second clean windows; zero
+clean failures; moderate rate within 5%; balanced Layer-1 accounting and zero
+drops; trace parsing and classifiability; ratios in range; exact 72+8 trace
+accounting; clock/load snapshots present; no other-user process; drain
+quarantine below 20%; non-identical pattern results; and exact reproduction of
+the 36.76% A-P3-6 operational result. **No data-sanity red flag remains.** H1a
+is inconclusive because of declared Layer-2 validity limits; H1b passes on five
+complete contrasts; the compound hypothesis is partial and cannot be refuted.

docs/opprof/phase4-optimization-plan.md

@@ -0,0 +1,306 @@
+# OpProf Phase 4 measured optimization plan
+
+Status: **PROPOSAL FROM ACCEPTED PHASE-3 DATA; NO UNMEASURED GAIN CLAIMS**.
+
+Date: 2026-07-12. This plan uses only the accepted Phase-3 40/52-run,
+20/24-cell dataset and the single optional Phase-4 capture-size validation.
+Phase-3 protocol and results are frozen. Bounds below are ceilings in the
+units actually measured; padding or raggedness percentages are not relabeled
+as end-to-end throughput gains.
+
+## Pinned implementation context
+
+- Model/hardware: Qwen3-30B-A3B BF16, one H20, TP1 primary.
+- vLLM source: accepted OpProf tip
+  `23450fb21ac255b0cf710f4ee965ee694921975d` on v0.24.0.
+- vLLM 0.24.0 exposes `--cudagraph-capture-sizes`,
+  `--max-cudagraph-capture-size`, `--max-num-seqs`, and
+  `--max-num-batched-tokens` (`vllm/engine/arg_utils.py:1390-1467`).
+- An explicit capture-size list replaces the inferred list. The default is
+  `[1,2,4]`, multiples of eight below 256, then multiples of sixteen through
+  the maximum, normally 512 (`vllm/config/vllm.py:1669-1792`).
+- Chunked prefill remains enabled. vLLM schedules decode first, then fills the
+  remaining MBT budget with prefill and chunks an over-budget prefill
+  (`docs/configuration/optimization.md:45-59`).
+
+The ranking is by measured opportunity, readiness, and downside together—not
+by a normalized composite score. Items that share the same raggedness bound
+are explicitly non-additive.
+
+## Ranked optimization list
+
+| Rank | Tier | Target and affected regime | Measured bound | Owner | Decision |
+|---:|---|---|---|---|---|
+| 1 | Config now / scheduler backlog | Preserve prefix affinity and prefix caching for P08-like shared-prefix traffic | P08 vs matched P07: **+82.14% saturation req/s**, 80% prefix-hit ratio, **62.29% fewer prefill tokens** | Serving/config owner now; cache-aware dispatch upstream | Deploy behind a workload classifier; do not apply to no-sharing traffic |
+| 2 | Scheduler | Length-aware cohorting/admission for ragged P10/P09/P06 | At most **44.79 pp** R64 contrast and **44.69%** measured efficiency gap on P10; 39.62 pp/8.32% on P09; 35.44 pp/22.85% on P06 | Upstream vLLM scheduler | Highest structural backlog item; preserve fairness and arrival semantics |
+| 3 | Config now | Add exact small CUDAGraph sizes for P09/P10-like moderate decode batches | Distributional bound **4.98 pp P09 / 5.26 pp P10** padding; P09 validation achieved **4.980 pp** | Serving/config owner | Mechanism confirmed; canary only because p95 was +3.01% in one pair |
+| 4 | Config now | Route P06/P10-like pools to MNS=64 | Saturation throughput **+3.37% P06 / +3.70% P10** versus C00 | Serving/config owner | Pattern-specific trial only; same setting was −24.27% on P01 |
+| 5 | Kernel backlog | Ragged-aware MoE GEMM and attention over the measured shape stream | Shares the rank-2 ceiling; no independent additive gain. Descriptive MoE share is 54.04–75.00% on P10/P09/P06 moderate | Engine/kernel colleagues | Optimize exact weighted shapes below; require serving confirmation |
+| 6 | Config guardrail | Keep MBT=8192 for short/high-throughput and long-prefill classes; do not globally set 2048 | Avoided saturation regressions up to **11.64% P03**, 6.53% P01, 5.82% P10 | Serving/config owner | Encode as a policy guardrail, not a positive optimization claim |
+
+## Tier A — configuration-level actions deployable today
+
+### A1. Prefix-affine routing with prefix caching
+
+**Measured regime.** P07 and P08 have the same 1,280-token prompt length,
+512-token output, burst-of-eight arrival, C00 config, and seed. P08 alone uses
+eight 1,024-token shared prefixes plus a unique 256-token suffix. With prefix
+caching already enabled:
+
+| Metric, saturation | P07 no sharing | P08 high sharing | Delta |
+|---|---:|---:|---:|
+| Prefix-query hit ratio | 0.00 | 0.80 | +0.80 |
+| Clean prefill tokens | 1,528,968 | 576,512 | **−62.29%** |
+| Completed throughput | 5.1083 req/s | 9.3042 req/s | **+82.14%** |
+
+**Root mechanism.** The matched cell changes only controlled prefix sharing;
+the observed hit and prefill-token changes directly identify cache reuse. The
+82.14% throughput delta is a point observation and the maximum evidence-backed
+gain for this exact regime, not a fleet-wide forecast.
+
+**Action now.** Keep `--enable-prefix-caching`; hash an application-known
+stable prefix or conversation identity to a replica so related requests do
+not destroy affinity through round-robin load balancing. Apply only when the
+online prefix-query hit ratio resembles P08, not P07.
+
+**Verification.** Interleave affinity ON/OFF on the same replicas and fixed
+request stream. Primary gates are prefix-hit ratio, prefill-token reduction,
+completed req/s, TTFT p95, per-replica queue imbalance, and KV occupancy. A
+throughput gain with queue/fairness or KV-capacity regression does not pass.
+
+### A2. Measured CUDAGraph capture sizes
+
+**Measured regime.** P09 moderate has decode-batch p50/p95/max 4/16/25. P10
+moderate has 1/4/7. Default captures skip sizes 3, 5, 6, 7, and 9, so those
+batches pad upward. Replaying the Phase-3 hit distribution predicts that
+adding exactly `{3,5,6,7,9}` to the complete default list can remove:
+
+- **4.9776 percentage points** of P09's 8.5421% graph-hit padding; and
+- **5.2579 points** of P10's 5.5652% padding.
+
+The list must be `default ∪ {3,5,6,7,9}`; passing only five sizes would replace
+and discard the rest of the default list.
+
+**Closed-loop result.** One P09 moderate ON/OFF pair used fresh servers, the
+same fixed seed and accepted saturation-rate source, 60 excluded warm-up
+seconds, and 240 clean seconds per arm. No Layer-2 profiler ran.
+
+| Metric | ON: exact sizes | OFF: default | ON relative to OFF |
+|---|---:|---:|---:|
+| Graph-hit padding | **3.5659%** | 8.5456% | **−4.9798 pp; −58.27%** |
+| Useful tokens/model-step ms | 4.56222 | 4.55405 | +0.179% |
+| Completed throughput | 4.9583 req/s | 4.9333 req/s | +0.507% |
+| Mean E2E latency | 1.6123 s | 1.6812 s | −4.10% |
+| p95 E2E latency | 3.9930 s | 3.8763 s | **+3.01%** |
+| Clean failures | 0 | 0 | equal |
+
+The observed padding reduction differs from the Phase-3 bound by only 0.0022
+percentage points, confirming the mechanism. It does **not** establish a broad
+performance win: token efficiency and throughput moved less than 1%, p95 moved
+the wrong way, and there is one ordered pair with no CI.
+
+Operational cost also matters: ON captured 56 FULL and 56 PIECEWISE sizes
+versus 51/51 OFF, estimated graph memory increased 0.64→0.68 GiB, and server
+ownership was 76.9 seconds longer. Deploy only as a pattern-specific canary;
+require interleaved replication with a p95 non-regression gate before rollout.
+
+### A3. Pattern-specific MNS pools
+
+The complete saturation comparisons show that `--max-num-seqs 64` is an
+interaction, not a globally better default:
+
+| Pattern | C10 MNS=64 vs C00 | Interpretation |
+|---|---:|---|
+| P01 short/short | **−24.27% req/s** | Reject for dense short traffic |
+| P03 long/short | −1.41% | No measured benefit |
+| P06 bimodal/long burst | **+3.37%** | Candidate pattern pool |
+| P10 real long-context | **+3.70%** | Candidate pattern pool |
+
+The evidence identifies the config×pattern interaction but not a lower-level
+cause. Do not attribute it to a particular kernel or queue effect without a
+new bisection. Verification is five interleaved saturation pairs per intended
+class plus TTFT, queue depth, preemption, KV usage, and exact-work checks.
+
+### A4. MBT policy guardrail
+
+`--max-num-batched-tokens 2048` versus the default 8192 changed saturation
+throughput by −6.53% P01, −11.64% P03, +0.35% P06, and −5.82% P10. The combined
+MNS64/MBT2048 setting was −30.22% on P01. Phase 3 therefore supports retaining
+MBT8192 for these classes and rejects a global MBT2048 rollout. The bound is
+an avoided regression, not new speedup.
+
+## Tier B — upstream scheduler changes
+
+### B1. Length-aware cohorting without starvation
+
+**Measured mechanism.** R64 is the rectangular padding fraction of the exact
+arrival-order prompt stream. It is 0.6923 for P10, 0.7648 for P09, and 0.5988
+for P06. Their passing control contrasts are:
+
+| Irregular pattern | Control | R64 excess | Useful-token efficiency loss |
+|---|---|---:|---:|
+| P10 | P03 | **44.79 pp** | **44.69%** |
+| P10 | P04 | **44.79 pp** | **14.26%** |
+| P09 | P01 | **39.62 pp** | **8.32%** |
+| P06 | P02 | **23.01 pp** | **11.61%** |
+| P06 | P04 | **35.44 pp** | **22.85%** |
+
+These are upper bounds on waste a length-aware path could avoid. R64 is not
+observed GPU time, and efficiency association is not causal. The scheduler
+change should maintain several ready queues by remaining prompt/context band,
+select a less-ragged cohort subject to the existing decode-first token budget,
+and impose a finite age/fairness bound. It must not rewrite request arrivals or
+drop long requests.
+
+**Verification.** Add a runtime per-step raggedness counter rather than using
+manifest R64 as a surrogate. Compare fixed-arrival ON/OFF runs for useful
+tokens/model-step ms, TTFT/E2E p95, queue age, starvation count, preemption,
+KV occupancy, and the full length histogram. The gain cannot exceed the
+corresponding R64/efficiency bounds above, and it is non-additive with a
+ragged-aware kernel.
+
+### B2. Automatic cache-aware dispatch
+
+The upstream form of A1 is a scheduler/replica dispatcher that chooses a live
+prefix-cache owner while respecting load. Its collaboration contract is the
+measured P07/P08 tuple: 1,024 shared + 256 unique prompt tokens, eight prefix
+IDs, burst size eight, output 512, target hit ratio 0.80. The load-balancing
+penalty and lost cache hits must be reported together; a synthetic cache hit
+increase without end-to-end balance is insufficient.
+
+### B3. Histogram-driven capture-list generation
+
+Static A2 proves that Layer-1 can choose useful sizes. An upstream controller
+could select a bounded number of exact sizes from padding contribution
+`count(size) * (next_bucket-size)`, while retaining the default list and a
+memory/startup budget. P09's top five `{3,5,6,7,9}` recovered 4.98 points at a
+0.04-GiB graph-memory and 76.9-second server-lifetime cost in this run. The
+selector must freeze its list before measurement and never continually tune on
+the scored window.
+
+## Tier C — kernel-level backlog and collaboration interface
+
+H1a is inconclusive, so Phase 3 does not prove a universal top operator. The
+available moderate windows are still useful shape inputs: descriptive MoE-GEMM
+shares are 57.52% P06, 75.00% P09, and 54.04% P10; attention shares are 31.60%,
+16.19%, and 30.61%. Only P04's operator windows pass inference gates, where
+attention is 47.88% and MoE GEMM 40.64%. Kernel work must therefore claim
+shape-local improvement, not a resolved global bottleneck.
+
+### Exact shape stream for kernel engineers
+
+`P`, `D`, and `N` below are per-step prefill tokens, decode tokens, and
+scheduled requests. Counts are from clean C00-moderate Layer-1 records.
+Context mix is the fraction of scheduled request-context observations in
+`<=1024 / 1025–8192 / 8193–32768 / >32768` bins.
+
+| Pattern | Model steps: decode / mixed / prefill | N p50 / p95 / max | Dominant exact `(P,D,N): count` | Context mix | Chunk signal |
+|---|---|---|---|---|---|
+| P01 | 5,760: 114 / 5,646 / 0 | 69 / 74 / 77 | `(0,66,66):18`, `(0,67,67):18`; mixed P p50=334, D p50=68 | 100.00 / 0 / 0 / 0% | 6,235 unsplit; sizes 129–512 dominate |
+| P04 | 12,455: 12,291 / 141 / 23 | 8 / 8 / 16 | `(0,8,8):12,129`, `(8191,1,3):23` | 0.03 / 94.62 / 5.35 / 0% | 238/319 chunks >2,048; first/final 122/123 |
+| P06 | 17,464: 17,310 / 152 / 2 | 8 / 16 / 16 | `(0,8,8):13,103`, `(0,16,16):4,098` | 55.11 / 42.93 / 1.96 / 0% | 174/422 >2,048; 136 in 257–512 |
+| P09 | 12,837: 11,783 / 1,054 / 0 | 4 / 16 / 25 | `(0,3,3):3,397`, `(0,2,2):1,549`, `(0,4,4):1,526`, `(0,5,5):1,050` | 51.70 / 48.26 / 0.04 / 0% | 228/1,193 >2,048; 345 in 1,025–2,048 |
+| P10 | 18,130: 17,941 / 92 / 97 | 1 / 4 / 7 | `(0,1,1):13,572`, `(0,2,2):2,675`, `(0,3,3):792`, `(0,4,4):524`, `(8192,0,1):38` | 12.25 / 39.94 / 47.76 / 0.05% | 138/191 >2,048; first/middle/final/unsplit 49/29/49/64 |
+
+The exported kernel-benchmark interface should be a prompt-free weighted table
+with `(P,D,N,context_bin,chunk_class,chunk_size_bin,runtime_mode,count)` plus
+step duration and useful tokens. Use the frozen histogram edges already emitted
+by Layer 1: context `128..131072` powers of two and chunk `16..2048` powers of
+two. Preserve the joint tuples; independent marginal sampling would erase the
+mixed-batch structure.
+
+### Kernel targets and acceptance
+
+1. **Ragged MoE GEMM:** accept variable token counts without padding every
+   expert/layer tile to the largest sequence. Weight microbenchmarks by the P06,
+   P09, and P10 tuples above. The ceiling is the same R64/efficiency opportunity
+   as B1, not an additional gain.
+2. **Attention:** retain P04 `(D,N)=(8,8)` as the valid long rectangular control
+   and test P10's mostly 1–4 decode batches plus 8,192-token chunks. Report
+   useful-token time, workspace, and graph compatibility.
+3. **Serving confirmation:** kernel time must improve on the exact weighted
+   stream, then pass a fixed-arrival serving A/B for throughput, TTFT/p95,
+   memory, and correctness. A rectangular-only kernel win does not close the
+   Phase-3 finding.
+
+`moe_expert_load` was unavailable in Phase 3. No expert-imbalance mechanism or
+gain is claimed; expert-specific packing requires a new low-overhead route
+histogram before implementation.
+
+## Honest limits and Phase-5 measurement requirement
+
+- H1a remains inconclusive: only P04 had two representative/recovered operator
+  windows. Eight other completed moderate patterns failed window validity even
+  though kernel classifiability was 97.05–99.64%.
+- A Phase 5 operator study needs longer, time-stratified samples that reproduce
+  clean scheduled-token, prefill-fraction, decode-batch, and graph-mode
+  distributions, plus a lower-perturbation per-op timer. It must demonstrate
+  overhead before using shares for optimization; the Phase-2 Kineto active
+  window perturbed throughput by 51.3%.
+- Confirmation runs are absent. The MNS and MBT config effects are single-run
+  point estimates and require replication before production decisions.
+- R64 is an offline rectangular-padding upper bound, not measured GPU idle
+  time. H1b's efficiency association does not establish causality.
+- Mixed-batch interference was N/A because no cell retained 30 supported mixed
+  steps inside both leave-one-pattern-out pure-fit supports.
+- Results cover one model, BF16, H20, mostly TP1, and 20/24 cells. P03/C11,
+  P05/C00, P10/C00-TP2, and P11/C00 are absent.
+- The capture validation is one ordered ON/OFF pair. Its padding endpoint is
+  mechanism-valid, but performance deltas have no CI and p95 regressed.
+- Layer 1 did not collect expert-route identities; kernel engineers cannot infer
+  routed-expert imbalance from these artifacts.
+
+## Verification and stop rules for Phase 4 work
+
+Every proposed experiment keeps the original fixed manifest/seed/work, excludes
+warm-up, records Layer-1 accounting, and changes one mechanism. A candidate
+stops on clean failure, footer imbalance/drop, output mismatch, GPU
+contamination, memory regression beyond its declared budget, or violation of
+the 16-H20-hour campaign cap. Throughput, latency, memory, and correctness are
+reported together; no metric shopping or silent pattern substitution is
+allowed.
+
+## GPU accounting
+
+The optional capture pair consumed **0.296389 H20-hours**, taking cumulative
+campaign use from 14.025875 to **14.322265 H20-hours**. Remaining headroom is
+**1.677735 H20-hours**. Both arms returned GPU0 to zero, all eight GPUs were
+0 MiB/0% at final inspection, and no other-user process appeared.
+
+Artifacts are under
+`runs/opprof-phase3/phase4/capture-p09/`; `result.json` SHA-256 is
+`5bb91df28790f6f3c34e4e9ed8e35a1cb8100f93086a4286689d587fd732f2a4`.
+
+## Final ranked one-liners
+
+1. **Prefix affinity (config now):** P08's measured ceiling is **+82.14% req/s** with 62.29% fewer prefill tokens versus matched P07.
+2. **Length-aware scheduler:** raggedness ceiling is **44.79 pp R64 / 44.69% efficiency gap** on P10; smaller confirmed bounds apply to P09/P06.
+3. **Exact capture sizes (config now):** ceiling **5.26 pp P10 / 4.98 pp P09 padding**; P09 validation removed 4.980 pp but did not prove p95 gain.
+4. **MNS64 pattern pools (config now):** measured ceiling **+3.70% req/s P10 / +3.37% P06**, with a −24.27% P01 counterexample.
+5. **Ragged kernels (kernel backlog):** share rank 2's bound; no additive E2E bound is supported while H1a is inconclusive.
+6. **MBT8192 guardrail (config now):** avoids measured regressions up to **11.64%**; MBT2048 has no general positive case.
+
+## Data sanity block
+
+| Numeric family | n | finite | missing | min | max | distinct | Invariant/result |
+|---|---:|---:|---:|---:|---:|---:|---|
+| Ranked items | 6 | 6 | 0 | rank 1 | rank 6 | 6 | Three tiers represented; bounds not summed |
+| Sentinel saturation config deltas | 11 | 11 | 0 | −30.216% | +3.704% | 11 | Both gains and regressions retained |
+| Passing R64 contrast effects | 5 | 5 | 0 | 0.230148 | 0.447872 | 4 | Ratios in [0,1]; duplicate P10 controls expected |
+| Capture-arm padding fraction | 2 | 2 | 0 | 0.035659 | 0.085456 | 2 | Non-negative; ON < OFF |
+| Capture-arm token efficiency | 2 | 2 | 0 | 4.554051 | 4.562222 | 2 | Positive; +0.179% ON |
+| Capture-arm throughput (req/s) | 2 | 2 | 0 | 4.933333 | 4.958333 | 2 | Same offered rate 4.920833 req/s |
+| Capture-arm clean failures | 2 | 2 | 0 | 0 | 0 | 1 expected | Exact 240 s and zero failures |
+| Capture-arm Layer-1 records | 2 | 2 | 0 | 16,491 | 17,579 | 2 | Every footer/sidecar invariant true; zero drops |
+| Optional validation GPU-hours | 1 | 1 | 0 | 0.296389 | 0.296389 | 1 | Positive; cumulative 14.322265 < 16 |
+| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | Cleanup passed |
+
+Checked invariants: Phase-3 metrics remain frozen; every cited number resolves
+to accepted metrics or the checksum-recorded validation; config comparisons
+use saturation rather than normalized moderate throughput; padding/raggedness
+bounds are not presented as throughput; duplicate/non-independent bounds are
+not added; both validation arms use identical work and offered rate; clean
+failures are zero; output work, Layer-1 schema, step continuity, footer/sidecar
+balance, and zero drops pass; ratios lie in their declared domains; all GPU
+memory returned to zero; and cumulative GPU use stays below 16 H20-hours. No
+data-sanity red flag remains.

docs/opprof/phase5-protocol.md

@@ -0,0 +1,640 @@
+# OpProf Phase 5 pre-registered mechanism-decomposition protocol
+
+Status: **ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED**.
+
+Date frozen: 2026-07-12 (Asia/Singapore). This document specifies Phase 5 only.
+It does not authorize a GPU launch, helper implementation, trace transfer, or
+change to the accepted vLLM patch series. Any change to an estimand, request
+set, service order, arrival transform, capture-size set, load, validity gate,
+or decision threshold requires a dated amendment before the affected run.
+
+## Approved dispositions (orchestrator; 2026-07-12)
+
+All five open decisions below are approved before execution. These dispositions
+are normative and close the protocol review gate without changing any frozen
+estimand, command delta, validity threshold, or budget:
+
+1. The recorded-arrival **bridge ledger** is approved. Every machine and human
+   output must state that it decomposes the recorded-arrival P5 gap anchored to
+   P3 controls, not literally P3's already-uniform P10 gap; A3 supplies the
+   explicit bridge back to P3.
+2. The dual P03/P04 control ledgers are approved. Both are always reported and
+   a dominant-mechanism call must pass the frozen rule under both denominators.
+3. A1 is approved exactly as specified: 142 requests, 32-request reorder
+   blocks, 16-request analysis cohorts, frozen bins, and a 64-second fairness
+   cap.
+4. Three P10 replicates per arm, P3 control reuse behind the 3% bridge gate,
+   the conditional control reruns, and optional-tier ordering under the 6.0
+   H20-hour hard cap are approved.
+5. Layer-1-only primary measurement is approved. Routed-expert telemetry is
+   analysis-only, private, optional, and never receives a causal ledger share.
+
+This approval authorizes the later execution turn only when its preflight,
+echo-before-launch, detached-controller, long-context, privacy, accounting,
+cleanup, and budget gates all pass.
+
+## Amendment A-P5-1 — rate-following cold-start gate (orchestrator; 2026-07-12)
+
+The first Phase-5 wave correctly hard-stopped because all four offered-load
+arms failed the inherited A-P3-6 throughput-drift gate: recorded base/A4 drift
+was approximately 216.5%, A1 was 190.6%, and uniform A3 was 13.23%, versus the
+frozen 10% limit. Their 240-second clean windows, output work, offered rates,
+Layer-1 accounting, and drains otherwise passed. The gate was semantically
+wrong for these arms: a rate-following run's scheduled-token throughput follows
+its arrival process by design, so recorded non-stationarity is treatment signal,
+not cold-start contamination. The large recorded-versus-uniform drift gap is
+retained as direct arrival-mechanism evidence. Throughput-drift stationarity
+remains appropriate and unchanged for saturation arms.
+
+For every **rate-following/offered-load** arm, A-P3-6 is replaced by all three
+of the following cold-start-artifact gates:
+
+1. Every logged torch.compile or CUDA-graph capture first-occurrence event must
+   precede the clean boundary. Match server-log event messages containing
+   `torch.compile took`, `Directly load AOT compilation`, `Compiling`, or
+   `Capturing CUDA graphs` (case-insensitive). Timestamped events are compared
+   against `t0_wall_ns + 60 s`. vLLM's capture progress lines have no timestamp;
+   they count as pre-client only when their log-line order precedes the server
+   ready/startup-complete marker, which itself precedes client `t0`. Any matching
+   event after readiness must have a parseable timestamp and precede clean;
+   otherwise the run is invalid.
+2. At least 16 requests must complete successfully in `[0,60 s)`, including at
+   least one request whose recorded `input_tokens >= 8192`.
+3. No first-occurrence capture event may appear inside `[60,300 s)`. Parse the
+   configured startup capture-size set and the completed FULL/PIECEWISE startup
+   capture passes from the server log. From Layer 1, define a captured replay
+   descriptor as `(runtime_mode,bucket_tokens)` for every model-executed
+   `cudagraph.hit=true` step. Every clean descriptor must be covered by the
+   startup-captured mode and bucket set, and no server-log compile/capture event
+   may occur inside clean. Warm and clean descriptor sets are both reported;
+   a descriptor's first **replay** in clean is not mislabeled as a first capture
+   when startup logs prove it was already captured. An uncovered descriptor or
+   clean-window capture/compile event invalidates that run only.
+
+The report records matched server-log events, warm-up completion/long-request
+counts, warm-up and clean descriptor sets, and clean-only descriptors per run.
+Absence of any required log timestamp, Layer-1 interval, or request record is a
+gate failure. The original 10% A-P3-6 drift criterion remains mandatory for
+closed-loop saturation arms. This amendment changes no request set, arrival
+transform, service order, server configuration, clean interval, metric,
+bootstrap, share estimator, dominance rule, control-reuse gate, or GPU budget.
+
+## Goal, system boundary, and success criterion
+
+Phase 3 measured a total useful-token-efficiency gap between irregular patterns
+and rectangular controls but did not causally allocate it. Phase 5 asks how much
+of the P10 gap is recovered when one treatment at a time removes:
+
+1. intra-cohort input-length raggedness;
+2. CUDA-graph decode-batch capture-bucket mismatch;
+3. recorded arrival burstiness; or
+4. usable natural-prefix structure.
+
+The primary system remains Qwen3-30B-A3B BF16, patched vLLM 0.24.0, C00, TP1,
+one H20 per server on dash0. The primary load is the P3 P10 rate
+`lambda = 0.60 * 0.7875 = 0.4725 request/s`; it is held fixed across arms so an
+ablation changes one treatment, not offered demand. The 240-second clean window
+and Layer-1 definition of
+
+```text
+E_token = sum(prefill_tokens + decode_tokens) / sum(model-step duration_ms)
+```
+
+are inherited unchanged. Layer 2 is not needed for the causal ledger and is not
+enabled in primary throughput runs.
+
+Success is a mechanism ledger with an absolute `E_token` for every arm, an
+un-normalized share and bootstrap confidence interval for every mechanism, and
+an explicit residual/interaction line. A null or negative share is a valid
+result. Merely recovering the expected sign is not success.
+
+## Pinned Phase-3 evidence and the arrival-estimand discrepancy
+
+The frozen P3 C00-TP1 moderate values are:
+
+| Cell | `E_token` (tokens/ms) | Role |
+|---|---:|---|
+| P10 | 2.6191132083 | irregular P3 base |
+| P03 | 4.7355997154 | long-input/short-output rectangular control |
+| P04 | 3.0547035940 | long-input/long-output rectangular control |
+
+P10 therefore lost 44.693% versus P03 and 14.260% versus P04. P3 used both
+controls, so Phase 5 reports two parallel ledgers, one per frozen control. It
+never selects the denominator that makes a mechanism look largest. A mechanism
+is called control-robust only when its decision agrees under both ledgers.
+
+There is one blocking provenance discrepancy. The private source contains
+`timestamp` and `source_timestamp`, but P3's materializer discarded them and
+set every P10 row to `arrival=steady`; the P3 client then admitted requests at
+exactly `1/lambda`. Thus P3 has no recorded-arrival burstiness to remove.
+
+The recommended resolution, pending orchestrator approval, is a **P5 bridge
+ledger**: the P5 base replays the same P10 requests at rate-normalized recorded
+timestamps, A3 uniformizes those timestamps, and P03/P04 remain the frozen P3
+controls. A3 also acts as a bridge back to the P3 steady workload. This meets
+the requested arrival ablation but decomposes a recorded-arrival P5 gap, not
+literally the already-uniform P3 gap. The report must show both
+`E_A3 - E_P3_P10` and its CI before relating the P5 ledger to P3.
+
+If the orchestrator rejects this rebase, the P3-exact alternative is mandatory:
+A3's share is `0 / N/A by construction`, and recorded arrival is reported only
+as a stress sensitivity, not as a mechanism share. It is scientifically invalid
+to call the reverse, burstiness-injecting treatment an ablation that removes
+arrival dynamics. No GPU work may begin before this decision is recorded.
+
+## Common request set and exact arrival transforms
+
+The primary request set is the first **142** rows of the frozen 4,011-row P10
+selection in original source order. This is exactly the number of admissions at
+`lambda=0.4725` over `[0,300 s)`: scheduled times are
+`0, 1/lambda, ..., 141/lambda`. All five arms contain the same 142 request IDs,
+prompts, per-request input/output lengths, and aggregate input/output token
+totals. There is no wrap, replacement, cancellation, or resampling.
+
+The Phase-5 materializer must preserve `timestamp` as private metadata. Let
+`z_i` be its stable source-order timestamp for request `i`, and let `N=142`.
+The two arrival vectors are frozen as:
+
+```text
+recorded-scaled: a_i = (z_i-z_0) * ((N-1) / (lambda*(z_(N-1)-z_0)))
+uniformized:     a_i = i / lambda
+```
+
+`z_(N-1)` must exceed `z_0`; timestamps must be finite and nondecreasing.
+Ties remain ties. Both vectors start at zero, end at `141/lambda`, have the same
+mean rate, and use the same request order except in A1. The client schedules
+against `a_i` directly and does not add jitter. This preserves the recorded
+inter-arrival shape while preventing mean-rate differences from masquerading as
+an arrival mechanism.
+
+The protocol requires a small `scripts/opprof_phase5_client.py` extension in a
+later, no-GPU implementation turn. Before execution, CPU-only tests must prove:
+
+- exact 142-row identity and token sums across all manifests;
+- timestamp normalization endpoints and nonnegative gaps;
+- uniform gaps equal `1/0.4725` within 1 microsecond;
+- the A1 fairness bound and deterministic ordering;
+- fixed 60+240-second timing, no wrap, exact output work, and text redaction.
+
+Its reviewed SHA-256 and all manifest SHA-256 values are frozen in the detached
+controller before GPU use.
+
+## Falsifiable mechanism estimands
+
+For arm `m` and rectangular control `c in {P03,P04}`:
+
+```text
+gap_c       = E_control,c - E_base
+delta_m     = E_ablated,m - E_base
+share_m,c   = delta_m / gap_c
+```
+
+The same base and same control are used for all four mechanisms within a
+ledger. No share is clipped to `[0,1]`. Shares may be negative, exceed one, or
+sum above/below one because single-factor interventions can overlap or interact.
+
+The arithmetic residual/interaction line is
+
+```text
+share_residual+interaction,c = 1 - sum_m share_m,c
+```
+
+with a joint bootstrap CI. This is bookkeeping, not proof that the remainder is
+one separable mechanism. It may contain unmeasured mechanisms, non-additivity,
+double-counting, MoE routing, chunked-prefill interference, and measurement
+error. Individual shares are never renormalized to total 100%; the residual is
+never clipped to make the table visually close.
+
+### A1 — length-binned service order
+
+**Hypothesis.** Length-homogeneous local cohorts reduce ragged-attention/SM
+imbalance, so `E_A1 > E_base`. The hypothesis is falsified if the registered
+manipulation check fails or the Holm-corrected efficiency contrast is not
+positive.
+
+Starting from consecutive **32-request reorder blocks** in original P10 order,
+assign each request to the fixed input-length bins
+
+```text
+[0,512], [513,1024], [1025,2048], [2049,4096],
+[4097,8192], [8193,16384], [16385,32768]
+```
+
+and stable-sort each block by `(bin_id, input_tokens, original_index)`. This
+creates two more homogeneous 16-request analysis cohorts per complete reorder
+block. Assign the sorted requests to the block's unchanged recorded-scaled
+arrival slots. If that assignment would delay any request by more than **64
+seconds** relative
+to its original slot, choose the earliest-deadline request first until all
+deadlines are feasible, then resume the length order. Early movement is allowed;
+late movement is capped. Ties are stable and no sorting crosses a 32-request
+block.
+
+On consecutive complete 16-request cohorts of evaluation-slice service order,
+define `R16 = 1 - sum(L_i) / sum(16*max_cohort(L_i))`; the incomplete final
+cohort is excluded and reported. Frozen pre-run values are base `R16=0.641744`
+and sorted `R16=0.473409`, a 0.168334 absolute (26.23% relative) reduction;
+plain sorting's maximum added delay is 62.744 seconds. The manipulation passes
+only if regenerated values match these within `1e-6`, `R16` falls by at least
+20% relative and 0.15 absolute, and no request violates the 64-second delay
+bound.
+Arrival-slot timestamps, request/content multiset, per-request output lengths,
+total tokens, server config, and prefix-caching setting are identical to base.
+
+A1 estimates the total effect of changing cohort composition. It does not
+claim to isolate a particular attention kernel: service order can mediate
+decode-batch composition, chunked-prefill mixing, cache locality, and
+content-bound MoE routing. If the prefix-query hit ratio changes by more than
+one percentage point, or the normalized inter-arrival vector changes at all,
+the arm is labeled confounded and has no publishable raggedness share.
+
+### A2 — measured decode-B capture sizes (config-tier deliverable)
+
+**Hypothesis.** Exact capture sizes for P10's observed pure-decode batch support
+remove decode-bucket slack, so pure-decode padding falls and `E_A2 > E_base`.
+No recovery falsifies the efficiency hypothesis; failure to remove the targeted
+padding invalidates the ablation rather than supporting a null mechanism.
+
+The P3 P10/C00/rho=0.60 clean Layer-1 stream has SHA-256
+`51ad4be12178da91d2af484d0946a2274afd3bcbbee33f37940cfe0ff2ea7fa7`.
+Its 17,941 pure-decode steps have the exact decode-B histogram:
+
+| B | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+|---:|---:|---:|---:|---:|---:|---:|---:|
+| Steps | 13,572 | 2,675 | 792 | 524 | 161 | 202 | 15 |
+
+Defaults already contain 1, 2, 4, and 8. A2 therefore adds exactly
+`{3,5,6,7}` and freezes the complete server list as:
+
+```text
+1 2 3 4 5 6 7 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128
+136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256
+272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512
+```
+
+This covers 100% of P3's observed P10 pure-decode B support. The A2
+manipulation requires at least 99% support coverage in the new clean runs and a
+90% reduction in pure-decode padding tokens versus base. It targets decode
+capture-bucket mismatch; it does not remove prefill-token padding, eager
+overflow, graph launch overhead, or length raggedness itself. Capture startup
+time and memory are reported as config cost, not treated as free.
+
+### A3 — recorded arrival to uniform arrival
+
+**Hypothesis.** Uniformizing the same rate and request order reduces burst-driven
+decode-batch/queue variance, so `E_A3 > E_base`. The mechanism is falsified if
+the clean 5-second decode-B CV and waiting-queue CV do not fall, or if the
+Holm-corrected efficiency contrast is not positive.
+
+A3 changes only `recorded-scaled` arrival slots to `i/lambda`. Content, order,
+input/output lengths, aggregate tokens, server config, prefix caching, and
+capture sizes are unchanged. It estimates the total service effect of arrival
+shape, including its legitimate downstream changes to batching and queueing.
+It does not isolate a scheduler instruction cost. Under the P3-exact fallback,
+this arm is the base-equivalent bridge and its arrival share is N/A as described
+above.
+
+### A4 — natural prefix caching disabled
+
+**Hypothesis.** Direction is deliberately two-sided. Natural P10 reuse may
+increase `E_token` through KV reuse, in which case disabling it gives a negative
+share; alternatively, low-value/fragmented cache structure may impose overhead
+or alter batching, giving a positive share. Either direction is publishable.
+
+A4 omits only `--enable-prefix-caching`. Prompts, natural repeated-prefix
+structure, recorded arrival slots, request order, token totals, scheduler limits,
+and capture sizes remain identical to base. Required manipulation checks are
+zero local prefix cache hits/queries in the disabled arm and unchanged prompt
+hashes. This estimates the contribution of exploiting C structure, not the
+intrinsic content similarity of the prompts.
+
+### Mechanisms without a clean ablation
+
+MoE routing skew cannot be removed while preserving P10 content and model
+semantics: changing tokens, router weights, top-k, or expert placement changes
+more than routing skew. It receives no causal share.
+
+If budget remains, one **analysis-only**, separately started P10 sample may add
+`--enable-return-routed-experts`. It is excluded from every `E_token` clean
+window and ledger numerator. Offline analysis reports per-layer expert-count
+entropy, Gini coefficient, coefficient of variation, max/mean load, and their
+association with same-step token-normalized duration. The arrays remain private.
+This telemetry is sampled and perturbing, can consume a large scheduler-side
+buffer, and provides correlation rather than causal attribution; it can only
+help interpret the residual/interaction line. Phase 3's MoE layer-duration CV
+was N/A, so it cannot substitute for these routed-expert counts.
+
+## Exact commands and config deltas
+
+The following interface is normative for the later implementation. Variables:
+
+```bash
+P5C='python scripts/opprof_phase5_client.py'
+PRIVATE=/home/admin/cpfs/wjh/opprof-phase5-private/manifests
+P3PRIVATE=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl
+P3SOURCE=/home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/chat_w20260311_1000.jsonl
+MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
+RATE=0.4725
+```
+
+Materialize the five private manifests without printing prompt text:
+
+```bash
+$P5C transform --in "$P3PRIVATE" --take-first 142 \
+  --timestamp-source "$P3SOURCE" --join-key source_index \
+  --timestamp-field timestamp --arrival recorded-scaled --target-rate "$RATE" \
+  --service-order original --out "$PRIVATE/P10-base.jsonl"
+$P5C transform --in "$P3PRIVATE" --take-first 142 \
+  --timestamp-source "$P3SOURCE" --join-key source_index \
+  --timestamp-field timestamp --arrival recorded-scaled --target-rate "$RATE" \
+  --service-order length-binned --reorder-block-size 32 \
+  --analysis-cohort-size 16 \
+  --length-bin-edges 512,1024,2048,4096,8192,16384,32768 \
+  --max-added-delay-seconds 64 --out "$PRIVATE/P10-A1.jsonl"
+$P5C transform --in "$P3PRIVATE" --take-first 142 \
+  --timestamp-source "$P3SOURCE" --join-key source_index \
+  --timestamp-field timestamp --arrival uniform --target-rate "$RATE" \
+  --service-order original --out "$PRIVATE/P10-A3.jsonl"
+ln -s P10-base.jsonl "$PRIVATE/P10-A2.jsonl"
+ln -s P10-base.jsonl "$PRIVATE/P10-A4.jsonl"
+```
+
+The symlinks make unchanged request bytes explicit; the controller hashes the
+resolved content and requires A2/A4 hashes to equal base. A1/A3 require equal
+sorted request-ID sets and equal input/output token sums.
+
+Common server command (`ARM` is `base`, `A1`, `A2`, `A3`, or `A4`):
+
+```bash
+taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU" \
+  VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
+  vllm serve "$MODEL" --host 127.0.0.1 --port "$PORT" \
+  --tensor-parallel-size 1 --enable-chunked-prefill \
+  --enable-prefix-caching --shutdown-timeout 600
+```
+
+- A2 adds
+  `--cudagraph-capture-sizes 1 2 3 4 5 6 7 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512`.
+- A4 removes `--enable-prefix-caching`.
+- Base, A1, and A3 have no server delta.
+
+Every moderate client command is:
+
+```bash
+taskset -c "$CPUSET" $P5C run \
+  --manifest "$PRIVATE/P10-$ARM.jsonl" \
+  --base-url "http://127.0.0.1:$PORT" --model "$MODEL" \
+  --load-point moderate --fixed-request-rate "$RATE" \
+  --max-concurrency 256 --ignore-eos --temperature 0 \
+  --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 \
+  --post-clean-seconds 0 --drain-timeout-seconds 600 \
+  --workload-seed 20260712 --server-seed 20260712 \
+  --result-dir "$RUN_DIR/client"
+```
+
+There are no profiler endpoint calls. Exact commands, effective config, startup
+capture list, compile-cache key, hashes, clocks, host load, and GPU process list
+are recorded per run.
+
+## Execution and validity discipline
+
+### Placement, order, and detached ownership
+
+A-P3-1 already rejected eight-way and authorized four-way placement. Phase 5
+therefore uses at most four simultaneous TP1 servers, GPU0--GPU3, with the
+frozen disjoint CPU masks `0-19`, `20-39`, `40-59`, and `60-79`. Topology must
+still match P3. A changed host topology, source/patch/runtime hash, or clock
+policy invalidates reuse of the placement gate and stops for review.
+
+Three independent replicates are run for each of the five primary arms (15
+measured runs). Sort all `(replicate,arm)` assignments by SHA-256 of
+`"20260715:<replicate>:<arm>"`, pack them into waves of 4, 4, 4, and 3, and
+rotate GPU assignments. The final three-target wave uses the frozen P06/C00
+saturation background in the fourth slot so measurements retain the validated
+four-way host regime. No wave contains two replicates of the same arm; if the
+hash order would do so, stable-swap the later item with the first legal item and
+record the resolved order before launch. Background data are not analyzed.
+
+As required by A-P3-2, a dash0-resident `setsid`/`nohup` controller owns every
+process, has `--resume`, atomically records state, and skips only fully validated
+runs with matching hashes. Interactive SSH never owns a server. Before each
+wave it logs one echo line with resolved arms, GPUs/CPUs, manifests, rate,
+paths, reserved H20-hours, expected duration, and disk headroom.
+
+Use three unmeasured 60-second burn-ins before measured arms: common C00,
+A2 capture-list C00, and prefix-cache-off C00. They warm compile/AOT artifacts
+but are not measurements. Per-run-unique OpProf directories must remain ignored
+by vLLM compile factors, preserving the accepted Phase-2 fix.
+
+### Long-context-safe warm-up and drain gates
+
+These gates are active from the first run; no short-context default is tried
+first.
+
+- Warm-up is exactly 60 seconds and excluded. P10 passes with at least 32
+  successful warm-up completions, **or** at least 16 completions plus the exact
+  A-P3-6 stabilization test: model-executed steps in `[45,50)`, `[50,55)`, and
+  `[55,60)`; at least 16 steps/bin; positive scheduled-token rates `R_j`; and
+  OLS drift `abs(slope)*15/mean(R_j) <= 0.10`. Missing bins, discontinuity, or
+  accounting failure invalidates the run.
+- Drain timeout is 600 seconds. A timeout marks the run drain-quarantined but
+  does not invalidate an otherwise valid clean window. More than 20% of primary
+  runs quarantined stops Phase 5 for review.
+- The clean window is exactly three contiguous 80-second segments. Admission
+  and completion accounting follows P3. Offered rate must be within 5% of
+  0.4725 request/s, with zero clean failures and exact output tokens.
+- Layer-1 JSONL/footer/sidecar accounting, zero drops, contiguous steps, no
+  profile leakage, clock/load capture, other-user-process absence, and final
+  zero GPU memory are hard gates inherited from P3.
+
+No semantic failure is retried with altered parameters. One exact retry is
+allowed for an infrastructure artifact failure, and both attempts remain in
+the operational findings.
+
+## Control reuse, secondary scope, and budget
+
+P03/P04 P3 controls may be reused because they already use the same model,
+patch, C00-TP1 config, normalized load, 240-second clean metric, placement
+regime, and Layer-1 schema. Reuse avoids spending GPU time without changing the
+denominator.
+
+Reuse is valid only if source, patch, model, runtime, clocks, and placement
+hashes match and fresh A3-uniformized P10 differs from frozen P3 P10 by at most
+3% in `E_token` with a bootstrap CI containing zero difference. If this bridge
+gate fails, temporal/runtime drift is plausible: rerun P03 and P04 under their
+exact P3 manifests and commands, three replicates each, before computing any
+share. A failed bridge never licenses rescaling old controls.
+
+After the complete P10 ledger, optional work is ordered as follows and starts
+only if the controller's conservative reservation remains below the 6.0
+H20-hour hard cap:
+
+1. one saturation run per P10 arm, using the full 4,011-row manifest, for
+   descriptive mechanism persistence only;
+2. one five-arm moderate ledger each for P09 and P06, reported as exploratory
+   within-run-bootstrap evidence, not equal replication to P10; and
+3. one routed-expert analysis-only sample.
+
+For secondary A2, the precomputed P3 pure-decode support additions are P09
+`{3,5,6,7,9,10,11,12,13,14,15,17,18,19,20,21,22,23,25}` and P06
+`{3,9,10,11,12,13,14,15}`; default sizes are retained. P09 A3 is a no-change
+steady negative control, while P06 A3 changes its registered burst-8 arrivals
+to uniform spacing at the same mean rate. P09/P06 conclusions are always
+labeled secondary.
+
+Expected accounting, including server-owned startup/shutdown time:
+
+| Tier | New measured runs | Expected H20-hours | Expected 4-way wall |
+|---|---:|---:|---:|
+| P10: 5 arms x 3 replicates | 15 | 1.6-1.9 | 35-55 min |
+| Three compile burn-ins | 0 | 0.1-0.2 | 3-6 min |
+| Conditional P03/P04 reruns | 6 | 0.6-0.8 | 15-25 min |
+| Optional P10 saturation | 5 | 0.5-0.8 | 15-25 min |
+| Optional P09/P06, 5 arms each | 10 | 0.9-1.2 | 20-35 min |
+| Optional routed-expert sample | 1 analysis-only | 0.1-0.2 | 5-10 min |
+| **Maximum planned** | **36 ledger + 1 analysis-only** | **3.8-5.1 expected** | **about 1.5-2.5 h** |
+
+The hard cap is **6.0 H20-hours new Phase-5 spend**, not a target. Actual time
+while a server owns GPU memory is charged. Optional tiers are skipped rather
+than overrunning the cap. The 600-second drain allowance is a watchdog, not an
+assumption in the expected estimate; repeated long drains consume the optional
+budget first.
+
+With no Kineto traces, primary P10 artifacts are estimated at 0.4-0.6 GB and
+all planned public artifacts below 1.5 GB. Stop if public Phase-5 artifacts
+exceed 3 GB or CPFS free space falls below 100 GB. Prompt-bearing manifests and
+routed-expert arrays remain in mode-0700 private storage and are not counted as
+public deliverables.
+
+## Statistical analysis and decision rules
+
+Use 5-second moving-block bootstrap over clean time, 100,000 resamples, seed
+20260716. For P10, first resample the three run IDs, then resample 5-second
+blocks within each selected run; arms and reused P3 controls are resampled
+independently. Every `E`, `delta`, share, share sum, and residual/interaction
+gets a percentile 95% CI. Absolute `E` and delta accompany every ratio.
+
+Bootstrap ratio draws are never deleted because their denominator is
+inconvenient. If either control gap has a point estimate `<=0`, its CI includes
+zero, or more than 5% of bootstrap denominator draws are `<=0`, that control's
+ledger is **INCONCLUSIVE (unstable denominator)**.
+
+The confirmatory family is the four two-sided tests of `E_ablated-E_base=0` on
+P10. Apply Holm correction at family-wise alpha 0.05 across A1--A4. The same
+corrected delta test serves both control ledgers; duplicating denominators does
+not create eight tests. A1--A3 only support the expected recovery claim when
+their corrected contrast is positive. A4 may be significant in either
+direction. Manipulation-check failure makes the corresponding share N/A even
+if efficiency changes.
+
+A mechanism is **dominant** only if, under both P03 and P04 ledgers:
+
+- point `share >= 0.30`;
+- the 95% share CI excludes 0.15 on the high side (`CI_low > 0.15`); and
+- its Holm-corrected efficiency contrast is significant in the expected
+  direction (two-sided for A4).
+
+Meeting the rule under only one control is reported as **control-sensitive**,
+not dominant.
+
+The primary ledger is **publishable** when all five P10 arms have three valid
+replicates, both control denominators are stable, all four manipulation checks
+are evaluable, every share/residual has a finite CI, the fresh A3/P3 bridge is
+reported, and no privacy/data-sanity red flag exists. It may be publishable
+with no dominant mechanism. It is **inconclusive** if any primary arm is
+missing, a denominator is unstable, the arrival-rebase decision is unresolved,
+or two or more share CIs have width greater than 0.50. One failed mechanism
+manipulation yields a publishable partial ledger only if that line is explicitly
+N/A and the headline claim excludes it.
+
+No additive causal claim is made from `sum(share_m)`. Pairwise and higher-order
+interactions are not identified by this one-factor-at-a-time matrix; a combined
+all-off arm would be a new experiment requiring amendment, not an improvised
+way to close the ledger.
+
+## Deliverable and artifact contract
+
+Execution, if later approved, must produce:
+
+- `docs/opprof/phase5-results.md`;
+- `runs/opprof-phase5/phase5/metrics.json` with schema, arm/run values,
+  bootstrap draws' seed and summary, Holm results, dual-control ledgers,
+  residual/interaction, gates, and GPU accounting;
+- per-run exact commands, environment/provenance, client records, Layer-1
+  stream/footer/sidecar, monitor data, and machine-readable sanity JSON; and
+- an **Operational findings** section covering warm-up stabilization, drains,
+  compile/capture startup cost, contamination, retries, and any mismatch between
+  expected and realized mechanism removal.
+
+Every raw-run summary, aggregate table, and final metrics file ends with the P3
+sanity schema: `n`, finite/missing, min, max, distinct count, and applicable
+invariants. Red flags are reported first and stop inferential analysis. Public
+artifacts may contain request IDs, hashes, lengths, counts, and timing only;
+prompt, messages, content, generated text, source substrings, and routed-expert
+arrays are forbidden.
+
+## Final ablation table
+
+| Mechanism | What changes | What is preserved |
+|---|---|---|
+| Base | Recorded P10 timestamps are rate-normalized and replayed in source order | Frozen 142 requests, content/tokens, C00-TP1, prefix cache on, default capture list, `lambda=0.4725` |
+| A1: length raggedness | Stable length-bin sort within 32-request reorder blocks into 16-request cohorts, 64-second late cap | Requests/content/token totals, arrival-slot vector, output lengths, server config, prefix setting |
+| A2: capture mismatch | Add exact P10 decode-B sizes `{3,5,6,7}` to the full default capture list | Manifest/order/arrivals/content/tokens, scheduler limits, prefix setting |
+| A3: arrival dynamics | Recorded-scaled slots become uniform `i/0.4725` slots | Request order/content/tokens, mean rate, server/capture/prefix config |
+| A4: prefix structure | Remove `--enable-prefix-caching` | Natural prompt structure, request order/arrivals/content/tokens, capture/scheduler config |
+| Residual + interactions | No extra run; joint arithmetic remainder `1-sum(shares)` | Raw unnormalized mechanism shares; no clipping or forced 100% allocation |
+
+## Final run count and GPU estimate
+
+Primary commitment: **15 new measured P10 runs plus three unmeasured burn-ins,
+1.7-2.1 expected H20-hours, 35-60 minutes four-way wall, and 0.4-0.6 GB public
+disk**. Conditional control reruns add 6 runs; all optional tiers bring the
+maximum to **36 ledger runs plus one analysis-only run, 3.8-5.1 expected
+H20-hours, about 1.5-2.5 hours wall, and less than 1.5 GB public disk**. New
+Phase-5 GPU use stops at **6.0 H20-hours** under all circumstances.
+
+## Resolved decisions from the orchestrator
+
+1. **Approved:** use the recommended recorded-arrival P5 bridge ledger, with
+   its explicit limitation that it is not a literal decomposition of P3's
+   already-uniform P10 gap; otherwise select the P3-exact A3=N/A fallback.
+2. **Approved:** dual P03/P04 control ledgers and the requirement that “dominant” hold
+   under both, rather than selecting one P3 denominator.
+3. **Approved:** the 142-request slice, 32-request reorder blocks, 16-request analysis
+   cohorts, fixed bins, and 64-second fairness cap as A1's isolation/latency
+   tradeoff.
+4. **Approved:** three P10 replicates, reuse of P3 controls behind the 3% bridge gate,
+   and the optional-tier order within the 6.0-H20-hour cap.
+5. **Approved:** Layer-1-only primary runs and treating routed-expert telemetry as
+   private analysis-only evidence with no causal share.
+
+## Protocol sanity block
+
+| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---|
+| P3 control/base `E_token` | 3 | 2.619113 | 4.735600 | 3 | Finite, positive, not identical |
+| P3 P10 control gaps | 2 | 0.435590 | 2.116487 | 2 | Positive; dual denominators retained |
+| P10 request rows/arm | 5 arms | 142 | 142 | 1 expected | Same IDs and input/output token sums required |
+| Selected source timestamps (s) | 142 | 0.014 | 21.445 | 142 | Finite, nondecreasing; gap min/max 0.002/0.851 s, 127 distinct gaps |
+| Primary offered rate (req/s) | 5 arms | 0.4725 | 0.4725 | 1 expected | Positive; achieved rate must be within 5% |
+| Warm-up / clean / drain gates (s) | 3 values | 60 | 600 | 3 | Clean is exactly `3*80=240`; long-context drain is 600 |
+| A1 input-length bins | 7 | 0 | 32768 | 7 intervals | Ordered, contiguous, cover frozen P10 range |
+| A1 frozen `R16` values | 2 | 0.473409 | 0.641744 | 2 | Sorted is lower by 0.168334 absolute / 26.23% relative; not identical |
+| A1 added-delay values (s) | 142 | 0 | 62.744 | >1 expected | Non-negative and all below 64-second cap |
+| P3 P10 pure-decode steps | 17,941 | B=1 | B=7 | 7 B values | Counts sum to 17,941; non-negative; stream SHA pinned |
+| P10 A2 added capture sizes | 4 | 3 | 7 | 4 | Exactly missing observed support `{3,5,6,7}`; 100% P3 support covered |
+| Primary measured runs | 15 | 3/arm | 3/arm | 1 expected | `5 arms * 3 replicates`; no arm omitted |
+| Maximum GPU analysis/ledger runs | 1 plan | 37 | 37 | 1 expected | `15+6+5+10+1=37`; burn-ins excluded |
+| Expected H20-hours | 1 plan | 3.8 | 5.1 | 2 bounds | Finite, non-negative, below hard cap 6.0 |
+| Share domain | 5 ledger lines/control | Unbounded | Unbounded | N/A | No clipping; ratios may be negative or >1 |
+| Bootstrap resamples | 1 setting | 100,000 | 100,000 | 1 expected | Seed 20260716; no denominator-draw deletion |
+| Phase-5 GPU runs in this protocol turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied |
+
+Checked invariants: `0.60*0.7875=0.4725`; P3 control gaps are positive;
+the seven decode-B counts sum to 17,941; default plus `{3,5,6,7}` covers all
+observed P10 pure-decode B values; `5*3=15` primary runs; maximum planned count
+is `15+6+5+10+1=37`; expected GPU use remains below the 6.0-hour hard cap;
+ratios are not constrained to `[0,1]`; expected constants are labeled; and no
+GPU command, helper change, manifest transform, or experiment was executed in
+this protocol-only turn. The unresolved P3-arrival/P5-arrival estimand mismatch
+is reported first as a blocking decision rather than hidden in the ledger.

docs/opprof/phase5-results.md

@@ -0,0 +1,223 @@
+# OpProf Phase 5 dash0 results
+
+Status: **FINAL — INCONCLUSIVE MECHANISM LEDGER; NO DOMINANCE CALL**.
+
+Date: 2026-07-12. All 15 registered P10 primary runs completed with three
+replicates per arm and passed A-P5-1. The fresh A3 bridge passed, so the frozen
+P3 P03/P04 controls were reused. The ledger is nevertheless inconclusive:
+P04's reused-control denominator is unstable, and A2, A3, and A4 fail their
+predeclared manipulation checks. The protocol therefore permits one official
+share (A1 under P03), requires the other lines to be N/A, and forbids a
+dual-control dominance conclusion.
+
+The machine result is `runs/opprof-phase5/phase5/metrics.json` (SHA-256
+`fe16934866806ac542f7974cac061beeb1f1c58f5bfb36698efd8aeaa99b8cb7`).
+The final analyzer SHA-256 is
+`8b0ff8658614227ee00cf8d10c82c4b5edf6f6210db172fd6efc9cc149e51e10`;
+its no-GPU tools and analysis tests pass.
+
+This remains explicitly a **recorded-arrival P5 bridge ledger anchored to P3
+controls**, not a literal decomposition of P3's already-uniform P10 gap.
+
+## Data-sanity result first
+
+**RED FLAG: the P04 control denominator is unstable.** P5 recorded-arrival
+base has `E_token=3.013752`, while reused P3 P04 has `E_token=3.054704`, leaving
+only a `0.040952` gap with bootstrap 95% CI `[-0.649096, 0.752594]`; 45.353% of
+denominator draws are nonpositive. This violates every registered stability
+form: the CI contains zero and the nonpositive fraction exceeds 5%.
+
+P03 remains stable: its gap is `1.721848`, CI `[1.459092, 1.988772]`, with zero
+nonpositive draws. All execution-validity checks pass: 15/15 primary runs,
+three replicates per arm, exact 240-second clean windows, zero clean failures,
+offered rates within 5%, continuous/balanced Layer-1 accounting, identical
+request IDs and token sums, all A-P5-1 cold-start gates, no drain quarantine,
+and GPU spend below the cap. The red flag is reported before the ledger and no
+conclusion is built on P04's raw ratio draws.
+
+## Bridge and absolute efficiency
+
+A3 reproduces the P3 uniform-arrival P10 base: `delta_E=+0.007041`, 95% CI
+`[-0.329799, 0.354859]`, or 0.269% absolute relative difference. The CI contains
+zero and the point difference is below 3%, so the bridge passes and P3 control
+reuse is valid under the frozen rule. Conditional control reruns were not
+launched.
+
+| Arm | `E_token` | Bootstrap 95% CI | Delta from P5 base |
+|---|---:|---:|---:|
+| Base, recorded | 3.013752 | [2.754462, 3.267964] | — |
+| A1 length-binned | 3.078999 | [2.801659, 3.349307] | +0.065247 |
+| A2 capture sizes | 3.062128 | [2.802812, 3.314276] | +0.048376 |
+| A3 uniform arrival | 2.626154 | [2.451852, 2.792684] | -0.387598 |
+| A4 prefix cache off | 3.020959 | [2.760494, 3.274105] | +0.007207 |
+
+## Manipulation checks and Holm family
+
+| Arm | Manipulation result | Efficiency delta 95% CI | Raw p | Holm p |
+|---|---|---:|---:|---:|
+| A1 | **PASS:** R16 0.641744 -> 0.473409; max delay 62.743 s; prefix-hit-ratio delta -0.296 pp | [-0.309461, 0.442565] | 0.73108 | 1.00000 |
+| A2 | **FAIL:** support coverage 98.879% <99%; padding reduction 82.541% <90% | [-0.315361, 0.411446] | 0.79022 | 1.00000 |
+| A3 | **FAIL:** decode-B CV falls 0.8706 -> 0.6551, but waiting CV rises 6.0145 -> 6.8557 | [-0.693444, -0.078141] | 0.01384 | 0.05536 |
+| A4 | **FAIL:** disabled arm still reports 3,093,324 local prefix queries and 52,896 hits, not zero | [-0.357637, 0.369309] | 0.97230 | 1.00000 |
+
+Per protocol, a failed manipulation makes that mechanism's official share N/A
+even if its raw efficiency contrast is nonzero. A3's uncorrected contrast is
+negative, and it also misses family-wise significance after Holm correction.
+
+## A-P5-1 arrival evidence
+
+The four originally invalidated arms supplied the evidence for A-P5-1. Under
+the superseded throughput-drift calculation, recorded base/A1/A4 span
+190.55%-216.50% normalized drift, while uniform A3 is 13.23%. This gap is
+retained as direct evidence that the old gate measured arrival shape in a
+rate-following experiment; it is not converted into an efficiency share.
+
+All 15 replacement/continued primary runs pass the amended cold-start gate:
+25-28 warm-up completions, 12-16 completions with input at least 8192 tokens,
+all compile/capture first occurrences before clean, zero clean-window
+first-capture events, and zero uncovered Layer-1 graph descriptors. The 10%
+drift gate was retained unchanged for the P06 saturation background arm.
+
+## Operational findings
+
+- The detached controller completed three cache burn-ins, four four-way waves,
+  all drains, validation, budget accounting, and cleanup. The four invalidated
+  first-wave arms were rerun exactly; none of their superseded measurements
+  enters the ledger.
+- The first exact rerun produced complete clean artifacts but hit a post-hoc
+  validator `NameError` from a missing `math` import. Immutable artifacts were
+  CPU-re-adjudicated under A-P5-1, the import was repaired, and no third GPU run
+  was charged.
+- CPU analysis repairs admitted legitimate idle fixed-time blocks as `[0,0]`,
+  supported the older P3 control schema, converted one NumPy boolean at the
+  JSON boundary, and explicitly separated diagnostic from reportable shares.
+  These changes alter no GPU data, estimator, seed, threshold, or resample
+  count.
+- P10 drains range from 0.676 to 6.004 seconds. The separate P06 saturation
+  background run drained naturally in 87.903 seconds, below the 600-second
+  gate, with 564/564 clean completions and zero failures.
+- Public Phase-5 artifacts occupy 286,205,365 bytes (304 MiB), below the 3 GB
+  stop threshold; CPFS retained 1,286 GB free.
+- MoE routing skew remains content-bound and receives no causal share. The
+  optional routed-expert analysis-only run was not required for this primary
+  ledger and was not launched.
+
+## Launch echo
+
+```text
+LAUNCH_ECHO utc=2026-07-12T12:07:54Z host=dash0 gpus=0-3 cpus=0-79 source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0@4b253fd manifests=/home/admin/cpfs/wjh/opprof-phase5-private/manifests outputs=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5 runs=3burnin+15primary+1background rate=0.4725 warmup=60s clean=240s drain=600s est_wall=35-60min est_gpu=1.7-2.1_H20h hard_cap=6.0_H20h conditional_controls=6_if_bridge_fails
+```
+
+Resume after A-P5-1 reused the completed burn-ins and compile/cudagraph caches.
+The first launch through final controller completion took approximately 57.9
+minutes wall time, including the adjudication/resume interval.
+
+## Run completion stats
+
+| Item | Count/result |
+|---|---:|
+| Valid primary P10 runs | **15/15** |
+| Replicates | **3 per arm** |
+| Valid P06 background runs | 1 |
+| Completed burn-ins | 3 |
+| Superseded pre-amendment arms | 4 |
+| Conditional fresh controls | 0; bridge passed, P3 reused |
+| Optional secondary/routed-expert runs | 0 |
+| A-P5-1 primary passes | 15/15 |
+| Clean failures | 0 |
+| Drain quarantines | 0 |
+
+## Mechanism ledger
+
+Official shares follow the frozen N/A rules. Values in parentheses are raw
+diagnostic ratios retained for audit, not causal/reportable shares.
+
+| Mechanism | P03 share (95% CI) | P04 share (95% CI) | Disposition |
+|---|---|---|---|
+| A1 length raggedness | **0.0379 [-0.2011, 0.2344]** | N/A; unstable denominator (diagnostic 1.5933 [-6.8925, 7.0636]) | Manipulation passes; no significant recovery |
+| A2 capture mismatch | N/A; manipulation failed (diagnostic 0.0281 [-0.2049, 0.2170]) | N/A; unstable denominator (diagnostic 1.1813 [-6.2871, 6.4924]) | Config did not meet 99%/90% removal gates |
+| A3 arrival dynamics | N/A; manipulation failed (diagnostic -0.2251 [-0.4625, -0.0403]) | N/A; unstable denominator (diagnostic -9.4647 [-17.2450, 17.4102]) | Queue-variance gate failed; Holm p=0.05536 |
+| A4 prefix structure | N/A; manipulation failed (diagnostic 0.0042 [-0.2340, 0.1948]) | N/A; unstable denominator (diagnostic 0.1760 [-6.2590, 6.4925]) | Zero-query/hit gate failed |
+
+Shares are neither clipped nor renormalized. P04's extreme raw ratios are the
+registered consequence of dividing by a near-zero, sign-unstable denominator,
+not evidence of large mechanisms.
+
+## Dominance verdicts
+
+| Mechanism | P03 rule | P04 rule | Dual-control verdict |
+|---|---|---|---|
+| A1 | Fails share/CI/Holm thresholds | Not evaluable | **NOT EVALUABLE** |
+| A2 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
+| A3 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
+| A4 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
+
+No mechanism may be called dominant because the frozen rule requires both
+control denominators. This is an inconclusive verdict, not evidence that every
+mechanism is small.
+
+## Residual + interaction line
+
+The official residual is N/A under both controls because the official share
+ledger is incomplete. The unchanged arithmetic over all raw diagnostic shares
+is reported only for transparency:
+
+| Control | Diagnostic share sum | Diagnostic residual + interaction (95% CI) | Status |
+|---|---:|---:|---|
+| P03 | -0.1549 | 1.1549 [0.5536, 1.9397] | Diagnostic only |
+| P04 | -6.5142 | 7.5142 [-21.3965, 22.8767] | Diagnostic only; unstable denominator |
+
+Nothing is forced to 100%. The residual can include interactions, unablated
+MoE routing, chunked-prefill effects, double-counting, and measurement error.
+
+## Config-tier A2 measured delta
+
+A2 added the P3-derived sizes `{3,5,6,7}` to the full default capture list.
+Its measured `delta_E` is **+0.048376**, 95% CI
+`[-0.315361, 0.411446]`, or **+1.605%** relative to base. Pure-decode padding
+falls from 13.7405% to 2.3989%, an 82.541% reduction. New recorded-arrival
+runs expose decode-B support 1-13, so uncaptured 9-13 leave coverage at 98.879%;
+both this and the sub-90% padding reduction invalidate the causal A2 share.
+
+The config cost is reproducible: base graph capture is 11 seconds and 0.80 GiB
+in all three runs; A2 is 12-14 seconds and 0.84 GiB, a 1-3 second and 0.04 GiB
+increase. The measured config-tier result is therefore a small, statistically
+uncertain efficiency gain with a failed preregistered removal gate.
+
+## GPU total
+
+New Phase-5 spend is **2.4388167443 H20-hours** of the 6.0-hour cap, including
+the superseded first wave, exact reruns, continued primary matrix, burn-ins,
+and background arm. **3.5611832557 H20-hours remain unspent.** Final state on
+all eight H20s is 0 MiB and 0% utilization with zero compute processes.
+
+## Sanity block
+
+**Anomaly first:** P04's gap CI contains zero and 45.353% of its denominator
+draws are nonpositive; the dual-control ledger is inconclusive and inferential
+claims stop there.
+
+| Numeric family | n | Finite | Missing | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---:|---:|---|
+| Primary run `E_token` | 15 | 15 | 0 | 2.623534 | 3.083263 | 15 | Positive; per-arm results not all identical |
+| Clean Layer-1 steps | 15 | 15 | 0 | 11,178 | 18,400 | 15 | Non-negative; continuous/balanced per run |
+| Clean duration (s) | 15 | 15 | 0 | 240 | 240 | 1 expected | Exact `3*80`; zero clean failures |
+| Offered rate (req/s) | 15 | 15 | 0 | 0.470833 | 0.487500 | 2 | Every run within 5% of 0.4725 |
+| Warm-up completions | 15 | 15 | 0 | 25 | 28 | 2 | Every run >=16 |
+| Warm-up long completions | 15 | 15 | 0 | 12 | 16 | 3 | Every run >=1 with input >=8192 |
+| Clean first-capture events | 15 | 15 | 0 | 0 | 0 | 1 expected | Zero events and zero uncovered descriptors |
+| P10 drain duration (s) | 15 | 15 | 0 | 0.676181 | 6.003959 | 15 | Non-negative; all below 600 |
+| Control gap `E_token` | 2 | 2 | 0 | 0.040952 | 1.721848 | 2 | P03 stable; **P04 unstable red flag** |
+| Raw diagnostic shares | 8 | 8 | 0 | -9.464746 | 1.593268 | 8 | Finite; intentionally not constrained to [0,1] |
+| Diagnostic residuals | 2 | 2 | 0 | 1.154931 | 7.514211 | 2 | Finite; never forced to close |
+| New H20-hours | 1 | 1 | 0 | 2.438817 | 2.438817 | 1 | Non-negative; below 6.0 |
+
+Checked invariants: source/helper/protocol provenance; identical request IDs,
+content/token totals, and prompt-preserving manifests; C00-TP1 four-way
+placement; detached controller ownership; exact clean windows; zero failures;
+offered-rate tolerance; A-P5-1 compile/capture ordering and coverage; Layer-1
+footer/sidecar balance, continuous step indices, token composition, and zero
+drops; non-negative counters; drain gates; bridge decision; no public prompt or
+generated-text leakage; public-disk/free-space limits; GPU hard-cap compliance;
+and zero-process/zero-memory cleanup. Manipulation failures are A2, A3, and A4;
+the sole machine sanity red flag is P04 denominator instability.

docs/opprof/phase6-protocol.md

@@ -0,0 +1,638 @@
+# OpProf Phase 6 pre-registered cross-version churn protocol
+
+Status: **ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED**.
+
+Date frozen: 2026-07-12 (Asia/Singapore). Phase 6 asks whether the vLLM 0.20.0
+best configuration and ranking on the C1
+`TP={1,2,4} x max-num-seqs={8,16,32,64}` surface survive an upgrade to patched
+vLLM 0.24.0. This is a paired historical-anchor experiment, not a fresh tuner
+search. It measures churn at the exact recorded C1 request sets and reports
+where the recorded local frontier moved.
+
+## Approved dispositions (orchestrator; 2026-07-12)
+
+All five open decisions are approved exactly as proposed, without changing an
+anchor, threshold, estimator, validity gate, or budget:
+
+1. The upgrade-path estimand is approved. Resolved default changes are part of
+   observed churn; dash1->dash0 and co-location remain explicit limitations and
+   the result is not described as a pure engine-version causal effect.
+2. The adaptive 25-anchor subset is approved: every cell receives its old peak
+   plus the direction-relevant adjacent anchor, and TP4/MNS16 receives both
+   neighbors.
+3. The 5% material-frontier threshold, `tau-b>=0.8` ranking-survival threshold,
+   and zero-anchored floor-bucket argmax/trap rules are approved.
+4. One primary observation per historical anchor plus the 0.35-H20-hour
+   confirmation reserve is approved.
+5. The A-P5-1-class warm-up long tier is approved as raw input `>4096` tokens
+   for this 0-8192 workload.
+
+This approval authorizes the later execution only after pinned-input, detached
+ownership, cold-start, echo-before-launch, placement, privacy, cleanup, and
+projected-budget gates pass. The 3.0-H20-hour cap remains absolute.
+
+## Amendment A-P6-1 — checkpoint-sidecar accounting and conditional cap
+
+The first Phase-6 W1 request replays were incorrectly rejected by requiring a
+clean in-stream footer and `final=true` sidecar after a non-graceful shutdown.
+That contradicts the already accepted A-P3-5/P5 accounting rule. Before any
+resume, all four W1 cell streams and their eight measured anchor intervals must
+be CPU-re-adjudicated against the latest atomic checkpoint sidecar. With no
+in-stream footer, the sidecar is authoritative when every complete JSONL line
+decodes, step indices are contiguous, on-disk data-record count equals
+`written_records`, final data-line step equals `last_step_index`,
+`encoded_records = written_records + dropped_records`, `dropped_records=0`, and
+the checkpoint is within the configured one-second flush interval (plus the
+accepted 100-ms scheduling tolerance) of the recorded shutdown boundary. At
+most one post-checkpoint flush interval may be absent. A balanced stream accepts
+all covered W1 replay intervals without a GPU rerun.
+
+All subsequent waves use the graceful P5 path: start `vllm serve` with positive
+`--shutdown-timeout`, signal each API parent with SIGINT, wait up to 150 seconds
+for EngineCore drain/finalization, and use process-group SIGTERM/SIGKILL only as
+fallback. The preferred result is one in-stream footer plus an agreeing
+`final=true` sidecar; the atomic checkpoint rule remains the crash fallback.
+
+Only if any W1 sidecar fails the CPU balance/coverage gate does A-P6-1c activate:
+the hard cap becomes **3.5 H20-hours**, the activation is logged before launch,
+and W1 is rerun under graceful shutdown. If all W1 sidecars balance, A-P6-1c is
+not activated, W1 is accepted in place, its already charged 0.299393 H20-hours
+is retained, and execution resumes at W2 under the original 3.0-hour cap.
+
+## Amendment A-P6-2 — authoritative solo frontier tier and 6.0-hour cap
+
+The user raises the cumulative Phase-6 hard cap from **3.0 to 6.0 H20-hours**.
+This amendment responds to the W2/W3 confirmation evidence: when neighboring
+wave clients became idle, pass rate changed from `0.412 -> 1.000` and `0.028 ->
+0.957` at TP2/MNS32 and from `0.036 -> 1.000` at TP4/MNS16, while Layer-1
+waiting means collapsed from `1.22 -> 0.00`, `8.50 -> 0.35`, and `28.33 ->
+0.06`. Because the v0.20 C1 baseline ran one cell at a time on dash1,
+co-located SLO feasibility is not a valid authoritative tier for the paired
+frontier. Co-location remains an explicitly reported, metric-dependent
+methodological ablation; its throughput-only observations are indicative.
+
+Every SLO-frontier-decisive Phase-6 anchor is therefore run **solo**: exactly
+one vLLM server and one replay client are active on dash0, no other Phase-6
+server/client is resident, and only the cell's `TP` GPUs are allocated. Each
+solo cell gets a fresh server process, the frozen exact-selection client, an
+independent 16-request long-tier warm-up, A-P5-1 cold-start gates, Layer-1
+telemetry, and graceful A-P6-1 shutdown. The solo result is authoritative for
+feasibility, frontier, floor bucket, argmax, tau-b, and trap decisions. A
+co-located-only result may not fill a missing solo frontier or support a paper
+claim.
+
+The mandatory solo set has no cell exclusions:
+
+| Cell set | Mandatory solo anchors | Inclusion reason |
+|---|---|---|
+| TP1/MNS8 | existing L+P | co-located peak failed |
+| TP1/MNS16,32,64 | existing P+U | co-located U passed and censored the frontier |
+| TP2/MNS8,16 | existing L+P | co-located peak failed; L distinguishes bracket vs left censoring |
+| TP2/MNS32 | existing L+P | old argmax; both anchors split primary/confirmation |
+| TP2/MNS64 | existing L+P | the indicative `>29.4% down` bound is unquotable until solo-confirmed |
+| TP4/MNS8 | existing L+P | co-located peak failed; trap-neighbor comparison |
+| TP4/MNS16 | existing L+P+U | named trap; peak split and L failed co-located |
+| TP4/MNS32,64 | P plus direction-relevant L/U | W4 completion; unmeasured under the 3.0-hour cap |
+
+After these 25 anchors, the controller may crawl only the pinned 92-anchor
+history: upward while the highest solo anchor passes or downward while the
+lowest solo anchor fails, stopping at the first opposite-feasibility anchor.
+There is no interpolation and no unrecorded request set. A pass rate in
+`[0.93,0.97]`, or a solo/co-located disagreement that changes argmax/trap or a
+material-frontier claim, triggers a same-solo-placement confirmation; a split
+gets a third trial for the frozen 2-of-3 rule if projected budget permits.
+Unresolved or history-edge frontiers remain explicitly censored.
+
+Priority is W4, TP2/MNS32, TP2/MNS64, TP4/MNS16, then the remaining failed or
+censored cells, so a budget stop preserves the most decision-relevant evidence.
+All twelve cells are nevertheless included. The planning envelope is 3.44 new
+H20-hours for serialized startup, warm-up, 25 mandatory anchors, bounded crawl,
+graceful drain, and a 0.20-hour safety reserve. Added to the already charged
+2.291173 hours, the launch projection is **5.731173/6.0 H20-hours**. Before
+each one-cell wave, the controller echoes cell, anchors, GPU placement, input
+paths, charged spend, remaining projection, and cap. Projected or actual spend
+reaching 6.0 stops further GPU work.
+
+Final analysis preserves the complete W1-W3 co-located attempt history and
+reports a solo-versus-co-located delta table for every exact anchor pair.
+ARGMAX, RANKING, and TRAP use solo values only. RANKING remains inconclusive
+and tau-b non-evaluable unless all 12 solo frontiers are bounded under the
+pinned history.
+
+## Headline claim and falsifiable outcomes
+
+The paper-facing question is:
+
+> Does the C1 vLLM 0.20.0 per-GPU SLO-feasible ranking, especially global best
+> TP2/MNS32 and local trap TP4/MNS16, remain valid on vLLM 0.24.0 when the exact
+> historical request-selection and SLO semantics are replayed?
+
+Phase 6 can produce four distinct outcomes:
+
+1. **SURVIVES:** TP2/MNS32 remains in the top floor bucket, the full-surface
+   Kendall tau-b is at least 0.8, and the TP4/MNS16 trap persists.
+2. **ARGMAX MOVED:** TP2/MNS32 is not in the top v0.24 measured-frontier bucket.
+3. **RANKING MOVED WITHOUT ARGMAX MOVEMENT:** the old best survives but tau-b is
+   below 0.8 or the trap changes status.
+4. **INCONCLUSIVE/PARTIAL:** a decision-critical frontier is unbracketed, an
+   anchor is not reproducible, repeated boundary verdicts disagree, the 12-cell
+   surface is incomplete, or the 3.0-H20-hour cap stops required work.
+
+The experiment quantifies observed upgrade churn. Because engine version,
+host, resolved defaults, and co-location differ together, it does not identify
+a pure causal effect of one vLLM commit.
+
+## Frozen vLLM 0.20.0 surface
+
+The historical objective is the maximum measured SLO-feasible offered rate
+divided by TP:
+
+```text
+f20(c) = max feasible selected_request_count / 60 seconds / TP
+```
+
+| TP | MNS8 | MNS16 | MNS32 | MNS64 |
+|---:|---:|---:|---:|---:|
+| 1 | 2.1000 | 2.3500 | 2.2833 | 2.2833 |
+| 2 | 2.2750 | 2.2750 | **3.2833** | 3.2583 |
+| 4 | 1.2833 | **2.4417** | 2.4417 | 2.4417 |
+
+Values are request/s/GPU. The global best is TP2/MNS32. TP4/MNS16 is the
+registered local trap: it is the first point on a TP4 plateau, has no improving
+adjacent MNS move, but is below the global best.
+
+The pinned ground-truth asset is
+`/home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json`, SHA-256
+`23ff3e6f6b88df8632bf37d37c0ff87bfef9d1676db81592948c08e898f7a670`.
+It contains 92 probe observations: eight anchors for each TP1/TP2 cell and seven
+for each TP4 cell.
+
+## Comparability contract
+
+### Exact workload and selection semantics
+
+The private materialized workload is
+`trace_windows/traces/chat_w20260311_1000.jsonl`, 32,606 rows, SHA-256
+`f539f38eb0ee0f750e3c23ff47df6eed3faf723a25f1444d55665a85871750b9`.
+Its window record is `trace_windows/windows.json`, SHA-256
+`23c432e7439508b07991cabfe1db9977ebb55f8ddc39ca214a627fc5f5ae4725`:
+window `[0,600]` seconds becomes `[0,60]` under replay scale 0.1.
+
+The later helper must call or byte-for-byte reproduce the following pinned
+AITuner implementations, rather than use the Phase-3/5 client:
+
+| Source | SHA-256 | Frozen role |
+|---|---|---|
+| `src/aituner/trace.py` | `6bb19f0e3eb42b1aecc6969a5f20d65eb505a91564dd117dd149155a6114eb35` | load, stable time sort, cap/downsample, threshold selection |
+| `src/aituner/worker.py` | `18337fe273a48d1cbb2bc364f774d100f8ffa2be5f9cd6a7c58c8ed4717c5e48` | rate-following submit loop, early stop, denominator accounting |
+| `src/aituner/slo.py` | `1d2e5b1ed9d0d6d6c07f8b5b6fa56d36ed9bb62710be54c125e9b2890f4f96d5` | stepped SLO evaluation |
+| `src/aituner/http_client.py` | `213b80d5b37cc74333517f46ab4e8cdba1a274bdbfabdfc14207b314f61799ac` | urllib/SSE TTFT, TPOT, usage accounting |
+
+The exact order is normative:
+
+1. Read all JSONL rows in file order. Retain raw `input_length` in `[0,8192]`.
+2. Build the same chat body and stable-sort by scaled arrival time only. Stable
+   ties retain source order; sorting by `(timestamp,sampling_u,...)` is forbidden.
+3. For TP1/TP2, even-downsample the full filtered, time-sorted list to 512
+   entries **before** threshold selection, using index
+   `floor(i*N/512)` for `i=0..511`. TP4 is uncapped.
+4. Select exactly the requests with `sampling_u <= anchor`. Preserve their
+   selected service order. Do not resample, wrap, shuffle, or interpolate.
+5. Submit at each row's materialized `timestamp*0.1`; do not uniformize,
+   rebase individual gaps, or replace the recorded burst process.
+6. Send the original messages through `/v1/chat/completions` with `stream=true`,
+   `stream_options.include_usage=true`, and the row's temperature when present.
+   Set both `min_tokens=128` and `max_tokens=128`.
+7. Use maximum concurrency 64. Client request timeout remains 900 seconds.
+8. Verify `usage.completion_tokens==128`; missing usage, a different token count,
+   HTTP failure, timeout, and every request not submitted because the SLO verdict
+   became unrecoverable are failures in the original selected-request denominator.
+9. Preserve original early-stop logic: after more than
+   `N-ceil(0.95*N)` failed evaluations, synthesize failures for the remaining
+   selected requests. Disable adaptive Stop-A; it was absent in C1.
+
+An offline, prompt-free preflight must regenerate all 92 historical request
+counts exactly. The local reconstruction already yields 17,710 filtered rows,
+512 capped rows, and **92/92 count matches**. Any future mismatch in count,
+request-ID hash, arrival hash, raw-length hash, or requested-token sum stops the
+GPU launch.
+
+### Exact SLO and score
+
+Request feasibility uses the original raw trace input length, not tokenizer
+usage after chat templating:
+
+```text
+raw input <= 4096:  TTFT <= 2000 ms
+raw input <= 32768: TTFT <= 4000 ms
+otherwise:          TTFT <= 6000 ms
+all requests:       TPOT <= 50 ms
+anchor feasible:    SLO-passing selected requests / all selected requests >= 0.95
+```
+
+Although the C1 filter limits inputs to 8192, the complete three-step rule is
+retained. TTFT is client wall time from request start to first nonempty content.
+TPOT is `(last_content_time-first_content_time)/(usage_completion_tokens-1)`.
+The paired surface score is always selected count divided by 60 seconds and TP;
+completed throughput is a diagnostic and never replaces the historical score.
+
+### Serving configuration
+
+The v0.24 command repeats the explicit C1 launch surface:
+
+```bash
+MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
+VENV=/tmp/wjh-opprof-phase2-dash0-20260711/.venv
+RUN_ROOT=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6
+
+taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU_IDS" \
+  VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
+  "$VENV/bin/vllm" serve "$MODEL" \
+  --host 127.0.0.1 --port "$PORT" \
+  --served-model-name qwen3-30b-a3b-community \
+  --max-num-batched-tokens 8192 \
+  --max-num-seqs "$MNS" \
+  --tensor-parallel-size "$TP" \
+  --shutdown-timeout 120
+```
+
+`--shutdown-timeout` is operational and outside the measured interval. As in
+C1, do **not** explicitly set block size, GPU-memory utilization, prefix
+caching, chunked prefill, CUDA-graph capture sizes, DP, or EP. Their resolved
+v0.24 values must be parsed from startup logs and reported. Allowing version
+defaults to change is intentional: this experiment asks whether an operator's
+old explicit configuration remains good after upgrading, not whether internal
+defaults can be manually made identical.
+
+Preflight records `vllm.__version__`, source commit, patch checksums, Python,
+PyTorch, CUDA, NCCL, Triton, driver, model/tokenizer file hashes, MoE backend,
+resolved scheduler/cache/graph config, compile-cache key, and exact command.
+The expected source is the existing Phase-3/5 patched vLLM 0.24 environment;
+any missing OpProf patch or unexpected model path is a stop condition.
+
+### Client equivalence, not a new client covariate
+
+The later `scripts/opprof_phase6_client.py` is permitted only as a thin anchor
+driver around the four pinned AITuner modules. Its interface is normative:
+
+```bash
+P6C='python scripts/opprof_phase6_client.py'
+
+$P6C preflight \
+  --trace trace_windows/traces/chat_w20260311_1000.jsonl \
+  --windows trace_windows/windows.json --window-id chat_w20260311_1000 \
+  --ground-truth /home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json
+
+$P6C run-anchor --cell "$CELL" --anchor "$U" \
+  --base-url "http://127.0.0.1:$PORT" \
+  --model qwen3-30b-a3b-community --completion-tokens 128 \
+  --replay-time-scale 0.1 --max-concurrency 64 \
+  --max-requests "$CAP_OR_NONE" --target-pass-rate 0.95 \
+  --ttft-step-ms '4096:2000,32768:4000,inf:6000' --tpot-ms 50 \
+  --request-timeout-seconds 900 --probe-deadline-ceiling-seconds 900 \
+  --result-dir "$ANCHOR_DIR"
+```
+
+Before GPU execution, a no-server golden test must prove identical selected IDs,
+order, arrivals, raw input lengths, serialized bodies excluding model/URL, and
+SLO decisions for synthetic outcomes. The wrapper may add interval markers and
+sanity JSON, but may not change HTTP transport, scheduling, completion checks,
+early stop, or latency formulas.
+
+### Covariate ledger and estimand limit
+
+| Covariate | C1 v0.20 | Phase 6 v0.24 | Treatment/handling |
+|---|---|---|---|
+| Engine | community vLLM 0.20.0 | patched vLLM 0.24.0 | Primary intended churn dimension; exact builds recorded |
+| Host | dash1 | dash0 | Both 8x H20 96 GB class; CPU/NUMA/clocks/driver/kernel recorded, not assumed identical |
+| GPU visibility | TP1 C1 env exposed GPUs `0,1`, TP1 consumed one | expose exactly TP GPUs | Declared operational covariate; H20-hour charge follows TP |
+| Trial placement | recovered C1 harness trials | up to four co-located servers | CPU/NUMA pinned; reject outside GPU processes and report co-location limitation |
+| HTTP client | pinned AITuner urllib/SSE | same code paths through thin wrapper | Eliminated as a semantic covariate by golden tests |
+| Workload | materialized C1 chat window | byte-identical file/hash | Fixed |
+| Selection/cap | cap-before-threshold; TP4 uncapped | exact same | Fixed and 92-count preflighted |
+| SLO length | raw input length | raw input length | Fixed; server usage length is diagnostic only |
+| Explicit flags | TP, MNS, MBT8192 | same | Fixed |
+| Unspecified defaults | v0.20 defaults | v0.24 defaults | Intentionally allowed to churn; resolved values reported |
+| Telemetry | no Layer 1 | zero-overhead OpProf Layer 1 | Added measurement; overhead evidence -0.04196%, CI [-0.17443%, 0.04550%] |
+
+Therefore “version churn” means the observed paired upgrade path under these
+documented platform covariates. It must not be rewritten as a same-host causal
+microbenchmark.
+
+## Budget-constrained anchor subset
+
+### Adaptive adjacent-anchor design
+
+The primary subset improves on measuring next-infeasible anchors only for a
+chosen top four:
+
+1. Measure the recorded v0.20 peak anchor for all 12 cells.
+2. For every non-trap cell, select exactly one adjacent recorded anchor by a
+   result-independent rule fixed now:
+   - if the v0.20 peak remains feasible on v0.24, measure the nearest higher
+     recorded anchor, which was infeasible on v0.20;
+   - if the peak becomes infeasible, measure the nearest lower recorded anchor,
+     which was feasible on v0.20.
+3. For TP4/MNS16, measure lower, peak, and higher anchors unconditionally. This
+   directly brackets the registered trap.
+
+Thus exactly 25 primary anchors are measured: TP1 eight, TP2 eight, and TP4
+nine. The design covers every cell and spends the second anchor in the direction
+where its frontier could have moved. It also includes the suggested top-four
+test. Under the floor-bucket rule, the nominal top-four cutoff expands to five
+cells because TP4/MNS16/32/64 share one bucket; all five receive an upward test
+whenever their old peak still passes.
+
+If an adjacent anchor does not bracket the transition—higher also passes or
+lower also fails—the cell is censored. Remaining budget may crawl one recorded
+anchor farther outward, in this priority order: old global best TP2/MNS32,
+TP2/MNS64, the TP4 plateau MNS16/32/64, then other cells by old score. Crawling
+uses only that cell's recorded history and stops at the first transition. No
+new `sampling_u` value may be invented. Failure to close a decision-critical
+bracket before the budget stop makes the applicable argmax/ranking verdict
+inconclusive.
+
+### Boundary confirmations
+
+The original surface has one observation per anchor. Phase 6 keeps that paired
+unit but reserves at most 0.35 H20-hours for decision-triggered confirmation.
+Repeat an anchor if its v0.24 pass rate lies in `[0.93,0.97]`, or if its single
+verdict alone changes argmax or trap status. Priority is argmax, trap, material
+frontier calls, then other cells. Agreement across two runs freezes the verdict;
+disagreement requests a third run if budget permits and uses 2-of-3. Without a
+third run, that anchor and every dependent headline verdict are inconclusive.
+Request outcomes are arrival-correlated, so a binomial CI is descriptive only
+and cannot replace these repeat rules.
+
+## Execution plan
+
+### Preflight and detached ownership
+
+The Phase-6 controller follows A-P3-2 detached ownership: one controller PID,
+per-stage atomic state, process-group/environment markers, resumable completed
+anchors, exact command logs, and cleanup restricted to its own markers. Before
+launch it must verify dash0 has eight H20s, no outside compute processes on
+assigned GPUs, at least 100 GB CPFS free, pinned source/workload/helper hashes,
+all 92 selection counts, legal TP placement, and a projected total below 3.0
+H20-hours.
+
+The autonomous launch log must echo one resolved line before starting:
+
+```text
+LAUNCH_ECHO host=dash0 engine=vllm-0.24.0-patched model=Qwen3-30B-A3B trace_sha=f539f38e... cells=12 primary_anchors=25 waves=4 gpus=0-7 warmup=16 clean_horizon=60s drain=derived<=120s est_wall=25-35min est_gpu=2.65_H20h confirm_reserve=0.35_H20h hard_cap=3.0_H20h
+```
+
+### Rate-following cold-start and drain gates
+
+The throughput-drift stationarity gate is forbidden for these rate-following
+anchors. A-P5-1-class cold-start gates are applied instead:
+
+1. Every torch.compile/CUDA-graph first-occurrence event must precede the first
+   measured anchor, verified from server logs and Layer 1.
+2. A deterministic unmeasured warm set must complete at least 16 exact-output
+   requests, including at least one request with raw input `>4096` tokens. The
+   peak-selected sets contain 43-184 such requests, so this gate is feasible.
+3. No first-occurrence compile/capture event may occur in any measured anchor;
+   all graph-hit `(runtime_mode,bucket_tokens)` descriptors must be covered by
+   startup capture logs. A violation invalidates that anchor only.
+4. Before the next anchor, all selected requests must be accounted for and three
+   consecutive controller samples must show running/waiting/deferred queues at
+   zero. Prefix cache is not explicitly enabled; nevertheless a full drain is
+   required.
+
+The probe deadline reproduces `_probe_drain_deadline`: last selected arrival +
+raw-length TTFT budget + `128*50 ms` + 30 seconds, capped by the historical
+900-second ceiling. With raw inputs <=8192 this is about 100.4 seconds from
+probe start; the controller has a 120-second class watchdog for cleanup. A
+deadline miss is an SLO-infeasible anchor, not permission to omit denominator
+requests. The server then shuts down through the official 120-second path.
+
+### Four-wave packing
+
+| Wave | Cells | Placement | Per-server measured anchors | Planned wall |
+|---|---|---|---:|---:|
+| W1 | TP1/MNS8,16,32,64 | four servers on GPUs 0,1,2,3; GPUs 4-7 idle | 2 each | 4-6 min |
+| W2 | TP2/MNS8,16,32,64 | four servers on GPU pairs 0-1,2-3,4-5,6-7 | 2 each | 4-6 min |
+| W3 | TP4/MNS8 and TP4/MNS16 trap | GPU quads 0-3 and 4-7 | 2 and 3 | 5-7 min |
+| W4 | TP4/MNS32 and TP4/MNS64 | GPU quads 0-3 and 4-7 | 2 each | 4-6 min |
+
+Each server starts once, passes warm-up once, runs its peak first, drains, then
+runs the predeclared adjacent anchor. Trap lower/upper order after peak is
+counterbalanced by execution seed 20260717. Anchor confirmation occurs before
+that server is released when possible. The controller records GPU clocks,
+power, utilization, memory, CPU load, NUMA placement, and outside processes
+through every wave. Any contamination invalidates the whole co-located wave.
+
+Actual H20-hours are charged from server launch through verified cleanup for
+every allocated GPU, including idle time while a paired TP4 cell finishes.
+Before each wave or confirmation, the controller recomputes
+`spent + current + conservative_remaining`; it stops before launch if the
+projection can reach 3.0. Optional crawling and confirmations are sacrificed
+before any hard-cap violation.
+
+## Measurement and mechanism notes
+
+For every anchor, report selected/admitted/completed/exact-output/SLO-pass
+counts; early-stop reason; offered and completed req/s; TTFT/TPOT mean and
+p50/p90/p95/p99; failures by raw-length TTFT bucket; elapsed/drain time; and
+peak GPU memory/utilization. The surface objective remains offered req/s/GPU at
+the highest measured feasible recorded anchor.
+
+Layer 1 is active over the complete anchor service interval, from the first
+submission through the last selected-request accounting event. Aggregate:
+
+- prefill, decode, and mixed-step counts and token shares;
+- decode-batch distribution and 5-second CV;
+- FULL/PIECEWISE/NONE graph-mode shares, bucket padding, and uncovered support;
+- running/waiting/deferred queues and queue CV;
+- preemption count/rate;
+- KV blocks used/total, mean/max usage, and allocation pressure; and
+- prefix queries/hits as a resolved-default diagnostic.
+
+Mechanism notes compare v0.24 cells/anchors and may relate a frontier change to
+preemptions, KV pressure, batching, or graph modes. C1 lacks Layer-1 telemetry,
+so these notes cannot claim that a measured composition field itself changed
+from v0.20 or caused the version drift. Historical v0.20 TTFT/TPOT failure
+reasons may be paired directly; device-mechanism attribution remains
+descriptive.
+
+## Statistical and ranking analysis
+
+### Floor buckets
+
+For each unrounded score vector `s`, use the SimFid inventory rule:
+
+```text
+tol(s) = max(1e-9, 1e-6 * max_c(abs(s(c))))
+bucket_s(c) = floor(s(c) / tol(s))
+```
+
+Two scores tie only when their integer buckets match. Bucketing occurs before
+display rounding, pair signs, tau-b, top selection, or trap evaluation. Report
+`tol`, bucket integers, and occupied intervals `[j*tol,(j+1)*tol)`. For v0.20,
+`tol20=3.283333333333333e-6`; the TP4 MNS16/32/64 tie makes nominal top-4 an
+effective top-5 set.
+
+### Per-cell frontier
+
+For each cell, define the v0.24 measured recorded-anchor frontier:
+
+```text
+f24(c) = max_{measured u with accepted feasible verdict} N_c(u) / 60 / TP
+drift(c) = (f24(c) - f20(c)) / f20(c)
+```
+
+The predeclared material threshold is **X=5% relative**, large enough to avoid
+calling one-request anchor quantization a paper-level churn effect. Report two
+orthogonal labels:
+
+- **BOUNDARY DOWN/UP:** old peak flips feasible->infeasible, or the old adjacent
+  infeasible anchor flips to feasible, respectively;
+- **MATERIAL FRONTIER MOVED:** `abs(drift)>5%`; otherwise report the boundary
+  flip as sub-material.
+
+If peak passes and higher fails, and `abs(drift)<=5%`, the local recorded
+frontier is **STABLE**. If peak and lower both fail or peak and every affordable
+higher anchor pass, report a directional bound and **UNBRACKETED**, never an
+invented peak.
+
+### Argmax, ranking, and trap
+
+- **ARGMAX MOVED** when TP2/MNS32 is not in the top `bucket_f24` among all 12
+  decision-bounded cells. **ARGMAX SURVIVED** requires it to share the top
+  bucket and no censored competitor capable of overtaking it. Otherwise it is
+  inconclusive.
+- Compute Kendall tau-b over all 12 floor-bucketed `f20/f24` scores and report
+  concordant, discordant, v0.20-tied, v0.24-tied, and comparable-pair counts.
+  **RANKING SURVIVES** requires an evaluable 12-cell surface, argmax survival,
+  `tau-b>=0.8`, and no >5% top-bucket pair reversal. `tau-b<0.8` is **RANKING
+  MOVED**. A partial surface receives no suite-level tau claim.
+- **TRAP PERSISTS** when TP4/MNS16 is in a bucket at least as high as both
+  adjacent TP4 cells MNS8 and MNS32, yet below the global-best bucket. It
+  **ESCAPES** if an adjacent TP4 move is strictly better, and **CEASES TO BE A
+  TRAP** if it joins the global-best bucket. Missing/unbracketed inputs make trap
+  status inconclusive.
+
+No arithmetic average of normalized scores is reported. Absolute rates,
+relative drift, the paired table, and every excluded/censored anchor accompany
+all ranking summaries.
+
+## Later deliverables and artifact contract
+
+Execution, after approval, must produce:
+
+- `docs/opprof/phase6-results.md`;
+- `runs/opprof-phase6/phase6/metrics.json` containing the complete paired
+  surface, anchor/run values, selection hashes, feasibility, floor buckets,
+  frontier/argmax/ranking/trap verdicts, confirmation history, Layer-1 notes,
+  covariates, and GPU accounting;
+- exact commands, controller state, environment/machine fingerprints, server
+  logs, client outcomes, Layer-1 JSONL/footer/sidecar, monitor samples, and
+  per-anchor sanity JSON; and
+- an operational-findings section covering startup/capture gates, drains,
+  early stops, co-location, default changes, retries, contamination, and budget.
+
+The results begin with data sanity and stop inferential claims on any red flag.
+Every numeric family ends with `n`, finite/missing, min, max, distinct count,
+and applicable invariants. Prompt text, messages, generated content, source
+substrings, and private request bodies stay in mode-0700 storage. Public files
+contain only hashes, IDs, lengths, timestamps, outcomes, counters, and summaries.
+
+## Anchor-subset table
+
+Notation is `u / selected N / recorded req/s/GPU`. `P` is always measured. For
+non-trap cells, measure `U` if P passes on v0.24, otherwise `L`. TP4/MNS16
+measures all `L+P+U`. Planning cost per anchor includes the 60-second replay and
+20 seconds expected drain/accounting; shared startup/warm-up is budgeted below.
+
+| Cell | L: nearest recorded feasible below P | P: recorded peak | U: nearest recorded infeasible above P | Primary selection | H20-hour cost each |
+|---|---|---|---|---|---:|
+| TP1/MNS8 | 0.21875 / 121 / 2.0167 | 0.2265625 / 126 / 2.1000 | 0.23046875 / 130 / 2.1667 | P + (U if pass else L) | 0.0222 |
+| TP1/MNS16 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | 0.25 / 143 / 2.3833 | P + (U if pass else L) | 0.0222 |
+| TP1/MNS32 | 0.234375 / 132 / 2.2000 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | P + (U if pass else L) | 0.0222 |
+| TP1/MNS64 | 0.234375 / 132 / 2.2000 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | P + (U if pass else L) | 0.0222 |
+| TP2/MNS8 | 0.4921875 / 269 / 2.2417 | 0.49609375 / 273 / 2.2750 | 0.5 / 276 / 2.3000 | P + (U if pass else L) | 0.0444 |
+| TP2/MNS16 | 0.4921875 / 269 / 2.2417 | 0.49609375 / 273 / 2.2750 | 0.5 / 276 / 2.3000 | P + (U if pass else L) | 0.0444 |
+| TP2/MNS32 | 0.75 / 391 / 3.2583 | 0.75390625 / 394 / **3.2833** | 0.7578125 / 396 / 3.3000 | P + (U if pass else L) | 0.0444 |
+| TP2/MNS64 | 0.5 / 276 / 2.3000 | 0.75 / 391 / 3.2583 | 0.75390625 / 394 / 3.2833 | P + (U if pass else L) | 0.0444 |
+| TP4/MNS8 | 0.016055910008 / 301 / 1.2542 | 0.016591107009 / 308 / 1.2833 | 0.017126304009 / 317 / 1.3208 | P + (U if pass else L) | 0.0889 |
+| TP4/MNS16 trap | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / **2.4417** | 0.034252608017 / 600 / 2.5000 | **L + P + U** | 0.0889 |
+| TP4/MNS32 | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / 2.4417 | 0.034252608017 / 600 / 2.5000 | P + (U if pass else L) | 0.0889 |
+| TP4/MNS64 | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / 2.4417 | 0.034252608017 / 600 / 2.5000 | P + (U if pass else L) | 0.0889 |
+
+## Total GPU estimate versus 3.0 cap
+
+The 25 primary anchors contain 8 TP1, 8 TP2, and 9 TP4 replays. At the
+80-second per-anchor planning charge they consume `0.1778 + 0.3556 + 0.8000 =
+1.3334` H20-hours. Four shared startup/warm-up/cleanup envelopes plus runtime
+variance are budgeted at 1.3166 hours, for a **2.65-H20-hour primary plan**.
+The remaining **0.35 H20-hours** is reserved for boundary confirmations or
+decision-critical outward crawling. The absolute launch cap is **3.0
+H20-hours**; projected or actual spend reaching the cap stops all further work.
+Expected wall time is 25-35 minutes over four waves. Expected public disk is
+0.3-0.6 GB and must remain below 3 GB.
+
+## Decision rules
+
+1. Anchor feasibility is exact-output request pass rate `>=0.95` using raw-length
+   stepped TTFT and 50-ms TPOT; absent/early-stopped requests remain failures.
+2. Boundary movement is an old peak feasible->infeasible flip or an old adjacent
+   infeasible->feasible flip. Material frontier churn is `abs(drift)>5%`.
+3. Apply zero-anchored floor buckets with set-wide `tol=max(1e-9,1e-6*max|s|)`
+   before every tie/rank decision.
+4. ARGMAX MOVED iff TP2/MNS32 is outside the v0.24 top bucket on a bounded
+   12-cell subset; censored contenders make survival inconclusive.
+5. RANKING SURVIVES only with all 12 cells, argmax survival, tau-b >=0.8, and no
+   >5% top-bucket pair reversal. Tau-b <0.8 is RANKING MOVED.
+6. TP4/MNS16 trap persists only when it is no worse than adjacent MNS8/MNS32 and
+   remains below the global best; an improving neighbor means escape.
+7. Boundary `[0.93,0.97]` or verdict-changing anchors require confirmation when
+   budget permits. Disagreement without a 2-of-3 resolution is inconclusive.
+
+## Open decisions for orchestrator
+
+1. Approve the **upgrade-path estimand**: resolved default changes are part of
+   churn, while dash1->dash0 and co-location remain explicit limitations rather
+   than pretending this is a pure engine-version causal effect.
+2. Approve the **adaptive 25-anchor subset**, which gives every cell a
+   direction-relevant neighbor and the trap both neighbors, instead of spending
+   next-infeasible anchors only on a tie-ambiguous top four.
+3. Approve **5% material frontier drift**, `tau-b>=0.8`, and the stated
+   argmax/trap floor-bucket rules.
+4. Approve one primary observation per historical anchor with the 0.35-hour
+   reserve for `[0.93,0.97]`/verdict-changing confirmations, rather than reducing
+   surface coverage to replicate every anchor.
+5. Approve the A-P5-1-class warm-up adaptation from `input>=8192` to
+   **raw input >4096**, the longest SLO-relevant tier available reliably in this
+   0-8192 C1 workload.
+
+## Sanity block
+
+| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---|
+| C1 cells | 12 | TP=1,MNS=8 | TP=4,MNS=64 | 12 | Exact 3x4 Cartesian surface |
+| Historical probe observations | 92 | 7/cell | 8/cell | 2 per-cell counts | `8*8 + 4*7 = 92` |
+| Materialized trace rows | 1 file | 32,606 | 32,606 | 1 expected | SHA pinned; prompt content not emitted |
+| Raw-length-filtered rows | 1 reconstruction | 17,710 | 17,710 | 1 expected | Raw length 22-8192; non-negative |
+| Capped TP1/TP2 source rows | 1 set | 512 | 512 | 1 expected | Stable time sort, then even downsample |
+| Reconstructed historical counts | 92 | 66 | 600 | 34 | 92/92 match ground truth |
+| C1 peak scores (req/s/GPU) | 12 | 1.283333 | 3.283333 | 8 | Finite, positive, not all identical |
+| v0.20 floor tolerance | 1 vector | 3.283333e-6 | 3.283333e-6 | 1 | Applied before display rounding |
+| Nominal top-4 effective size | 1 cutoff | 5 | 5 | 1 | TP4 three-way cutoff tie retained |
+| Primary peak anchors | 12 | 1/cell | 1/cell | 1 expected | No cell omitted |
+| Realized adjacent/trap anchors | 13 | 1/non-trap | 2/trap | 2 | Predeclared direction rule; total 25 |
+| Primary anchors by TP | 25 | 8 at TP1/TP2 | 9 at TP4 | 2 counts | `8+8+9=25` |
+| Selected requests in anchor candidates | 36 candidates | 121 | 600 | >1 expected | TP1/2 cap-before-threshold; TP4 uncapped |
+| Per-anchor planning cost (H20-hour) | 3 TP classes | 0.0222 | 0.0889 | 3 | Proportional to TP for 80 seconds |
+| Primary planned H20-hours | 1 plan | 2.65 | 2.65 | 1 | Non-negative; below 3.0 |
+| Confirmation/crawl reserve | 1 plan | 0.35 | 0.35 | 1 | Primary plus reserve equals hard cap |
+| GPU work in this protocol turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied |
+
+Checked invariants: `12=3*4`; `92=8*8+4*7`; all 92 local selection counts
+match; cap precedes threshold; arrivals span the fixed 60-second replay; raw
+input length, not retokenized usage, selects TTFT thresholds; completion is
+exactly 128; SLO denominator includes all selected requests; surface scores use
+60 seconds and divide by TP; floor buckets precede ranking; the top-four cutoff
+expands to five; `25=12+11+2`; planned `2.65+0.35=3.0`; ratios and counters have
+their declared domains; and no GPU run, helper, manifest, remote artifact, or
+results file was created in this protocol-only turn.

docs/opprof/phase6-results.md

@@ -0,0 +1,226 @@
+# OpProf Phase 6 dash0 results
+
+Status: **COMPLETE, DATA-VALID A-P6-2 SOLO TIER; ARGMAX/RANKING/TRAP INCONCLUSIVE UNDER THE FROZEN RULES**.
+
+Phase 6 completed all 12 C1 cells on patched vLLM
+`0.24.1.dev3+g668cfb7e2` (`4b253fd`). A-P6-2 makes the serialized solo tier
+authoritative because the v0.20 baseline was solo and the original W2/W3
+confirmations exposed large pass-rate and queue changes after neighboring wave
+clients became idle. The authoritative campaign contains 37 primary anchors,
+29 same-placement confirmations, and 12 valid Layer-1 streams.
+
+The machine-readable result is
+`runs/opprof-phase6/phase6/metrics.json`, SHA-256
+`290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16`.
+Co-located W1-W3 artifacts are preserved beside the solo tree rather than
+overwritten. Public artifacts contain no prompt or generated text.
+
+## Data sanity first
+
+There are **no data-sanity red flags**. All 37 authoritative primary anchors
+reproduce the pinned v0.20 request count exactly; all client invariants pass;
+all 12 solo cells have one graceful footer and agreeing final sidecar; Layer-1
+indices are contiguous with zero drops; every anchor interval is represented;
+and all cold-start compile/capture events precede measurement.
+
+The scientific decision gate is separate from data validity. Eight cells have
+bounded monotonic solo frontiers. TP2/MNS16, TP4/MNS32, and TP4/MNS64 pass the
+highest recorded request set and are right-censored at the pinned-history edge.
+TP4/MNS16 is non-monotonic: L resolves feasible, P resolves infeasible, and U
+resolves feasible. These four cells prohibit a full-surface floor-bucket rank,
+tau-b, and trap decision; no value is imputed beyond the recorded history.
+
+## Launch echo
+
+```text
+A-P6-2_LAUNCH_ECHO host=dash0 engine=vllm-0.24.1.dev3+g668cfb7e2@4b253fd authoritative=solo cells=12 mandatory_anchors=25 crawl=recorded-only confirmations=2of3 prior_h20h=2.291173 new_est_h20h=3.440 cumulative_est_h20h=5.731173 cap_h20h=6.0 est_wall=45-70min
+SOLO_WAVE_ECHO order=TP4/MNS32,TP4/MNS64,TP2/MNS32,TP2/MNS64,TP4/MNS16,TP2/MNS8,TP2/MNS16,TP4/MNS8,TP1/MNS8,TP1/MNS16,TP1/MNS32,TP1/MNS64 placement=one_server_one_client
+FINAL_ACCOUNTING status=complete cells=12 primary=37 confirmations=29 solo_h20h=3.353371 campaign_h20h=5.644544 cap_h20h=6.0
+```
+
+The exact per-cell echoes, GPU placement, trace and ground-truth paths, spend,
+and projection are retained in
+`runs/opprof-phase6/phase6/solo-authoritative/launch-echo.log`.
+
+## Run completion stats
+
+| Item | Result |
+|---|---:|
+| Historical selections reconstructed | 92/92 exact |
+| Authoritative solo cells | 12/12 |
+| Solo primary anchors | 37 |
+| Solo confirmations | 29 |
+| Solo anchor trials | 66 |
+| Solo warm-ups | 12; exactly 16 completions each |
+| Long prompts per warm-up | 2-8 raw inputs `>4096` |
+| Bounded solo frontiers | 8/12 |
+| History-edge right-censored cells | 3 |
+| Non-monotonic cells | 1 |
+| Prior co-located primary/confirm trials | 21 + 3 |
+| Total accepted anchor trials across attempts | 90 |
+| Co-located spend before A-P6-2 | 2.291173 H20-hours |
+| A-P6-2 solo spend | 3.353371 H20-hours |
+| Total campaign spend | **5.644544 H20-hours** |
+| Artifact footprint | 467 MiB, below 3 GiB |
+
+## Final paired surface
+
+`A*` marks the authoritative A-P6-2 solo tier. Exact values are bounded
+frontiers. `>=` is a recorded-history lower bound. The TP4/MNS16 value is not a
+frontier because its feasibility sequence is non-monotonic.
+
+| Cell | Tier | v0.20 frontier | v0.24 authoritative result | Drift | Status |
+|---|---|---:|---:|---:|---|
+| TP1/MNS8 | A* solo | 2.1000 | **2.3833** | **+13.49%** | bounded up |
+| TP1/MNS16 | A* solo | 2.3500 | 2.3833 | +1.42% | bounded up |
+| TP1/MNS32 | A* solo | 2.2833 | 2.3833 | +4.38% | bounded up |
+| TP1/MNS64 | A* solo | 2.2833 | 2.3833 | +4.38% | bounded up |
+| TP2/MNS8 | A* solo | 2.2750 | 2.2417 | -1.47% | bounded down |
+| TP2/MNS16 | A* solo | 2.2750 | >=2.3000 | >=+1.10% | right-censored history edge |
+| TP2/MNS32 | A* solo | **3.2833** | **3.2583** | -0.76% | bounded down |
+| TP2/MNS64 | A* solo | 3.2583 | **2.3000** | **-29.41%** | bounded down |
+| TP4/MNS8 | A* solo | 1.2833 | 1.3208 | +2.92% | bounded up |
+| TP4/MNS16 | A* solo | 2.4417 | N/E; observed max feasible 2.5000 | N/E | **non-monotonic: L pass, peak fail, U pass** |
+| TP4/MNS32 | A* solo | 2.4417 | >=2.5000 | >=+2.39% | right-censored history edge |
+| TP4/MNS64 | A* solo | 2.4417 | >=2.5000 | >=+2.39% | right-censored history edge |
+
+The solo tier validates two material frontier changes under the predeclared 5%
+threshold: TP1/MNS8 improves by 13.49%, while TP2/MNS64 declines by 29.41%.
+The previous TP2/MNS64 decline is now quotable because both its passing 2.3000
+anchor and failing 3.2583 anchor were repeated solo.
+
+## Frozen decision-rule verdicts
+
+- **ARGMAX: INCONCLUSIVE.** TP2/MNS32 is the highest bounded observed frontier
+  at 3.2583 req/s/GPU, but the frozen rule requires a bounded 12-cell subset.
+  TP2/MNS16 and TP4/MNS32/64 terminate at passing history-edge anchors, and
+  TP4/MNS16 has no coherent frontier. The top floor bucket is therefore not
+  defined for all contenders.
+- **RANKING: INCONCLUSIVE.** Only 8/12 frontiers are bounded. Per protocol,
+  Kendall tau-b is **not evaluable** (`tau_b=null`) rather than computed from
+  censored lower bounds. The `tau-b>=0.8` survival rule cannot be applied.
+- **TRAP: INCONCLUSIVE.** TP4/MNS16 is non-monotonic and its TP4/MNS32 neighbor
+  is right-censored. Neither trap persistence nor escape satisfies the frozen
+  floor-bucket rule.
+
+## Solo versus co-located exact-anchor deltas
+
+The co-located column is the simultaneous-wave primary. The solo column is the
+accepted median after same-placement 2-of-3 adjudication. Delta is percentage
+points. `P/F` are SLO feasible/infeasible at 0.95.
+
+| Cell | Anchor | Co-located primary | Solo authoritative | Delta | State change |
+|---|---:|---:|---:|---:|---|
+| TP1/MNS8 | 0.21875000 | 0.992 | 0.992 | +0.00pp | P->P |
+| TP1/MNS8 | 0.22656250 | 0.206 | 0.992 | **+78.57pp** | F->P |
+| TP1/MNS16 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP1/MNS16 | 0.25000000 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP1/MNS32 | 0.24218750 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP1/MNS32 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP1/MNS64 | 0.24218750 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP1/MNS64 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP2/MNS8 | 0.49218750 | 0.691 | 1.000 | +30.86pp | F->P |
+| TP2/MNS8 | 0.49609375 | 0.092 | 0.581 | +48.90pp | F->F |
+| TP2/MNS16 | 0.49218750 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP2/MNS16 | 0.49609375 | 0.088 | 1.000 | **+91.21pp** | F->P |
+| TP2/MNS32 | 0.75000000 | 0.412 | 1.000 | +58.82pp | F->P |
+| TP2/MNS32 | 0.75390625 | 0.028 | 0.575 | +54.70pp | F->F |
+| TP2/MNS64 | 0.50000000 | 0.460 | 1.000 | +53.99pp | F->P |
+| TP2/MNS64 | 0.75000000 | 0.141 | 0.561 | +42.07pp | F->F |
+| TP4/MNS8 | 0.016055910008 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP4/MNS8 | 0.016591107009 | 0.071 | 1.000 | **+92.86pp** | F->P |
+| TP4/MNS16 | 0.033182214016 | 0.609 | 1.000 | +39.13pp | F->P |
+| TP4/MNS16 | 0.033717411016 | 0.036 | 0.238 | +20.22pp | F->F |
+| TP4/MNS16 | 0.034252608017 | 0.847 | 1.000 | +15.33pp | F->P |
+
+The deltas range from 0 to +92.86pp across 21 exact pairs. Four TP1/MNS16-64
+pairs are unchanged, while several TP2/TP4 and TP1/MNS8 anchors flip. Thus
+co-location validity is **metric- and cell-dependent**, not a universal offset.
+Throughput-only values may co-locate, but SLO-frontier feasibility cannot be
+mixed across placement tiers.
+
+## Layer-1 mechanism notes for material drift
+
+Layer-1 is mechanism context, not a matched v0.20 causal decomposition. P3 P10
+is TP1 at a different workload/output contract. All 37 solo primaries have
+zero preemptions.
+
+| Cell/anchor | State | Waiting mean/max | Decode-B mean | KV mean | `NONE` graph share | Padding |
+|---|---|---:|---:|---:|---:|---:|
+| TP1/MNS8, 0.25 | frontier pass | 0.74 / 8 | 4.25 | 0.0407 | 3.26% | 0.34% |
+| TP1/MNS8, 0.50 | next fail | 11.66 / 27 | 7.19 | 0.0644 | 5.04% | 0.02% |
+| TP2/MNS64, 0.50 | frontier pass | 0.00 / 0 | 3.28 | 0.0084 | 0.00% | 18.88% |
+| TP2/MNS64, 0.75 | old-peak fail, primary | 0.00 / 0 | 18.05 | 0.0448 | 9.47% | 0.52% |
+| TP2/MNS64, 0.75 | old-peak fail, confirm | 0.0002 / 1 | 11.52 | 0.0285 | 4.57% | 1.13% |
+
+For TP1/MNS8, the new frontier remains feasible despite a larger decode batch
+and modest queueing; failure at 0.5 coincides with a 16x waiting-mean increase,
+higher KV usage, and more eager/non-full graph execution. For TP2/MNS64, the
+passing lower anchor has small decode batches and full/piecewise graphs despite
+high padding, whereas both old-peak trials fail with much larger decode batches
+and graph fallback. Zero preemptions and near-zero queueing rule out preemption
+and server waiting as the main TP2/MNS64 explanation, but the unmatched v0.20
+telemetry prevents assigning causality uniquely to graph mode or batch shape.
+
+## Operational findings and full attempt history
+
+1. The initial ownership-monitor false start charged 0.132918 H20-hours without
+   measurement. The repaired W1 charged 0.299393; A-P6-1 later accepted its
+   eight anchors from balanced checkpoint sidecars without rerun.
+2. Co-located W2/W3 completed 13 primary anchors and three confirmations but
+   exposed placement-sensitive SLO results. The 3.0-hour projection stopped W4
+   at 2.291173 total; those values remain indicative and preserved.
+3. A-P6-2 completed W4 and remeasured the full surface solo. Every solo server
+   used `--shutdown-timeout 120`, emitted a graceful footer, and passed Layer-1
+   accounting.
+4. Placement was not the only transient. TP4/MNS32 and TP4/MNS64 first full
+   replays failed at 0.031/0.114 but subsequent identical solo trials passed;
+   TP4/MNS16 produced an order-dependent non-monotonic sequence. The 16-request
+   A-P5-1 gate removes compile/capture cold start but not necessarily the first
+   full-trace transient. The predeclared 2-of-3 rule prevented single-trial
+   conclusions, but a future protocol should use a full-trace burn-in or
+   randomized/reversed anchor order.
+5. Two cleanup checks briefly observed 1-4 MiB at 0% with zero compute PIDs.
+   Memory decayed to 0 MiB within seconds, graceful footers validated, and no
+   rerun was needed. Cleanup now polls for 30 seconds rather than treating
+   transient driver bookkeeping as a failed experiment.
+
+## GPU total
+
+The complete campaign charged **5.6445441643 H20-hours** against the
+user-approved **6.0** cap:
+
+- original co-located attempts: 2.2911728495;
+- A-P6-2 authoritative solo tier: 3.3533713148;
+- remaining headroom: 0.3554558357 H20-hours.
+
+Final dash0 state is 0 MiB and 0% utilization on all eight H20s, with zero
+controller, client, vLLM, EngineCore, or worker processes.
+
+## Sanity block
+
+| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---|
+| Historical selection checks | 92 | 66 requests | 600 requests | 34 | 92/92 exact |
+| Solo primary pass rates | 37 | 0.0307 | 1.0000 | 17 | Ratios in `[0,1]`; not identical |
+| Solo selected requests | 37 | 121 | 600 | 18 | Exact v0.20 count per anchor |
+| Layer-1 steps per primary | 37 | 343 | 12,103 | 37 | Contiguous; non-negative; zero drops |
+| Layer-1 records per cell | 12 | 14,174 | 58,725 | 12 | Footer/sidecar balanced |
+| Solo preemptions | 37 | 0 | 0 | 1 expected | Zero throughout |
+| Reported v0.24 frontier values/bounds | 11 | 1.3208 | 3.2583 | 6 | TP4/MNS16 omitted as non-monotonic |
+| Frontier drift values/bounds | 12 | -29.41% | +13.49% | 9 | 11 finite; one non-monotonic missing |
+| Bounded/censored cells | 12 | 8 bounded | 4 unbounded | 2 | Censoring never imputed |
+| Exact solo/co-located deltas | 21 | +0.00pp | +92.86pp | 13 | Same request set and anchor |
+| Solo confirmation pass rates | 29 | 0.4130 | 1.0000 | 10 | Frozen 2-of-3 applied |
+| Solo cell H20-hours | 12 | 0.0845 | 0.5534 | 12 | Non-negative; serialized placement |
+| Total campaign H20-hours | 1 | 5.644544 | 5.644544 | 1 | Below 6.0 hard cap |
+| Final GPU memory/utilization | 8 | 0 MiB / 0% | 0 MiB / 0% | 1 expected | Zero compute processes |
+
+Checked invariants: exact cap-before-threshold selection; raw-length SLO
+evaluation; exact 128-token completions or counted failures; nondecreasing
+arrivals; all selected outcomes represented in the denominator; one solo
+server/client at a time; compile/capture outside measured intervals; 12/12
+graceful footer balances; non-negative counters; pass ratios in range; no prompt
+text in public artifacts; complete attempt-history retention; authoritative
+tier separation; explicit censoring/non-monotonicity; tau-b suppression when
+the full surface is unbounded; hard-cap compliance; and complete GPU cleanup.

docs/opprof_campaign_state.md

@@ -0,0 +1,80 @@
+# OpProf campaign state — pattern-conditioned operator profiling (vLLM 0.24.0 / Qwen3-30B-A3B / H20)
+
+Started 2026-07-11. Orchestrator: Claude (session a42d23fb). Workers: codex via companion.
+Discipline inherited from SimFid campaign (replayserve/docs/simfid_campaign_state.md):
+pre-registered acceptance gates per phase, codex implement + independent review,
+orchestrator verifies, echo-before-expensive-action, GPU actions logged.
+
+## User decisions (2026-07-11)
+- vLLM pinned to **0.24.0** (latest community); record commit + sha256 at clone time.
+- GPU: **dash0 8×H20, approved for the whole campaign** (Phases 2-4). Orchestrator
+  still echoes load/duration before each launch. Single H20 sufficient for TP1.
+- Observation patch: **dual-layer** — always-on lightweight per-iteration composition
+  telemetry (<3% overhead budget) + sampled heavy torch.profiler/CUPTI windows.
+- L-C-A is REFERENCE ONLY in this campaign — pattern axes defined operationally
+  (input-len dist / output len / arrival shape / prefix sharing), no L-C-A dependency.
+
+## Phase plan (pre-registered)
+- P0 recon (local): clone v0.24.0, inventory built-in observability, patch design doc.
+  Gate: USER reviews patch design before P1 implementation.
+- P1 patch dev + no-GPU tests (local). Gate: strict review + orchestrator verify.
+- P2 single-H20 smoke on dash0: artifacts complete; measured overhead (on/off same
+  load) within budget; op-time sums ≈ iteration time; composition matches request-level
+  ground truth. Hard gate before P3.
+- P3 pattern matrix (~8-12 patterns × 2-3 configs, short replays, dash0): per-pattern
+  operator time breakdown + composition distributions + waste accounting
+  (padding% / graph-miss / ragged imbalance / mixed-batch interference).
+  Hypothesis go/no-go: does operator bottleneck ranking change across patterns?
+- P4 optimization proposal (ranked, measured bounds) + one cheap config-level
+  closed-loop validation (e.g., cudagraph capture sizes vs measured B distribution).
+- Non-goals: no new kernels; no L-C-A ablation (separate track); no production clusters.
+
+## Artifact layout
+- vLLM source: /home/gahow/phd/vllm-v0.24.0 (clone, pinned)
+- Patches: aituner patches/vllm-0.24.0-opprof/ (patch files + apply script + no-GPU tests)
+- Docs: aituner docs/opprof/ (phase0-recon.md, patch-design.md, later phase reports)
+
+## Job log
+- P0 recon dispatched: task-mrg3nm1f-spyl85 (2026-07-11T08:25:42Z). Scope: clone v0.24.0 to ~/phd/vllm-v0.24.0, observability inventory with file:line, dual-layer patch design doc. Gate: user reviews design before P1.
+- P0 ACCEPTED (task-mrg3nm1f, 34m54s): pin ee0da84a=v0.24.0 verified; VLLM_TORCH_PROFILER_DIR-absent claim verified; profiler moved to ProfilerConfig + entrypoints/serve/profile/api_router.py (verified); design doc 421 lines. USER APPROVED design with 5 delegated decisions (JSONL+msgspec+8192 queue; expert loads L2-only; 2+8 profiler window; 3% gate on 95% CI upper bound; reject --disable-log-stats) + checkpoint decision: community BF16, TP1 primary, TP2/4 counterpoints.
+- P1 dispatching: implement per approved design; branch in ~/phd/vllm-v0.24.0 + exported patches/vllm-0.24.0-opprof/ in aituner; no-GPU tests must pass locally (opprof core import-light).
+- P1 completed (task-mrg588j2, 26m36s): branch opprof tip c60e7eeb, +560/-1 across 5 files (within ±30% of design). Orchestrator pre-checks: pytest 11/11 reproduced; patch series applies cleanly on pristine v0.24.0 worktree and applied tree is byte-identical to branch (diff=0); scheduler/model_runner integration diff reviewed — surgical, matches design hooks. Strict review dispatching.
+- Strict review (task-mrg68puk, 14m59s): FAIL — 0 Blocking / 3 Major / 1 Minor. M1 writer-thread silent death + shutdown deadlock (opprof.py:106/145); M2 apply.sh already-applied bypasses base/dirty checks; M3 foreground raw lists violate no-raw-list hotspot contract (opprof.py:182). Minor: missing zero-token+None-stat test; true Scheduler integration test DEFERRED to P2 smoke gate (no local vllm install). Reviewer verified OK: fail-fast log_stats ValueError, pairing across sync/batch-queue/PP/spec-decode, schedule-time snapshots, DP-unique paths, patch-id equality 3/3. Orchestrator adjudication: ACCEPT all; fix worker dispatching (fresh task — resume-last would hit reviewer thread).
+- P1 fix round CLOSED (task-mrg6tnum, 13m34s): branch tip 668cfb7e (3 clean commits on ee0da84a). Orchestrator closure: 14/14 tests reproduced; M3 raw-lists-gone verified (bisect accumulation); BOTH apply.sh refusal cases independently reproduced (dirty exit=1; wrong-base-with-identical-patches exit=1); M1 writer failure handling inspected (one-time log, liveness check, bounded close, drop visibility). P1 ACCEPTED.
+- ECHO P2 (GPU, dash0): deploy patched vLLM 0.24.0 (python-only patch, VLLM_USE_PRECOMPILED path) on dash0, serve Qwen3-30B-A3B BF16 TP1 on ONE free H20; artifacts smoke + overhead gate (interleaved 5xON/5xOFF identical load, 3% gate on 95% CI upper bound) + one 2+8 Layer-2 profiler window + integration assertions (records==steps, no pending leak, footer accounting). Est 1.5-2.5h wall, single GPU. Deferred review item (real Scheduler integration) is covered by these assertions.
+- P2 dispatched: task-mrg7erq5 (dash0 smoke; ssh dash0 ONLY authorized; hard gates: artifacts/schema/footer/no-pending-leak, overhead 95%CI-upper <=3%, Layer-2 window loadable). Est 1.5-2.5h.
+- P2 first run (task-mrg7erq5, 1h37m): overhead gate FAIL honored (13.11%, CI [6.24,19.53]) — but measurement judged INVALID by orchestrator: OFF-arm spread 29% across identical runs, one pair NEGATIVE overhead, 10s window with cold-start per run, ON-first order confound (run 1 = coldest = ON). Physical bound: per-step cost is bisect+encode+enqueue at ~11 steps/s. Artifacts gate passed except PIECEWISE not exercised by smoke load. 14 remote tests pass; TRITON backend 12/12; cleanup verified. AMENDMENT A-P2-1 (pre-registered before rerun): warmup excluded, >=120s steady-state window, ignore_eos fixed output lengths, ABBA counterbalanced order + discard first pair, host load recorded, recorder microbenchmark as secondary evidence, one PIECEWISE-exercising artifact run. GATE UNCHANGED (3% on 95% CI upper).
+- P2 A-P2-1 rerun (task-mrgdbfju + predecessor, valid measurement): overhead gate REAL FAIL — 4.1816%, CI [3.1364, 4.7117], gate 3%. ON tight (28.18-28.26), OFF (28.80-29.64), pairs 2.07-4.74%. Artifacts PASS incl. PIECEWISE addendum (129 records, accounting balanced). Layer-2 correctly not run. LOCALIZATION PUZZLE: producer microbenchmark 29.1µs/step x 9.7 steps/s ≈ 0.04s/run vs observed ON-OFF gap 5.9s — direct recording cost explains <1% of overhead; structural suspect (capture scan @200 concurrency, writer-thread GIL, queue locking, flush I/O locus). GPU spend so far 117 H20-min. Diagnostic+fix round dispatching (stage-bisection first, minimal fix, gate rerun).
+- Bisection round (task-mrge67ke, 34m9s): recorder pipeline exonerated — all 4 stages ~0.5% vs stage-off; BUT all stages kept VLLM_OPPROF_DIR set → gpu_model_runner CUDAGraphStat construction/propagation path active in all; cross-attempt (attempt-2 true-OFF 29.45 vs bisection off 28.39) localizes ~3.6% to that path. Worker honored outside-OpProf stop boundary; temp diffs reversed; 14/14 tests both sides. Orchestrator hypothesis for next round: CUDAGraphStat in ModelRunnerOutput poisons msgspec IPC fast path (pickle fallback for whole output) OR upstream cudagraph_metrics feature carries inherent cost. Next: STATIC analysis first, then 3-trial confirm (true-baseline / upstream-metrics-only / env-set-recorder-off), then targeted fix.
+- P2 CLOSED: ALL GATES PASS (task-mrgfgedp, 2h17m). ROOT CAUSE of the 4.18%: VLLM_OPPROF_DIR entered compile_factors() (envs.py:2011-2105) → hashed into torch.compile cache path (backends.py:1024-1065) → every unique ON dir = cold graph/AOT cache (compile 36.41s vs 6.07s) and ~4% slower artifacts. Recording itself was NEVER the cost (recorder 29.1µs/step; serializer-poisoning hypothesis WRONG — TP1 in-process executor, msgspec handles CUDAGraphStat natively). 3-trial confirmation: baseline 29.72 / upstream-metrics-only 29.64 (0.25%) / env-set 28.45 (4.25%). Fix: +1 production line (ignore-list) + tests; tip bbfa717; 5-patch series; 15/15 tests x3 environments. FINAL GATE: overhead -0.042%, CI [-0.174, +0.046] — PASS. Layer-2 PASS (16.4MB Kineto, 2+8 window, 790,843 events, 7,367 kernel events; active-window perturbation 51.3% — confirms sampled-only design). Artifacts PASS (PIECEWISE 129/139). Orchestrator verified: fix commits, ignore-list diff, 15/15 local rerun, 5 patches. Caveat noted: attempt-4 JSONLs footerless (simultaneous shutdown) — excluded from accounting. UPSTREAM FINDING worth reporting: any per-run-unique env var hashed into compile factors silently causes cold-compile + slower artifacts.
+- P3 dispatching: protocol-first (worker drafts docs/opprof/phase3-protocol.md, STOPS for orchestrator review before execution).
+- USER DIRECTIVE (2026-07-12): parallelize P3 across 8xH20, one experiment per GPU (~8x speedup). Orchestrator caveat: no GPU contention but shared host CPU/memory-bandwidth (API server/tokenizer/scheduler/bench are host-side; P2 showed host-side effects can fabricate %-level differences; saturation-point calibration depends on absolute throughput). P3 protocol must include: pre-registered co-location validity check (same cell solo vs alongside 7 neighbors; throughput + op-share deltas <2-3% to authorize 8-way; fallback 4-way + revalidate), CPU affinity pinning per server/client pair, and host load recording per run.
+- P3 protocol (task-mrh6vfqp, 27m13s, 835 lines): ACCEPTED with orchestrator amendments. Dispositions on 6 open decisions: (1) fixed-duration client build+test APPROVED; (2) P10 private-trace transfer to dash0 CPFS wjh APPROVED (sampling_u<=0.125, input<=32768, output cap 256; no prompt text in artifacts; data returns to same org cluster it came from); (3) MNS{64,1024}xMBT{2048,8192} sparse factorial APPROVED; (4) P10/C00 sole TP2 counterpoint APPROVED; (5) 60% load / burst-8 / 240s / 4 windows APPROVED; (6) kernel mapping + 70% classifiability gate + thresholds APPROVED. AMENDMENT A-P3-1 (user directive): 8-way GPU parallelism, one cell per GPU; pre-registered co-location validity check (same cell solo vs with 7 neighbors, throughput AND op-share deltas <3% to authorize; fallback 4-way + revalidate); CPU affinity pinning (disjoint core sets per server+client); saturation calibration under the SAME co-location regime as measurement; host load + clocks per run; revised wall estimate ~1.5-2h. AMENDMENT A-P3-2 (E2 lesson): execution via detached resumable controller script on dash0 (setsid, --resume, state file) so runs survive codex turn deaths.
+- P3 E-a ACCEPTED (task-mrh7wd5x, 1h27m, 3.96 H20-h): co-location gate → 8-way FAIL (throughput Δ=0.000% but op-share shifts up to 4.2pp > 3% threshold), 4-way PASS (shares ≤0.075pp) — AUTHORIZED 4-WAY. Client+controller 12/12 no-GPU tests; provenance hashed. P10 transfer verified (4,011 rows, tokenizer parity 4011/4011 exact, 0600/0700 perms, no prompt text public). Orchestrator decisions: (1) 4-way verdict accepted despite validation-stream footer absence (shares from Kineto, unaffected); (2) shutdown-fix (API-parent-first) GPU quick-verify REQUIRED before E-b matrix; (3) A-P3-3: per-class drain budget — 240s for output-512 burst cells (P04-class), 120s others; (4) A-P3-4: P3 hard stop raised 8→16 H20-hours (user blanket dash0 approval; E-a consumed 3.96).
+- E-b step1 STOP (task-mrhb2dxw, 10m42s): shutdown-fix verification FAILED hard footer gate — vLLM 0.24.0 API shutdown selected mode=abort timeout=0s and SIGTERMed EngineCore before scheduler/opprof teardown; finally-footer never runs. Run itself healthy (5268 reqs, 43.9 req/s, 1701 records, 0 drops, step range contiguous). Matrix NOT launched (correct). Orchestrator decision: footer must not depend on graceful shutdown — (a) 10-min static check for an official graceful path in 0.24.0; if unreliable, (b) checkpoint-footer sidecar in opprof.py (atomic write each flush cadence; accounting gate accepts sidecar when stream footer absent; bounded loss = last flush interval), new commit + re-export + tests, GPU re-verify incl. abort-kill case, then matrix.
+- E-b round 2 (task-mrhbhj9j, ~1h5m): sidecar fix DONE (commit f8b68f24, +195/-7, patch tip 23450fb2, 7 patches, 18/18+12/12 tests; graceful path found: vllm serve --shutdown-timeout N; dual GPU verification PASS incl. SIGKILL with 24ms checkpoint sidecar balancing). Matrix launched, STOPPED at pre-registered drain gate: P10/C01 saturation drain 288.6s > 120s budget (clean window + accounting VALID; drain is post-measurement hang-watchdog only; 0.74 req/s saturated 32k-context queue drains slowly by nature). 3/4 first-wave runs PASS with plausible ordering. GPU 5.47/16. AMENDMENT A-P3-5: P10-class drain budget 600s; drain violations = quarantine-run-and-continue (stop only if >20% of runs violate); P10/C01 existing data re-adjudicated under amended rule if accounting balanced.
+- NETWORK OUTAGE (~19:2xZ 2026-07-12 onward): dash0 AND dash1 unreachable from local (pre-auth timeout) — ingress/network fault, not host failure. Detached matrix controller on dash0 unaffected by design (A-P3-2); worker holds 1/min read-only reachability probe with retained artifact hashes for post-recovery integrity check. On reconnect: verify controller state (expect matrix advanced or complete), resume worker thread if its turn died.
+- E-b round 3 blocked cleanly on ssh outage (task-mrhdu37w): resumable handoff at runs/opprof-phase3/phase3/access-blocker-20260712.json; last durable 8/52 accepted, 0 quarantine, 0 clean-failures, 6.82 H20-h; controller PID 2237019 untouched, likely still advancing locally. Also delivered mid-flight: P03 profile-control starvation client fix (13/13 tests) + frozen analyzer analyze_phase3.py (4/4 tests). Reconnection watcher armed.
+- NETWORK RESTORED (~21:1xZ per user). Controller advanced 8→12/52 during outage then FAILED ~2.2h ago (idle since). GPU 8.13/16. E-b worker resumed (task-mrhkbq8s): integrity check vs retained hashes → diagnose controller failure → minimal fix → --resume matrix (40 runs remain) → frozen analysis.
+- E-b round 4 (task-mrhkbq8s, ~2h): 40/52 accepted (20/24 cells), 0 clean-window failures, 542,350 L1 records, 72 L2 traces, GPU 13.31/16. Two aux-gate fixes shipped (+58/-11, 15/15+4/4 tests): profiler-control connection, clean-window failure boundary (post-clean disconnects wrongly counted), 3-sample GPU preflight. FINAL BLOCKER: P10/TP2 warm-up gate demands 32/32 completions; long-context prompts cannot finish in budget (15/32, 21/32) though clean windows are valid. Same failure class as drain budget: auxiliary gate miscalibrated for long-context pattern. AMENDMENT A-P3-6: P10-class warm-up gate = 32 completions OR (>=16 completions AND Layer-1-verified throughput stabilization over trailing warm-up steps), with mandatory post-hoc stability evidence in the report. Finish remaining cells within 16h cap; if cap hits, stop and analyze with documented gaps.
+- E-b round 5 (task-mrhnw1rm, 23m46s): P10/TP2 FAILED frozen A-P3-6 stabilization on fresh data (drift 36.76% vs 10%; bins 11/10/16 vs 16) — the cell genuinely does not stabilize; clean window itself valid. Worker correctly refused to relax. GPU 14.03/16. ORCHESTRATOR DECISION: freeze matrix at 40/52 runs / 20/24 cells; abandon remaining 12 runs. AMENDMENT A-P3-7: per-contrast evaluation — each frozen contrast evaluates iff both cells complete; missing → NOT EVALUABLE (never imputed); H1a/H1b use existential logic on completed cells (CONFIRM possible, REFUTE NOT possible at 20/24 — must be stated explicitly). P10/TP2 non-stabilization recorded as a pattern-conditioned operational FINDING.
+- PHASE 3 FINAL (task-mrhos386, 19m37s): **H1a INCONCLUSIVE / H1b PASS (5 of 6 evaluable contrasts) / compound PARTIAL** under A-P3-7. H1b effects large + Holm-corrected p≈0: P10 real trace R64 +44.79pp vs both long rectangular controls (efficiency −14.3%/−44.7%); P09 production mix R64 +39.62pp, padding +6.67pp (−8.3%); P06 bimodal burst R64 +23.0/+35.4pp (−11.6%/−22.8%). H1a inconclusive because only 1/9 patterns Layer-2 windows passed representativeness gates (descriptive: P06/P10 flip attention-led↔MoE-GEMM-led across load points). Orchestrator verified metrics.json per-contrast records match report exactly. GPU final 14.03/16 (1.97 unused). P4 dispatching.
+
+## Phase 5 — mechanism-decomposition ablations (USER-APPROVED 2026-07-12)
+- Motivation (user challenge): the sign of H1b is obvious; the contribution is the mechanism LEDGER. vLLM does no request-level padding (continuous batching); measured bucket padding (+5.4-6.7pp) explains ~1/6 of the 14-45% E_token gap; the remaining ~30pp attribution (ragged-attention SM imbalance / cudagraph bucket mismatch / chunked-prefill mix interference / MoE routing skew / scheduler batch-variance) is unmeasured in the literature.
+- Design skeleton (protocol to formalize): controlled ablations each isolating ONE mechanism on P10 (primary) + P09/P06 (secondary): (i) length-sorted/binned replay [kills intra-cohort raggedness, keeps content+totals]; (ii) capture sizes matched to measured decode-B distribution [kills bucket mismatch; doubles as the config-tier deliverable]; (iii) arrival-shape ablation [steady vs burst, same lengths]; (iv) prefix-caching on/off [C structure]. Accounting must include an explicit interaction/residual term — shares are NOT forced to sum to 100%.
+- Metric: E_token under the P3 discipline (C00-TP1, rho=0.60, 240s clean windows). New GPU cap: +6 H20-hours for P5 (P3 cap closed at 14.03/16).
+- Gates: protocol-first (orchestrator review before any GPU run); P4 acceptance re-scoped — speculative recovery claims in P4 must defer to P5 measured shares.
+- P5 protocol (task-mrhq3ud3, 13m49s): ACCEPTED. Data-grounded specifics verified by worker from P3 raw Layer-1: P10 decode-B support is {1..7} (17,941 pure-decode steps) → A2 capture sizes {3,5,6,7}; A1 = 32-request reorder blocks (R16 0.6417→0.4734, −26.2% rel), 142-request slice @0.4725 req/s, 64s fairness cap. Orchestrator dispositions on 5 open decisions: (1) BLOCKING → approve recorded-arrival BRIDGE ledger (production-faithful estimand; explicit not-literal-P3-decomposition limitation); (2) dual P03/P04 control ledgers, dominant must hold under both — approved; (3) A1 parameters — approved; (4) 3 replicates/arm, P3-control reuse behind 3% bridge gate, optional tier within 6.0 H20-h — approved; (5) Layer-1-only primary, routed-expert telemetry analysis-only no causal share — approved. Execution authorized.
+- P5 wave 1 (detached exec, 30min, 0.65 H20-h): HARD STOP correct — all 4 attempted arms failed frozen warm-up stabilization (drift 190.6-216.5% recorded arms, 13.2% uniform arm, gate 10%; completions 25-28/32). ROOT CAUSE: semantic mismatch — throughput-drift gate designed for saturation/steady loads is wrong for rate-following arms whose throughput follows a non-stationary recorded arrival BY DESIGN. Purpose of warm-up = no cold-start contamination, not stationarity. Bonus signal: 190% vs 13% drift gap is itself direct arrival-mechanism evidence. AMENDMENT A-P5-1: rate-following arms use cold-start-artifact gates (all compile/capture events before clean window per Layer-1 graph-mode series + >=16 warm-up completions incl >=1 long-context + zero first-occurrence capture events inside clean window); drift gate retained for saturation arms only. Budget 5.35 remaining.
+- P5 round 2 (detached, ~70min, 2.44/6.0 H20-h): 15/15 primary valid under A-P5-1, bridge PASS (0.269%). OFFICIAL ledger INCONCLUSIVE per frozen rules (only A1-under-P03 evaluable: share 0.038, CI [-0.20,0.23]; A2/A3/A4 manipulation-failed; P04 denominator unstable 45% nonpositive draws). REAL FINDINGS: (1) P3 P10-vs-P04 gap was largely a MATERIALIZATION ARTIFACT — P5 recorded-arrival base E=3.014 ≈ P04 control 3.055 (gap 0.041 CI spans 0), while uniformizing arrival (as P3 did) drops E to 2.626 (−12.9%, raw p=0.014, Holm 0.055): burstiness HELPS at low rate via batch formation (decode-B CV 0.87→0.66 under uniform but waiting CV rises); P5 A3 reproduces P3 base within 0.27% — bridge design caught our own artifact. (2) All four intuitive mechanisms ≈0 at rho=0.60: raggedness 3.8% n.s., capture-fix +1.6% n.s. (P4 independent validation +0.18%, padding bound matched within 0.002pp), prefix ~0. Remaining P10-vs-P03 gap (~36%) explained by NONE — workload physics, not recoverable waste at this regime. (3) P4 ranked list: #1 prefix affinity +82.14% ceiling (P08 vs P07), #4 pattern-specific MNS64 has a −24.27% P01 counterexample (sign flips by pattern). OpProf campaign phases P0-P5 all complete; ~16.5 H20-h total.
+
+## Phase 6 — cross-version churn quantification (USER-APPROVED 2026-07-13 "请你测试一下")
+- Question: does the vLLM 0.20.0 C1 12-cell TP×MNS optimum/ranking survive on 0.24.0? Paired-surface churn evidence for the paper motivation (matrix × real-eval cost × churn).
+- Ground truth: C1 recovered stores (verified): TP1 [2.10,2.35,2.283,2.283], TP2 [2.275,2.275,3.283,3.258], TP4 [1.283,2.442,2.442,2.442] req/s/GPU; best TP2/MNS32=3.2833; trap TP4/MNS16; per-cell recorded anchors + request counts in replayserve docs/assets/simfid_s2r/ground_truth.json; workload = materialized chat_w20260311_1000 (local + dash0 CPFS), output override 128, scale 0.1, pass>=0.95, stepped TTFT 2/4/6s, TPOT 50ms; caps 512 TP1/TP2, uncapped TP4.
+- Budget: 3.0 H20-hours hard cap → anchor-subset design required (TP4 cells cost 4 GPU-h per wall-hour). Covariate to note: C1 ran on dash1, P6 on dash0 (same H20 class).
+- Gate: protocol-first, orchestrator review before GPU.
+- P6 protocol ACCEPTED: 92/92 historical anchor counts reproduced locally pre-GPU (comparability verified); adaptive 25-anchor design (12 peak + 11 adjacent + 2 trap-bracket), 2.65 primary + 0.35 confirmation reserve = 3.0 cap exactly. Decision rules: feasibility flip / >5% drift / floor-bucket argmax / tau-b>=0.8 ranking survival / trap escape rule / 2-of-3 confirmation. All 5 open decisions APPROVED incl. upgrade-path estimand (resolved defaults = churn; dash1→dash0 + colocation = stated limitations) and warmup long-tier adaptation raw>4096. Execution dispatched.
+- P6 W1 hard stop (0.43 H20-h spent): 8 replays REJECTED for missing in-stream footers (wrong shutdown path) + redo projection 3.03 > 3.0 cap. Orchestrator diagnosis: P6 validator failed to apply the P5-ACCEPTED sidecar-footer accounting rule (records == sidecar counters within one flush interval). AMENDMENT A-P6-1: (a) sidecar footers are valid accounting per P5 rule — re-adjudicate W1 artifacts WITHOUT rerun if sidecars balance; (b) controller must use the graceful path (vllm serve --shutdown-timeout, found in P5) for subsequent waves; (c) IF W1 salvage fails, cap raised 3.0→3.5 as fallback (worker recommendation, within user posture).
+- P6 r2 partial (2.29/3.0 H20-h, hard stop before W4): 21/25 anchors, 10/12 cells. Signals: TP2/MNS64 >=29.4% DOWN (left-censored), TP1 cells right-censored UP, TP2 family down — surface shape moved opposite directions. CONFOUND (worker confirmation runs): co-location flips SLO-frontier pass rates (0.41→1.00, 0.03→0.96 solo; waiting 8.5→0.35) — co-location valid for throughput/op-shares (P3) but NOT for pass-rate cliffs; 0.20 baseline was solo. USER DECISION: cap raised to 6.0; A-P6-2 = frontier anchors SOLO placement; finish W4 + solo re-confirm key cells.
+- P6 FINAL (A-P6-2 solo tier complete, 5.64/6.0 H20-h, 66 authoritative trials, 12/12 cells): formal verdicts INCONCLUSIVE per frozen letter (4 cells censored/non-monotonic prevent bounded 12-cell surface; tau_b=null, never manufactured). SUBSTANTIVE PAIRED SURFACE (all solo-authoritative): TP2/MNS64 −29.41% CONFIRMED (3.2583→2.3000 bounded; mechanism: decode-B 11.5-18 + 4.6-9.5% NONE-graph at old anchor); TP1/MNS8 +13.49% and TP1 plateau converges to 2.3833; old best TP2/MNS32 −0.76% (held, highest bounded); TP4 family right-censored UP >=2.50; TP4/MNS16 frontier became NON-MONOTONIC in load. CO-LOCATION DELTA TABLE: up to +92.86pp pass-rate flips (21 exact-request anchor pairs) — co-location validity is metric-dependent (standalone methodological finding). Orchestrator verified −0.2941 drift in metrics.json. Campaign total incl. P6: ~22.2 H20-h.

docs/simulator-fidelity-frontier-20260711.md

@@ -0,0 +1,134 @@
+# Simulator-based tuning 保真度实验总结（Frontier / H20）— 供 review
+
+日期：2026-07-11。目的：判断现有实验数据是否**严格**支撑"simulator-based
+tuning 会导致错误的 rank 排序和最终性能 gap"这一论文主张。全部原始材料在
+`~/phd/replayserve`（见文末指针）；本文只做总结，不新增任何数字。
+
+## 一句话结论
+
+在给足模拟器一切有利条件（同模型同硬件的 H20 算子 profile、逐 token 精确的
+真实 workload、在独立数据上冻结的逐 TP 吞吐校准）之后，Frontier 在预注册判定
+规则下仍未通过：在真实调优器实测过的 12-cell TP×MNS 面上，模拟器选出的配置
+比真实最优差 **30.46%**（top-1 real-evaluated regret），关键交互关系只复现
+3/6。**若按模拟器结果部署，你会选 TP1/MNS64（真实 2.283 req/s/GPU），放弃
+TP2/MNS32（真实 3.283）——这就是"最终 gap"的操作性含义。**
+
+## 实验设置（三个不可辩驳性设计）
+
+- **Ground truth 是真实调优研究本身**：C1 交互研究（dash1 恢复的 store，
+  vLLM 0.20.0，H20，`chat_w20260311_1000` 窗口，scale 0.1），12 个
+  TP{1,2,4}×MNS{8,16,32,64} cell 的 SLO-feasible peak（pass≥0.95，阶梯
+  TTFT 2/4/6s，TPOT 50ms）。每个数字从原始 state.json/engine.log 独立复算。
+- **EXACT workload**：materialized JSONL（32,606 行，含真实 prompt 文本）
+  逐 token 重建——tokenize 后逐行断言等于 `input_length`（17,710/17,710 全
+  过），chat 模板开销恒为 +8 token 并分解到具体 token ID，block-16 前缀 hash
+  按 vLLM 0.20.0 的 `hash_block_tokens` 源码（vendored、逐字节比对）计算。
+  每个 cell 用它自己实测过的锚点（92 个锚点，选中请求数 92/92 与真实记录
+  相等），不插值、不补网格。
+- **预注册**：判定规则在任何模拟跑之前冻结于
+  `replayserve/docs/simfid_s2r_protocol.md` §6——(1) 最坏 top-1 regret ≤5%；
+  (2) TP4/MNS16 陷阱 6 条关系全复现；(3) 92 次留一锚点复检全过。四种读法中
+  只有"冻结校准 + 吞吐代理"承载结论，其余为诊断。校准系数
+  a_tp = {TP1: 0.7235, TP2: 0.4681, TP4: 0.3521} 来自独立的 3 配置切片
+  （S2-E），S2-R 数据上不重拟合。
+
+执行：184 次 Frontier CPU run（92 锚点 × 未校准/校准），184/184 通过、
+0 失败，sanity 无红旗。
+
+## 核心证据：12-cell 分数表（req/s/GPU）
+
+| Cell | 真实 SLO peak | 校准后 sim 吞吐 | 校准后 sim SLO |
+|---|---:|---:|---:|
+| tp1_mns8  | 2.100 | 2.173 | 1.717 |
+| tp1_mns16 | 2.350 | 3.242 | 2.383 |
+| tp1_mns32 | 2.283 | 4.297 | 2.383 |
+| tp1_mns64 | 2.283 | **4.357** ← sim 判为全局最优 | 2.383 |
+| tp2_mns8  | 2.275 | 2.039 | 1.742 |
+| tp2_mns16 | 2.275 | 2.244 | 2.300 |
+| tp2_mns32 | **3.283** ← 真实全局最优 | 3.650 | 3.750 |
+| tp2_mns64 | 3.258 | 3.650 | 3.750 |
+| tp4_mns8  | 1.283 | 1.545 | 1.321 |
+| tp4_mns16 | 2.442 | 2.437 | 2.500 |
+| tp4_mns32 | 2.442 | 2.462 | 2.500 |
+| tp4_mns64 | 2.442 | 2.462 | 2.500 |
+
+**失败机理**（不是尺度错，是形状错）：真实 TP1 在 MNS16 后随 SLO 边界饱和
+（2.35→2.28），sim 认为吞吐随 MNS 单调上升（3.24→4.30→4.36）。逐 TP 校准
+已消掉尺度误差，负载响应形状仍然错——来源是排队/尾延迟/调度开销，不是算子
+时间表。佐证：sim TTFT p95 仅为真实的 0.30–0.38，TPOT p95 0.63–0.79
+（S2-E 持出集）。
+
+排序质量：校准吞吐读法 τ-b = 0.448（未校准 0.236），成对方向正确率
+68–73%。远低于可用水平。
+
+## 什么已被严格证明（本文主张的边界内）
+
+1. **在被测面上，simulator-only 的吞吐排序会给出 30.46% 的真实 gap**——
+   端到端、预注册、workload 逐 token 精确、同硬件 profile、校准冻结。链条
+   里没有"我们没给模拟器机会"的空隙。
+2. **未做延迟校准的模拟器做 SLO 判断不可信**：S2-E 直接反例（sim TPOT p95
+   46.43ms vs 真实 71.38ms，横跨 50ms SLO → sim 判可行、真实不可行）；
+   S2-R-b 校准 SLO 读法在 92 锚点上有 21 假可行 / 7 假不可行。
+3. **覆盖缺口独立于精度成立**：MoE EP>1 无 profile 支持、GMU 惰性、
+   CUDA-graph 未接、scheduler-delay 不可表达——这些 knob 模拟器根本无法
+   评估（replayserve/docs/simfid_inventory.md 旋钮矩阵）。
+
+## 什么还没被严格证明（review 时请重点判断这三条）
+
+1. **跨引擎版本 profile**（最大攻击面）：H20 profile 来自 vLLM 0.11.1 时代
+   的对齐工作，ground truth 是 0.20.0。防守方可以说"按 0.20.0 重新 profile
+   就好了"。我们的回应有二：(a) 冻结校准已吸收逐 TP 尺度误差，剩余的是形状
+   误差，其来源（排队/调度/CUDA-graph）不在算子表里；(b) 反身性论证——若
+   每个引擎版本 × 硬件都要重 profile + 重校准，profiling 本身占用同款 GPU
+   跑真实负载，模拟器的成本优势即被churn吃掉。但 (a) 目前是机理论证 +
+   间接证据，不是实验闭环。
+2. **SLO 门控读法的意外成功**：校准 + SLO 门控（预注册为仅诊断）达到
+   regret 0–0.76%、τ-b 0.967、陷阱 6/6。审稿人可主张"你选错了读法"。
+   我们的回应：该读法建立在错误的逐锚点判定上（21 假可行/7 假不可行），
+   正确性来自误差在逐 cell 峰值处的部分抵消，无法保证泛化；且它是事后
+   观察，预注册规则不允许换读法。**但要诚实：这条把可主张的结论从
+   "模拟器必然错排"弱化为"模拟器排序不可信、必须真实验证"。**
+3. **单面、单模型、单硬件、单 workload 窗口**：无跨 workload 复制。且论文
+   审稿人点名的多半是 Vidur，我们测的是 Frontier（同类：算子 profile +
+   事件驱动调度模拟；需论证类代表性或列为 limitation）。
+
+## 建议的论文主张口径（可辩护版本）
+
+> 不主张"simulator 总是错排"，主张：在我们的真实调优器必须绕开局部陷阱的
+> 那个交互面上，一个被给足条件的模拟器（同硬件 profile + 吞吐校准 + 精确
+> workload）在其预注册的最优读法下产生 30% 的部署 gap，且其 SLO 判定在
+> 无延迟校准时存在跨边界的假可行；因此 simulator-only tuning 的结果不经
+> 真实评估不可信——而真实评估的成本控制正是 AITuner 的贡献。混合设计
+> （模拟器粗筛 + 真实终判）是被我们的诊断数据支持的未来方向，且"何时必须
+> 真实评估"仍由相似度度量回答。
+
+## 可选补强（按闭环价值排序）
+
+1. **负载响应形状分析**（零新模拟，复用 184 run + 真实 probe history）：
+   逐 cell 对比归一化吞吐/延迟-锚点曲线。若形状系统性不匹配，则证明任意
+   逐配置尺度校准（= 任意精度的算子 profile）原理上无法恢复排序，直接
+   封死上面第 1 条攻击面的一半。
+2. **vLLM 0.20.0 重 profile**（需 GPU 时间 + 审批）：实验性封死跨版本
+   质疑，但注意这同时演示了 churn 成本，输赢都有叙事价值。
+3. **第二个 workload 窗口**（如 coder 或 2200 slot）：复制性。
+
+## 数据 sanity block
+
+- 12-cell 真实向量：n=12，min/max=1.283/3.283，8 个 distinct 值。
+- 校准 sim 吞吐向量：n=12，min/max=1.545/4.357，10 个 distinct。
+- 执行：n=184，失败 0，请求数 min/max=66/600（34 distinct）。
+- regret=30.456853% 由 1−2.2833/3.2833 独立复算精确一致；a_tp 三值与
+  S2-E 冻结清单一致；所有比率在 [0,1]；无逐配置全同向量。
+- 已知异常（保留未修饰）：真实 TP2/MNS32 与 MNS64 的 pass-rate 非单调，
+  但 0.95 可行性截断有序。
+
+## 材料指针（全部在 ~/phd/replayserve）
+
+- 最终综合报告：`docs/simfid_s3_fidelity_report.md`
+- 12-cell 结果与全部指标：`docs/simfid_s2rb_results.md`、
+  `runs/simfid_s2rb/results/metrics.json`
+- 预注册协议 + 修正案：`docs/simfid_s2r_protocol.md`
+- 吞吐校准与延迟反例：`docs/simfid_s2e_report.md`
+- 数据清单与旋钮覆盖矩阵：`docs/simfid_inventory.md`
+- 全程决策/验收台账：`docs/simfid_campaign_state.md`
+- 恢复的 EXACT workload：`~/phd/aituner/trace_windows/traces/chat_w20260311_1000.jsonl`

patches/vllm-0.24.0-opprof/0001-Add-lightweight-per-step-OpProf-telemetry.patch

@@ -0,0 +1,487 @@
+From f6f1cacbce0e39992d04843f652c1adda373ae43 Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sat, 11 Jul 2026 17:29:02 +0800
+Subject: [PATCH 1/5] Add lightweight per-step OpProf telemetry
+
+Assisted-by: OpenAI Codex
+---
+ vllm/envs.py                       |   4 +
+ vllm/v1/core/sched/scheduler.py    |  28 +++
+ vllm/v1/opprof.py                  | 337 +++++++++++++++++++++++++++++
+ vllm/v1/worker/gpu_model_runner.py |   6 +-
+ 4 files changed, 374 insertions(+), 1 deletion(-)
+ create mode 100644 vllm/v1/opprof.py
+
+diff --git a/vllm/envs.py b/vllm/envs.py
+index 27a85bb..b3093e9 100755
+--- a/vllm/envs.py
++++ b/vllm/envs.py
+@@ -45,6 +45,7 @@ if TYPE_CHECKING:
+     VLLM_LOGGING_COLOR: str = "auto"
+     NO_COLOR: bool = False
+     VLLM_LOG_STATS_INTERVAL: float = 10.0
++    VLLM_OPPROF_DIR: str = ""
+     VLLM_TRACE_FUNCTION: int = 0
+     VLLM_USE_FLASHINFER_SAMPLER: bool = True
+     VLLM_PP_LAYER_PARTITION: str | None = None
+@@ -786,6 +787,9 @@ environment_variables: dict[str, Callable[[], Any]] = {
+         if (val := float(os.getenv("VLLM_LOG_STATS_INTERVAL", "10."))) > 0.0
+         else 10.0
+     ),
++    # Directory for per-step OpProf JSONL telemetry.
++    # Empty disables OpProf.
++    "VLLM_OPPROF_DIR": lambda: os.getenv("VLLM_OPPROF_DIR", ""),
+     # Trace function calls
+     # If set to 1, vllm will trace function calls
+     # Useful for debugging
+diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py
+index 90d93a1..303c562 100644
+--- a/vllm/v1/core/sched/scheduler.py
++++ b/vllm/v1/core/sched/scheduler.py
+@@ -7,6 +7,7 @@ from collections.abc import Iterable
+ from dataclasses import replace
+ from typing import Any
+ 
++import vllm.envs as envs
+ from vllm.compilation.cuda_graph import CUDAGraphStat
+ from vllm.config import VllmConfig
+ from vllm.distributed.ec_transfer.ec_connector.base import (
+@@ -55,6 +56,7 @@ from vllm.v1.engine import EngineCoreEventType, EngineCoreOutput, EngineCoreOutp
+ from vllm.v1.kv_cache_interface import KVCacheConfig
+ from vllm.v1.metrics.perf import ModelMetrics, PerfStats
+ from vllm.v1.metrics.stats import PrefixCacheStats, SchedulerStats
++from vllm.v1.opprof import OpProfRecorder
+ from vllm.v1.outputs import DraftTokenIds, KVConnectorOutput, ModelRunnerOutput
+ from vllm.v1.request import Request, RequestStatus, StreamingUpdate
+ from vllm.v1.spec_decode.dynamic.utils import build_dynamic_sd_schedule_lookup
+@@ -271,6 +273,12 @@ class Scheduler(SchedulerInterface):
+         if self.connector is not None:
+             self.connector.bind_gpu_block_pool(self.kv_cache_manager.block_pool)
+ 
++        self.opprof = OpProfRecorder.create(
++            output_dir=envs.VLLM_OPPROF_DIR,
++            dp_rank=self.parallel_config.data_parallel_index,
++            log_stats=self.log_stats,
++        )
++
+         self.use_pp = self.parallel_config.pipeline_parallel_size > 1
+         self.use_v2_model_runner = vllm_config.use_v2_model_runner
+         # Scheduler iteration counter. Drives the V2+PP+async decode-throttle
+@@ -386,6 +394,9 @@ class Scheduler(SchedulerInterface):
+         return num_new_tokens
+ 
+     def schedule(self, throttle_prefills: bool = False) -> SchedulerOutput:
++        opprof_start = (
++            self.opprof.capture_start(self) if self.opprof is not None else None
++        )
+         self.current_step += 1
+         # NOTE(woosuk) on the scheduling algorithm:
+         # There's no "decoding phase" nor "prefill phase" in the scheduler.
+@@ -1090,6 +1101,14 @@ class Scheduler(SchedulerInterface):
+             )
+             scheduler_output.ec_connector_metadata = ec_meta
+ 
++        if self.opprof is not None:
++            assert opprof_start is not None
++            self.opprof.begin(
++                scheduler=self,
++                output=scheduler_output,
++                start=opprof_start,
++            )
++
+         # Advance the fence only for non-empty steps (those that actually
+         # write KV and have their output processed later in update_from_output).
+         if self.defer_block_free and total_num_scheduled_tokens > 0:
+@@ -1800,6 +1819,12 @@ class Scheduler(SchedulerInterface):
+                 engine_core_outputs[0] = eco = EngineCoreOutputs()
+             eco.scheduler_stats = stats
+ 
++        if self.opprof is not None:
++            self.opprof.finalize(
++                output=scheduler_output,
++                cudagraph_stat=cudagraph_stats,
++            )
++
+         return engine_core_outputs
+ 
+     @staticmethod
+@@ -2292,6 +2317,9 @@ class Scheduler(SchedulerInterface):
+         if self.ec_connector is not None:
+             self.ec_connector.shutdown()
+ 
++        if self.opprof is not None:
++            self.opprof.close()
++
+         logger.debug_once("[shutdown] Scheduler: complete")
+ 
+     ########################################################################
+diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py
+new file mode 100644
+index 0000000..f0330d0
+--- /dev/null
++++ b/vllm/v1/opprof.py
+@@ -0,0 +1,337 @@
++# SPDX-License-Identifier: Apache-2.0
++# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
++import atexit
++import logging
++import os
++import queue
++import threading
++import time
++from bisect import bisect_left
++from pathlib import Path
++from typing import Any
++
++import msgspec
++
++logger = logging.getLogger(__name__)
++
++SCHEMA_VERSION = 1
++CONTEXT_LENGTH_EDGES = tuple(1 << exponent for exponent in range(7, 18))
++CHUNK_SIZE_EDGES = tuple(1 << exponent for exponent in range(4, 12))
++DEFAULT_QUEUE_CAPACITY = 8192
++_CLOSE_TIMEOUT_SECONDS = 1.0
++_STOP = object()
++_PREFIX_FIELDS = (  # noqa: SIM905
++    "requests queries hits preempted_requests preempted_queries preempted_hits"
++).split()
++
++
++ScheduleStart = tuple[int, int, dict[str, dict[str, int] | None]]
++
++
++def classify_chunk(was_chunk: bool, end: int, target: int) -> str:
++    assert 0 <= end <= target
++    if was_chunk:
++        return "middle" if end < target else "final"
++    return "first" if end < target else "unsplit"
++
++
++def _prefix_values(stats: Any | None) -> dict[str, int] | None:
++    if stats is None:
++        return None
++    return {name: int(getattr(stats, name, 0)) for name in _PREFIX_FIELDS}
++
++
++def _prefix_snapshot(scheduler: Any) -> dict[str, dict[str, int] | None]:
++    return {
++        "local": _prefix_values(scheduler.kv_cache_manager.prefix_cache_stats),
++        "external": _prefix_values(scheduler.connector_prefix_cache_stats),
++    }
++
++
++def _prefix_delta(
++    before: dict[str, int] | None, after: dict[str, int] | None
++) -> dict[str, int] | None:
++    if before is None and after is None:
++        return None
++    before = before or dict.fromkeys(_PREFIX_FIELDS, 0)
++    after = after or dict.fromkeys(_PREFIX_FIELDS, 0)
++    delta = {name: after[name] - before[name] for name in _PREFIX_FIELDS}
++    assert all(value >= 0 for value in delta.values())
++    return delta
++
++
++class JSONLWriter:
++    def __init__(
++        self,
++        path: Path,
++        capacity: int = DEFAULT_QUEUE_CAPACITY,
++        start: bool = True,
++    ) -> None:
++        self._queue: queue.Queue[Any] = queue.Queue(capacity)
++        self._encoder = msgspec.json.Encoder()
++        self._file = path.open("xb", buffering=1 << 20)
++        self._thread = threading.Thread(target=self._run, daemon=True)
++        self._failure_lock = threading.Lock()
++        self._started = self._closed = False
++        self.failed = False
++        self.failure: Exception | None = None
++        self.encoded_records = self.written_records = 0
++        self.dropped_records = self._unreported_drops = 0
++        if start:
++            self.start()
++
++    def start(self) -> None:
++        if not self._started:
++            self._started = True
++            self._thread.start()
++
++    def _record_failure(self, error: Exception) -> None:
++        with self._failure_lock:
++            if self.failed:
++                return
++            self.failed = True
++            self.failure = error
++            logger.error("OpProf writer failed: %s", error)
++
++    def _writer_unavailable(self) -> bool:
++        if self.failed:
++            return True
++        if self._started and not self._thread.is_alive():
++            self._record_failure(RuntimeError("writer thread stopped unexpectedly"))
++            return True
++        return False
++
++    def _drop(self, pending: int) -> bool:
++        self.dropped_records += 1
++        self._unreported_drops = pending + 1
++        return False
++
++    def submit(self, record: dict[str, Any]) -> bool:
++        if self._closed:
++            raise RuntimeError("OpProf writer is closed")
++        pending = self._unreported_drops
++        record["dropped_records_before"] = pending
++        if self._writer_unavailable():
++            return self._drop(pending)
++        payload = self._encoder.encode(record) + b"\n"
++        self.encoded_records += 1
++        if self._writer_unavailable():
++            return self._drop(pending)
++        try:
++            self._queue.put_nowait(payload)
++        except queue.Full:
++            return self._drop(pending)
++        self._unreported_drops = 0
++        return True
++
++    def _run(self) -> None:
++        buffered = 0
++        last_flush = time.monotonic()
++        try:
++            while True:
++                try:
++                    item = self._queue.get(timeout=1.0)
++                except queue.Empty:
++                    self._file.flush()
++                    buffered = 0
++                    last_flush = time.monotonic()
++                    continue
++                try:
++                    if item is _STOP:
++                        break
++                    self._file.write(item)
++                    buffered += len(item)
++                    self.written_records += 1
++                    now = time.monotonic()
++                    if buffered >= 1 << 20 or now - last_flush >= 1.0:
++                        self._file.flush()
++                        buffered = 0
++                        last_flush = now
++                finally:
++                    self._queue.task_done()
++        except Exception as error:
++            self._record_failure(error)
++        finally:
++            footer = dict(
++                schema=SCHEMA_VERSION,
++                record_type="footer",
++                encoded_records=self.encoded_records,
++                written_records=self.written_records,
++                dropped_records=self.dropped_records,
++            )
++            try:
++                self._file.write(self._encoder.encode(footer) + b"\n")
++                self._file.flush()
++            except Exception as error:
++                self._record_failure(error)
++            finally:
++                try:
++                    self._file.close()
++                except Exception as error:
++                    self._record_failure(error)
++
++    def close(self) -> None:
++        if self._closed:
++            return
++        self._closed = True
++        self.start()
++        if not self._thread.is_alive():
++            return
++        try:
++            self._queue.put(_STOP, timeout=_CLOSE_TIMEOUT_SECONDS)
++        except queue.Full:
++            if not self._thread.is_alive():
++                return
++            try:
++                self._queue.put_nowait(_STOP)
++            except queue.Full:
++                self._record_failure(
++                    TimeoutError("timed out enqueueing writer stop sentinel")
++                )
++                return
++        self._thread.join(timeout=_CLOSE_TIMEOUT_SECONDS)
++        if self._thread.is_alive():
++            self._record_failure(TimeoutError("timed out joining writer thread"))
++
++
++class OpProfRecorder:
++    def __init__(self, engine_id: str, writer: JSONLWriter) -> None:
++        self.engine_id = engine_id
++        self.writer = writer
++        self._next_step = 0
++        self._pending: dict[int, dict[str, Any]] = {}
++        atexit.register(self.close)
++
++    @classmethod
++    def create(
++        cls, output_dir: str, dp_rank: int, log_stats: bool
++    ) -> "OpProfRecorder | None":
++        if not output_dir:
++            return None
++        if not log_stats:
++            raise ValueError("VLLM_OPPROF_DIR requires log stats to be enabled")
++        directory = Path(output_dir).expanduser()
++        if not directory.is_absolute():
++            raise ValueError("VLLM_OPPROF_DIR must be an absolute path")
++        directory.mkdir(parents=True, exist_ok=True)
++        engine_id = f"dp{dp_rank}-pid{os.getpid()}"
++        name = f"opprof-v{SCHEMA_VERSION}-{engine_id}-{time.time_ns()}.jsonl"
++        return cls(engine_id, JSONLWriter(directory / name))
++
++    @staticmethod
++    def capture_start(scheduler: Any) -> ScheduleStart:
++        return time.time_ns(), time.monotonic_ns(), _prefix_snapshot(scheduler)
++
++    def begin(self, scheduler: Any, output: Any, start: ScheduleStart) -> None:
++        key = id(output)
++        assert key not in self._pending, "duplicate OpProf begin"
++        new_ids = {request.req_id for request in output.scheduled_new_reqs}
++        prefill_requests = prefill_tokens = decode_requests = decode_tokens = 0
++        context_length_hist = [0] * (len(CONTEXT_LENGTH_EDGES) + 1)
++        chunk_size_hist = [0] * (len(CHUNK_SIZE_EDGES) + 1)
++        chunks: dict[str, Any] = dict.fromkeys(
++            ("first", "middle", "final", "unsplit"), 0
++        )
++        for req_id, num_tokens in output.num_scheduled_tokens.items():
++            request = scheduler.requests[req_id]
++            end = request.num_computed_tokens + num_tokens
++            assert end >= 0
++            context_length_hist[bisect_left(CONTEXT_LENGTH_EDGES, end)] += 1
++            is_prefill = (
++                req_id in new_ids
++                or output.scheduled_cached_reqs.is_context_phase(req_id)
++            )
++            if is_prefill:
++                prefill_requests += 1
++                prefill_tokens += num_tokens
++                assert num_tokens >= 0
++                chunk_size_hist[bisect_left(CHUNK_SIZE_EDGES, num_tokens)] += 1
++                target = request.num_tokens + request.num_output_placeholders
++                chunks[classify_chunk(request.is_prefill_chunk, end, target)] += 1
++            else:
++                decode_requests += 1
++                decode_tokens += num_tokens
++        prefix_after = _prefix_snapshot(scheduler)
++        block_pool = scheduler.kv_cache_manager.block_pool
++        total_blocks = block_pool.num_gpu_blocks - 1
++        free_blocks = block_pool.get_num_free_blocks()
++        assert 0 <= free_blocks <= total_blocks
++        chunks["tokens"] = prefill_tokens
++        chunks["chunk_size_hist"] = chunk_size_hist
++        values = dict(
++            schema=SCHEMA_VERSION,
++            engine_id=self.engine_id,
++            step_index=self._next_step,
++            submit_wall_ns=start[0],
++            submit_mono_ns=start[1],
++            model_executed=output.total_num_scheduled_tokens > 0,
++            scheduled_requests=len(output.num_scheduled_tokens),
++            decode_batch_size=decode_requests,
++            prefill_requests=prefill_requests,
++            prefill_tokens=prefill_tokens,
++            decode_tokens=decode_tokens,
++            chunked_prefill=chunks,
++            context_length_hist=context_length_hist,
++            preemptions=len(output.preempted_req_ids or ()),
++            queues=dict(
++                running=len(scheduler.running),
++                waiting=len(scheduler.waiting),
++                deferred=len(scheduler.skipped_waiting),
++            ),
++            kv=dict(
++                total_blocks=total_blocks,
++                free_blocks=free_blocks,
++                used_blocks=total_blocks - free_blocks,
++                usage=scheduler.kv_cache_manager.usage,
++            ),
++            prefix=dict(
++                local=_prefix_delta(start[2]["local"], prefix_after["local"]),
++                external=_prefix_delta(start[2]["external"], prefix_after["external"]),
++            ),
++        )
++        assert prefill_tokens + decode_tokens == output.total_num_scheduled_tokens
++        self._pending[key] = values
++        self._next_step += 1
++
++    def finalize(self, output: Any, cudagraph_stat: Any | None) -> bool:
++        try:
++            values = self._pending.pop(id(output))
++        except KeyError:
++            raise AssertionError("missing or already finalized OpProf step") from None
++        if cudagraph_stat is None:
++            assert not values["model_executed"]
++            cudagraph = dict(
++                hit=False,
++                runtime_mode="NONE",
++                unpadded_tokens=0,
++                bucket_tokens=0,
++                padding_tokens=0,
++            )
++        else:
++            mode = str(cudagraph_stat.runtime_mode).rsplit(".", 1)[-1]
++            cudagraph = dict(
++                hit=mode != "NONE",
++                runtime_mode=mode,
++                unpadded_tokens=cudagraph_stat.num_unpadded_tokens,
++                bucket_tokens=cudagraph_stat.num_padded_tokens,
++                padding_tokens=cudagraph_stat.num_paddings,
++            )
++        record = dict(
++            values,
++            complete_mono_ns=time.monotonic_ns(),
++            cudagraph=cudagraph,
++            moe_expert_load=None,
++            dropped_records_before=0,
++        )
++        return self.writer.submit(record)
++
++    def close(self) -> None:
++        self.writer.close()
++
++    @property
++    def failed(self) -> bool:
++        return self.writer.failed
++
++    @property
++    def failure(self) -> Exception | None:
++        return self.writer.failure
+diff --git a/vllm/v1/worker/gpu_model_runner.py b/vllm/v1/worker/gpu_model_runner.py
+index 74938a8..c11d773 100644
+--- a/vllm/v1/worker/gpu_model_runner.py
++++ b/vllm/v1/worker/gpu_model_runner.py
+@@ -437,6 +437,7 @@ class GPUModelRunner(
+         self.scheduler_config = vllm_config.scheduler_config
+         self.speculative_config = vllm_config.speculative_config
+         self.observability_config = vllm_config.observability_config
++        self.opprof_enabled = bool(envs.VLLM_OPPROF_DIR)
+ 
+         model_config = self.model_config
+         cache_config = self.cache_config
+@@ -3917,7 +3918,10 @@ class GPUModelRunner(
+                 assert batch_descriptor.num_tokens == num_tokens_padded
+ 
+         cudagraph_stats = None
+-        if self.vllm_config.observability_config.cudagraph_metrics:
++        if (
++            self.vllm_config.observability_config.cudagraph_metrics
++            or self.opprof_enabled
++        ):
+             cudagraph_stats = CUDAGraphStat(
+                 num_unpadded_tokens=num_tokens,
+                 num_padded_tokens=batch_descriptor.num_tokens,
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0002-Add-standalone-OpProf-telemetry-tests.patch

@@ -0,0 +1,417 @@
+From 4f4ee674f217698436b00c3ab6357f59a792477a Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sat, 11 Jul 2026 17:29:02 +0800
+Subject: [PATCH 2/5] Add standalone OpProf telemetry tests
+
+Assisted-by: OpenAI Codex
+---
+ tests/v1/core/test_opprof.py | 397 +++++++++++++++++++++++++++++++++++
+ 1 file changed, 397 insertions(+)
+ create mode 100644 tests/v1/core/test_opprof.py
+
+diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
+new file mode 100644
+index 0000000..9bfbfcc
+--- /dev/null
++++ b/tests/v1/core/test_opprof.py
+@@ -0,0 +1,397 @@
++# SPDX-License-Identifier: Apache-2.0
++# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
++"""Standalone tests: this file intentionally does not import the vllm package."""
++
++import errno
++import importlib.util
++import logging
++import sys
++import threading
++from pathlib import Path
++from types import SimpleNamespace
++
++import msgspec
++import pytest
++
++_ROOT = Path(__file__).parents[3]
++_SPEC = importlib.util.spec_from_file_location(
++    "opprof_standalone", _ROOT / "vllm" / "v1" / "opprof.py"
++)
++assert _SPEC is not None and _SPEC.loader is not None
++opprof = importlib.util.module_from_spec(_SPEC)
++sys.modules[_SPEC.name] = opprof
++_SPEC.loader.exec_module(opprof)
++
++
++class CachedRequests:
++    def __init__(self, context_ids=()):
++        self.context_ids = set(context_ids)
++
++    def is_context_phase(self, req_id):
++        return req_id in self.context_ids
++
++
++def prefix_stats(**overrides):
++    values = dict.fromkeys(opprof._PREFIX_FIELDS, 0)
++    values.update(overrides)
++    return SimpleNamespace(**values)
++
++
++def request(computed, total, was_chunk=False, placeholders=0):
++    return SimpleNamespace(
++        num_computed_tokens=computed,
++        num_tokens=total,
++        num_output_placeholders=placeholders,
++        is_prefill_chunk=was_chunk,
++    )
++
++
++def scheduler(requests, local=None, external=None):
++    block_pool = SimpleNamespace(
++        num_gpu_blocks=101,
++        get_num_free_blocks=lambda: 40,
++    )
++    kv_manager = SimpleNamespace(
++        prefix_cache_stats=local or prefix_stats(),
++        block_pool=block_pool,
++        usage=0.6,
++    )
++    return SimpleNamespace(
++        requests=requests,
++        kv_cache_manager=kv_manager,
++        connector_prefix_cache_stats=external,
++        running=list(range(len(requests))),
++        waiting=[0, 1],
++        skipped_waiting=[0],
++    )
++
++
++def schedule_output(tokens, context_ids=(), new_ids=(), preempted=()):
++    return SimpleNamespace(
++        scheduled_new_reqs=[SimpleNamespace(req_id=req_id) for req_id in new_ids],
++        scheduled_cached_reqs=CachedRequests(context_ids),
++        num_scheduled_tokens=tokens,
++        total_num_scheduled_tokens=sum(tokens.values()),
++        preempted_req_ids=set(preempted),
++    )
++
++
++def graph(mode="FULL", unpadded=1, padded=1):
++    return SimpleNamespace(
++        runtime_mode=mode,
++        num_unpadded_tokens=unpadded,
++        num_padded_tokens=padded,
++        num_paddings=padded - unpadded,
++    )
++
++
++def recorder(tmp_path, *, capacity=8192, start=True):
++    path = tmp_path / "opprof.jsonl"
++    writer = opprof.JSONLWriter(path, capacity=capacity, start=start)
++    return opprof.OpProfRecorder("dp0-pid1", writer), path
++
++
++def emit(rec, sched, output, cg=None):
++    start = rec.capture_start(sched)
++    rec.begin(sched, output, start)
++    return rec.finalize(output, cg or graph())
++
++
++def read_jsonl(path):
++    return [msgspec.json.decode(line) for line in path.read_bytes().splitlines()]
++
++
++def test_import_light_and_approved_constants():
++    assert "torch" not in sys.modules
++    assert "vllm" not in sys.modules
++    assert opprof.DEFAULT_QUEUE_CAPACITY == 8192
++    assert tuple(1 << i for i in range(7, 18)) == opprof.CONTEXT_LENGTH_EDGES
++    assert tuple(1 << i for i in range(4, 12)) == opprof.CHUNK_SIZE_EDGES
++
++
++def test_schema_and_invariants(tmp_path):
++    sched = scheduler(
++        {
++            "first": request(0, 100),
++            "final": request(64, 100, was_chunk=True),
++            "decode": request(1024, 1025),
++        }
++    )
++    output = schedule_output(
++        {"first": 64, "final": 36, "decode": 1},
++        context_ids={"final"},
++        new_ids={"first"},
++        preempted={"old"},
++    )
++    rec, path = recorder(tmp_path)
++    start = rec.capture_start(sched)
++    sched.kv_cache_manager.prefix_cache_stats = prefix_stats(
++        requests=1, queries=100, hits=64
++    )
++    rec.begin(sched, output, start)
++    assert rec.finalize(output, graph("FULL", 101, 128))
++    rec.close()
++
++    record, footer = read_jsonl(path)
++    assert record["schema"] == 1
++    assert record["scheduled_requests"] == 3
++    assert record["prefill_requests"] == 2
++    assert record["decode_batch_size"] == 1
++    assert record["prefill_tokens"] + record["decode_tokens"] == 101
++    assert sum(record["context_length_hist"]) == 3
++    assert len(record["context_length_hist"]) == 12
++    assert sum(record["chunked_prefill"]["chunk_size_hist"]) == 2
++    assert len(record["chunked_prefill"]["chunk_size_hist"]) == 9
++    assert record["chunked_prefill"]["first"] == 1
++    assert record["chunked_prefill"]["final"] == 1
++    assert record["preemptions"] == 1
++    assert record["kv"] == {
++        "total_blocks": 100,
++        "free_blocks": 40,
++        "used_blocks": 60,
++        "usage": 0.6,
++    }
++    assert record["prefix"]["local"]["hits"] == 64
++    assert record["moe_expert_load"] is None
++    assert record["complete_mono_ns"] >= record["submit_mono_ns"]
++    assert footer["record_type"] == "footer"
++    assert footer["written_records"] == 1
++
++
++def test_capture_record_matches_pre_refactor_golden(tmp_path, monkeypatch):
++    sched = scheduler(
++        {
++            "edge": request(0, 128),
++            "after": request(65, 129, was_chunk=True),
++            "decode": request(256, 257),
++        }
++    )
++    output = schedule_output(
++        {"edge": 128, "after": 64, "decode": 1},
++        context_ids={"after"},
++        new_ids={"edge"},
++        preempted={"old"},
++    )
++    rec, path = recorder(tmp_path)
++    zero_prefix = dict.fromkeys(opprof._PREFIX_FIELDS, 0)
++    start = (100, 200, {"local": zero_prefix, "external": None})
++    monkeypatch.setattr(opprof.time, "monotonic_ns", lambda: 300)
++
++    rec.begin(sched, output, start)
++    assert rec.finalize(output, graph("FULL", 193, 256))
++    rec.close()
++
++    record = read_jsonl(path)[0]
++    assert record == {
++        "schema": 1,
++        "engine_id": "dp0-pid1",
++        "step_index": 0,
++        "submit_wall_ns": 100,
++        "submit_mono_ns": 200,
++        "model_executed": True,
++        "scheduled_requests": 3,
++        "decode_batch_size": 1,
++        "prefill_requests": 2,
++        "prefill_tokens": 192,
++        "decode_tokens": 1,
++        "chunked_prefill": {
++            "first": 0,
++            "middle": 0,
++            "final": 1,
++            "unsplit": 1,
++            "tokens": 192,
++            "chunk_size_hist": [0, 0, 1, 1, 0, 0, 0, 0, 0],
++        },
++        "context_length_hist": [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
++        "preemptions": 1,
++        "queues": {"running": 3, "waiting": 2, "deferred": 1},
++        "kv": {
++            "total_blocks": 100,
++            "free_blocks": 40,
++            "used_blocks": 60,
++            "usage": 0.6,
++        },
++        "prefix": {"local": zero_prefix, "external": None},
++        "complete_mono_ns": 300,
++        "cudagraph": {
++            "hit": True,
++            "runtime_mode": "FULL",
++            "unpadded_tokens": 193,
++            "bucket_tokens": 256,
++            "padding_tokens": 63,
++        },
++        "moe_expert_load": None,
++        "dropped_records_before": 0,
++    }
++
++
++@pytest.mark.parametrize(
++    ("was_chunk", "end", "target", "expected"),
++    [
++        (False, 64, 100, "first"),
++        (True, 80, 100, "middle"),
++        (True, 100, 100, "final"),
++        (False, 100, 100, "unsplit"),
++    ],
++)
++def test_chunk_classification(was_chunk, end, target, expected):
++    assert opprof.classify_chunk(was_chunk, end, target) == expected
++
++
++def test_async_pairing_out_of_order_and_double_finalize(tmp_path):
++    sched = scheduler({"a": request(10, 11), "b": request(200, 201)})
++    first = schedule_output({"a": 1})
++    second = schedule_output({"b": 1})
++    rec, path = recorder(tmp_path)
++    rec.begin(sched, first, rec.capture_start(sched))
++    rec.begin(sched, second, rec.capture_start(sched))
++    assert rec.finalize(second, graph())
++    assert rec.finalize(first, graph("NONE"))
++    with pytest.raises(AssertionError, match="already finalized"):
++        rec.finalize(second, graph())
++    assert not rec._pending
++    rec.close()
++    records = read_jsonl(path)[:-1]
++    assert [record["step_index"] for record in records] == [1, 0]
++    assert records[0]["context_length_hist"][1] == 1
++    assert records[1]["context_length_hist"][0] == 1
++
++
++def test_disabled_noop_and_log_stats_fail_fast(tmp_path):
++    assert opprof.OpProfRecorder.create("", dp_rank=0, log_stats=False) is None
++    with pytest.raises(ValueError, match="requires log stats"):
++        opprof.OpProfRecorder.create(str(tmp_path), dp_rank=0, log_stats=False)
++    assert not list(tmp_path.iterdir())
++
++
++def test_bounded_queue_drop_accounting(tmp_path):
++    sched = scheduler({str(i): request(i, i + 1) for i in range(3)})
++    rec, path = recorder(tmp_path, capacity=1, start=False)
++    assert emit(rec, sched, schedule_output({"0": 1}))
++    assert not emit(rec, sched, schedule_output({"1": 1}))
++    rec.writer.start()
++    rec.writer._queue.join()
++    assert emit(rec, sched, schedule_output({"2": 1}))
++    rec.close()
++
++    first, after_drop, footer = read_jsonl(path)
++    assert first["step_index"] == 0
++    assert after_drop["step_index"] == 2
++    assert after_drop["dropped_records_before"] == 1
++    assert footer["encoded_records"] == 3
++    assert footer["written_records"] == 2
++    assert footer["dropped_records"] == 1
++
++
++def test_writer_enospc_is_exposed_and_shutdown_is_bounded(
++    tmp_path, monkeypatch, caplog
++):
++    sched = scheduler(
++        {
++            "first": request(0, 1),
++            "after_failure": request(1, 2),
++        }
++    )
++    rec, _ = recorder(tmp_path, capacity=1, start=False)
++    assert emit(rec, sched, schedule_output({"first": 1}))
++
++    real_file = rec.writer._file
++
++    def fail_enospc(*_args, **_kwargs):
++        raise OSError(errno.ENOSPC, "No space left on device")
++
++    failing_file = SimpleNamespace(
++        write=fail_enospc,
++        flush=fail_enospc,
++        close=real_file.close,
++    )
++    monkeypatch.setattr(rec.writer, "_file", failing_file)
++    caplog.set_level(logging.ERROR, logger=opprof.__name__)
++
++    rec.writer.start()
++    rec.writer._thread.join(timeout=1.0)
++    assert not rec.writer._thread.is_alive()
++
++    producer_result = emit(
++        rec, sched, schedule_output({"after_failure": 1})
++    )
++
++    closer = threading.Thread(target=rec.close, daemon=True)
++    closer.start()
++    closer.join(timeout=1.0)
++    assert not closer.is_alive(), "OpProf close blocked after writer failure"
++    assert not producer_result
++    assert rec.writer.dropped_records == 1
++    assert rec.failed
++    assert isinstance(rec.failure, OSError)
++    assert rec.failure.errno == errno.ENOSPC
++    errors = [
++        record
++        for record in caplog.records
++        if "OpProf writer failed" in record.getMessage()
++    ]
++    assert len(errors) == 1
++
++
++def test_shutdown_flush_is_idempotent(tmp_path):
++    sched = scheduler({"decode": request(8, 9)})
++    rec, path = recorder(tmp_path)
++    assert emit(rec, sched, schedule_output({"decode": 1}))
++    rec.close()
++    rec.close()
++    record, footer = read_jsonl(path)
++    assert record["step_index"] == 0
++    assert footer["written_records"] == 1
++    assert path.stat().st_size > 0
++
++
++def test_piecewise_cudagraph_record_preserved(tmp_path):
++    sched = scheduler({"decode": request(4096, 4097)})
++    output = schedule_output({"decode": 1})
++    rec, path = recorder(tmp_path)
++    assert emit(rec, sched, output, graph("PIECEWISE", 513, 520))
++    rec.close()
++    record = read_jsonl(path)[0]
++    assert record["cudagraph"] == {
++        "hit": True,
++        "runtime_mode": "PIECEWISE",
++        "unpadded_tokens": 513,
++        "bucket_tokens": 520,
++        "padding_tokens": 7,
++    }
++
++
++def test_zero_scheduled_tokens_finalize_without_cudagraph(tmp_path):
++    sched = scheduler({})
++    output = schedule_output({})
++    rec, path = recorder(tmp_path)
++    rec.begin(sched, output, rec.capture_start(sched))
++    assert rec._pending
++
++    assert rec.finalize(output, cudagraph_stat=None)
++    assert not rec._pending
++    rec.close()
++
++    record = read_jsonl(path)[0]
++    assert record["model_executed"] is False
++    assert record["scheduled_requests"] == 0
++    assert record["prefill_requests"] == 0
++    assert record["prefill_tokens"] == 0
++    assert record["decode_batch_size"] == 0
++    assert record["decode_tokens"] == 0
++    assert record["context_length_hist"] == [0] * 12
++    assert record["chunked_prefill"] == {
++        "first": 0,
++        "middle": 0,
++        "final": 0,
++        "unsplit": 0,
++        "tokens": 0,
++        "chunk_size_hist": [0] * 9,
++    }
++    assert record["cudagraph"] == {
++        "hit": False,
++        "runtime_mode": "NONE",
++        "unpadded_tokens": 0,
++        "bucket_tokens": 0,
++        "padding_tokens": 0,
++    }
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0003-Log-the-OpProf-output-path-at-startup.patch

@@ -0,0 +1,52 @@
+From 668cfb7e27e488454dbf09a4927b8a60d6d49b40 Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sat, 11 Jul 2026 17:32:27 +0800
+Subject: [PATCH 3/5] Log the OpProf output path at startup
+
+Assisted-by: OpenAI Codex
+---
+ tests/v1/core/test_opprof.py    | 1 +
+ vllm/v1/core/sched/scheduler.py | 2 ++
+ vllm/v1/opprof.py               | 1 +
+ 3 files changed, 4 insertions(+)
+
+diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
+index 9bfbfcc..79c1fae 100644
+--- a/tests/v1/core/test_opprof.py
++++ b/tests/v1/core/test_opprof.py
+@@ -336,6 +336,7 @@ def test_writer_enospc_is_exposed_and_shutdown_is_bounded(
+ def test_shutdown_flush_is_idempotent(tmp_path):
+     sched = scheduler({"decode": request(8, 9)})
+     rec, path = recorder(tmp_path)
++    assert rec.writer.path == path
+     assert emit(rec, sched, schedule_output({"decode": 1}))
+     rec.close()
+     rec.close()
+diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py
+index 303c562..769a02a 100644
+--- a/vllm/v1/core/sched/scheduler.py
++++ b/vllm/v1/core/sched/scheduler.py
+@@ -278,6 +278,8 @@ class Scheduler(SchedulerInterface):
+             dp_rank=self.parallel_config.data_parallel_index,
+             log_stats=self.log_stats,
+         )
++        if self.opprof is not None:
++            logger.info("OpProf telemetry enabled: %s", self.opprof.writer.path)
+ 
+         self.use_pp = self.parallel_config.pipeline_parallel_size > 1
+         self.use_v2_model_runner = vllm_config.use_v2_model_runner
+diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py
+index f0330d0..75f63de 100644
+--- a/vllm/v1/opprof.py
++++ b/vllm/v1/opprof.py
+@@ -68,6 +68,7 @@ class JSONLWriter:
+         start: bool = True,
+     ) -> None:
+         self._queue: queue.Queue[Any] = queue.Queue(capacity)
++        self.path = path
+         self._encoder = msgspec.json.Encoder()
+         self._file = path.open("xb", buffering=1 << 20)
+         self._thread = threading.Thread(target=self._run, daemon=True)
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0004-Exclude-OpProf-output-path-from-compile-cache-key.patch

@@ -0,0 +1,64 @@
+From 335da4abe60e0177872e0b7751e86eeec6756a2b Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sat, 11 Jul 2026 22:32:30 +0800
+Subject: [PATCH 4/5] Exclude OpProf output path from compile cache key
+
+---
+ tests/v1/core/test_opprof.py | 21 ++++++++++++++++++++-
+ vllm/envs.py                 |  1 +
+ 2 files changed, 21 insertions(+), 1 deletion(-)
+
+diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
+index 79c1fae..a820e7e 100644
+--- a/tests/v1/core/test_opprof.py
++++ b/tests/v1/core/test_opprof.py
+@@ -8,7 +8,7 @@ import logging
+ import sys
+ import threading
+ from pathlib import Path
+-from types import SimpleNamespace
++from types import ModuleType, SimpleNamespace
+ 
+ import msgspec
+ import pytest
+@@ -23,6 +23,25 @@ sys.modules[_SPEC.name] = opprof
+ _SPEC.loader.exec_module(opprof)
+ 
+ 
++def test_output_dir_is_not_a_compile_factor(monkeypatch: pytest.MonkeyPatch):
++    spec = importlib.util.spec_from_file_location(
++        "envs_standalone", _ROOT / "vllm" / "envs.py"
++    )
++    assert spec is not None and spec.loader is not None
++    envs = importlib.util.module_from_spec(spec)
++    monkeypatch.setitem(sys.modules, spec.name, envs)
++    spec.loader.exec_module(envs)
++
++    monkeypatch.setitem(sys.modules, "vllm", ModuleType("vllm"))
++    monkeypatch.setitem(sys.modules, "vllm.config", ModuleType("vllm.config"))
++    config_utils = ModuleType("vllm.config.utils")
++    config_utils.__dict__["normalize_value"] = lambda value: value
++    monkeypatch.setitem(sys.modules, "vllm.config.utils", config_utils)
++    monkeypatch.setenv("VLLM_OPPROF_DIR", "/tmp/opprof")
++
++    assert "VLLM_OPPROF_DIR" not in envs.compile_factors()
++
++
+ class CachedRequests:
+     def __init__(self, context_ids=()):
+         self.context_ids = set(context_ids)
+diff --git a/vllm/envs.py b/vllm/envs.py
+index b3093e9..5634708 100755
+--- a/vllm/envs.py
++++ b/vllm/envs.py
+@@ -2044,6 +2044,7 @@ def compile_factors() -> dict[str, object]:
+         "VLLM_LOGGING_CONFIG_PATH",
+         "VLLM_LOGGING_COLOR",
+         "VLLM_LOG_STATS_INTERVAL",
++        "VLLM_OPPROF_DIR",
+         "VLLM_DEBUG_LOG_API_SERVER_RESPONSE",
+         "VLLM_TUNED_CONFIG_FOLDER",
+         "VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR",
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0005-Keep-compile-factor-regression-import-light.patch

@@ -0,0 +1,24 @@
+From bbfa7176a6a3686a88ee66696f1ad8d754559d96 Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sat, 11 Jul 2026 22:38:08 +0800
+Subject: [PATCH 5/5] Keep compile-factor regression import-light
+
+---
+ tests/v1/core/test_opprof.py | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
+index a820e7e..0b8a8a1 100644
+--- a/tests/v1/core/test_opprof.py
++++ b/tests/v1/core/test_opprof.py
+@@ -37,6 +37,7 @@ def test_output_dir_is_not_a_compile_factor(monkeypatch: pytest.MonkeyPatch):
+     config_utils = ModuleType("vllm.config.utils")
+     config_utils.__dict__["normalize_value"] = lambda value: value
+     monkeypatch.setitem(sys.modules, "vllm.config.utils", config_utils)
++    monkeypatch.setitem(sys.modules, "torch", ModuleType("torch"))
+     monkeypatch.setenv("VLLM_OPPROF_DIR", "/tmp/opprof")
+ 
+     assert "VLLM_OPPROF_DIR" not in envs.compile_factors()
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0006-Checkpoint-OpProf-accounting-across-hard-kills.patch

@@ -0,0 +1,306 @@
+From f8b68f2452c424d22de4a69527a427207dcfbca5 Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sun, 12 Jul 2026 12:56:39 +0800
+Subject: [PATCH] Checkpoint OpProf accounting across hard kills
+
+---
+ tests/v1/core/test_opprof.py | 118 +++++++++++++++++++++++++++++++++++
+ vllm/v1/opprof.py            |  84 ++++++++++++++++++++++---
+ 2 files changed, 195 insertions(+), 7 deletions(-)
+
+diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
+index 0b8a8a1..007c5bb 100644
+--- a/tests/v1/core/test_opprof.py
++++ b/tests/v1/core/test_opprof.py
+@@ -5,6 +5,8 @@
+ import errno
+ import importlib.util
+ import logging
++import os
++import subprocess
+ import sys
+ import threading
+ from pathlib import Path
+@@ -121,6 +123,12 @@ def read_jsonl(path):
+     return [msgspec.json.decode(line) for line in path.read_bytes().splitlines()]
+ 
+ 
++def read_sidecar(path):
++    return msgspec.json.decode(
++        path.with_name(f"{path.name}.footer.json").read_bytes()
++    )
++
++
+ def test_import_light_and_approved_constants():
+     assert "torch" not in sys.modules
+     assert "vllm" not in sys.modules
+@@ -366,6 +374,116 @@ def test_shutdown_flush_is_idempotent(tmp_path):
+     assert path.stat().st_size > 0
+ 
+ 
++def test_sidecar_updates_are_atomic(tmp_path, monkeypatch):
++    writer = opprof.JSONLWriter(tmp_path / "atomic.jsonl", start=False)
++    writer._write_sidecar(
++        encoded_records=1,
++        written_records=1,
++        dropped_records=0,
++        last_step_index=0,
++        final=False,
++    )
++    original = writer.sidecar_path.read_bytes()
++    real_replace = os.replace
++    replacements = []
++
++    def inspect_replace(source, destination):
++        assert Path(destination) == writer.sidecar_path
++        assert writer.sidecar_path.read_bytes() == original
++        candidate = msgspec.json.decode(Path(source).read_bytes())
++        assert candidate["written_records"] == 2
++        replacements.append(candidate)
++        real_replace(source, destination)
++
++    monkeypatch.setattr(opprof.os, "replace", inspect_replace)
++    writer._write_sidecar(
++        encoded_records=3,
++        written_records=2,
++        dropped_records=1,
++        last_step_index=2,
++        final=False,
++    )
++    monkeypatch.setattr(opprof.os, "replace", real_replace)
++
++    assert len(replacements) == 1
++    assert read_sidecar(writer.path)["encoded_records"] == 3
++    assert not list(tmp_path.glob(".*.tmp-*"))
++    writer.close()
++
++
++def test_hard_kill_sidecar_balances_last_flush(tmp_path):
++    path = tmp_path / "hard-kill.jsonl"
++    child = """
++import importlib.util
++import sys
++import time
++from pathlib import Path
++
++source, output = sys.argv[1:]
++spec = importlib.util.spec_from_file_location("opprof_hard_kill", source)
++module = importlib.util.module_from_spec(spec)
++sys.modules[spec.name] = module
++spec.loader.exec_module(module)
++writer = module.JSONLWriter(Path(output))
++for step in range(3):
++    assert writer.submit({"schema": 1, "step_index": step})
++writer._queue.join()
++while not writer.sidecar_path.exists():
++    time.sleep(0.01)
++print("READY", flush=True)
++time.sleep(60)
++"""
++    process = subprocess.Popen(
++        [
++            sys.executable,
++            "-c",
++            child,
++            str(_ROOT / "vllm/v1/opprof.py"),
++            str(path),
++        ],
++        stdout=subprocess.PIPE,
++        text=True,
++    )
++    try:
++        assert process.stdout is not None
++        assert process.stdout.readline().strip() == "READY"
++        process.kill()
++        assert process.wait(timeout=5) < 0
++    finally:
++        if process.poll() is None:
++            process.kill()
++            process.wait(timeout=5)
++
++    records = read_jsonl(path)
++    sidecar = read_sidecar(path)
++    assert len(records) == 3
++    assert all(record.get("record_type") != "footer" for record in records)
++    assert sidecar["final"] is False
++    assert sidecar["written_records"] == len(records)
++    assert sidecar["encoded_records"] == (
++        sidecar["written_records"] + sidecar["dropped_records"]
++    )
++    assert sidecar["last_step_index"] == records[-1]["step_index"] == 2
++
++
++def test_clean_footer_and_final_sidecar_agree(tmp_path):
++    sched = scheduler({"decode": request(8, 9)})
++    rec, path = recorder(tmp_path)
++    assert emit(rec, sched, schedule_output({"decode": 1}))
++    rec.close()
++
++    record, footer = read_jsonl(path)
++    sidecar = read_sidecar(path)
++    assert sidecar["record_type"] == "footer_checkpoint"
++    assert sidecar["stream"] == path.name
++    assert sidecar["final"] is True
++    assert sidecar["last_step_index"] == record["step_index"] == 0
++    assert sidecar["checkpoint_wall_ns"] > 0
++    assert sidecar["flush_interval_seconds"] == opprof.FLUSH_INTERVAL_SECONDS
++    for counter in ("encoded_records", "written_records", "dropped_records"):
++        assert sidecar[counter] == footer[counter]
++
++
+ def test_piecewise_cudagraph_record_preserved(tmp_path):
+     sched = scheduler({"decode": request(4096, 4097)})
+     output = schedule_output({"decode": 1})
+diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py
+index 75f63de..28d9635 100644
+--- a/vllm/v1/opprof.py
++++ b/vllm/v1/opprof.py
+@@ -7,6 +7,7 @@ import queue
+ import threading
+ import time
+ from bisect import bisect_left
++from contextlib import suppress
+ from pathlib import Path
+ from typing import Any
+ 
+@@ -18,6 +19,7 @@ SCHEMA_VERSION = 1
+ CONTEXT_LENGTH_EDGES = tuple(1 << exponent for exponent in range(7, 18))
+ CHUNK_SIZE_EDGES = tuple(1 << exponent for exponent in range(4, 12))
+ DEFAULT_QUEUE_CAPACITY = 8192
++FLUSH_INTERVAL_SECONDS = 1.0
+ _CLOSE_TIMEOUT_SECONDS = 1.0
+ _STOP = object()
+ _PREFIX_FIELDS = (  # noqa: SIM905
+@@ -69,6 +71,7 @@ class JSONLWriter:
+     ) -> None:
+         self._queue: queue.Queue[Any] = queue.Queue(capacity)
+         self.path = path
++        self.sidecar_path = path.with_name(f"{path.name}.footer.json")
+         self._encoder = msgspec.json.Encoder()
+         self._file = path.open("xb", buffering=1 << 20)
+         self._thread = threading.Thread(target=self._run, daemon=True)
+@@ -78,6 +81,9 @@ class JSONLWriter:
+         self.failure: Exception | None = None
+         self.encoded_records = self.written_records = 0
+         self.dropped_records = self._unreported_drops = 0
++        self._checkpoint_encoded_records = 0
++        self._checkpoint_dropped_records = 0
++        self._last_written_step_index: int | None = None
+         if start:
+             self.start()
+ 
+@@ -119,33 +125,90 @@ class JSONLWriter:
+         if self._writer_unavailable():
+             return self._drop(pending)
+         try:
+-            self._queue.put_nowait(payload)
++            self._queue.put_nowait(
++                (
++                    payload,
++                    self.encoded_records,
++                    self.dropped_records,
++                    int(record["step_index"]),
++                )
++            )
+         except queue.Full:
+             return self._drop(pending)
+         self._unreported_drops = 0
+         return True
+ 
++    def _write_sidecar(
++        self,
++        *,
++        encoded_records: int,
++        written_records: int,
++        dropped_records: int,
++        last_step_index: int | None,
++        final: bool,
++    ) -> None:
++        sidecar = dict(
++            schema=SCHEMA_VERSION,
++            record_type="footer_checkpoint",
++            stream=self.path.name,
++            encoded_records=encoded_records,
++            written_records=written_records,
++            dropped_records=dropped_records,
++            last_step_index=last_step_index,
++            checkpoint_wall_ns=time.time_ns(),
++            flush_interval_seconds=FLUSH_INTERVAL_SECONDS,
++            final=final,
++        )
++        temporary_path = self.sidecar_path.with_name(
++            f".{self.sidecar_path.name}.tmp-{os.getpid()}-{threading.get_ident()}"
++        )
++        try:
++            with temporary_path.open("wb") as temporary_file:
++                temporary_file.write(self._encoder.encode(sidecar) + b"\n")
++                temporary_file.flush()
++            os.replace(temporary_path, self.sidecar_path)
++        finally:
++            with suppress(FileNotFoundError):
++                temporary_path.unlink()
++
++    def _flush_checkpoint(self) -> None:
++        self._file.flush()
++        self._write_sidecar(
++            encoded_records=self._checkpoint_encoded_records,
++            written_records=self.written_records,
++            dropped_records=self._checkpoint_dropped_records,
++            last_step_index=self._last_written_step_index,
++            final=False,
++        )
++
+     def _run(self) -> None:
+         buffered = 0
+         last_flush = time.monotonic()
+         try:
+             while True:
+                 try:
+-                    item = self._queue.get(timeout=1.0)
++                    item = self._queue.get(timeout=FLUSH_INTERVAL_SECONDS)
+                 except queue.Empty:
+-                    self._file.flush()
++                    self._flush_checkpoint()
+                     buffered = 0
+                     last_flush = time.monotonic()
+                     continue
+                 try:
+                     if item is _STOP:
+                         break
+-                    self._file.write(item)
+-                    buffered += len(item)
++                    payload, encoded, dropped, step_index = item
++                    self._file.write(payload)
++                    buffered += len(payload)
+                     self.written_records += 1
++                    self._checkpoint_encoded_records = encoded
++                    self._checkpoint_dropped_records = dropped
++                    self._last_written_step_index = step_index
+                     now = time.monotonic()
+-                    if buffered >= 1 << 20 or now - last_flush >= 1.0:
+-                        self._file.flush()
++                    if (
++                        buffered >= 1 << 20
++                        or now - last_flush >= FLUSH_INTERVAL_SECONDS
++                    ):
++                        self._flush_checkpoint()
+                         buffered = 0
+                         last_flush = now
+                 finally:
+@@ -163,6 +226,13 @@ class JSONLWriter:
+             try:
+                 self._file.write(self._encoder.encode(footer) + b"\n")
+                 self._file.flush()
++                self._write_sidecar(
++                    encoded_records=footer["encoded_records"],
++                    written_records=footer["written_records"],
++                    dropped_records=footer["dropped_records"],
++                    last_step_index=self._last_written_step_index,
++                    final=True,
++                )
+             except Exception as error:
+                 self._record_failure(error)
+             finally:
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0007-Recreate-scheduled-torch-profiler-between-windows.patch

@@ -0,0 +1,27 @@
+From 23450fb21ac255b0cf710f4ee965ee694921975d Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sun, 12 Jul 2026 13:12:52 +0800
+Subject: [PATCH] Recreate scheduled torch profiler between windows
+
+---
+ vllm/v1/worker/gpu_worker.py | 4 ++++
+ 1 file changed, 4 insertions(+)
+
+diff --git a/vllm/v1/worker/gpu_worker.py b/vllm/v1/worker/gpu_worker.py
+index 5e266a3..0058f96 100644
+--- a/vllm/v1/worker/gpu_worker.py
++++ b/vllm/v1/worker/gpu_worker.py
+@@ -978,6 +978,10 @@ class Worker(WorkerBase):
+                 logger.warning("Profiler was not started, nothing to stop.")
+                 return
+             self.profiler.stop()
++            # A scheduled torch.profiler.profile does not reset its schedule
++            # after stop(). Recreate it for the next /start_profile window.
++            if isinstance(self.profiler, TorchProfilerWrapper):
++                self.profiler = None
+ 
+     def execute_dummy_batch(self) -> None:
+         num_tokens = getattr(self.model_runner, "uniform_decode_query_len", 1)
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/README.md

@@ -0,0 +1,119 @@
+# vLLM 0.24.0 OpProf patch series
+
+## Goal
+
+Apply the accepted OpProf Layer-1 instrumentation to exactly vLLM `v0.24.0`
+at base commit `ee0da84ab9e04ac7610e28580af62c365e898389`. The series adds one
+scheduler-owned composition record per step without installing new runtime
+dependencies or changing GPU kernels.
+
+## Contents
+
+- `0001-Add-lightweight-per-step-OpProf-telemetry.patch`: adds the environment
+  switch, import-light JSONL recorder/writer, scheduler hooks, and reuse of the
+  existing CUDA-graph stat. Writer failures are exposed without blocking
+  producers or shutdown, and request histograms are accumulated in-place.
+- `0002-Add-standalone-OpProf-telemetry-tests.patch`: adds CPU-only tests that
+  load the recorder directly without importing or installing vLLM or torch,
+  including ENOSPC, golden-record, and zero-token regressions.
+- `0003-Log-the-OpProf-output-path-at-startup.patch`: logs the resolved JSONL
+  output path and covers it in the standalone shutdown test.
+- `0004-Exclude-OpProf-output-path-from-compile-cache-key.patch`: prevents the
+  per-run telemetry destination from invalidating vLLM's torch.compile/AOT
+  cache and adds an import-light regression test.
+- `0005-Keep-compile-factor-regression-import-light.patch`: isolates the new
+  regression from torch in full vLLM test environments.
+- `0006-Checkpoint-OpProf-accounting-across-hard-kills.patch`: atomically
+  checkpoints balanced writer counters beside each JSONL stream once per
+  flush interval, with clean-close and hard-kill regressions.
+- `0007-Recreate-scheduled-torch-profiler-between-windows.patch`: discards a
+  stopped scheduled torch-profiler wrapper so each subsequent official profile
+  endpoint call receives a fresh 2+8 schedule and emits its own trace.
+- `apply.sh`: verifies the exact base, refuses dirty/wrong revisions, applies
+  all numbered patches with `git am`, and exits successfully only when the
+  exact series is already applied directly on the required base.
+- `pytest-evidence.txt`: exact isolated test command, dependency versions, and
+  all-pass summary.
+
+The source branch tip used to generate the patches is
+`23450fb21ac255b0cf710f4ee965ee694921975d` (`opprof`).
+
+## Apply
+
+Prerequisite: a clean checkout whose `HEAD` is the exact base commit.
+
+```bash
+./patches/vllm-0.24.0-opprof/apply.sh /path/to/vllm-v0.24.0
+```
+
+Running the command again is a no-op only when the five matching patch commits
+are rooted directly at the required base. A partially applied series, dirty
+tree, unrelated commit, or any other `HEAD` is rejected instead of being
+guessed around.
+
+## Enable and output
+
+Set an absolute output directory before starting vLLM:
+
+```bash
+export VLLM_OPPROF_DIR=/absolute/path/to/run/opprof
+```
+
+Unset or empty disables the feature before recorder construction. Combining it
+with `--disable-log-stats` fails fast, as approved.
+
+Each EngineCore/DP scheduler writes one file named approximately
+`opprof-v1-dp0-pid1234-<start_ns>.jsonl`. Records contain schema/engine/step and
+timestamps; scheduled prefill/decode composition; first/middle/final/unsplit
+prefill chunks; 12-bin context and 9-bin chunk-size histograms; preemptions;
+running/waiting/deferred queues; KV blocks/usage; local/external prefix deltas;
+CUDA-graph hit/mode/bucket/padding; explicit null Layer-1 MoE load; and drop-gap
+accounting. A clean close writes a final writer-count footer in the stream.
+
+Every JSONL flush also atomically replaces
+`<stream>.footer.json` through a same-directory temporary file. The sidecar
+contains the encoded, written, and dropped counts through that durable flush,
+the last written step index, a wall-clock timestamp, the one-second flush
+interval, and whether it is final. Queue entries carry their submission
+ordinal and cumulative drops, so a periodic checkpoint always satisfies
+`encoded = written + dropped` without decoding records in the writer thread.
+On clean close the in-stream footer is authoritative and the final sidecar must
+agree with all three counters. If a hard kill prevents the in-stream footer,
+the latest sidecar is authoritative: the decoded data-line count must equal
+its `written_records`, its final data-line step must equal
+`last_step_index`, and its counters must balance. Data after that checkpoint
+may be lost, bounded by at most the configured one-second flush interval.
+
+The bounded queue holds 8192 encoded records. Producers never wait for disk;
+full queues or a failed writer drop the new record and report the gap on the
+next successful record. A writer I/O failure is exposed through recorder state,
+logged once, and cannot make shutdown wait indefinitely. The writer flushes at
+1 MiB, one second, or shutdown.
+
+## Test
+
+Only pytest and msgspec are required. `--confcutdir` prevents vLLM's global
+test configuration from importing its full dependency stack.
+
+```bash
+cd /path/to/vllm-v0.24.0
+uv run --no-project --with pytest --with msgspec \
+  pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q
+```
+
+Expected summary:
+
+```text
+18 passed in 1.09s
+```
+
+## Caveats
+
+- Layer 1 intentionally records no expert-load arrays. Exact routed experts
+  remain a separate Layer-2 run.
+- `PIECEWISE` means graph-wrapped compiled regions, not full-step graph replay.
+- Phase 2 must measure the always-on overhead; acceptance requires the upper
+  bound of the 95% confidence interval to remain below 3% for every primary
+  serving metric.
+- Primary campaign topology is TP1 on community BF16 Qwen3-30B-A3B, with TP2
+  and TP4 counterpoints. Record the selected MoE backend log every run.

patches/vllm-0.24.0-opprof/apply.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+base_commit=ee0da84ab9e04ac7610e28580af62c365e898389
+script_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
+repo=${1:-.}
+patches=("$script_dir"/0*.patch)
+
+git -C "$repo" rev-parse --git-dir >/dev/null
+if [[ -n $(git -C "$repo" status --porcelain) ]]; then
+    echo "Refusing to apply to a dirty worktree." >&2
+    exit 1
+fi
+if ((${#patches[@]} == 0)) || [[ ! -f ${patches[0]} ]]; then
+    echo "No numbered patch files found in $script_dir" >&2
+    exit 1
+fi
+
+head_commit=$(git -C "$repo" rev-parse HEAD)
+mapfile -t recent_commits < <(
+    git -C "$repo" rev-list --max-count="${#patches[@]}" --reverse HEAD
+)
+patches_match=true
+((${#recent_commits[@]} == ${#patches[@]})) || patches_match=false
+for i in "${!patches[@]}"; do
+    $patches_match || break
+    patch_id=$(git patch-id --stable <"${patches[$i]}" | awk '{print $1}')
+    commit_id=$(
+        git -C "$repo" show --pretty=format: "${recent_commits[$i]}" |
+            git patch-id --stable | awk '{print $1}'
+    )
+    [[ $patch_id == "$commit_id" ]] || patches_match=false
+done
+first_parent=""
+if ((${#recent_commits[@]} > 0)); then
+    first_parent=$(
+        git -C "$repo" rev-parse --verify "${recent_commits[0]}^" 2>/dev/null || true
+    )
+fi
+if $patches_match && [[ $first_parent == "$base_commit" ]]; then
+    echo "OpProf patch series is already applied."
+    exit 0
+fi
+
+if [[ $head_commit == "$base_commit" ]]; then
+    git -C "$repo" am "${patches[@]}"
+    echo "Applied OpProf patch series to $repo"
+    exit 0
+fi
+if $patches_match; then
+    echo "Refusing to treat OpProf patches as already applied:" >&2
+    echo "first patch parent is $first_parent, expected $base_commit" >&2
+    exit 1
+fi
+
+echo "Refusing to apply: HEAD is $head_commit; expected $base_commit" >&2
+echo "or the exact OpProf patch series rooted at that base." >&2
+exit 1

patches/vllm-0.24.0-opprof/pytest-evidence.txt

@@ -0,0 +1,16 @@
+OpProf standalone pytest evidence
+Date: 2026-07-12
+Source branch: opprof
+Source tip: 23450fb21ac255b0cf710f4ee965ee694921975d
+Base: ee0da84ab9e04ac7610e28580af62c365e898389 (v0.24.0)
+Environment: Python 3.11.13, pytest 9.1.1, msgspec 0.21.1
+vLLM installed: no
+torch installed in isolated test environment: no
+GPU/remote access: no
+
+Command:
+uv run --no-project --with pytest --with msgspec pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q
+
+Output:
+..................                                                       [100%]
+18 passed in 1.09s

runs/fidelity-headroom/analyze_existing.py

@@ -0,0 +1,508 @@
+#!/usr/bin/env python3
+"""Retrospective headroom audit for a fidelity-aware tuning harness.
+
+This analysis intentionally separates two questions:
+
+1. How many real cell evaluations does a simulator top-k shortlist already
+   need to recover the real optimum on the frozen SimFid surface?
+2. On the P6 anchor ladder, do Layer-1 engine features predict the next
+   anchor's feasibility better than outcome-only features from the same
+   current anchor?
+
+The second question is diagnostic rather than decision-bearing: it uses a
+small, already-observed single-workload surface and full current-anchor
+summaries.  It is a premise check for a future prospective early-probe study.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable
+
+import numpy as np
+
+
+SCHEMA = "fidelity-headroom-v1"
+DEFAULT_REGULARIZATION = 1.0
+REGULARIZATION_SENSITIVITY = (0.1, 1.0, 10.0)
+BOOTSTRAP_SEED = 20260714
+BOOTSTRAP_REPLICATES = 10_000
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def numeric(values: Iterable[float | int]) -> dict[str, Any]:
+    array = [float(value) for value in values]
+    return {
+        "n": len(array),
+        "min": min(array) if array else None,
+        "max": max(array) if array else None,
+        "distinct_n": len(set(array)),
+    }
+
+
+def score_buckets(scores: dict[str, float], tolerance: float) -> dict[str, int]:
+    if tolerance <= 0:
+        raise ValueError("score tolerance must be positive")
+    return {cell: math.floor(float(score) / tolerance) for cell, score in scores.items()}
+
+
+def topk_curve(
+    real_scores: dict[str, float],
+    simulated_scores: dict[str, float],
+    tolerance: float,
+) -> dict[str, Any]:
+    if set(real_scores) != set(simulated_scores):
+        raise ValueError("real and simulator score cells differ")
+    buckets = score_buckets(simulated_scores, tolerance)
+    ordered = sorted(
+        simulated_scores,
+        key=lambda cell: (-buckets[cell], -float(simulated_scores[cell]), cell),
+    )
+    real_best = max(float(value) for value in real_scores.values())
+    points = []
+    for nominal_k in range(1, len(ordered) + 1):
+        cutoff_bucket = buckets[ordered[nominal_k - 1]]
+        candidates = [cell for cell in ordered if buckets[cell] >= cutoff_bucket]
+        selected = max(candidates, key=lambda cell: (float(real_scores[cell]), cell))
+        selected_score = float(real_scores[selected])
+        points.append(
+            {
+                "nominal_k": nominal_k,
+                "expanded_k": len(candidates),
+                "candidates": candidates,
+                "selected_cell_after_real_final": selected,
+                "selected_real_score": selected_score,
+                "real_regret": 1.0 - selected_score / real_best,
+            }
+        )
+
+    minimum_k = {}
+    for name, threshold in (("zero", 1e-15), ("one_percent", 0.01), ("five_percent", 0.05)):
+        eligible = [point for point in points if point["real_regret"] <= threshold]
+        minimum_k[name] = (
+            {
+                "nominal_k": eligible[0]["nominal_k"],
+                "expanded_k": eligible[0]["expanded_k"],
+            }
+            if eligible
+            else None
+        )
+    return {
+        "real_best": real_best,
+        "minimum_k": minimum_k,
+        "points": points,
+    }
+
+
+@dataclass(frozen=True)
+class Transition:
+    cell: str
+    current_anchor: float
+    next_anchor: float
+    external: tuple[float, ...]
+    instrumentation: tuple[float, ...]
+    next_feasible: int
+
+
+EXTERNAL_FEATURES = (
+    "log_current_rate_per_gpu",
+    "log_next_over_current_rate",
+    "log2_tp",
+    "log2_mns",
+    "current_pass_rate",
+    "ttft_max_over_6s",
+    "tpot_max_over_50ms",
+    "exact_output_fraction",
+    "early_stopped",
+)
+
+INSTRUMENTATION_FEATURES = (
+    "waiting_mean",
+    "waiting_max",
+    "decode_batch_mean",
+    "decode_batch_cv",
+    "kv_usage_mean",
+    "kv_usage_max",
+    "graph_none_share",
+    "graph_full_share",
+    "padding_fraction",
+    "prefill_token_fraction",
+    "model_steps_per_second",
+)
+
+
+def _finite(value: float | int | None) -> float:
+    if value is None:
+        return 0.0
+    result = float(value)
+    if not math.isfinite(result):
+        raise ValueError(f"non-finite feature: {value}")
+    return result
+
+
+def build_transitions(phase6: dict[str, Any]) -> list[Transition]:
+    transitions = []
+    for cell, cell_result in sorted(phase6["cells"].items()):
+        anchors = sorted(cell_result["anchors"], key=lambda item: float(item["anchor"]))
+        for current, following in zip(anchors, anchors[1:]):
+            if following["accepted_feasible"] is None:
+                continue
+            primary = current["primary"]
+            next_primary = following["primary"]
+            layer = current["layer1"]
+            rate = float(primary["selection"]["offered_req_s_per_gpu"])
+            next_rate = float(next_primary["selection"]["offered_req_s_per_gpu"])
+            selected_count = int(primary["selection"]["count"])
+            if rate <= 0 or next_rate <= 0 or selected_count <= 0:
+                raise ValueError("rates and selected counts must be positive")
+            external = (
+                math.log(rate),
+                math.log(next_rate / rate),
+                math.log2(float(cell_result["tp"])),
+                math.log2(float(cell_result["mns"])),
+                float(primary["pass_rate"]),
+                _finite(primary["ttft_ms"]["max"]) / 6000.0,
+                _finite(primary["tpot_ms"]["max"]) / 50.0,
+                float(primary["exact_output_count"]) / selected_count,
+                float(bool(primary["early_stopped"])),
+            )
+            graph_shares = layer.get("graph_mode_shares", {})
+            prefill_tokens = _finite(layer["prefill_tokens"])
+            decode_tokens = _finite(layer["decode_tokens"])
+            instrumentation = (
+                _finite(layer["waiting_mean"]),
+                _finite(layer["waiting_max"]),
+                _finite(layer["decode_B_mean"]),
+                _finite(layer["decode_B_cv"]),
+                _finite(layer["kv_usage_mean"]),
+                _finite(layer["kv_usage_max"]),
+                float(graph_shares.get("NONE", 0.0)),
+                float(graph_shares.get("FULL", 0.0)),
+                _finite(layer["padding_fraction"]),
+                prefill_tokens / max(1.0, prefill_tokens + decode_tokens),
+                _finite(layer["model_steps"]) / float(primary["interval"]["elapsed_s"]),
+            )
+            transitions.append(
+                Transition(
+                    cell=cell,
+                    current_anchor=float(current["anchor"]),
+                    next_anchor=float(following["anchor"]),
+                    external=external,
+                    instrumentation=instrumentation,
+                    next_feasible=int(bool(following["accepted_feasible"])),
+                )
+            )
+    return transitions
+
+
+def _sigmoid(values: np.ndarray) -> np.ndarray:
+    clipped = np.clip(values, -30.0, 30.0)
+    return 1.0 / (1.0 + np.exp(-clipped))
+
+
+def _fit_logistic(x: np.ndarray, y: np.ndarray, regularization: float) -> np.ndarray:
+    weights = np.zeros(x.shape[1], dtype=np.float64)
+    penalty = np.eye(x.shape[1], dtype=np.float64)
+    penalty[0, 0] = 0.0
+    for _ in range(100):
+        probability = _sigmoid(x @ weights)
+        gradient = x.T @ (probability - y) / len(y)
+        gradient += regularization * penalty @ weights / len(y)
+        curvature = probability * (1.0 - probability)
+        hessian = (x.T * curvature) @ x / len(y)
+        hessian += regularization * penalty / len(y)
+        step = np.linalg.lstsq(hessian, gradient, rcond=None)[0]
+        weights -= step
+        if float(np.max(np.abs(step))) < 1e-9:
+            break
+    return weights
+
+
+def _classification_metrics(y: np.ndarray, probability: np.ndarray) -> dict[str, Any]:
+    if np.any(probability < 0.0) or np.any(probability > 1.0):
+        raise ValueError("classification probabilities must be in [0, 1]")
+    prediction = probability >= 0.5
+    true_positive = int(np.sum(prediction & (y == 1)))
+    true_negative = int(np.sum(~prediction & (y == 0)))
+    false_positive = int(np.sum(prediction & (y == 0)))
+    false_negative = int(np.sum(~prediction & (y == 1)))
+    positive_total = true_positive + false_negative
+    negative_total = true_negative + false_positive
+    balanced = 0.5 * (
+        true_positive / positive_total + true_negative / negative_total
+    )
+    clipped = np.clip(probability, 1e-12, 1.0 - 1e-12)
+    return {
+        "accuracy": float(np.mean(prediction == y)),
+        "balanced_accuracy": float(balanced),
+        "brier": float(np.mean((probability - y) ** 2)),
+        "log_loss": float(np.mean(-(y * np.log(clipped) + (1 - y) * np.log(1 - clipped)))),
+        "confusion": {
+            "true_positive": true_positive,
+            "true_negative": true_negative,
+            "false_positive": false_positive,
+            "false_negative": false_negative,
+        },
+    }
+
+
+def _mcnemar_exact_p(outcome_only_correct: int, instrumentation_only_correct: int) -> float:
+    discordant = outcome_only_correct + instrumentation_only_correct
+    if discordant == 0:
+        return 1.0
+    tail = sum(
+        math.comb(discordant, value)
+        for value in range(min(outcome_only_correct, instrumentation_only_correct) + 1)
+    ) / (2**discordant)
+    return min(1.0, 2.0 * tail)
+
+
+def grouped_predictions(
+    transitions: list[Transition],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> tuple[np.ndarray, np.ndarray, list[str]]:
+    probabilities = []
+    labels = []
+    test_cells = []
+    for held_out in sorted({transition.cell for transition in transitions}):
+        train = [transition for transition in transitions if transition.cell != held_out]
+        test = [transition for transition in transitions if transition.cell == held_out]
+
+        def row(transition: Transition) -> np.ndarray:
+            values = transition.external
+            if instrumentation_aware:
+                values += transition.instrumentation
+            return np.asarray((1.0, *values), dtype=np.float64)
+
+        x_train = np.stack([row(transition) for transition in train])
+        x_test = np.stack([row(transition) for transition in test])
+        y_train = np.asarray([transition.next_feasible for transition in train], dtype=np.float64)
+        mean = x_train[:, 1:].mean(axis=0)
+        standard_deviation = x_train[:, 1:].std(axis=0)
+        standard_deviation[standard_deviation < 1e-8] = 1.0
+        x_train[:, 1:] = (x_train[:, 1:] - mean) / standard_deviation
+        x_test[:, 1:] = (x_test[:, 1:] - mean) / standard_deviation
+        weights = _fit_logistic(x_train, y_train, regularization)
+        probabilities.extend(_sigmoid(x_test @ weights).tolist())
+        labels.extend(transition.next_feasible for transition in test)
+        test_cells.extend(held_out for _ in test)
+    return (
+        np.asarray(labels, dtype=np.int64),
+        np.asarray(probabilities, dtype=np.float64),
+        test_cells,
+    )
+
+
+def _group_bootstrap_delta(
+    y: np.ndarray,
+    outcome_probability: np.ndarray,
+    instrumentation_probability: np.ndarray,
+    cells: list[str],
+) -> dict[str, Any]:
+    groups = sorted(set(cells))
+    indices = {group: np.asarray([i for i, cell in enumerate(cells) if cell == group]) for group in groups}
+    random = np.random.default_rng(BOOTSTRAP_SEED)
+    accuracy_deltas = []
+    brier_deltas = []
+    for _ in range(BOOTSTRAP_REPLICATES):
+        sampled = random.choice(groups, size=len(groups), replace=True)
+        selected = np.concatenate([indices[group] for group in sampled])
+        selected_y = y[selected]
+        outcome = outcome_probability[selected]
+        instrumentation = instrumentation_probability[selected]
+        accuracy_deltas.append(
+            float(np.mean((instrumentation >= 0.5) == selected_y))
+            - float(np.mean((outcome >= 0.5) == selected_y))
+        )
+        brier_deltas.append(
+            float(np.mean((instrumentation - selected_y) ** 2))
+            - float(np.mean((outcome - selected_y) ** 2))
+        )
+    return {
+        "semantics": "group bootstrap over cells; diagnostic confidence interval",
+        "replicates": BOOTSTRAP_REPLICATES,
+        "seed": BOOTSTRAP_SEED,
+        "accuracy_delta_instrumentation_minus_outcome": {
+            "point": float(np.mean((instrumentation_probability >= 0.5) == y))
+            - float(np.mean((outcome_probability >= 0.5) == y)),
+            "ci95": [float(x) for x in np.percentile(accuracy_deltas, [2.5, 97.5])],
+        },
+        "brier_delta_instrumentation_minus_outcome": {
+            "point": float(np.mean((instrumentation_probability - y) ** 2))
+            - float(np.mean((outcome_probability - y) ** 2)),
+            "ci95": [float(x) for x in np.percentile(brier_deltas, [2.5, 97.5])],
+        },
+    }
+
+
+def transition_analysis(transitions: list[Transition]) -> dict[str, Any]:
+    sensitivity = {}
+    headline_payload = None
+    for regularization in REGULARIZATION_SENSITIVITY:
+        y, outcome_probability, cells = grouped_predictions(
+            transitions,
+            instrumentation_aware=False,
+            regularization=regularization,
+        )
+        instrumentation_y, instrumentation_probability, instrumentation_cells = grouped_predictions(
+            transitions,
+            instrumentation_aware=True,
+            regularization=regularization,
+        )
+        if not np.array_equal(y, instrumentation_y) or cells != instrumentation_cells:
+            raise AssertionError("model folds or labels differ")
+        outcome_correct = (outcome_probability >= 0.5) == y
+        instrumentation_correct = (instrumentation_probability >= 0.5) == y
+        payload = {
+            "outcome_only": _classification_metrics(y, outcome_probability),
+            "instrumentation_aware": _classification_metrics(y, instrumentation_probability),
+            "paired_correctness": {
+                "both_correct": int(np.sum(outcome_correct & instrumentation_correct)),
+                "outcome_only_correct": int(np.sum(outcome_correct & ~instrumentation_correct)),
+                "instrumentation_only_correct": int(np.sum(~outcome_correct & instrumentation_correct)),
+                "both_wrong": int(np.sum(~outcome_correct & ~instrumentation_correct)),
+            },
+            "bootstrap": _group_bootstrap_delta(
+                y,
+                outcome_probability,
+                instrumentation_probability,
+                cells,
+            ),
+        }
+        payload["paired_correctness"]["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+            payload["paired_correctness"]["outcome_only_correct"],
+            payload["paired_correctness"]["instrumentation_only_correct"],
+        )
+        sensitivity[str(regularization)] = payload
+        if regularization == DEFAULT_REGULARIZATION:
+            headline_payload = payload
+    assert headline_payload is not None
+    labels = [transition.next_feasible for transition in transitions]
+    accuracy_deltas = [
+        value["instrumentation_aware"]["accuracy"] - value["outcome_only"]["accuracy"]
+        for value in sensitivity.values()
+    ]
+    brier_deltas = [
+        value["instrumentation_aware"]["brier"] - value["outcome_only"]["brier"]
+        for value in sensitivity.values()
+    ]
+    return {
+        "status": "RETROSPECTIVE_DIAGNOSTIC_ONLY",
+        "estimand": "next-anchor feasibility from the full current-anchor summary",
+        "split": "leave-one-cell-out",
+        "model": "L2 logistic regression with train-fold standardization",
+        "external_features": list(EXTERNAL_FEATURES),
+        "instrumentation_features": list(INSTRUMENTATION_FEATURES),
+        "headline_regularization": DEFAULT_REGULARIZATION,
+        "headline": headline_payload,
+        "regularization_sensitivity": sensitivity,
+        "sensitivity_summary": {
+            "accuracy_delta_min_max": [min(accuracy_deltas), max(accuracy_deltas)],
+            "brier_delta_min_max": [min(brier_deltas), max(brier_deltas)],
+            "incremental_signal_verdict": "NEEDS_PROSPECTIVE_EVIDENCE",
+        },
+        "label_sanity": {
+            **numeric(labels),
+            "positive": sum(labels),
+            "negative": len(labels) - sum(labels),
+        },
+    }
+
+
+def analyze(simfid_path: Path, phase6_path: Path) -> dict[str, Any]:
+    simfid = json.loads(simfid_path.read_text())
+    phase6 = json.loads(phase6_path.read_text())
+    real_scores = {cell: float(score) for cell, score in simfid["real_scores"].items()}
+    topk = {}
+    for reading, payload in sorted(simfid["analyses"].items()):
+        tie = payload["metrics"]["tie_buckets"]["simulator"]
+        topk[reading] = topk_curve(
+            real_scores,
+            {cell: float(score) for cell, score in payload["simulated_scores"].items()},
+            float(tie["tolerance"]),
+        )
+    transitions = build_transitions(phase6)
+    transition_result = transition_analysis(transitions)
+    red_flags = []
+    if len(real_scores) != 12:
+        red_flags.append("unexpected_simfid_cell_count")
+    if len(transitions) == 0 or len(set(x.next_feasible for x in transitions)) != 2:
+        red_flags.append("transition_labels_missing_or_single_class")
+    if any(not math.isfinite(value) or value < 0 for value in real_scores.values()):
+        red_flags.append("invalid_real_score")
+    return {
+        "schema": SCHEMA,
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": "retrospective single-workload premise audit; not prospective contribution evidence",
+        "provenance": {
+            "simfid_metrics": str(simfid_path.resolve()),
+            "simfid_sha256": sha256_file(simfid_path),
+            "phase6_metrics": str(phase6_path.resolve()),
+            "phase6_sha256": sha256_file(phase6_path),
+        },
+        "topk_headroom": topk,
+        "next_anchor_prediction": transition_result,
+        "decision": {
+            "current_surface_can_show_selection_contribution": False,
+            "reason": (
+                "The strongest frozen-calibrated SLO reading reaches zero real regret "
+                "after real evaluation of its first two-cell tie bucket. A method that "
+                "requires one calibration probe and one final verification cannot use "
+                "this single task to demonstrate fewer real cell evaluations."
+            ),
+            "prospective_target": (
+                "Test whether internal features from a short, shared real probe reduce "
+                "the number or duration of full frontier evaluations relative to an "
+                "outcome-only model given the same probe."
+            ),
+        },
+        "sanity": {
+            "real_scores": numeric(real_scores.values()),
+            "simulator_readings": len(topk),
+            "transitions": len(transitions),
+            "transition_cells": len({transition.cell for transition in transitions}),
+            "red_flags": red_flags,
+            "invariants": {
+                "same_cells_all_readings": all(
+                    set(payload["simulated_scores"]) == set(real_scores)
+                    for payload in simfid["analyses"].values()
+                ),
+                "scores_nonnegative": all(value >= 0 for value in real_scores.values()),
+                "transition_features_finite": all(
+                    all(math.isfinite(value) for value in (*item.external, *item.instrumentation))
+                    for item in transitions
+                ),
+                "probabilities_bounded": True,
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--simfid-metrics", type=Path, required=True)
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(args.simfid_metrics, args.phase6_metrics)
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(json.dumps({"status": result["status"], "output": str(args.output)}, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_pilot.py

@@ -0,0 +1,293 @@
+#!/usr/bin/env python3
+"""Evaluate frozen outcome-only and instrumentation-aware policies on P1."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from analyze_existing import _classification_metrics, _mcnemar_exact_p
+from analyze_prefixes import (
+    PrefixExample,
+    _load_jsonl,
+    _prefix_features,
+    numeric,
+    policy_metrics,
+    predict_frozen_model,
+    sha256_file,
+)
+
+
+def result_path(run_root: Path, cell: str, level: str, replicate: int) -> Path:
+    return run_root / "cells" / cell / f"{level}-rep{replicate}" / "result.json"
+
+
+def requests_path(run_root: Path, cell: str, level: str, replicate: int) -> Path:
+    return run_root / "cells" / cell / f"{level}-rep{replicate}" / "requests.jsonl"
+
+
+def selection_for(
+    manifest: dict[str, Any], cell: str, level: str, replicate: int
+) -> dict[str, Any]:
+    role = f"{level}{replicate}"
+    return manifest["cells"][cell]["targets"][level]["selections"][role]
+
+
+def build_pilot_examples(
+    manifest: dict[str, Any], run_root: Path, cutoff_s: float
+) -> tuple[list[PrefixExample], list[dict[str, Any]], list[str]]:
+    examples = []
+    details = []
+    red_flags = []
+    for cell, config in sorted(manifest["cells"].items()):
+        stream_path = next((run_root / "cells" / cell / "opprof").glob("*.jsonl"))
+        stream = _load_jsonl(stream_path, require_key="submit_mono_ns")
+        for level in ("low", "high"):
+            results = [
+                json.loads(result_path(run_root, cell, level, replicate).read_text())
+                for replicate in (1, 2, 3)
+            ]
+            votes = [bool(result["feasible"]) for result in results]
+            adjudicated = sum(votes) >= 2
+            primary = results[0]
+            requests = _load_jsonl(requests_path(run_root, cell, level, 1))
+            exact_timestamps = sum(
+                request.get("completed_elapsed_s") is not None for request in requests
+            )
+            actual_outcomes = sum(
+                request.get("completed_mono_ns") is not None for request in requests
+            )
+            if exact_timestamps != actual_outcomes:
+                red_flags.append(f"timestamp_count_mismatch_{cell}_{level}")
+            expected = selection_for(manifest, cell, level, 1)
+            if int(primary["selection"]["count"]) != int(expected["selected_count"]):
+                red_flags.append(f"selection_count_mismatch_{cell}_{level}")
+            for result_key, manifest_key in (
+                ("request_id_order_sha256", "request_id_order_sha256"),
+                ("arrival_order_sha256", "arrival_order_sha256"),
+                ("raw_length_order_sha256", "input_length_order_sha256"),
+            ):
+                if primary["selection"][result_key] != expected[manifest_key]:
+                    red_flags.append(f"selection_hash_mismatch_{cell}_{level}_{result_key}")
+            start_ns = int(primary["interval"]["start_mono_ns"])
+            end_ns = start_ns + int(cutoff_s * 1e9)
+            records = [
+                record
+                for record in stream
+                if record.get("model_executed")
+                and start_ns <= int(record["submit_mono_ns"]) <= end_ns
+            ]
+            outcome, instrumentation, completion_source = _prefix_features(
+                primary=primary,
+                tp=int(config["tp"]),
+                max_num_seqs=int(config["mns"]),
+                requests=requests,
+                records=records,
+                cutoff_s=cutoff_s,
+            )
+            example = PrefixExample(
+                cell=cell,
+                anchor=float(primary["anchor"]),
+                cutoff_s=cutoff_s,
+                tp=int(config["tp"]),
+                full_elapsed_s=float(primary["interval"]["elapsed_s"]),
+                feasible=int(adjudicated),
+                primary_feasible=int(bool(primary["feasible"])),
+                outcome=outcome,
+                instrumentation=instrumentation,
+                completion_time_source=completion_source,
+            )
+            examples.append(example)
+            details.append(
+                {
+                    "cell": cell,
+                    "level": level,
+                    "anchor_rep1": primary["anchor"],
+                    "selected_count_rep1": primary["selection"]["count"],
+                    "votes": votes,
+                    "pass_rates": [result["pass_rate"] for result in results],
+                    "adjudicated_feasible": adjudicated,
+                    "primary_feasible": bool(primary["feasible"]),
+                    "actual_timestamped_outcomes": actual_outcomes,
+                    "selected_outcomes": len(requests),
+                    "prefix_layer1_records": len(records),
+                    "completion_time_source": completion_source,
+                }
+            )
+    return examples, details, red_flags
+
+
+def analyze(
+    manifest_path: Path,
+    model_path: Path,
+    run_root: Path,
+) -> dict[str, Any]:
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    models = json.loads(model_path.read_text(encoding="utf-8"))
+    state_path = run_root / "controller-state.json"
+    state = json.loads(state_path.read_text(encoding="utf-8"))
+    cutoff_s = float(models["cutoff_s"])
+    threshold = float(models["accept_probability"])
+    examples, details, red_flags = build_pilot_examples(manifest, run_root, cutoff_s)
+    labels = np.asarray([example.feasible for example in examples], dtype=np.int64)
+    outcome_probability = predict_frozen_model(models["models"]["outcome_only"], examples)
+    instrumentation_probability = predict_frozen_model(
+        models["models"]["instrumentation_aware"], examples
+    )
+    outcome_policy = policy_metrics(
+        examples, labels, outcome_probability, threshold
+    )
+    instrumentation_policy = policy_metrics(
+        examples, labels, instrumentation_probability, threshold
+    )
+    outcome_correct = (outcome_probability >= 0.5) == labels
+    instrumentation_correct = (instrumentation_probability >= 0.5) == labels
+    paired = {
+        "both_correct": int(np.sum(outcome_correct & instrumentation_correct)),
+        "outcome_only_correct": int(np.sum(outcome_correct & ~instrumentation_correct)),
+        "instrumentation_only_correct": int(np.sum(~outcome_correct & instrumentation_correct)),
+        "both_wrong": int(np.sum(~outcome_correct & ~instrumentation_correct)),
+    }
+    paired["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+        paired["outcome_only_correct"], paired["instrumentation_only_correct"]
+    )
+    for detail, outcome_p, instrumentation_p in zip(
+        details, outcome_probability, instrumentation_probability
+    ):
+        detail["outcome_probability_feasible"] = float(outcome_p)
+        detail["instrumentation_probability_feasible"] = float(instrumentation_p)
+
+    positive = int(np.sum(labels))
+    negative = len(labels) - positive
+    if state["status"] != "complete" or int(state["completed_cells"]) != 6:
+        red_flags.append("campaign_incomplete")
+    if positive < 3 or negative < 3:
+        red_flags.append("insufficient_label_balance")
+    if any(
+        detail["actual_timestamped_outcomes"] == 0 for detail in details
+    ):
+        red_flags.append("no_exact_request_timestamps")
+    if float(state["gpu_hours_total"]) >= float(state["hard_cap_h20_hours"]):
+        red_flags.append("hard_cap_exceeded")
+
+    outcome_errors = outcome_policy["false_accept"] + outcome_policy["false_reject"]
+    instrumentation_errors = (
+        instrumentation_policy["false_accept"]
+        + instrumentation_policy["false_reject"]
+    )
+    outcome_decisions = outcome_policy["early_accept"] + outcome_policy["early_reject"]
+    instrumentation_decisions = (
+        instrumentation_policy["early_accept"]
+        + instrumentation_policy["early_reject"]
+    )
+    outcome_reduction = outcome_policy["valid_cost_reduction_fraction"]
+    instrumentation_reduction = instrumentation_policy["valid_cost_reduction_fraction"]
+    cost_delta = (
+        instrumentation_reduction - outcome_reduction
+        if outcome_reduction is not None and instrumentation_reduction is not None
+        else None
+    )
+    data_valid = not red_flags
+    safety_gate = instrumentation_errors == 0 and instrumentation_errors <= outcome_errors
+    incremental_gate = (
+        instrumentation_decisions - outcome_decisions >= 3
+        or (cost_delta is not None and cost_delta >= 0.15)
+    )
+    pilot_pass = data_valid and safety_gate and incremental_gate
+
+    return {
+        "schema": "fidelity-prefix-pilot-result-v1",
+        "status": "PILOT_PASS" if pilot_pass else "PILOT_FAIL",
+        "scope": "held-out single-task gate; not paper-facing contribution evidence",
+        "provenance": {
+            "manifest": str(manifest_path.resolve()),
+            "manifest_sha256": sha256_file(manifest_path),
+            "frozen_models": str(model_path.resolve()),
+            "frozen_models_sha256": sha256_file(model_path),
+            "controller_state": str(state_path.resolve()),
+            "controller_state_sha256": sha256_file(state_path),
+        },
+        "cutoff_s": cutoff_s,
+        "threshold": threshold,
+        "examples": details,
+        "outcome_only": {
+            "classification": _classification_metrics(labels, outcome_probability),
+            "policy": outcome_policy,
+        },
+        "instrumentation_aware": {
+            "classification": _classification_metrics(labels, instrumentation_probability),
+            "policy": instrumentation_policy,
+        },
+        "paired_correctness": paired,
+        "gate": {
+            "data_valid": data_valid,
+            "safety_gate": safety_gate,
+            "incremental_gate": incremental_gate,
+            "additional_early_decisions": instrumentation_decisions - outcome_decisions,
+            "valid_cost_reduction_fraction_delta": cost_delta,
+            "opens_expanded_p2": pilot_pass,
+        },
+        "gpu": {
+            "actual_h20_hours": state["gpu_hours_total"],
+            "hard_cap_h20_hours": state["hard_cap_h20_hours"],
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "labels": {
+                **numeric(labels.tolist()),
+                "positive": positive,
+                "negative": negative,
+            },
+            "full_elapsed_s": numeric(example.full_elapsed_s for example in examples),
+            "remaining_h20_hours": numeric(
+                example.remaining_h20_hours for example in examples
+            ),
+            "outcome_probability": numeric(outcome_probability.tolist()),
+            "instrumentation_probability": numeric(
+                instrumentation_probability.tolist()
+            ),
+            "invariants": {
+                "examples_12": len(examples) == 12,
+                "cells_6": len({example.cell for example in examples}) == 6,
+                "ratios_bounded": bool(
+                    np.all((outcome_probability >= 0) & (outcome_probability <= 1))
+                    and np.all(
+                        (instrumentation_probability >= 0)
+                        & (instrumentation_probability <= 1)
+                    )
+                ),
+                "costs_nonnegative": all(
+                    example.remaining_h20_hours >= 0 for example in examples
+                ),
+                "all_cell_validations": all(
+                    all(cell["validation"]["invariants"].values())
+                    for cell in state["cells"].values()
+                ),
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--frozen-models", type=Path, required=True)
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(args.manifest, args.frozen_models, args.run_root)
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(json.dumps({
+        "status": result["status"],
+        "gate": result["gate"],
+        "sanity_red_flags": result["sanity"]["red_flags"],
+    }, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_prefixes.py

@@ -0,0 +1,629 @@
+#!/usr/bin/env python3
+"""Retrospective, leakage-bounded audit of short real-probe prefixes.
+
+The outcome-only and instrumentation-aware models receive the same trial
+prefix.  The latter differs only by Layer-1 engine state.  Existing Phase-6
+request artifacts predate exact completion timestamps, so their completion
+time is reconstructed from arrival + TTFT + token intervals and is explicitly
+marked approximate.  New artifacts use ``completed_elapsed_s`` directly.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable
+
+import numpy as np
+
+from analyze_existing import (
+    DEFAULT_REGULARIZATION,
+    REGULARIZATION_SENSITIVITY,
+    _classification_metrics,
+    _fit_logistic,
+    _group_bootstrap_delta,
+    _mcnemar_exact_p,
+    _sigmoid,
+)
+
+
+SCHEMA = "fidelity-prefix-v1"
+DEFAULT_CUTOFFS = (5.0, 10.0, 15.0, 20.0)
+POLICY_THRESHOLDS = (0.8, 0.9, 0.95)
+
+OUTCOME_FEATURES = (
+    "log_offered_rate_per_gpu",
+    "log2_tp",
+    "log2_max_num_seqs",
+    "admitted_fraction",
+    "completed_over_admitted",
+    "completed_pass_rate",
+    "completed_fail_fraction_of_total",
+    "outstanding_over_admitted",
+    "ttft_max_over_slo_max",
+    "ttft_mean_over_slo_max",
+    "tpot_max_over_slo",
+    "tpot_mean_over_slo",
+    "admitted_input_tokens_mean_over_limit",
+)
+
+INSTRUMENTATION_FEATURES = (
+    "model_steps_per_second",
+    "waiting_mean",
+    "waiting_max",
+    "waiting_nonzero_share",
+    "running_mean",
+    "running_max",
+    "decode_batch_mean",
+    "decode_batch_max",
+    "decode_batch_cv",
+    "kv_usage_mean",
+    "kv_usage_max",
+    "kv_usage_end_minus_start",
+    "graph_none_share",
+    "graph_full_share",
+    "padding_fraction",
+    "prefill_token_fraction",
+    "preemptions",
+)
+
+
+@dataclass(frozen=True)
+class PrefixExample:
+    cell: str
+    anchor: float
+    cutoff_s: float
+    tp: int
+    full_elapsed_s: float
+    feasible: int
+    primary_feasible: int
+    outcome: tuple[float, ...]
+    instrumentation: tuple[float, ...]
+    completion_time_source: str
+
+    @property
+    def remaining_h20_hours(self) -> float:
+        return self.tp * max(0.0, self.full_elapsed_s - self.cutoff_s) / 3600.0
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def numeric(values: Iterable[float | int]) -> dict[str, Any]:
+    array = [float(value) for value in values]
+    return {
+        "n": len(array),
+        "min": min(array) if array else None,
+        "max": max(array) if array else None,
+        "distinct_n": len(set(array)),
+    }
+
+
+def _cv(values: list[float]) -> float:
+    if not values:
+        return 0.0
+    array = np.asarray(values, dtype=np.float64)
+    mean = float(array.mean())
+    return float(array.std(ddof=0) / mean) if mean else 0.0
+
+
+def completion_elapsed_s(request: dict[str, Any]) -> tuple[float | None, str]:
+    exact = request.get("completed_elapsed_s")
+    if exact is not None:
+        value = float(exact)
+        if value < 0 or not math.isfinite(value):
+            raise ValueError(f"invalid completed_elapsed_s={exact}")
+        return value, "exact_monotonic"
+    if not request.get("success"):
+        return None, "unobserved_failure"
+    required = (
+        request.get("arrival_s"),
+        request.get("ttft_ms"),
+        request.get("tpot_ms"),
+        request.get("completion_tokens"),
+    )
+    if any(value is None for value in required):
+        return None, "unobserved_failure"
+    arrival_s, ttft_ms, tpot_ms, completion_tokens = required
+    value = float(arrival_s) + (
+        float(ttft_ms) + max(int(completion_tokens) - 1, 0) * float(tpot_ms)
+    ) / 1000.0
+    if value < 0 or not math.isfinite(value):
+        raise ValueError(f"invalid reconstructed completion time={value}")
+    return value, "reconstructed_from_latency"
+
+
+def _load_jsonl(path: Path, *, require_key: str | None = None) -> list[dict[str, Any]]:
+    records = []
+    with path.open(encoding="utf-8") as source:
+        for line in source:
+            item = json.loads(line)
+            if require_key is None or require_key in item:
+                records.append(item)
+    return records
+
+
+def _anchor_directory(cell_root: Path, anchor: float) -> Path:
+    matches = []
+    for result_path in cell_root.glob("anchor-*/result.json"):
+        payload = json.loads(result_path.read_text(encoding="utf-8"))
+        if math.isclose(float(payload["anchor"]), anchor, rel_tol=0.0, abs_tol=1e-15):
+            matches.append(result_path.parent)
+    if len(matches) != 1:
+        raise ValueError(f"expected one primary directory for anchor {anchor}: {matches}")
+    return matches[0]
+
+
+def _prefix_features(
+    *,
+    primary: dict[str, Any],
+    tp: int,
+    max_num_seqs: int,
+    requests: list[dict[str, Any]],
+    records: list[dict[str, Any]],
+    cutoff_s: float,
+) -> tuple[tuple[float, ...], tuple[float, ...], str]:
+    admitted = [request for request in requests if float(request["arrival_s"]) <= cutoff_s]
+    completed = []
+    sources = set()
+    for request in requests:
+        completed_s, source = completion_elapsed_s(request)
+        if completed_s is None or completed_s > cutoff_s:
+            continue
+        completed.append(request)
+        sources.add(source)
+    if not admitted or not records:
+        raise ValueError("prefix has no admitted requests or Layer-1 records")
+    if any(request not in admitted for request in completed):
+        raise ValueError("completed request was not admitted inside prefix")
+
+    total = len(requests)
+    passed = sum(bool(request["slo_pass"]) for request in completed)
+    ttft = [float(request["ttft_ms"]) for request in completed if request["ttft_ms"] is not None]
+    tpot = [float(request["tpot_ms"]) for request in completed if request["tpot_ms"] is not None]
+    offered_rate = float(primary["selection"]["offered_req_s_per_gpu"])
+    if offered_rate <= 0 or total <= 0:
+        raise ValueError("offered rate and selected request count must be positive")
+
+    outcome = (
+        math.log(offered_rate),
+        math.log2(float(tp)),
+        math.log2(float(max_num_seqs)),
+        len(admitted) / total,
+        len(completed) / len(admitted),
+        passed / max(1, len(completed)),
+        (len(completed) - passed) / total,
+        (len(admitted) - len(completed)) / len(admitted),
+        max(ttft, default=0.0) / 6000.0,
+        float(np.mean(ttft)) / 6000.0 if ttft else 0.0,
+        max(tpot, default=0.0) / 50.0,
+        float(np.mean(tpot)) / 50.0 if tpot else 0.0,
+        float(np.mean([float(request["raw_input_tokens"]) for request in admitted])) / 8192.0,
+    )
+
+    waiting = [float(record["queues"]["waiting"]) for record in records]
+    running = [float(record["queues"]["running"]) for record in records]
+    decode_batch = [float(record["decode_batch_size"]) for record in records]
+    kv_usage = [float(record["kv"]["usage"]) for record in records]
+    graph_modes = [str(record["cudagraph"]["runtime_mode"]) for record in records]
+    bucket_tokens = sum(int(record["cudagraph"]["bucket_tokens"]) for record in records)
+    padding_tokens = sum(int(record["cudagraph"]["padding_tokens"]) for record in records)
+    prefill_tokens = sum(int(record["prefill_tokens"]) for record in records)
+    decode_tokens = sum(int(record["decode_tokens"]) for record in records)
+    instrumentation = (
+        len(records) / cutoff_s,
+        float(np.mean(waiting)),
+        max(waiting),
+        sum(value > 0 for value in waiting) / len(waiting),
+        float(np.mean(running)),
+        max(running),
+        float(np.mean(decode_batch)),
+        max(decode_batch),
+        _cv(decode_batch),
+        float(np.mean(kv_usage)),
+        max(kv_usage),
+        kv_usage[-1] - kv_usage[0],
+        graph_modes.count("NONE") / len(graph_modes),
+        graph_modes.count("FULL") / len(graph_modes),
+        padding_tokens / max(1, bucket_tokens),
+        prefill_tokens / max(1, prefill_tokens + decode_tokens),
+        float(sum(int(record["preemptions"]) for record in records)),
+    )
+    completion_source = "+".join(sorted(sources)) if sources else "none_completed"
+    return outcome, instrumentation, completion_source
+
+
+def build_examples(
+    phase6: dict[str, Any],
+    raw_root: Path,
+    cutoff_s: float,
+) -> list[PrefixExample]:
+    examples = []
+    for cell, cell_result in sorted(phase6["cells"].items()):
+        cell_root = raw_root / cell
+        stream_path = next((cell_root / "opprof").glob("*.jsonl"))
+        stream = _load_jsonl(stream_path, require_key="submit_mono_ns")
+        for anchor in cell_result["anchors"]:
+            primary = anchor["primary"]
+            full_elapsed_s = float(primary["interval"]["elapsed_s"])
+            if full_elapsed_s + 1e-9 < cutoff_s:
+                continue
+            anchor_value = float(primary["anchor"])
+            anchor_root = _anchor_directory(cell_root, anchor_value)
+            requests = _load_jsonl(anchor_root / "requests.jsonl")
+            start_ns = int(primary["interval"]["start_mono_ns"])
+            end_ns = start_ns + int(cutoff_s * 1e9)
+            records = [
+                record
+                for record in stream
+                if record.get("model_executed")
+                and start_ns <= int(record["submit_mono_ns"]) <= end_ns
+            ]
+            outcome, instrumentation, source = _prefix_features(
+                primary=primary,
+                tp=int(cell_result["tp"]),
+                max_num_seqs=int(cell_result["mns"]),
+                requests=requests,
+                records=records,
+                cutoff_s=cutoff_s,
+            )
+            examples.append(
+                PrefixExample(
+                    cell=cell,
+                    anchor=anchor_value,
+                    cutoff_s=cutoff_s,
+                    tp=int(cell_result["tp"]),
+                    full_elapsed_s=full_elapsed_s,
+                    feasible=int(bool(anchor["accepted_feasible"])),
+                    primary_feasible=int(bool(primary["feasible"])),
+                    outcome=outcome,
+                    instrumentation=instrumentation,
+                    completion_time_source=source,
+                )
+            )
+    return examples
+
+
+def grouped_predictions(
+    examples: list[PrefixExample],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> tuple[np.ndarray, np.ndarray, list[str]]:
+    probabilities = []
+    labels = []
+    groups = []
+    for held_out in sorted({example.cell for example in examples}):
+        train = [example for example in examples if example.cell != held_out]
+        test = [example for example in examples if example.cell == held_out]
+
+        def row(example: PrefixExample) -> np.ndarray:
+            values = example.outcome
+            if instrumentation_aware:
+                values += example.instrumentation
+            return np.asarray((1.0, *values), dtype=np.float64)
+
+        x_train = np.stack([row(example) for example in train])
+        x_test = np.stack([row(example) for example in test])
+        y_train = np.asarray([example.feasible for example in train], dtype=np.float64)
+        if len(set(y_train.tolist())) != 2:
+            raise ValueError(f"training fold for {held_out} has a single label")
+        mean = x_train[:, 1:].mean(axis=0)
+        standard_deviation = x_train[:, 1:].std(axis=0)
+        standard_deviation[standard_deviation < 1e-8] = 1.0
+        x_train[:, 1:] = (x_train[:, 1:] - mean) / standard_deviation
+        x_test[:, 1:] = (x_test[:, 1:] - mean) / standard_deviation
+        weights = _fit_logistic(x_train, y_train, regularization)
+        probabilities.extend(_sigmoid(x_test @ weights).tolist())
+        labels.extend(example.feasible for example in test)
+        groups.extend(held_out for _ in test)
+    return (
+        np.asarray(labels, dtype=np.int64),
+        np.asarray(probabilities, dtype=np.float64),
+        groups,
+    )
+
+
+def fit_frozen_model(
+    examples: list[PrefixExample],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> dict[str, Any]:
+    def row(example: PrefixExample) -> np.ndarray:
+        values = example.outcome
+        if instrumentation_aware:
+            values += example.instrumentation
+        return np.asarray((1.0, *values), dtype=np.float64)
+
+    matrix = np.stack([row(example) for example in examples])
+    labels = np.asarray([example.feasible for example in examples], dtype=np.float64)
+    if len(set(labels.tolist())) != 2:
+        raise ValueError("frozen model requires both feasibility labels")
+    mean = matrix[:, 1:].mean(axis=0)
+    standard_deviation = matrix[:, 1:].std(axis=0)
+    standard_deviation[standard_deviation < 1e-8] = 1.0
+    standardized = matrix.copy()
+    standardized[:, 1:] = (standardized[:, 1:] - mean) / standard_deviation
+    weights = _fit_logistic(standardized, labels, regularization)
+    probabilities = _sigmoid(standardized @ weights)
+    names = list(OUTCOME_FEATURES)
+    if instrumentation_aware:
+        names.extend(INSTRUMENTATION_FEATURES)
+    return {
+        "instrumentation_aware": instrumentation_aware,
+        "regularization": regularization,
+        "feature_names": names,
+        "feature_mean": mean.tolist(),
+        "feature_standard_deviation": standard_deviation.tolist(),
+        "weights_with_intercept_first": weights.tolist(),
+        "training_classification": _classification_metrics(labels, probabilities),
+    }
+
+
+def predict_frozen_model(
+    model: dict[str, Any],
+    examples: list[PrefixExample],
+) -> np.ndarray:
+    instrumentation_aware = bool(model["instrumentation_aware"])
+    rows = []
+    for example in examples:
+        values = example.outcome
+        if instrumentation_aware:
+            values += example.instrumentation
+        rows.append((1.0, *values))
+    matrix = np.asarray(rows, dtype=np.float64)
+    mean = np.asarray(model["feature_mean"], dtype=np.float64)
+    standard_deviation = np.asarray(
+        model["feature_standard_deviation"], dtype=np.float64
+    )
+    weights = np.asarray(model["weights_with_intercept_first"], dtype=np.float64)
+    if matrix.shape[1] != len(weights) or matrix.shape[1] - 1 != len(mean):
+        raise ValueError("frozen model feature dimensions do not match examples")
+    matrix[:, 1:] = (matrix[:, 1:] - mean) / standard_deviation
+    return _sigmoid(matrix @ weights)
+
+
+def policy_metrics(
+    examples: list[PrefixExample],
+    labels: np.ndarray,
+    probabilities: np.ndarray,
+    threshold: float,
+) -> dict[str, Any]:
+    accept = probabilities >= threshold
+    reject = probabilities <= 1.0 - threshold
+    decide = accept | reject
+    prediction = accept.astype(np.int64)
+    correct = prediction == labels
+    remaining = np.asarray(
+        [example.remaining_h20_hours for example in examples], dtype=np.float64
+    )
+    full_cost = sum(example.tp * example.full_elapsed_s / 3600.0 for example in examples)
+    saved = float(np.sum(remaining[decide]))
+    correct_saved = float(np.sum(remaining[decide & correct]))
+    invalid_saved = float(np.sum(remaining[decide & ~correct]))
+
+    def describe(mask: np.ndarray) -> list[dict[str, Any]]:
+        return [
+            {
+                "cell": example.cell,
+                "anchor": example.anchor,
+                "label_feasible": bool(label),
+                "probability_feasible": float(probability),
+                "remaining_h20_hours": example.remaining_h20_hours,
+            }
+            for example, label, probability, selected in zip(
+                examples, labels, probabilities, mask
+            )
+            if selected
+        ]
+
+    return {
+        "threshold": threshold,
+        "early_accept": int(np.sum(accept)),
+        "early_reject": int(np.sum(reject)),
+        "abstain_continue_full": int(np.sum(~decide)),
+        "false_accept": int(np.sum(accept & (labels == 0))),
+        "false_reject": int(np.sum(reject & (labels == 1))),
+        "false_accept_examples": describe(accept & (labels == 0)),
+        "false_reject_examples": describe(reject & (labels == 1)),
+        "decision_coverage": float(np.mean(decide)),
+        "full_trial_h20_hours": float(full_cost),
+        "remaining_h20_hours_at_cutoff": float(np.sum(remaining)),
+        "saved_h20_hours_if_decisions_used": saved,
+        "correctly_saved_h20_hours": correct_saved,
+        "invalidly_saved_h20_hours": invalid_saved,
+        "valid_zero_error_policy": bool(np.all(correct[decide])),
+        "valid_cost_reduction_fraction": (
+            correct_saved / full_cost if invalid_saved == 0.0 and full_cost else None
+        ),
+    }
+
+
+def analyze_cutoff(examples: list[PrefixExample]) -> dict[str, Any]:
+    sensitivity = {}
+    headline = None
+    for regularization in REGULARIZATION_SENSITIVITY:
+        labels, outcome_probability, groups = grouped_predictions(
+            examples,
+            instrumentation_aware=False,
+            regularization=regularization,
+        )
+        instrument_labels, instrument_probability, instrument_groups = grouped_predictions(
+            examples,
+            instrumentation_aware=True,
+            regularization=regularization,
+        )
+        if not np.array_equal(labels, instrument_labels) or groups != instrument_groups:
+            raise AssertionError("paired folds or labels differ")
+        if groups != [example.cell for example in examples]:
+            raise AssertionError("prediction order differs from example order")
+        outcome_correct = (outcome_probability >= 0.5) == labels
+        instrument_correct = (instrument_probability >= 0.5) == labels
+        result = {
+            "outcome_only": {
+                "classification": _classification_metrics(labels, outcome_probability),
+                "policies": [
+                    policy_metrics(examples, labels, outcome_probability, threshold)
+                    for threshold in POLICY_THRESHOLDS
+                ],
+            },
+            "instrumentation_aware": {
+                "classification": _classification_metrics(labels, instrument_probability),
+                "policies": [
+                    policy_metrics(examples, labels, instrument_probability, threshold)
+                    for threshold in POLICY_THRESHOLDS
+                ],
+            },
+            "paired_correctness": {
+                "both_correct": int(np.sum(outcome_correct & instrument_correct)),
+                "outcome_only_correct": int(np.sum(outcome_correct & ~instrument_correct)),
+                "instrumentation_only_correct": int(np.sum(~outcome_correct & instrument_correct)),
+                "both_wrong": int(np.sum(~outcome_correct & ~instrument_correct)),
+            },
+            "bootstrap": _group_bootstrap_delta(
+                labels,
+                outcome_probability,
+                instrument_probability,
+                groups,
+            ),
+        }
+        paired = result["paired_correctness"]
+        paired["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+            paired["outcome_only_correct"], paired["instrumentation_only_correct"]
+        )
+        sensitivity[str(regularization)] = result
+        if regularization == DEFAULT_REGULARIZATION:
+            headline = result
+    assert headline is not None
+    labels = [example.feasible for example in examples]
+    return {
+        "examples": len(examples),
+        "cells": len({example.cell for example in examples}),
+        "label_sanity": {
+            **numeric(labels),
+            "positive": sum(labels),
+            "negative": len(labels) - sum(labels),
+            "primary_adjudicated_disagreements": sum(
+                example.feasible != example.primary_feasible for example in examples
+            ),
+        },
+        "completion_time_sources": {
+            source: sum(example.completion_time_source == source for example in examples)
+            for source in sorted({example.completion_time_source for example in examples})
+        },
+        "headline_regularization": DEFAULT_REGULARIZATION,
+        "headline": headline,
+        "regularization_sensitivity": sensitivity,
+        "remaining_h20_hours": numeric(
+            example.remaining_h20_hours for example in examples
+        ),
+    }
+
+
+def analyze(
+    phase6_path: Path,
+    raw_root: Path,
+    cutoffs: tuple[float, ...],
+) -> dict[str, Any]:
+    phase6 = json.loads(phase6_path.read_text(encoding="utf-8"))
+    by_cutoff = {}
+    red_flags = []
+    for cutoff in cutoffs:
+        examples = build_examples(phase6, raw_root, cutoff)
+        if len({example.feasible for example in examples}) != 2:
+            red_flags.append(f"single_label_at_{cutoff:g}s")
+            continue
+        by_cutoff[f"{cutoff:g}"] = analyze_cutoff(examples)
+        if len({example.cell for example in examples}) != 12:
+            red_flags.append(f"incomplete_cells_at_{cutoff:g}s")
+        if not all(
+            math.isfinite(value)
+            for example in examples
+            for value in (*example.outcome, *example.instrumentation)
+        ):
+            red_flags.append(f"nonfinite_features_at_{cutoff:g}s")
+
+    headline_deltas = {
+        cutoff: {
+            "accuracy": (
+                result["headline"]["instrumentation_aware"]["classification"]["accuracy"]
+                - result["headline"]["outcome_only"]["classification"]["accuracy"]
+            ),
+            "brier": (
+                result["headline"]["instrumentation_aware"]["classification"]["brier"]
+                - result["headline"]["outcome_only"]["classification"]["brier"]
+            ),
+        }
+        for cutoff, result in by_cutoff.items()
+    }
+    return {
+        "schema": SCHEMA,
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": (
+            "retrospective single-workload prefix diagnostic; model selection, "
+            "threshold choice, and contribution claims require held-out prospective tasks"
+        ),
+        "estimand": (
+            "2-of-3 adjudicated anchor feasibility from the first primary trial's "
+            "identical short real prefix"
+        ),
+        "split": "leave-one-configuration-cell-out",
+        "model": "same L2 logistic model and folds; instrumentation model appends Layer-1 features",
+        "outcome_features": list(OUTCOME_FEATURES),
+        "instrumentation_features": list(INSTRUMENTATION_FEATURES),
+        "provenance": {
+            "phase6_metrics": str(phase6_path.resolve()),
+            "phase6_metrics_sha256": sha256_file(phase6_path),
+            "raw_root": str(raw_root.resolve()),
+        },
+        "cutoffs_s": list(cutoffs),
+        "cutoffs": by_cutoff,
+        "headline_incremental_deltas": headline_deltas,
+        "decision": {
+            "contribution_established": False,
+            "reason": (
+                "This dataset contains one workload and reconstructed rather than exact request "
+                "completion times. Three TP4 primary trials also disagree with their 2-of-3 "
+                "labels. It can reject a missing-signal premise but cannot establish "
+                "generalization or a paper-facing cost reduction."
+            ),
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "cutoff_count": len(by_cutoff),
+            "invariants": {
+                "cutoffs_positive": all(cutoff > 0 for cutoff in cutoffs),
+                "paired_same_model_family": True,
+                "probabilities_checked_in_unit_interval": True,
+                "full_trial_label_not_used_as_feature": True,
+                "records_strictly_prefix_sliced": True,
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--raw-root", type=Path, required=True)
+    parser.add_argument("--cutoffs", type=float, nargs="+", default=DEFAULT_CUTOFFS)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(args.phase6_metrics, args.raw_root, tuple(args.cutoffs))
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    print(json.dumps({"status": result["status"], "output": str(args.output)}, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_strong_baseline.py

@@ -0,0 +1,298 @@
+#!/usr/bin/env python3
+"""Audit telemetry against a simulator-aware outcome calibration baseline.
+
+This is a retrospective headroom check.  It strengthens the earlier
+outcome-only baseline by giving both nested models the same per-anchor
+Frontier throughput and SLO predictions.  The only additional inputs to the
+larger model are real engine Layer-1 features.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from analyze_existing import (
+    DEFAULT_REGULARIZATION,
+    REGULARIZATION_SENSITIVITY,
+    _classification_metrics,
+    _fit_logistic,
+    _group_bootstrap_delta,
+    _mcnemar_exact_p,
+    _sigmoid,
+)
+from analyze_prefixes import (
+    INSTRUMENTATION_FEATURES,
+    OUTCOME_FEATURES,
+    PrefixExample,
+    build_examples,
+    numeric,
+    policy_metrics,
+    sha256_file,
+)
+
+
+SIMULATOR_FEATURES = (
+    "log_sim_completed_throughput_per_gpu",
+    "sim_slo_pass_rate",
+    "sim_slo_feasible",
+)
+
+
+def load_simulator_features(raw_root: Path) -> tuple[dict[tuple[str, float], tuple[float, ...]], str]:
+    features: dict[tuple[str, float], tuple[float, ...]] = {}
+    digest = hashlib.sha256()
+    paths = sorted(raw_root.glob("*/trial-0001/run_manifest.json"))
+    for manifest_path in paths:
+        manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+        run = manifest["run"]
+        if run["mode"] != "frozen-calibrated":
+            continue
+        scorer_path = manifest_path.parent / "scorer_output.json"
+        scorer = json.loads(scorer_path.read_text(encoding="utf-8"))
+        key = (str(run["cell_id"]), float(run["sampling_u"]))
+        if key in features:
+            raise ValueError(f"duplicate frozen simulator run: {key}")
+        throughput = float(scorer["throughput_requests_per_second_per_gpu"])
+        pass_rate = float(scorer["slo"]["pass_rate"])
+        if throughput <= 0 or not 0.0 <= pass_rate <= 1.0:
+            raise ValueError(f"invalid simulator output: {key}")
+        features[key] = (
+            math.log(throughput),
+            pass_rate,
+            float(bool(scorer["slo"]["feasible"])),
+        )
+        for path in (manifest_path, scorer_path):
+            digest.update(str(path.relative_to(raw_root)).encode())
+            digest.update(path.read_bytes())
+    return features, digest.hexdigest()
+
+
+def simulator_row(
+    example: PrefixExample,
+    features: dict[tuple[str, float], tuple[float, ...]],
+) -> tuple[float, ...]:
+    matches = [
+        values
+        for (cell, anchor), values in features.items()
+        if cell == example.cell
+        and math.isclose(anchor, example.anchor, rel_tol=0.0, abs_tol=1e-12)
+    ]
+    if len(matches) != 1:
+        raise ValueError(
+            f"expected one simulator match for {example.cell}/{example.anchor}: {len(matches)}"
+        )
+    return matches[0]
+
+
+def grouped_predictions(
+    examples: list[PrefixExample],
+    simulator: dict[tuple[str, float], tuple[float, ...]],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> tuple[np.ndarray, np.ndarray, list[str]]:
+    probabilities: list[float] = []
+    labels: list[int] = []
+    groups: list[str] = []
+    for held_out in sorted({example.cell for example in examples}):
+        train = [example for example in examples if example.cell != held_out]
+        test = [example for example in examples if example.cell == held_out]
+
+        def row(example: PrefixExample) -> np.ndarray:
+            values = example.outcome + simulator_row(example, simulator)
+            if instrumentation_aware:
+                values += example.instrumentation
+            return np.asarray((1.0, *values), dtype=np.float64)
+
+        x_train = np.stack([row(example) for example in train])
+        x_test = np.stack([row(example) for example in test])
+        y_train = np.asarray([example.feasible for example in train], dtype=np.float64)
+        mean = x_train[:, 1:].mean(axis=0)
+        standard_deviation = x_train[:, 1:].std(axis=0)
+        standard_deviation[standard_deviation < 1e-8] = 1.0
+        x_train[:, 1:] = (x_train[:, 1:] - mean) / standard_deviation
+        x_test[:, 1:] = (x_test[:, 1:] - mean) / standard_deviation
+        weights = _fit_logistic(x_train, y_train, regularization)
+        probabilities.extend(_sigmoid(x_test @ weights).tolist())
+        labels.extend(example.feasible for example in test)
+        groups.extend(held_out for _ in test)
+    return (
+        np.asarray(labels, dtype=np.int64),
+        np.asarray(probabilities, dtype=np.float64),
+        groups,
+    )
+
+
+def analyze(
+    phase6_path: Path,
+    phase6_raw_root: Path,
+    simulator_raw_root: Path,
+    simulator_metrics_path: Path,
+) -> dict[str, Any]:
+    phase6 = json.loads(phase6_path.read_text(encoding="utf-8"))
+    examples = build_examples(phase6, phase6_raw_root, 5.0)
+    simulator, simulator_raw_sha256 = load_simulator_features(simulator_raw_root)
+    red_flags = []
+    try:
+        matched = [simulator_row(example, simulator) for example in examples]
+    except ValueError as error:
+        matched = []
+        red_flags.append(str(error))
+
+    sensitivity = {}
+    if matched:
+        for regularization in REGULARIZATION_SENSITIVITY:
+            labels, baseline_probability, groups = grouped_predictions(
+                examples,
+                simulator,
+                instrumentation_aware=False,
+                regularization=regularization,
+            )
+            instrument_labels, instrument_probability, instrument_groups = grouped_predictions(
+                examples,
+                simulator,
+                instrumentation_aware=True,
+                regularization=regularization,
+            )
+            if not np.array_equal(labels, instrument_labels) or groups != instrument_groups:
+                raise AssertionError("nested baseline folds differ")
+            baseline_correct = (baseline_probability >= 0.5) == labels
+            instrument_correct = (instrument_probability >= 0.5) == labels
+            paired = {
+                "both_correct": int(np.sum(baseline_correct & instrument_correct)),
+                "sim_outcome_only_correct": int(
+                    np.sum(baseline_correct & ~instrument_correct)
+                ),
+                "instrumentation_only_correct": int(
+                    np.sum(~baseline_correct & instrument_correct)
+                ),
+                "both_wrong": int(np.sum(~baseline_correct & ~instrument_correct)),
+            }
+            paired["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+                paired["sim_outcome_only_correct"],
+                paired["instrumentation_only_correct"],
+            )
+            sensitivity[str(regularization)] = {
+                "sim_plus_outcome": {
+                    "classification": _classification_metrics(labels, baseline_probability),
+                    "policy_0p95": policy_metrics(
+                        examples, labels, baseline_probability, 0.95
+                    ),
+                },
+                "sim_plus_outcome_plus_instrumentation": {
+                    "classification": _classification_metrics(labels, instrument_probability),
+                    "policy_0p95": policy_metrics(
+                        examples, labels, instrument_probability, 0.95
+                    ),
+                },
+                "paired_correctness": paired,
+                "group_bootstrap": _group_bootstrap_delta(
+                    labels,
+                    baseline_probability,
+                    instrument_probability,
+                    groups,
+                ),
+            }
+
+    headline = sensitivity.get(str(DEFAULT_REGULARIZATION))
+    simulator_pass_rates = [row[1] for row in matched]
+    labels = [example.feasible for example in examples]
+    if len(examples) != 37:
+        red_flags.append("examples_not_37")
+    if len(simulator) != 92:
+        red_flags.append("frozen_simulator_runs_not_92")
+    if len(set(labels)) != 2:
+        red_flags.append("single_label")
+    if matched and not all(0.0 <= value <= 1.0 for value in simulator_pass_rates):
+        red_flags.append("simulator_pass_rate_out_of_range")
+
+    return {
+        "schema": "fidelity-strong-baseline-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": "retrospective one-task headroom audit; not contribution evidence",
+        "comparison": (
+            "same 5-second prefix, folds, logistic family, regularization, and frozen "
+            "Frontier outputs; the only nested difference is real Layer-1 engine state"
+        ),
+        "features": {
+            "shared_outcome": list(OUTCOME_FEATURES),
+            "shared_simulator": list(SIMULATOR_FEATURES),
+            "instrumentation_only": list(INSTRUMENTATION_FEATURES),
+        },
+        "headline_regularization": DEFAULT_REGULARIZATION,
+        "headline": headline,
+        "regularization_sensitivity": sensitivity,
+        "provenance": {
+            "phase6_metrics": str(phase6_path.resolve()),
+            "phase6_metrics_sha256": sha256_file(phase6_path),
+            "phase6_raw_root": str(phase6_raw_root.resolve()),
+            "simulator_metrics": str(simulator_metrics_path.resolve()),
+            "simulator_metrics_sha256": sha256_file(simulator_metrics_path),
+            "simulator_raw_root": str(simulator_raw_root.resolve()),
+            "frozen_simulator_manifest_scorer_set_sha256": simulator_raw_sha256,
+        },
+        "decision": {
+            "contribution_established": False,
+            "prospective_requirement": (
+                "repeat sim+outcome versus sim+outcome+instrumentation on complete held-out tasks"
+            ),
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "examples": numeric([1 for _ in examples]),
+            "labels": {
+                **numeric(labels),
+                "positive": sum(labels),
+                "negative": len(labels) - sum(labels),
+            },
+            "matched_simulator_pass_rate": numeric(simulator_pass_rates),
+            "frozen_simulator_runs": len(simulator),
+            "invariants": {
+                "all_examples_matched_once": len(matched) == len(examples),
+                "same_nested_folds": True,
+                "simulator_ratios_bounded": all(
+                    0.0 <= value <= 1.0 for value in simulator_pass_rates
+                ),
+                "labels_not_identical": len(set(labels)) == 2,
+                "per_config_results_not_all_identical": len(set(simulator_pass_rates)) > 1,
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--phase6-raw-root", type=Path, required=True)
+    parser.add_argument("--simulator-raw-root", type=Path, required=True)
+    parser.add_argument("--simulator-metrics", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(
+        args.phase6_metrics,
+        args.phase6_raw_root,
+        args.simulator_raw_root,
+        args.simulator_metrics,
+    )
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(
+        json.dumps(
+            {
+                "status": result["status"],
+                "output": str(args.output),
+                "red_flags": result["sanity"]["red_flags"],
+            },
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_strong_pilot.py

@@ -0,0 +1,398 @@
+#!/usr/bin/env python3
+"""Exploratory P1 audit against the strengthened simulator-aware baseline.
+
+P1 was already running when the strong baseline was added, so this script is
+not paper-facing prospective evidence.  It trains only on the historical
+Phase-6 task and evaluates the exact P1 primary probes.  Both nested models
+receive identical Frontier predictions; engine telemetry is the sole feature
+difference.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from analyze_existing import (
+    DEFAULT_REGULARIZATION,
+    REGULARIZATION_SENSITIVITY,
+    _classification_metrics,
+    _fit_logistic,
+    _mcnemar_exact_p,
+    _sigmoid,
+)
+from analyze_pilot import build_pilot_examples
+from analyze_prefixes import (
+    INSTRUMENTATION_FEATURES,
+    OUTCOME_FEATURES,
+    PrefixExample,
+    build_examples,
+    numeric,
+    policy_metrics,
+    sha256_file,
+)
+from analyze_strong_baseline import (
+    SIMULATOR_FEATURES,
+    load_simulator_features,
+    simulator_row,
+)
+
+
+def load_pilot_simulator(
+    path: Path,
+) -> tuple[dict[tuple[str, str], tuple[float, ...]], list[str]]:
+    payload = json.loads(path.read_text(encoding="utf-8"))
+    red_flags = []
+    if payload.get("status") != "PASS":
+        red_flags.append("pilot_simulator_not_pass")
+    features: dict[tuple[str, str], tuple[float, ...]] = {}
+    for item in payload.get("results", []):
+        key = (str(item["cell"]), str(item["role"]))
+        if key in features:
+            red_flags.append(f"duplicate_pilot_simulator_{key[0]}_{key[1]}")
+            continue
+        scorer = item["scorer"]
+        throughput = float(scorer["throughput_requests_per_second_per_gpu"])
+        pass_rate = float(scorer["slo"]["pass_rate"])
+        if throughput <= 0:
+            red_flags.append(f"nonpositive_pilot_simulator_throughput_{key[0]}_{key[1]}")
+        if not 0.0 <= pass_rate <= 1.0:
+            red_flags.append(f"pilot_simulator_ratio_out_of_range_{key[0]}_{key[1]}")
+        features[key] = (
+            math.log(throughput),
+            pass_rate,
+            float(bool(scorer["slo"]["feasible"])),
+        )
+    if len(features) != 12:
+        red_flags.append("pilot_simulator_entries_not_12")
+    return features, red_flags
+
+
+def fit_model(
+    examples: list[PrefixExample],
+    simulator: list[tuple[float, ...]],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> dict[str, Any]:
+    rows = []
+    for example, simulator_features in zip(examples, simulator):
+        values = example.outcome + simulator_features
+        if instrumentation_aware:
+            values += example.instrumentation
+        rows.append((1.0, *values))
+    matrix = np.asarray(rows, dtype=np.float64)
+    labels = np.asarray([example.feasible for example in examples], dtype=np.float64)
+    mean = matrix[:, 1:].mean(axis=0)
+    standard_deviation = matrix[:, 1:].std(axis=0)
+    standard_deviation[standard_deviation < 1e-8] = 1.0
+    standardized = matrix.copy()
+    standardized[:, 1:] = (standardized[:, 1:] - mean) / standard_deviation
+    weights = _fit_logistic(standardized, labels, regularization)
+    return {
+        "instrumentation_aware": instrumentation_aware,
+        "regularization": regularization,
+        "feature_mean": mean,
+        "feature_standard_deviation": standard_deviation,
+        "weights": weights,
+    }
+
+
+def predict_model(
+    model: dict[str, Any],
+    examples: list[PrefixExample],
+    simulator: list[tuple[float, ...]],
+) -> np.ndarray:
+    rows = []
+    for example, simulator_features in zip(examples, simulator):
+        values = example.outcome + simulator_features
+        if model["instrumentation_aware"]:
+            values += example.instrumentation
+        rows.append((1.0, *values))
+    matrix = np.asarray(rows, dtype=np.float64)
+    matrix[:, 1:] = (
+        matrix[:, 1:] - model["feature_mean"]
+    ) / model["feature_standard_deviation"]
+    return _sigmoid(matrix @ model["weights"])
+
+
+def comparison(
+    training_examples: list[PrefixExample],
+    training_simulator: list[tuple[float, ...]],
+    pilot_examples: list[PrefixExample],
+    pilot_simulator: list[tuple[float, ...]],
+    regularization: float,
+) -> dict[str, Any]:
+    labels = np.asarray([example.feasible for example in pilot_examples], dtype=np.int64)
+    baseline_model = fit_model(
+        training_examples,
+        training_simulator,
+        instrumentation_aware=False,
+        regularization=regularization,
+    )
+    instrument_model = fit_model(
+        training_examples,
+        training_simulator,
+        instrumentation_aware=True,
+        regularization=regularization,
+    )
+    baseline_probability = predict_model(
+        baseline_model, pilot_examples, pilot_simulator
+    )
+    instrument_probability = predict_model(
+        instrument_model, pilot_examples, pilot_simulator
+    )
+    baseline_correct = (baseline_probability >= 0.5) == labels
+    instrument_correct = (instrument_probability >= 0.5) == labels
+    paired = {
+        "both_correct": int(np.sum(baseline_correct & instrument_correct)),
+        "sim_outcome_only_correct": int(
+            np.sum(baseline_correct & ~instrument_correct)
+        ),
+        "instrumentation_only_correct": int(
+            np.sum(~baseline_correct & instrument_correct)
+        ),
+        "both_wrong": int(np.sum(~baseline_correct & ~instrument_correct)),
+    }
+    paired["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+        paired["sim_outcome_only_correct"], paired["instrumentation_only_correct"]
+    )
+    return {
+        "sim_plus_outcome": {
+            "classification": _classification_metrics(labels, baseline_probability),
+            "policy_0p95": policy_metrics(
+                pilot_examples, labels, baseline_probability, 0.95
+            ),
+            "probability": baseline_probability.tolist(),
+        },
+        "sim_plus_outcome_plus_instrumentation": {
+            "classification": _classification_metrics(labels, instrument_probability),
+            "policy_0p95": policy_metrics(
+                pilot_examples, labels, instrument_probability, 0.95
+            ),
+            "probability": instrument_probability.tolist(),
+        },
+        "paired_correctness": paired,
+    }
+
+
+def analyze(
+    phase6_path: Path,
+    phase6_raw_root: Path,
+    training_simulator_root: Path,
+    pilot_manifest_path: Path,
+    pilot_run_root: Path,
+    pilot_simulator_path: Path,
+) -> dict[str, Any]:
+    phase6 = json.loads(phase6_path.read_text(encoding="utf-8"))
+    pilot_manifest = json.loads(pilot_manifest_path.read_text(encoding="utf-8"))
+    training_examples = build_examples(phase6, phase6_raw_root, 5.0)
+    training_simulator_map, training_simulator_sha256 = load_simulator_features(
+        training_simulator_root
+    )
+    training_simulator = [
+        simulator_row(example, training_simulator_map)
+        for example in training_examples
+    ]
+    pilot_examples, pilot_details, red_flags = build_pilot_examples(
+        pilot_manifest, pilot_run_root, 5.0
+    )
+    pilot_simulator_map, simulator_red_flags = load_pilot_simulator(
+        pilot_simulator_path
+    )
+    red_flags.extend(simulator_red_flags)
+    pilot_simulator = []
+    for example, detail in zip(pilot_examples, pilot_details):
+        role = f"{detail['level']}1"
+        key = (example.cell, role)
+        if key not in pilot_simulator_map:
+            red_flags.append(f"missing_pilot_simulator_{example.cell}_{role}")
+            pilot_simulator.append((0.0, 0.0, 0.0))
+        else:
+            pilot_simulator.append(pilot_simulator_map[key])
+
+    sensitivity = {}
+    if not red_flags:
+        for regularization in REGULARIZATION_SENSITIVITY:
+            sensitivity[str(regularization)] = comparison(
+                training_examples,
+                training_simulator,
+                pilot_examples,
+                pilot_simulator,
+                regularization,
+            )
+    headline = sensitivity.get(str(DEFAULT_REGULARIZATION))
+    labels = [example.feasible for example in pilot_examples]
+    simulator_pass_rates = [row[1] for row in pilot_simulator]
+    simulator_labels = [int(row[2]) for row in pilot_simulator]
+    if len(training_examples) != 37:
+        red_flags.append("training_examples_not_37")
+    if len(pilot_examples) != 12:
+        red_flags.append("pilot_examples_not_12")
+    if len(set(labels)) != 2:
+        red_flags.append("pilot_single_label")
+    if len(set(simulator_pass_rates)) <= 1:
+        red_flags.append("pilot_simulator_results_identical")
+
+    if headline is None:
+        decision = {
+            "strong_incremental_gate": False,
+            "reason": "analysis red flag prevented nested comparison",
+        }
+    else:
+        baseline_policy = headline["sim_plus_outcome"]["policy_0p95"]
+        instrument_policy = headline[
+            "sim_plus_outcome_plus_instrumentation"
+        ]["policy_0p95"]
+        baseline_errors = baseline_policy["false_accept"] + baseline_policy["false_reject"]
+        instrument_errors = (
+            instrument_policy["false_accept"] + instrument_policy["false_reject"]
+        )
+        baseline_reduction = baseline_policy["valid_cost_reduction_fraction"]
+        instrument_reduction = instrument_policy["valid_cost_reduction_fraction"]
+        reduction_delta = (
+            instrument_reduction - baseline_reduction
+            if baseline_reduction is not None and instrument_reduction is not None
+            else None
+        )
+        per_lambda_safe_and_better = []
+        for item in sensitivity.values():
+            baseline = item["sim_plus_outcome"]["policy_0p95"]
+            instrument = item["sim_plus_outcome_plus_instrumentation"]["policy_0p95"]
+            base_errors = baseline["false_accept"] + baseline["false_reject"]
+            inst_errors = instrument["false_accept"] + instrument["false_reject"]
+            base_reduction = baseline["valid_cost_reduction_fraction"]
+            inst_reduction = instrument["valid_cost_reduction_fraction"]
+            per_lambda_safe_and_better.append(
+                inst_errors == 0
+                and inst_errors <= base_errors
+                and base_reduction is not None
+                and inst_reduction is not None
+                and inst_reduction > base_reduction
+            )
+        decision = {
+            "strong_incremental_gate": bool(
+                not red_flags
+                and instrument_errors == 0
+                and instrument_errors <= baseline_errors
+                and reduction_delta is not None
+                and reduction_delta >= 0.15
+            ),
+            "regularization_robust": all(per_lambda_safe_and_better),
+            "valid_cost_reduction_fraction_delta": reduction_delta,
+            "scope": "exploratory task; may choose P2 design but cannot establish contribution",
+        }
+
+    return {
+        "schema": "fidelity-strong-pilot-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": (
+            "post-amendment exploratory P1 audit; strong model was not frozen before "
+            "partial P1 outcomes, so this is not prospective contribution evidence"
+        ),
+        "features": {
+            "shared_outcome": list(OUTCOME_FEATURES),
+            "shared_simulator": list(SIMULATOR_FEATURES),
+            "instrumentation_only": list(INSTRUMENTATION_FEATURES),
+        },
+        "headline_regularization": DEFAULT_REGULARIZATION,
+        "headline": headline,
+        "regularization_sensitivity": sensitivity,
+        "simulator_only": {
+            "classification": _classification_metrics(
+                np.asarray(labels, dtype=np.int64),
+                np.asarray(simulator_labels, dtype=np.float64),
+            )
+            if labels
+            else None,
+            "predicted_feasible": simulator_labels,
+        },
+        "pilot_examples": [
+            {
+                **detail,
+                "sim_completed_throughput_per_gpu": math.exp(simulator[0]),
+                "sim_slo_pass_rate": simulator[1],
+                "sim_slo_feasible": bool(simulator[2]),
+            }
+            for detail, simulator in zip(pilot_details, pilot_simulator)
+        ],
+        "decision": decision,
+        "provenance": {
+            "phase6_metrics": str(phase6_path.resolve()),
+            "phase6_metrics_sha256": sha256_file(phase6_path),
+            "phase6_raw_root": str(phase6_raw_root.resolve()),
+            "training_simulator_root": str(training_simulator_root.resolve()),
+            "training_simulator_manifest_scorer_set_sha256": training_simulator_sha256,
+            "pilot_manifest": str(pilot_manifest_path.resolve()),
+            "pilot_manifest_sha256": sha256_file(pilot_manifest_path),
+            "pilot_run_root": str(pilot_run_root.resolve()),
+            "pilot_simulator": str(pilot_simulator_path.resolve()),
+            "pilot_simulator_sha256": sha256_file(pilot_simulator_path),
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "training_examples": numeric([1 for _ in training_examples]),
+            "pilot_labels": {
+                **numeric(labels),
+                "positive": sum(labels),
+                "negative": len(labels) - sum(labels),
+            },
+            "pilot_simulator_pass_rate": numeric(simulator_pass_rates),
+            "invariants": {
+                "training_examples_37": len(training_examples) == 37,
+                "pilot_examples_12": len(pilot_examples) == 12,
+                "pilot_cells_6": len({example.cell for example in pilot_examples}) == 6,
+                "pilot_both_labels": len(set(labels)) == 2,
+                "simulator_ratios_bounded": all(
+                    0.0 <= value <= 1.0 for value in simulator_pass_rates
+                ),
+                "per_config_not_all_identical": len(set(simulator_pass_rates)) > 1,
+                "all_prefixes_exact_monotonic": all(
+                    example.completion_time_source in {"exact_monotonic", "none_completed"}
+                    for example in pilot_examples
+                ),
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--phase6-raw-root", type=Path, required=True)
+    parser.add_argument("--training-simulator-root", type=Path, required=True)
+    parser.add_argument("--pilot-manifest", type=Path, required=True)
+    parser.add_argument("--pilot-run-root", type=Path, required=True)
+    parser.add_argument("--pilot-simulator", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(
+        args.phase6_metrics,
+        args.phase6_raw_root,
+        args.training_simulator_root,
+        args.pilot_manifest,
+        args.pilot_run_root,
+        args.pilot_simulator,
+    )
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(
+        json.dumps(
+            {
+                "status": result["status"],
+                "red_flags": result["sanity"]["red_flags"],
+                "decision": result["decision"],
+            },
+            sort_keys=True,
+        )
+    )
+    if result["status"] != "PASS":
+        raise RuntimeError(result["sanity"]["red_flags"])
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/freeze_models.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+"""Freeze the training-task prefix models before prospective GPU work."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from analyze_prefixes import (
+    DEFAULT_REGULARIZATION,
+    POLICY_THRESHOLDS,
+    build_examples,
+    fit_frozen_model,
+    sha256_file,
+)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--prefix-metrics", type=Path, required=True)
+    parser.add_argument("--raw-root", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+
+    cutoff_s = 5.0
+    threshold = 0.95
+    if threshold not in POLICY_THRESHOLDS:
+        raise AssertionError("frozen threshold is outside audited policy thresholds")
+    phase6 = json.loads(args.phase6_metrics.read_text(encoding="utf-8"))
+    examples = build_examples(phase6, args.raw_root, cutoff_s)
+    payload = {
+        "schema": "fidelity-prefix-model-v1",
+        "status": "FROZEN_BEFORE_PROSPECTIVE_RUN",
+        "cutoff_s": cutoff_s,
+        "accept_probability": threshold,
+        "reject_probability": 1.0 - threshold,
+        "regularization": DEFAULT_REGULARIZATION,
+        "label": "same-placement 2-of-3 adjudicated anchor feasibility",
+        "training_split_role": "historical training only; never headline test",
+        "training_examples": [
+            {
+                "cell": example.cell,
+                "anchor": example.anchor,
+                "label_feasible": bool(example.feasible),
+                "primary_feasible": bool(example.primary_feasible),
+                "completion_time_source": example.completion_time_source,
+            }
+            for example in examples
+        ],
+        "models": {
+            "outcome_only": fit_frozen_model(
+                examples,
+                instrumentation_aware=False,
+                regularization=DEFAULT_REGULARIZATION,
+            ),
+            "instrumentation_aware": fit_frozen_model(
+                examples,
+                instrumentation_aware=True,
+                regularization=DEFAULT_REGULARIZATION,
+            ),
+        },
+        "provenance": {
+            "phase6_metrics": str(args.phase6_metrics.resolve()),
+            "phase6_metrics_sha256": sha256_file(args.phase6_metrics),
+            "prefix_metrics": str(args.prefix_metrics.resolve()),
+            "prefix_metrics_sha256": sha256_file(args.prefix_metrics),
+            "raw_root": str(args.raw_root.resolve()),
+        },
+        "sanity": {
+            "n": len(examples),
+            "positive": sum(example.feasible for example in examples),
+            "negative": sum(not example.feasible for example in examples),
+            "cells": len({example.cell for example in examples}),
+            "invariants": {
+                "n_37": len(examples) == 37,
+                "cells_12": len({example.cell for example in examples}) == 12,
+                "both_labels": len({example.feasible for example in examples}) == 2,
+                "cutoff_5s": cutoff_s == 5.0,
+                "threshold_0.95": threshold == 0.95,
+            },
+        },
+    }
+    if not all(payload["sanity"]["invariants"].values()):
+        raise RuntimeError(f"model freeze invariants failed: {payload['sanity']}")
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    print(json.dumps({"status": payload["status"], "output": str(args.output)}))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/frozen-models.json

@@ -0,0 +1,515 @@
+{
+  "accept_probability": 0.95,
+  "cutoff_s": 5.0,
+  "label": "same-placement 2-of-3 adjudicated anchor feasibility",
+  "models": {
+    "instrumentation_aware": {
+      "feature_mean": [
+        0.8984976998643891,
+        0.8378378378378378,
+        4.324324324324325,
+        0.07552086023066117,
+        0.6758807403968693,
+        0.9459459459459459,
+        0.0,
+        0.3241192596031305,
+        0.04468545442770093,
+        0.025590516908533558,
+        0.23873649352596996,
+        0.1943716628122394,
+        0.4321792125178198,
+        112.70270270270272,
+        0.21087752102856197,
+        0.918918918918919,
+        0.055470351361483664,
+        4.904239530899751,
+        10.162162162162161,
+        4.822982150502539,
+        10.135135135135135,
+        0.43557131397798415,
+        0.031387890158936414,
+        0.05804311436894179,
+        0.03298678556030958,
+        0.030437119300455177,
+        0.9503065396705037,
+        0.07127076398319926,
+        0.6234198543231205,
+        0.0
+      ],
+      "feature_names": [
+        "log_offered_rate_per_gpu",
+        "log2_tp",
+        "log2_max_num_seqs",
+        "admitted_fraction",
+        "completed_over_admitted",
+        "completed_pass_rate",
+        "completed_fail_fraction_of_total",
+        "outstanding_over_admitted",
+        "ttft_max_over_slo_max",
+        "ttft_mean_over_slo_max",
+        "tpot_max_over_slo",
+        "tpot_mean_over_slo",
+        "admitted_input_tokens_mean_over_limit",
+        "model_steps_per_second",
+        "waiting_mean",
+        "waiting_max",
+        "waiting_nonzero_share",
+        "running_mean",
+        "running_max",
+        "decode_batch_mean",
+        "decode_batch_max",
+        "decode_batch_cv",
+        "kv_usage_mean",
+        "kv_usage_max",
+        "kv_usage_end_minus_start",
+        "graph_none_share",
+        "graph_full_share",
+        "padding_fraction",
+        "prefill_token_fraction",
+        "preemptions"
+      ],
+      "feature_standard_deviation": [
+        0.2953332526155246,
+        0.8546696378833459,
+        1.1402715194448103,
+        0.006588255148989237,
+        0.2751217635728275,
+        0.22612433149569594,
+        1.0,
+        0.27512176357282747,
+        0.048292427420964075,
+        0.02574874589991541,
+        0.1635381690436309,
+        0.14098674719611365,
+        0.02516276437103069,
+        61.39272994412853,
+        0.7234949448561444,
+        2.198013579605131,
+        0.18326586413988316,
+        2.4542471212960844,
+        6.08726412391018,
+        2.4074006634033043,
+        6.067913672185017,
+        0.12556414020947543,
+        0.03256962310836033,
+        0.054049610008010444,
+        0.048321850100969746,
+        0.04298231458641556,
+        0.041906068064246155,
+        0.08212268757576466,
+        0.4089385238422411,
+        1.0
+      ],
+      "instrumentation_aware": true,
+      "regularization": 1.0,
+      "training_classification": {
+        "accuracy": 0.972972972972973,
+        "balanced_accuracy": 0.9444444444444444,
+        "brier": 0.02820726479488704,
+        "confusion": {
+          "false_negative": 0,
+          "false_positive": 1,
+          "true_negative": 8,
+          "true_positive": 28
+        },
+        "log_loss": 0.11247563885308659
+      },
+      "weights_with_intercept_first": [
+        2.109507425802979,
+        -0.8372240489271802,
+        -0.2476229678897366,
+        0.18172257646801393,
+        -0.07076358054975332,
+        0.3035586906752765,
+        0.08500005412355496,
+        -7.754818242684634e-26,
+        -0.3035586906752766,
+        0.4014234393196892,
+        0.513218716194957,
+        -0.35161457106287,
+        0.10558147889556725,
+        0.5674345291616134,
+        0.15895995157114373,
+        -0.4274260624362057,
+        -0.048791959001756195,
+        -0.37221380985270663,
+        -0.35527537277290255,
+        0.20582736797173468,
+        -0.35837576944545413,
+        0.2342062515631318,
+        0.45071068059490843,
+        0.3326948315186803,
+        0.2698892549960913,
+        0.017868065865726347,
+        -0.1540209080477302,
+        0.3412427440368233,
+        0.5831011876762794,
+        -0.583920360300169,
+        0.0
+      ]
+    },
+    "outcome_only": {
+      "feature_mean": [
+        0.8984976998643891,
+        0.8378378378378378,
+        4.324324324324325,
+        0.07552086023066117,
+        0.6758807403968693,
+        0.9459459459459459,
+        0.0,
+        0.3241192596031305,
+        0.04468545442770093,
+        0.025590516908533558,
+        0.23873649352596996,
+        0.1943716628122394,
+        0.4321792125178198
+      ],
+      "feature_names": [
+        "log_offered_rate_per_gpu",
+        "log2_tp",
+        "log2_max_num_seqs",
+        "admitted_fraction",
+        "completed_over_admitted",
+        "completed_pass_rate",
+        "completed_fail_fraction_of_total",
+        "outstanding_over_admitted",
+        "ttft_max_over_slo_max",
+        "ttft_mean_over_slo_max",
+        "tpot_max_over_slo",
+        "tpot_mean_over_slo",
+        "admitted_input_tokens_mean_over_limit"
+      ],
+      "feature_standard_deviation": [
+        0.2953332526155246,
+        0.8546696378833459,
+        1.1402715194448103,
+        0.006588255148989237,
+        0.2751217635728275,
+        0.22612433149569594,
+        1.0,
+        0.27512176357282747,
+        0.048292427420964075,
+        0.02574874589991541,
+        0.1635381690436309,
+        0.14098674719611365,
+        0.02516276437103069
+      ],
+      "instrumentation_aware": false,
+      "regularization": 1.0,
+      "training_classification": {
+        "accuracy": 0.9459459459459459,
+        "balanced_accuracy": 0.8888888888888888,
+        "brier": 0.051887373873176545,
+        "confusion": {
+          "false_negative": 0,
+          "false_positive": 2,
+          "true_negative": 7,
+          "true_positive": 28
+        },
+        "log_loss": 0.184988719119571
+      },
+      "weights_with_intercept_first": [
+        1.8996338126233983,
+        -1.1536861934230125,
+        -0.3806404559018098,
+        0.5901136731733696,
+        0.022432085012851908,
+        0.5805554730881304,
+        0.25786307099613026,
+        -8.077935669463161e-27,
+        -0.5805554730881304,
+        -0.15413292402348447,
+        0.0986842306063204,
+        -0.5181573573074624,
+        0.06283513013708956,
+        0.911619634884147
+      ]
+    }
+  },
+  "provenance": {
+    "phase6_metrics": "/home/gahow/phd/aituner/runs/opprof-phase6/phase6/metrics.json",
+    "phase6_metrics_sha256": "290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16",
+    "prefix_metrics": "/home/gahow/phd/aituner/runs/fidelity-headroom/prefix-metrics.json",
+    "prefix_metrics_sha256": "cda821bcde1ae8427507aa4f03a1c116ccc7f7b8b717f73ca587bee3670a0340",
+    "raw_root": "/home/gahow/phd/aituner/runs/opprof-phase6/phase6/solo-authoritative/cells"
+  },
+  "regularization": 1.0,
+  "reject_probability": 0.050000000000000044,
+  "sanity": {
+    "cells": 12,
+    "invariants": {
+      "both_labels": true,
+      "cells_12": true,
+      "cutoff_5s": true,
+      "n_37": true,
+      "threshold_0.95": true
+    },
+    "n": 37,
+    "negative": 9,
+    "positive": 28
+  },
+  "schema": "fidelity-prefix-model-v1",
+  "status": "FROZEN_BEFORE_PROSPECTIVE_RUN",
+  "training_examples": [
+    {
+      "anchor": 0.24609375,
+      "cell": "tp1_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.2421875,
+      "cell": "tp1_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.24609375,
+      "cell": "tp1_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns32",
+      "completion_time_source": "none_completed",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.2421875,
+      "cell": "tp1_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.24609375,
+      "cell": "tp1_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns64",
+      "completion_time_source": "none_completed",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.21875,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.2265625,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.23046875,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.234375,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.4921875,
+      "cell": "tp2_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.49609375,
+      "cell": "tp2_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp2_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.75,
+      "cell": "tp2_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.75390625,
+      "cell": "tp2_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp2_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.75,
+      "cell": "tp2_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.4921875,
+      "cell": "tp2_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.49609375,
+      "cell": "tp2_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.033182214016,
+      "cell": "tp4_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.033717411016,
+      "cell": "tp4_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.033717411016,
+      "cell": "tp4_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.033717411016,
+      "cell": "tp4_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.016055910008,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.016591107009,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.017126304009,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    }
+  ],
+  "training_split_role": "historical training only; never headline test"
+}

runs/fidelity-headroom/pilot_controller.py

@@ -0,0 +1,461 @@
+#!/usr/bin/env python3
+"""Serialized dash0 controller for the exact-timestamp prefix pilot."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shlex
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+
+HERE = Path(__file__).resolve().parent
+PHASE6 = HERE.parent / "opprof-phase6"
+sys.path.insert(0, str(PHASE6))
+
+import opprof_phase6_controller as base  # noqa: E402
+
+
+ORDER = (
+    "tp1_mns8",
+    "tp1_mns64",
+    "tp2_mns8",
+    "tp2_mns64",
+    "tp4_mns16",
+    "tp4_mns64",
+)
+CELL_ESTIMATE_H20_HOURS = {1: 0.20, 2: 0.40, 4: 0.80}
+SAFETY_H20_HOURS = 0.20
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    base.atomic_json(path, payload)
+
+
+def wait_all_idle(timeout_s: float = 30.0) -> None:
+    deadline = time.monotonic() + timeout_s
+    last_error: Exception | None = None
+    while time.monotonic() < deadline:
+        try:
+            base.assert_all_idle()
+            return
+        except RuntimeError as error:
+            last_error = error
+            time.sleep(1.0)
+    raise last_error or RuntimeError("GPU idle timeout")
+
+
+def configure_base(args: argparse.Namespace, manifest: dict[str, Any]) -> None:
+    base.WORKDIR = args.run_root.parent
+    base.RUN_ROOT = args.run_root
+    base.STATE = args.run_root / "controller-state.json"
+    base.SOURCE = args.vllm_source
+    base.VENV = args.venv
+    base.AITUNER = args.aituner_root
+    base.MODEL = args.model
+    base.CLIENT = args.client
+    base.GPU_LIMIT = float(manifest["execution"]["hard_cap_h20_hours"])
+    base.MARKER = "fidelity-prefix-pilot-20260714"
+    base.CELLS = {
+        cell: {"tp": int(config["tp"]), "mns": int(config["mns"])}
+        for cell, config in manifest["cells"].items()
+    }
+
+
+def load_state(path: Path, hard_cap: float) -> dict[str, Any]:
+    if path.exists():
+        return json.loads(path.read_text(encoding="utf-8"))
+    return {
+        "schema": "fidelity-prefix-pilot-state-v1",
+        "status": "initialized",
+        "hard_cap_h20_hours": hard_cap,
+        "gpu_hours_total": 0.0,
+        "completed_cells": 0,
+        "cells": {},
+        "failures": [],
+        "started_at": time.time(),
+    }
+
+
+def save_state(path: Path, state: dict[str, Any]) -> None:
+    atomic_json(path, state)
+
+
+def append_echo(run_root: Path, line: str) -> None:
+    run_root.mkdir(parents=True, exist_ok=True)
+    with (run_root / "launch-echo.log").open("a", encoding="utf-8") as target:
+        target.write(line + "\n")
+    print(line, flush=True)
+
+
+def remaining_projection(manifest: dict[str, Any], index: int) -> float:
+    return sum(
+        CELL_ESTIMATE_H20_HOURS[int(manifest["cells"][cell]["tp"])]
+        for cell in ORDER[index:]
+    ) + SAFETY_H20_HOURS
+
+
+def start_server(
+    *,
+    cell: str,
+    index: int,
+    run_root: Path,
+) -> dict[str, Any]:
+    config = base.CELLS[cell]
+    gpus = tuple(range(int(config["tp"])))
+    cell_root = run_root / "cells" / cell
+    cell_root.mkdir(parents=True, exist_ok=True)
+    port = 8900 + index
+    command = base.server_command(cell, gpus, port)
+    with (cell_root / "commands.log").open("a", encoding="utf-8") as log:
+        log.write(f"SERVER {shlex.join(command)}\n")
+    server_log = (cell_root / "server.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update(
+        {
+            "CUDA_VISIBLE_DEVICES": ",".join(map(str, gpus)),
+            "VLLM_OPPROF_DIR": str(cell_root / "opprof"),
+            "OPPROF_PHASE6_MARKER": base.MARKER,
+            "AITUNER_ROOT": str(base.AITUNER),
+            "HF_HUB_OFFLINE": "1",
+            "TRANSFORMERS_OFFLINE": "1",
+            "PYTHONUNBUFFERED": "1",
+        }
+    )
+    server = subprocess.Popen(
+        command,
+        cwd=base.SOURCE,
+        env=environment,
+        stdout=server_log,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    base.OWNED_PGIDS.add(server.pid)
+    return {
+        "cell": cell,
+        "gpus": gpus,
+        "port": port,
+        "dir": cell_root,
+        "server": server,
+        "server_handle": server_log,
+        "spawned_at": time.time(),
+        "results": [],
+    }
+
+
+def selection_for(
+    manifest: dict[str, Any], cell: str, role: str
+) -> tuple[str, dict[str, Any]]:
+    level = "low" if role == "burnin" or role.startswith("low") else "high"
+    return level, manifest["cells"][cell]["targets"][level]["selections"][role]
+
+
+def client_command(
+    entry: dict[str, Any],
+    *,
+    role: str,
+    selection: dict[str, Any],
+    output: Path,
+    warmup: bool,
+) -> list[str]:
+    config = base.CELLS[entry["cell"]]
+    return [
+        "taskset",
+        "-c",
+        base.cpu_mask(entry["gpus"]),
+        str(base.VENV / "bin/python"),
+        str(base.CLIENT),
+        "warmup" if warmup else "run-anchor",
+        "--study",
+        str(selection["study"]),
+        "--cell",
+        entry["cell"],
+        "--anchor",
+        str(selection["anchor"]),
+        "--tp",
+        str(config["tp"]),
+        "--mns",
+        str(config["mns"]),
+        "--base-url",
+        f"http://127.0.0.1:{entry['port']}",
+        "--result-dir",
+        str(output),
+    ]
+
+
+def run_client(
+    *,
+    entry: dict[str, Any],
+    role: str,
+    selection: dict[str, Any],
+    output: Path,
+    state: dict[str, Any],
+    warmup: bool = False,
+) -> dict[str, Any]:
+    command = client_command(
+        entry, role=role, selection=selection, output=output, warmup=warmup
+    )
+    with (entry["dir"] / "commands.log").open("a", encoding="utf-8") as log:
+        log.write(f"CLIENT role={role} {shlex.join(command)}\n")
+    handle = (output.parent / f"{output.name}.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update({"AITUNER_ROOT": str(base.AITUNER), "PYTHONUNBUFFERED": "1"})
+    process = subprocess.Popen(
+        command,
+        cwd=base.WORKDIR,
+        env=environment,
+        stdout=handle,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    deadline = time.monotonic() + 180.0
+    try:
+        while process.poll() is None:
+            if time.monotonic() > deadline:
+                process.terminate()
+                raise TimeoutError(f"client timeout: {entry['cell']} {role}")
+            if entry["server"].poll() is not None:
+                raise RuntimeError(f"server exited during {entry['cell']} {role}")
+            base.assert_no_other_compute()
+            if state["gpu_hours_total"] + base.live_gpu_hours([entry]) >= base.GPU_LIMIT:
+                process.terminate()
+                raise RuntimeError("pilot H20-hour hard cap reached")
+            time.sleep(1.0)
+    finally:
+        handle.close()
+    if process.returncode:
+        raise RuntimeError(
+            f"client failed: cell={entry['cell']} role={role} rc={process.returncode}"
+        )
+    result = json.loads((output / "result.json").read_text(encoding="utf-8"))
+    validate_result_selection(
+        result=result,
+        selection=selection,
+        cell=entry["cell"],
+        role=role,
+        warmup=warmup,
+    )
+    entry["results"].append(
+        {"anchor": float(selection["anchor"]), "dir": str(output), "kind": result["kind"]}
+    )
+    return result
+
+
+def validate_result_selection(
+    *,
+    result: dict[str, Any],
+    selection: dict[str, Any],
+    cell: str,
+    role: str,
+    warmup: bool,
+) -> None:
+    if warmup:
+        if result["kind"] != "warmup" or int(result["selection"]["count"]) != 16:
+            raise RuntimeError(f"invalid warmup selection: {cell} {role}")
+        for key in ("warmup_16", "warmup_exact_16", "warmup_long"):
+            if not result["invariants"].get(key, False):
+                raise RuntimeError(f"warmup invariant {key} failed: {cell} {role}")
+        return
+
+    if int(result["selection"]["count"]) != int(selection["selected_count"]):
+        raise RuntimeError(f"selection count mismatch: {cell} {role}")
+    for key in (
+        "request_id_order_sha256",
+        "arrival_order_sha256",
+        "raw_length_order_sha256",
+    ):
+        manifest_key = (
+            "input_length_order_sha256" if key == "raw_length_order_sha256" else key
+        )
+        if result["selection"][key] != selection[manifest_key]:
+            raise RuntimeError(f"selection hash mismatch {key}: {cell} {role}")
+
+
+def execute_cell(
+    *,
+    index: int,
+    cell: str,
+    manifest: dict[str, Any],
+    run_root: Path,
+    state_path: Path,
+    state: dict[str, Any],
+) -> None:
+    if state["cells"].get(cell, {}).get("status") == "complete":
+        return
+    projection = remaining_projection(manifest, index)
+    if state["gpu_hours_total"] + projection > base.GPU_LIMIT:
+        state["status"] = "budget_projection_stop"
+        state["budget_stop"] = {
+            "before_cell": cell,
+            "spent_h20_hours": state["gpu_hours_total"],
+            "remaining_projection_h20_hours": projection,
+            "hard_cap_h20_hours": base.GPU_LIMIT,
+        }
+        save_state(state_path, state)
+        raise RuntimeError(f"projected pilot cost exceeds hard cap before {cell}")
+
+    config = manifest["cells"][cell]
+    echo = (
+        f"PILOT_CELL_ECHO cell={cell} tp={config['tp']} mns={config['mns']} "
+        f"gpus=0-{int(config['tp']) - 1} workload={manifest['source']['window_id']} "
+        f"roles=burnin+low1/high1/low2/high2/low3/high3 "
+        f"spent_h20h={state['gpu_hours_total']:.6f} "
+        f"remaining_projection_h20h={projection:.3f} cap_h20h={base.GPU_LIMIT:.1f} "
+        f"manifest={run_root / 'pilot-manifest.json'}"
+    )
+    append_echo(run_root, echo)
+    wait_all_idle()
+    cell_state = {
+        "status": "starting",
+        "tp": int(config["tp"]),
+        "mns": int(config["mns"]),
+        "started_at": time.time(),
+        "runs": [],
+    }
+    state["status"] = "running"
+    state["cells"][cell] = cell_state
+    save_state(state_path, state)
+    entry = start_server(cell=cell, index=index, run_root=run_root)
+    failure: Exception | None = None
+    try:
+        base.wait_ready(entry)
+        _level, burnin = selection_for(manifest, cell, "burnin")
+        cell_state["status"] = "warmup"
+        save_state(state_path, state)
+        warmup = run_client(
+            entry=entry,
+            role="burnin",
+            selection=burnin,
+            output=entry["dir"] / "warmup",
+            state=state,
+            warmup=True,
+        )
+        cell_state["warmup"] = {
+            "exact_output_count": warmup["exact_output_count"],
+            "long_gt4096": warmup["selection"]["long_gt4096"],
+        }
+        cell_state["status"] = "burnin"
+        save_state(state_path, state)
+        burnin_result = run_client(
+            entry=entry,
+            role="burnin",
+            selection=burnin,
+            output=entry["dir"] / "burnin",
+            state=state,
+        )
+        cell_state["burnin"] = {
+            "pass_rate": burnin_result["pass_rate"],
+            "feasible": burnin_result["feasible"],
+        }
+        role_order = manifest["execution"][
+            "even_cell_order" if index % 2 == 0 else "odd_cell_order"
+        ]
+        cell_state["status"] = "measured"
+        cell_state["role_order"] = role_order
+        save_state(state_path, state)
+        for role in role_order:
+            level, selection = selection_for(manifest, cell, role)
+            result = run_client(
+                entry=entry,
+                role=role,
+                selection=selection,
+                output=entry["dir"] / f"{level}-rep{role[-1]}",
+                state=state,
+            )
+            cell_state["runs"].append(
+                {
+                    "role": role,
+                    "level": level,
+                    "anchor": selection["anchor"],
+                    "selected_count": selection["selected_count"],
+                    "pass_rate": result["pass_rate"],
+                    "feasible": result["feasible"],
+                    "elapsed_s": result["interval"]["elapsed_s"],
+                }
+            )
+            save_state(state_path, state)
+        cell_state["status"] = "stopping"
+        save_state(state_path, state)
+    except Exception as error:  # noqa: BLE001
+        failure = error
+    finally:
+        try:
+            base.stop_entry(entry)
+        except Exception as error:  # noqa: BLE001
+            failure = failure or error
+        time.sleep(2.0)
+        try:
+            wait_all_idle()
+        except Exception as error:  # noqa: BLE001
+            failure = failure or error
+
+    cell_hours = base.live_gpu_hours([entry])
+    state["gpu_hours_total"] += cell_hours
+    cell_state["gpu_hours"] = cell_hours
+    if failure is not None:
+        cell_state["status"] = "failed"
+        cell_state["failure"] = repr(failure)
+        state["status"] = "failed"
+        state["failures"].append({"cell": cell, "failure": repr(failure)})
+        save_state(state_path, state)
+        raise failure
+    validation = base.validate_cell(entry)
+    cell_state["validation"] = validation
+    cell_state["status"] = "complete"
+    cell_state["completed_at"] = time.time()
+    state["completed_cells"] += 1
+    save_state(state_path, state)
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("--manifest", type=Path, required=True)
+    result.add_argument("--run-root", type=Path, required=True)
+    result.add_argument("--aituner-root", type=Path, required=True)
+    result.add_argument("--vllm-source", type=Path, required=True)
+    result.add_argument("--venv", type=Path, required=True)
+    result.add_argument("--model", type=Path, required=True)
+    result.add_argument("--client", type=Path, required=True)
+    return result
+
+
+def main() -> None:
+    args = parser().parse_args()
+    manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
+    if manifest["status"] != "PASS":
+        raise RuntimeError("pilot manifest did not pass preflight")
+    args.run_root.mkdir(parents=True, exist_ok=True)
+    copied_manifest = args.run_root / "pilot-manifest.json"
+    if not copied_manifest.exists():
+        atomic_json(copied_manifest, manifest)
+    configure_base(args, manifest)
+    state_path = args.run_root / "controller-state.json"
+    state = load_state(state_path, base.GPU_LIMIT)
+    state["status"] = "running"
+    save_state(state_path, state)
+    for index, cell in enumerate(ORDER):
+        execute_cell(
+            index=index,
+            cell=cell,
+            manifest=manifest,
+            run_root=args.run_root,
+            state_path=state_path,
+            state=state,
+        )
+    state["status"] = "complete"
+    state["completed_at"] = time.time()
+    save_state(state_path, state)
+    print(json.dumps({
+        "status": state["status"],
+        "completed_cells": state["completed_cells"],
+        "gpu_hours_total": state["gpu_hours_total"],
+    }, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/prepare_pilot.py

@@ -0,0 +1,355 @@
+#!/usr/bin/env python3
+"""Materialize session-disjoint pilot repeats and freeze attainable anchors.
+
+The private outputs retain prompt text and stay on the experiment host.  The
+public manifest contains only aggregate counts, hashes, paths, and parameters.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+
+AITUNER_ROOT = Path(os.environ.get("AITUNER_ROOT", Path(__file__).resolve().parents[2]))
+sys.path.insert(0, str(AITUNER_ROOT / "src"))
+
+from aituner.spec import load_study_spec  # noqa: E402
+from aituner.trace import load_trace_requests, select_requests_for_threshold  # noqa: E402
+
+
+ROLES = ("burnin", "low1", "high1", "low2", "high2", "low3", "high3")
+CELLS = {
+    "tp1_mns8": {"tp": 1, "mns": 8, "frontier_req_s_gpu": 2.3833333333333333},
+    "tp1_mns64": {"tp": 1, "mns": 64, "frontier_req_s_gpu": 2.3833333333333333},
+    "tp2_mns8": {"tp": 2, "mns": 8, "frontier_req_s_gpu": 2.2416666666666667},
+    "tp2_mns64": {"tp": 2, "mns": 64, "frontier_req_s_gpu": 2.3},
+    "tp4_mns16": {"tp": 4, "mns": 16, "frontier_req_s_gpu": 2.5},
+    "tp4_mns64": {"tp": 4, "mns": 64, "frontier_req_s_gpu": 2.5},
+}
+TARGET_MULTIPLIERS = {"low": 0.85, "high": 1.25}
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_suffix(path.suffix + ".tmp")
+    tmp.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    os.replace(tmp, path)
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def order_hash(values: list[str]) -> str:
+    return hashlib.sha256("\n".join(values).encode()).hexdigest()
+
+
+def resolve_source_trace(windows_path: Path, window_id: str) -> tuple[dict[str, Any], Path]:
+    payload = json.loads(windows_path.read_text(encoding="utf-8"))
+    for window in payload["windows"]:
+        if window["window_id"] != window_id:
+            continue
+        trace = Path(window["trace_file"])
+        if not trace.is_absolute():
+            trace = (windows_path.parent / trace).resolve()
+        return window, trace
+    raise ValueError(f"window not found: {window_id}")
+
+
+def materialize_bands(
+    source_trace: Path,
+    source_window: dict[str, Any],
+    private_root: Path,
+) -> tuple[Path, dict[str, Any]]:
+    traces_root = private_root / "traces"
+    traces_root.mkdir(parents=True, exist_ok=True)
+    temporary = {role: traces_root / f".{role}.jsonl.tmp" for role in ROLES}
+    final = {role: traces_root / f"{role}.jsonl" for role in ROLES}
+    handles = {role: temporary[role].open("w", encoding="utf-8") for role in ROLES}
+    stats = {
+        role: {
+            "rows": 0,
+            "sum_input_tokens": 0,
+            "min_timestamp": None,
+            "max_timestamp": None,
+        }
+        for role in ROLES
+    }
+    try:
+        with source_trace.open(encoding="utf-8") as source:
+            for line_number, line in enumerate(source):
+                row = json.loads(line)
+                value = float(row["sampling_u"])
+                if not 0.0 <= value <= 1.0:
+                    raise ValueError(f"sampling_u outside [0,1] at line {line_number}")
+                band = min(len(ROLES) - 1, int(value * len(ROLES)))
+                role = ROLES[band]
+                remapped = value * len(ROLES) - band
+                row["sampling_u"] = min(remapped, math.nextafter(1.0, 0.0))
+                row["fidelity_pilot_band"] = role
+                handles[role].write(json.dumps(row, ensure_ascii=False) + "\n")
+                timestamp = float(row["timestamp"])
+                item = stats[role]
+                item["rows"] += 1
+                item["sum_input_tokens"] += int(row.get("input_length") or 0)
+                item["min_timestamp"] = (
+                    timestamp if item["min_timestamp"] is None
+                    else min(float(item["min_timestamp"]), timestamp)
+                )
+                item["max_timestamp"] = (
+                    timestamp if item["max_timestamp"] is None
+                    else max(float(item["max_timestamp"]), timestamp)
+                )
+    finally:
+        for handle in handles.values():
+            handle.close()
+    for role in ROLES:
+        os.replace(temporary[role], final[role])
+        stats[role]["sha256"] = sha256_file(final[role])
+        stats[role]["bytes"] = final[role].stat().st_size
+
+    windows = []
+    for role in ROLES:
+        window = dict(source_window)
+        window["window_id"] = f"fidelity_pilot_{role}"
+        window["trace_file"] = f"traces/{role}.jsonl"
+        window["num_requests"] = stats[role]["rows"]
+        window["sum_input_length"] = stats[role]["sum_input_tokens"]
+        window["sampling_strategy"] = "session_uniform_seven_disjoint_bands_remapped"
+        window["fidelity_pilot_role"] = role
+        windows.append(window)
+    private_windows = private_root / "windows.json"
+    atomic_json(
+        private_windows,
+        {
+            "schema": "fidelity-pilot-private-windows-v1",
+            "roles": list(ROLES),
+            "windows": windows,
+        },
+    )
+    return private_windows, stats
+
+
+def write_studies(
+    *,
+    base_primary: Path,
+    base_tp4: Path,
+    private_windows: Path,
+    private_root: Path,
+) -> dict[str, dict[str, Path]]:
+    bases = {
+        "primary": json.loads(base_primary.read_text(encoding="utf-8")),
+        "tp4": json.loads(base_tp4.read_text(encoding="utf-8")),
+    }
+    result: dict[str, dict[str, Path]] = {}
+    for role in ROLES:
+        result[role] = {}
+        for tier, base in bases.items():
+            payload = json.loads(json.dumps(base))
+            payload["study_id"] = f"fidelity-prefix-pilot-{role}-{tier}"
+            payload["hardware"]["host_candidates"] = ["dash0"]
+            payload["engine"]["engine_version"] = "0.24.1.dev3+opprof"
+            payload["trace"]["windows_path"] = str(private_windows)
+            payload["trace"]["window_id"] = f"fidelity_pilot_{role}"
+            path = private_root / "studies" / f"{role}-{tier}.json"
+            atomic_json(path, payload)
+            result[role][tier] = path
+    return result
+
+
+def attainable_anchor(requests: list[Any], target_count: int) -> tuple[float, list[Any]]:
+    ordered = sorted(float(request.sampling_u) for request in requests)
+    if not ordered:
+        raise ValueError("no requests after study filtering")
+    candidate_indices = sorted({
+        max(0, min(len(ordered) - 1, target_count - 1)),
+        max(0, min(len(ordered) - 1, target_count)),
+    })
+    candidates = []
+    for index in candidate_indices:
+        anchor = ordered[index]
+        selected = select_requests_for_threshold(requests, threshold=anchor)
+        candidates.append((abs(len(selected) - target_count), len(selected), anchor, selected))
+    _error, _count, anchor, selected = min(candidates, key=lambda item: (item[0], item[1]))
+    return anchor, selected
+
+
+def selected_record(selected: list[Any], *, tp: int, duration_s: float) -> dict[str, Any]:
+    return {
+        "anchor": max(float(request.sampling_u) for request in selected),
+        "selected_count": len(selected),
+        "offered_req_s": len(selected) / duration_s,
+        "offered_req_s_per_gpu": len(selected) / duration_s / tp,
+        "request_id_order_sha256": order_hash([request.row_id for request in selected]),
+        "arrival_order_sha256": order_hash([f"{request.arrival_s:.12f}" for request in selected]),
+        "input_length_order_sha256": order_hash(
+            [str(request.prompt_tokens_hint) for request in selected]
+        ),
+    }
+
+
+def build_manifest(
+    *,
+    studies: dict[str, dict[str, Path]],
+    private_windows: Path,
+    band_stats: dict[str, Any],
+    source_trace: Path,
+    source_windows: Path,
+    source_window_id: str,
+) -> dict[str, Any]:
+    loaded = {}
+    durations = {}
+    for role, tiers in studies.items():
+        loaded[role] = {}
+        for tier, path in tiers.items():
+            study = load_study_spec(path)
+            window, requests = load_trace_requests(study, study_spec_path=path)
+            loaded[role][tier] = requests
+            durations[role] = float(window.window_end - window.window_start)
+
+    cells = {}
+    all_hashes = []
+    for cell, config in CELLS.items():
+        tp = int(config["tp"])
+        tier = "tp4" if tp == 4 else "primary"
+        targets = {}
+        for level, multiplier in TARGET_MULTIPLIERS.items():
+            target_rate = float(config["frontier_req_s_gpu"]) * multiplier
+            target_count = round(target_rate * durations["low1"] * tp)
+            roles = [
+                role
+                for role in ROLES
+                if role.startswith(level) or (level == "low" and role == "burnin")
+            ]
+            selections = {}
+            for role in roles:
+                anchor, selected = attainable_anchor(loaded[role][tier], target_count)
+                record = selected_record(selected, tp=tp, duration_s=durations[role])
+                record["anchor"] = anchor
+                record["study"] = str(studies[role][tier])
+                selections[role] = record
+                all_hashes.append(record["request_id_order_sha256"])
+            targets[level] = {
+                "multiplier": multiplier,
+                "target_req_s_per_gpu": target_rate,
+                "target_count": target_count,
+                "selections": selections,
+            }
+        cells[cell] = {**config, "targets": targets}
+
+    red_flags = []
+    for cell, config in cells.items():
+        for level, target in config["targets"].items():
+            if not target["selections"]:
+                red_flags.append(f"missing_{cell}_{level}")
+            for selection in target["selections"].values():
+                if selection["selected_count"] <= 0:
+                    red_flags.append(f"empty_{cell}_{level}")
+    per_cell_distinct = {}
+    for cell, config in cells.items():
+        hashes = [
+            selection["request_id_order_sha256"]
+            for target in config["targets"].values()
+            for selection in target["selections"].values()
+        ]
+        per_cell_distinct[cell] = len(hashes) == len(set(hashes))
+        if not per_cell_distinct[cell]:
+            red_flags.append(f"session_bands_overlap_{cell}")
+    return {
+        "schema": "fidelity-prefix-pilot-manifest-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "source": {
+            "windows": str(source_windows),
+            "window_id": source_window_id,
+            "trace": str(source_trace),
+            "trace_sha256": sha256_file(source_trace),
+        },
+        "private": {
+            "windows": str(private_windows),
+            "windows_sha256": sha256_file(private_windows),
+            "band_stats": band_stats,
+            "studies": {
+                role: {tier: str(path) for tier, path in tiers.items()}
+                for role, tiers in studies.items()
+            },
+        },
+        "roles": list(ROLES),
+        "cells": cells,
+        "execution": {
+            "cutoff_s": 5.0,
+            "replicates_per_level": 3,
+            "label": "2-of-3 session-disjoint repetitions",
+            "even_cell_order": ["low1", "high1", "high2", "low2", "low3", "high3"],
+            "odd_cell_order": ["high1", "low1", "low2", "high2", "high3", "low3"],
+            "hard_cap_h20_hours": 3.5,
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "n_cells": len(cells),
+            "n_roles": len(ROLES),
+            "selected_sets": len(all_hashes),
+            "distinct_selected_sets": len(set(all_hashes)),
+            "per_cell_selected_sets_distinct": per_cell_distinct,
+            "invariants": {
+                "cells_6": len(cells) == 6,
+                "roles_7": len(ROLES) == 7,
+                "band_rows_nonzero": all(stats["rows"] > 0 for stats in band_stats.values()),
+                "session_bands_disjoint_per_cell": all(per_cell_distinct.values()),
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--source-windows", type=Path, required=True)
+    parser.add_argument("--source-window-id", default="chat_w20260312_1000")
+    parser.add_argument("--base-primary-study", type=Path, required=True)
+    parser.add_argument("--base-tp4-study", type=Path, required=True)
+    parser.add_argument("--private-root", type=Path, required=True)
+    parser.add_argument("--public-manifest", type=Path, required=True)
+    args = parser.parse_args()
+
+    source_window, source_trace = resolve_source_trace(
+        args.source_windows, args.source_window_id
+    )
+    private_windows, band_stats = materialize_bands(
+        source_trace, source_window, args.private_root
+    )
+    studies = write_studies(
+        base_primary=args.base_primary_study,
+        base_tp4=args.base_tp4_study,
+        private_windows=private_windows,
+        private_root=args.private_root,
+    )
+    manifest = build_manifest(
+        studies=studies,
+        private_windows=private_windows,
+        band_stats=band_stats,
+        source_trace=source_trace,
+        source_windows=args.source_windows,
+        source_window_id=args.source_window_id,
+    )
+    atomic_json(args.public_manifest, manifest)
+    print(json.dumps({
+        "status": manifest["status"],
+        "manifest": str(args.public_manifest),
+        "sanity": manifest["sanity"],
+    }, sort_keys=True))
+    if manifest["status"] != "PASS":
+        raise RuntimeError(f"pilot preflight failed: {manifest['sanity']['red_flags']}")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/prepare_pilot_simulator.py

@@ -0,0 +1,331 @@
+#!/usr/bin/env python3
+"""Prepare exact Frontier fixtures for the P1 primary low/high probes.
+
+Prompt-bearing band traces remain under ``--private-root``.  The emitted
+fixtures and public manifest contain token IDs, block IDs, hashes, and
+aggregate metadata, but no prompt text.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import math
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+from transformers import AutoTokenizer
+
+
+HERE = Path(__file__).resolve().parent
+AITUNER_ROOT = HERE.parents[1]
+sys.path.insert(0, str(HERE))
+
+import prepare_pilot as pilot  # noqa: E402
+
+
+PRIMARY_ROLES = ("low1", "high1")
+
+
+def load_module(path: Path):
+    spec = importlib.util.spec_from_file_location("simfid_s2rb_prepare", path)
+    if spec is None or spec.loader is None:
+        raise ImportError(path)
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def order_hash(values: list[str]) -> str:
+    return hashlib.sha256("\n".join(values).encode()).hexdigest()
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def git_capture(root: Path, *arguments: str) -> str:
+    return subprocess.run(
+        ["git", "-C", str(root), *arguments],
+        check=True,
+        text=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    ).stdout
+
+
+def raw_rows(path: Path) -> dict[int, dict[str, Any]]:
+    result = {}
+    with path.open(encoding="utf-8") as source:
+        for index, line in enumerate(source):
+            if line.strip():
+                result[index] = json.loads(line)
+    return result
+
+
+def selected_hashes(
+    selected: list[Any], rows: dict[int, dict[str, Any]]
+) -> dict[str, str]:
+    identifiers = []
+    arrivals = []
+    lengths = []
+    for item in selected:
+        row = rows[item.row_index]
+        identifiers.append(str(row.get("request_id") or row.get("id") or item.row_index))
+        arrivals.append(f"{float(item.timestamp) * 0.1:.12f}")
+        lengths.append(str(int(item.input_length)))
+    return {
+        "request_id_order_sha256": order_hash(identifiers),
+        "arrival_order_sha256": order_hash(arrivals),
+        "input_length_order_sha256": order_hash(lengths),
+    }
+
+
+def kv_blocks(raw_root: Path, cell: str) -> int:
+    stream = next((raw_root / cell / "opprof").glob("*.jsonl"))
+    with stream.open(encoding="utf-8") as source:
+        for line in source:
+            record = json.loads(line)
+            if "step_index" in record:
+                return int(record["kv"]["total_blocks"])
+    raise ValueError(f"no Layer-1 record for {cell}")
+
+
+def source_window(windows_path: Path, window_id: str) -> tuple[dict[str, Any], Path]:
+    return pilot.resolve_source_trace(windows_path, window_id)
+
+
+def prepare(args: argparse.Namespace) -> dict[str, Any]:
+    simulator = load_module(args.replayserve_root / "tools/simfid_s2rb_prepare.py")
+    manifest = json.loads(args.pilot_manifest.read_text(encoding="utf-8"))
+    window, trace = source_window(args.source_windows, args.source_window_id)
+    if args.band_root is not None:
+        role_paths = {
+            role: (args.band_root / f"{role}.jsonl").resolve()
+            for role in PRIMARY_ROLES
+        }
+        band_stats = {
+            role: manifest["private"]["band_stats"][role]
+            for role in PRIMARY_ROLES
+        }
+        for role, path in role_paths.items():
+            if sha256_file(path) != band_stats[role]["sha256"]:
+                raise ValueError(f"pre-materialized band hash mismatch: {role}")
+        private_windows = None
+    else:
+        private_windows, all_band_stats = pilot.materialize_bands(
+            trace, window, args.private_root
+        )
+        private_payload = json.loads(private_windows.read_text(encoding="utf-8"))
+        role_paths = {
+            item["fidelity_pilot_role"]: (
+                private_windows.parent / item["trace_file"]
+            ).resolve()
+            for item in private_payload["windows"]
+        }
+        band_stats = {
+            role: all_band_stats[role]
+            for role in PRIMARY_ROLES
+        }
+
+    tokenizer = AutoTokenizer.from_pretrained(
+        args.tokenizer, local_files_only=True, use_fast=True
+    )
+    fixture_root = args.output / "fixtures"
+    config_root = args.output / "configs"
+    fixture_root.mkdir(parents=True, exist_ok=True)
+    config_root.mkdir(parents=True, exist_ok=True)
+    entries = []
+    red_flags = []
+    for role in PRIMARY_ROLES:
+        trace_path = role_paths[role]
+        retained, trace_stats = simulator.scan_trace(trace_path)
+        rows = raw_rows(trace_path)
+        primary_pool = [retained[(index * len(retained)) // 512] for index in range(512)]
+        selections: dict[str, list[Any]] = {}
+        selected_union: set[int] = set()
+        for cell, cell_manifest in sorted(manifest["cells"].items()):
+            level = "low" if role.startswith("low") else "high"
+            expected = cell_manifest["targets"][level]["selections"][role]
+            pool = retained if int(cell_manifest["tp"]) == 4 else primary_pool
+            selected = [item for item in pool if item.sampling_u <= float(expected["anchor"])]
+            selections[cell] = selected
+            selected_union.update(item.row_index for item in selected)
+            hashes = selected_hashes(selected, rows)
+            if len(selected) != int(expected["selected_count"]):
+                red_flags.append(f"selection_count_{cell}_{role}")
+            for key, value in hashes.items():
+                if value != expected[key]:
+                    red_flags.append(f"selection_hash_{cell}_{role}_{key}")
+
+        token_gates, selected_records, block_stats = simulator.tokenize_and_hash(
+            trace=trace_path,
+            tokenizer=tokenizer,
+            retained=retained,
+            selected_union=selected_union,
+        )
+        if any(gate["status"] != "pass" for gate in token_gates.values()):
+            red_flags.append(f"token_gate_{role}")
+        for cell, selected in selections.items():
+            cell_manifest = manifest["cells"][cell]
+            level = "low" if role.startswith("low") else "high"
+            expected = cell_manifest["targets"][level]["selections"][role]
+            fixture_id = f"fidelity_p1_{cell}_{role}"
+            cell_record = {
+                "cell_id": cell,
+                "tensor_parallel_size": int(cell_manifest["tp"]),
+                "max_num_seqs": int(cell_manifest["mns"]),
+                "store_role": "companion" if int(cell_manifest["tp"]) == 4 else "primary",
+                "kv_capacity": {
+                    "block_size_tokens": 16,
+                    "num_blocks": kv_blocks(args.phase6_raw_root, cell),
+                },
+            }
+            probe = {
+                "probe_index": 0 if role == "low1" else 1,
+                "sampling_u": float(expected["anchor"]),
+            }
+            fixture = simulator.create_fixture(
+                fixture_root=fixture_root,
+                fixture_id=fixture_id,
+                cell=cell_record,
+                probe=probe,
+                row_indexes=[item.row_index for item in selected],
+                meta_by_index={item.row_index: item for item in retained},
+                selected_records=selected_records,
+            )
+            config_path = config_root / f"{fixture_id}.json"
+            config = simulator.build_config(
+                path=config_path,
+                cell=cell_record,
+                mode="frozen-calibrated",
+                fixture_ids=[fixture_id],
+                frontier_root=args.frontier_root,
+                cache_dir=args.cache_dir,
+            )
+            entries.append(
+                {
+                    "cell": cell,
+                    "role": role,
+                    "level": level,
+                    "anchor": expected["anchor"],
+                    "selected_count": len(selected),
+                    "fixture_id": fixture_id,
+                    "fixture_manifest": str(
+                        (fixture_root / fixture_id / "fixture_manifest.json").resolve()
+                    ),
+                    "frontier_csv": fixture["frontier_csv"]["path"],
+                    "sidecar": fixture["sidecar_jsonl"]["path"],
+                    "config": str(config_path.resolve()),
+                    "calibration_scale": config["calibration"]["a_tp"],
+                }
+            )
+        if block_stats["selected_union_records"] != len(selected_union):
+            red_flags.append(f"selected_union_{role}")
+        if trace_stats["retained_inclusive_0_8192"] < 512:
+            red_flags.append(f"retained_too_small_{role}")
+
+    selected_counts = [int(entry["selected_count"]) for entry in entries]
+    calibration = [float(entry["calibration_scale"]) for entry in entries]
+    result = {
+        "schema": "fidelity-p1-frontier-prepared-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "source": {
+            "pilot_manifest": str(args.pilot_manifest.resolve()),
+            "source_windows": str(args.source_windows.resolve()),
+            "source_window_id": args.source_window_id,
+            "source_trace": str(trace.resolve()),
+            "private_windows": (
+                str(private_windows.resolve()) if private_windows is not None else None
+            ),
+            "pre_materialized_band_root": (
+                str(args.band_root.resolve()) if args.band_root is not None else None
+            ),
+            "band_stats": band_stats,
+        },
+        "simulator": {
+            "replayserve_root": str(args.replayserve_root.resolve()),
+            "frontier_root": str(args.frontier_root.resolve()),
+            "tokenizer": str(args.tokenizer.resolve()),
+            "mode": "frozen-calibrated",
+        },
+        "generator": {
+            "script": str(Path(__file__).resolve()),
+            "script_sha256": sha256_file(Path(__file__).resolve()),
+            "aituner_git_head": git_capture(AITUNER_ROOT, "rev-parse", "HEAD").strip(),
+            "aituner_git_status_short": git_capture(AITUNER_ROOT, "status", "--short"),
+        },
+        "entries": entries,
+        "sanity": {
+            "red_flags": red_flags,
+            "n": len(entries),
+            "selected_count": {
+                "n": len(selected_counts),
+                "min": min(selected_counts),
+                "max": max(selected_counts),
+                "distinct_n": len(set(selected_counts)),
+            },
+            "calibration_scale": {
+                "n": len(calibration),
+                "min": min(calibration),
+                "max": max(calibration),
+                "distinct_n": len(set(calibration)),
+            },
+            "invariants": {
+                "entries_12": len(entries) == 12,
+                "roles_2": {entry["role"] for entry in entries} == set(PRIMARY_ROLES),
+                "cells_6": len({entry["cell"] for entry in entries}) == 6,
+                "selected_nonnegative": all(value > 0 for value in selected_counts),
+                "per_config_not_identical": len(set(selected_counts)) > 1,
+            },
+        },
+    }
+    args.public_manifest.parent.mkdir(parents=True, exist_ok=True)
+    args.public_manifest.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    return result
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("--pilot-manifest", type=Path, required=True)
+    result.add_argument("--source-windows", type=Path, required=True)
+    result.add_argument("--source-window-id", required=True)
+    result.add_argument("--private-root", type=Path, required=True)
+    result.add_argument("--band-root", type=Path)
+    result.add_argument("--output", type=Path, required=True)
+    result.add_argument("--public-manifest", type=Path, required=True)
+    result.add_argument("--phase6-raw-root", type=Path, required=True)
+    result.add_argument("--replayserve-root", type=Path, required=True)
+    result.add_argument("--frontier-root", type=Path, required=True)
+    result.add_argument("--cache-dir", type=Path, required=True)
+    result.add_argument("--tokenizer", type=Path, required=True)
+    return result
+
+
+def main() -> None:
+    result = prepare(parser().parse_args())
+    print(
+        json.dumps(
+            {
+                "status": result["status"],
+                "entries": len(result["entries"]),
+                "red_flags": result["sanity"]["red_flags"],
+            },
+            sort_keys=True,
+        )
+    )
+    if result["status"] != "PASS":
+        raise RuntimeError(result["sanity"]["red_flags"])
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/run_pilot_simulator.py

@@ -0,0 +1,292 @@
+#!/usr/bin/env python3
+"""Run and score the 12 frozen Frontier P1 primary probes, CPU only."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import os
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+
+def load_module(name: str, path: Path):
+    spec = importlib.util.spec_from_file_location(name, path)
+    if spec is None or spec.loader is None:
+        raise ImportError(path)
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix(path.suffix + ".tmp")
+    temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    os.replace(temporary, path)
+
+
+def git_capture(root: Path, *arguments: str) -> str:
+    return subprocess.run(
+        ["git", "-C", str(root), *arguments],
+        check=True,
+        text=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    ).stdout
+
+
+def execute(args: argparse.Namespace) -> dict[str, Any]:
+    prepared = json.loads(args.prepared_manifest.read_text(encoding="utf-8"))
+    if prepared["status"] != "PASS":
+        raise RuntimeError("prepared simulator manifest did not pass")
+    driver = load_module(
+        "simfid_execution_driver",
+        args.replayserve_root
+        / "runs/simfid_s2rb/results/execution_driver.py",
+    )
+    head = git_capture(args.frontier_root, "rev-parse", "HEAD").strip()
+    status_short = git_capture(args.frontier_root, "status", "--short")
+    aituner_root = Path(__file__).resolve().parents[2]
+    aituner_head = git_capture(aituner_root, "rev-parse", "HEAD").strip()
+    aituner_status_short = git_capture(aituner_root, "status", "--short")
+    results = []
+    failures = []
+    gpu_visibility_disabled = True
+    for sequence, entry in enumerate(prepared["entries"]):
+        run_root = args.output / f"{sequence:02d}_{entry['fixture_id']}"
+        scorer_path = run_root / "scorer_output.json"
+        if scorer_path.is_file() and args.resume:
+            scorer = json.loads(scorer_path.read_text(encoding="utf-8"))
+            results.append({**entry, "sequence": sequence, "scorer": scorer, "resumed": True})
+            continue
+        run_root.mkdir(parents=True, exist_ok=True)
+        config_path = Path(entry["config"])
+        config = json.loads(config_path.read_text(encoding="utf-8"))
+        fixture_manifest_path = Path(entry["fixture_manifest"])
+        fixture = json.loads(fixture_manifest_path.read_text(encoding="utf-8"))
+        trace_path = Path(entry["frontier_csv"])
+        sidecar_path = Path(entry["sidecar"])
+        metrics_root = run_root / "frontier_metrics"
+        run_id = f"fidelity_p1_frontier_{sequence:02d}_{entry['cell']}_{entry['role']}"
+        knobs = config["frontier"]["knobs"]
+        command = driver.build_command(
+            trace_path=trace_path,
+            metrics_root=metrics_root,
+            run_id=run_id,
+            knobs=knobs,
+        )
+        driver.audit_command(command, knobs)
+        row = {
+            "hook_path": config["calibration"]["hook_path"],
+            "applied_a_tp": config["calibration"]["a_tp"],
+            "sidecar_path": str(sidecar_path),
+            "request_count": int(fixture["request_count"]),
+            "tensor_parallel_size": int(fixture["tensor_parallel_size"]),
+        }
+        environment = driver.environment_for(row)
+        gpu_visibility_disabled = gpu_visibility_disabled and (
+            environment.get("CUDA_VISIBLE_DEVICES") == ""
+            and environment.get("NVIDIA_VISIBLE_DEVICES") == "void"
+        )
+        run_manifest = {
+            "schema": "fidelity-p1-frontier-run-v1",
+            "sequence": sequence,
+            "cell": entry["cell"],
+            "role": entry["role"],
+            "anchor": entry["anchor"],
+            "request_count": entry["selected_count"],
+            "frontier": {
+                "root": str(args.frontier_root.resolve()),
+                "git_head": head,
+                "git_status_short": status_short,
+            },
+            "runner": {
+                "script": str(Path(__file__).resolve()),
+                "script_sha256": sha256_file(Path(__file__).resolve()),
+                "aituner_git_head": aituner_head,
+                "aituner_git_status_short": aituner_status_short,
+            },
+            "inputs": {
+                "config": str(config_path),
+                "config_sha256": sha256_file(config_path),
+                "fixture_manifest": str(fixture_manifest_path),
+                "fixture_manifest_sha256": sha256_file(fixture_manifest_path),
+                "frontier_csv": str(trace_path),
+                "frontier_csv_sha256": sha256_file(trace_path),
+                "sidecar": str(sidecar_path),
+                "sidecar_sha256": sha256_file(sidecar_path),
+            },
+            "environment": {
+                key: environment[key]
+                for key in (
+                    "PYTHONPATH",
+                    "FRONTIER_EXECUTION_TIME_SCALE",
+                    "CUDA_VISIBLE_DEVICES",
+                    "NVIDIA_VISIBLE_DEVICES",
+                    "FRONTIER_LOG_LEVEL",
+                )
+            },
+            "command": command,
+            "contains_prompt_text": False,
+        }
+        atomic_json(run_root / "run_manifest.json", run_manifest)
+        start = time.time()
+        with (run_root / "stdout.log").open("w", encoding="utf-8") as stdout, (
+            run_root / "stderr.log"
+        ).open("w", encoding="utf-8") as stderr:
+            try:
+                process = subprocess.run(
+                    command,
+                    cwd=args.frontier_root,
+                    env=environment,
+                    stdout=stdout,
+                    stderr=stderr,
+                    timeout=args.timeout_s,
+                )
+                return_code = int(process.returncode)
+            except subprocess.TimeoutExpired:
+                return_code = 124
+        runtime = time.time() - start
+        if return_code != 0:
+            failure = {
+                "sequence": sequence,
+                "cell": entry["cell"],
+                "role": entry["role"],
+                "return_code": return_code,
+                "runtime_s": runtime,
+            }
+            failures.append(failure)
+            atomic_json(run_root / "failure.json", failure)
+            break
+        system_path, request_path = driver.find_metrics(run_root)
+        scorer = driver.score_trial(row, system_path, request_path)
+        scorer["runtime_s"] = runtime
+        atomic_json(scorer_path, scorer)
+        results.append({**entry, "sequence": sequence, "scorer": scorer, "resumed": False})
+        print(
+            json.dumps(
+                {
+                    "sequence": sequence,
+                    "cell": entry["cell"],
+                    "role": entry["role"],
+                    "runtime_s": runtime,
+                    "sim_pass_rate": scorer["slo"]["pass_rate"],
+                    "sim_feasible": scorer["slo"]["feasible"],
+                },
+                sort_keys=True,
+            ),
+            flush=True,
+        )
+
+    pass_rates = [float(item["scorer"]["slo"]["pass_rate"]) for item in results]
+    throughputs = [
+        float(item["scorer"]["throughput_requests_per_second_per_gpu"])
+        for item in results
+    ]
+    runtimes = [float(item["scorer"]["runtime_s"]) for item in results]
+    red_flags = []
+    if failures:
+        red_flags.append("frontier_run_failure")
+    if len(results) != 12:
+        red_flags.append("runs_not_12")
+    if any(not 0.0 <= value <= 1.0 for value in pass_rates):
+        red_flags.append("pass_rate_out_of_range")
+    if any(value <= 0 for value in throughputs):
+        red_flags.append("nonpositive_throughput")
+    result = {
+        "schema": "fidelity-p1-frontier-result-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "prepared_manifest": str(args.prepared_manifest.resolve()),
+        "prepared_manifest_sha256": sha256_file(args.prepared_manifest),
+        "frontier": {
+            "root": str(args.frontier_root.resolve()),
+            "git_head": head,
+            "git_status_short": status_short,
+        },
+        "runner": {
+            "script": str(Path(__file__).resolve()),
+            "script_sha256": sha256_file(Path(__file__).resolve()),
+            "aituner_git_head": aituner_head,
+            "aituner_git_status_short": aituner_status_short,
+        },
+        "results": results,
+        "failures": failures,
+        "sanity": {
+            "red_flags": red_flags,
+            "n": len(results),
+            "pass_rate": {
+                "n": len(pass_rates),
+                "min": min(pass_rates) if pass_rates else None,
+                "max": max(pass_rates) if pass_rates else None,
+                "distinct_n": len(set(pass_rates)),
+            },
+            "throughput_per_gpu": {
+                "n": len(throughputs),
+                "min": min(throughputs) if throughputs else None,
+                "max": max(throughputs) if throughputs else None,
+                "distinct_n": len(set(throughputs)),
+            },
+            "runtime_s": {
+                "n": len(runtimes),
+                "min": min(runtimes) if runtimes else None,
+                "max": max(runtimes) if runtimes else None,
+                "distinct_n": len(set(runtimes)),
+            },
+            "invariants": {
+                "runs_12": len(results) == 12,
+                "zero_failures": not failures,
+                "ratios_bounded": all(0.0 <= value <= 1.0 for value in pass_rates),
+                "nonnegative_metrics": all(value > 0 for value in throughputs),
+                "per_config_not_identical": len(set(pass_rates)) > 1,
+                "gpu_visibility_disabled": gpu_visibility_disabled,
+            },
+        },
+    }
+    atomic_json(args.output / "metrics.json", result)
+    return result
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("--prepared-manifest", type=Path, required=True)
+    result.add_argument("--output", type=Path, required=True)
+    result.add_argument("--replayserve-root", type=Path, required=True)
+    result.add_argument("--frontier-root", type=Path, required=True)
+    result.add_argument("--timeout-s", type=float, default=900.0)
+    result.add_argument("--resume", action="store_true")
+    return result
+
+
+def main() -> None:
+    result = execute(parser().parse_args())
+    print(
+        json.dumps(
+            {
+                "status": result["status"],
+                "runs": len(result["results"]),
+                "red_flags": result["sanity"]["red_flags"],
+            },
+            sort_keys=True,
+        )
+    )
+    if result["status"] != "PASS":
+        raise RuntimeError(result["sanity"]["red_flags"])
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/strong-baseline-metrics.json

@@ -0,0 +1,477 @@
+{
+  "comparison": "same 5-second prefix, folds, logistic family, regularization, and frozen Frontier outputs; the only nested difference is real Layer-1 engine state",
+  "decision": {
+    "contribution_established": false,
+    "prospective_requirement": "repeat sim+outcome versus sim+outcome+instrumentation on complete held-out tasks"
+  },
+  "features": {
+    "instrumentation_only": [
+      "model_steps_per_second",
+      "waiting_mean",
+      "waiting_max",
+      "waiting_nonzero_share",
+      "running_mean",
+      "running_max",
+      "decode_batch_mean",
+      "decode_batch_max",
+      "decode_batch_cv",
+      "kv_usage_mean",
+      "kv_usage_max",
+      "kv_usage_end_minus_start",
+      "graph_none_share",
+      "graph_full_share",
+      "padding_fraction",
+      "prefill_token_fraction",
+      "preemptions"
+    ],
+    "shared_outcome": [
+      "log_offered_rate_per_gpu",
+      "log2_tp",
+      "log2_max_num_seqs",
+      "admitted_fraction",
+      "completed_over_admitted",
+      "completed_pass_rate",
+      "completed_fail_fraction_of_total",
+      "outstanding_over_admitted",
+      "ttft_max_over_slo_max",
+      "ttft_mean_over_slo_max",
+      "tpot_max_over_slo",
+      "tpot_mean_over_slo",
+      "admitted_input_tokens_mean_over_limit"
+    ],
+    "shared_simulator": [
+      "log_sim_completed_throughput_per_gpu",
+      "sim_slo_pass_rate",
+      "sim_slo_feasible"
+    ]
+  },
+  "headline": {
+    "group_bootstrap": {
+      "accuracy_delta_instrumentation_minus_outcome": {
+        "ci95": [
+          0.0,
+          0.18181818181818188
+        ],
+        "point": 0.08108108108108103
+      },
+      "brier_delta_instrumentation_minus_outcome": {
+        "ci95": [
+          -0.04292727744470806,
+          0.019924730979981074
+        ],
+        "point": -0.010145365131402809
+      },
+      "replicates": 10000,
+      "seed": 20260714,
+      "semantics": "group bootstrap over cells; diagnostic confidence interval"
+    },
+    "paired_correctness": {
+      "both_correct": 30,
+      "both_wrong": 4,
+      "instrumentation_only_correct": 3,
+      "mcnemar_exact_two_sided_p": 0.25,
+      "sim_outcome_only_correct": 0
+    },
+    "sim_plus_outcome": {
+      "classification": {
+        "accuracy": 0.8108108108108109,
+        "balanced_accuracy": 0.7242063492063493,
+        "brier": 0.1058226346682949,
+        "confusion": {
+          "false_negative": 3,
+          "false_positive": 4,
+          "true_negative": 5,
+          "true_positive": 25
+        },
+        "log_loss": 0.3011048455679668
+      },
+      "policy_0p95": {
+        "abstain_continue_full": 17,
+        "correctly_saved_h20_hours": 0.5429431818208333,
+        "decision_coverage": 0.5405405405405406,
+        "early_accept": 16,
+        "early_reject": 4,
+        "false_accept": 0,
+        "false_accept_examples": [],
+        "false_reject": 0,
+        "false_reject_examples": [],
+        "full_trial_h20_hours": 1.0669595034675,
+        "invalidly_saved_h20_hours": 0.0,
+        "remaining_h20_hours_at_cutoff": 0.957237281245278,
+        "saved_h20_hours_if_decisions_used": 0.5429431818208333,
+        "threshold": 0.95,
+        "valid_cost_reduction_fraction": 0.5088695307144538,
+        "valid_zero_error_policy": true
+      }
+    },
+    "sim_plus_outcome_plus_instrumentation": {
+      "classification": {
+        "accuracy": 0.8918918918918919,
+        "balanced_accuracy": 0.8154761904761905,
+        "brier": 0.0956772695368921,
+        "confusion": {
+          "false_negative": 1,
+          "false_positive": 3,
+          "true_negative": 6,
+          "true_positive": 27
+        },
+        "log_loss": 0.288823031828762
+      },
+      "policy_0p95": {
+        "abstain_continue_full": 12,
+        "correctly_saved_h20_hours": 0.7360063646722222,
+        "decision_coverage": 0.6756756756756757,
+        "early_accept": 20,
+        "early_reject": 5,
+        "false_accept": 0,
+        "false_accept_examples": [],
+        "false_reject": 0,
+        "false_reject_examples": [],
+        "full_trial_h20_hours": 1.0669595034675,
+        "invalidly_saved_h20_hours": 0.0,
+        "remaining_h20_hours_at_cutoff": 0.957237281245278,
+        "saved_h20_hours_if_decisions_used": 0.7360063646722222,
+        "threshold": 0.95,
+        "valid_cost_reduction_fraction": 0.6898165884274738,
+        "valid_zero_error_policy": true
+      }
+    }
+  },
+  "headline_regularization": 1.0,
+  "provenance": {
+    "frozen_simulator_manifest_scorer_set_sha256": "833842d96ecaa0b059ef99852621752f7989e63d100118b6025425fb119b7a55",
+    "phase6_metrics": "/home/gahow/phd/aituner/runs/opprof-phase6/phase6/metrics.json",
+    "phase6_metrics_sha256": "290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16",
+    "phase6_raw_root": "/home/gahow/phd/aituner/runs/opprof-phase6/phase6/solo-authoritative/cells",
+    "simulator_metrics": "/home/gahow/phd/replayserve/runs/simfid_s2rb/results/metrics.json",
+    "simulator_metrics_sha256": "55edb37d5692e979ab6f6dc6c65913a9db0aa0a836c350e4c05d9c38eee78206",
+    "simulator_raw_root": "/home/gahow/phd/replayserve/runs/simfid_s2rb/results/raw"
+  },
+  "regularization_sensitivity": {
+    "0.1": {
+      "group_bootstrap": {
+        "accuracy_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            -0.17500000000000004,
+            0.0
+          ],
+          "point": -0.08108108108108103
+        },
+        "brier_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            -0.026383192545085435,
+            0.0607951286646285
+          ],
+          "point": 0.019228316404518567
+        },
+        "replicates": 10000,
+        "seed": 20260714,
+        "semantics": "group bootstrap over cells; diagnostic confidence interval"
+      },
+      "paired_correctness": {
+        "both_correct": 30,
+        "both_wrong": 4,
+        "instrumentation_only_correct": 0,
+        "mcnemar_exact_two_sided_p": 0.25,
+        "sim_outcome_only_correct": 3
+      },
+      "sim_plus_outcome": {
+        "classification": {
+          "accuracy": 0.8918918918918919,
+          "balanced_accuracy": 0.8154761904761905,
+          "brier": 0.10990776306815446,
+          "confusion": {
+            "false_negative": 1,
+            "false_positive": 3,
+            "true_negative": 6,
+            "true_positive": 27
+          },
+          "log_loss": 0.328357763455984
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 12,
+          "correctly_saved_h20_hours": 0.7402314096841667,
+          "decision_coverage": 0.6756756756756757,
+          "early_accept": 20,
+          "early_reject": 5,
+          "false_accept": 0,
+          "false_accept_examples": [],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.0,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.7402314096841667,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": 0.6937764809990414,
+          "valid_zero_error_policy": true
+        }
+      },
+      "sim_plus_outcome_plus_instrumentation": {
+        "classification": {
+          "accuracy": 0.8108108108108109,
+          "balanced_accuracy": 0.7619047619047619,
+          "brier": 0.12913607947267303,
+          "confusion": {
+            "false_negative": 4,
+            "false_positive": 3,
+            "true_negative": 6,
+            "true_positive": 24
+          },
+          "log_loss": 0.4373556318820343
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 9,
+          "correctly_saved_h20_hours": 0.7469523484622221,
+          "decision_coverage": 0.7567567567567568,
+          "early_accept": 22,
+          "early_reject": 6,
+          "false_accept": 2,
+          "false_accept_examples": [
+            {
+              "anchor": 0.49609375,
+              "cell": "tp2_mns8",
+              "label_feasible": false,
+              "probability_feasible": 0.9869795738005246,
+              "remaining_h20_hours": 0.010117910306111111
+            },
+            {
+              "anchor": 0.033717411016,
+              "cell": "tp4_mns16",
+              "label_feasible": false,
+              "probability_feasible": 0.9855364057197005,
+              "remaining_h20_hours": 0.023106262014444445
+            }
+          ],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.03322417232055556,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.7801765207827777,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": null,
+          "valid_zero_error_policy": false
+        }
+      }
+    },
+    "1.0": {
+      "group_bootstrap": {
+        "accuracy_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            0.0,
+            0.18181818181818188
+          ],
+          "point": 0.08108108108108103
+        },
+        "brier_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            -0.04292727744470806,
+            0.019924730979981074
+          ],
+          "point": -0.010145365131402809
+        },
+        "replicates": 10000,
+        "seed": 20260714,
+        "semantics": "group bootstrap over cells; diagnostic confidence interval"
+      },
+      "paired_correctness": {
+        "both_correct": 30,
+        "both_wrong": 4,
+        "instrumentation_only_correct": 3,
+        "mcnemar_exact_two_sided_p": 0.25,
+        "sim_outcome_only_correct": 0
+      },
+      "sim_plus_outcome": {
+        "classification": {
+          "accuracy": 0.8108108108108109,
+          "balanced_accuracy": 0.7242063492063493,
+          "brier": 0.1058226346682949,
+          "confusion": {
+            "false_negative": 3,
+            "false_positive": 4,
+            "true_negative": 5,
+            "true_positive": 25
+          },
+          "log_loss": 0.3011048455679668
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 17,
+          "correctly_saved_h20_hours": 0.5429431818208333,
+          "decision_coverage": 0.5405405405405406,
+          "early_accept": 16,
+          "early_reject": 4,
+          "false_accept": 0,
+          "false_accept_examples": [],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.0,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.5429431818208333,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": 0.5088695307144538,
+          "valid_zero_error_policy": true
+        }
+      },
+      "sim_plus_outcome_plus_instrumentation": {
+        "classification": {
+          "accuracy": 0.8918918918918919,
+          "balanced_accuracy": 0.8154761904761905,
+          "brier": 0.0956772695368921,
+          "confusion": {
+            "false_negative": 1,
+            "false_positive": 3,
+            "true_negative": 6,
+            "true_positive": 27
+          },
+          "log_loss": 0.288823031828762
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 12,
+          "correctly_saved_h20_hours": 0.7360063646722222,
+          "decision_coverage": 0.6756756756756757,
+          "early_accept": 20,
+          "early_reject": 5,
+          "false_accept": 0,
+          "false_accept_examples": [],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.0,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.7360063646722222,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": 0.6898165884274738,
+          "valid_zero_error_policy": true
+        }
+      }
+    },
+    "10.0": {
+      "group_bootstrap": {
+        "accuracy_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            -0.13333333333333341,
+            0.05555555555555558
+          ],
+          "point": -0.027027027027027084
+        },
+        "brier_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            -0.03091105649870874,
+            0.01684192005239855
+          ],
+          "point": -0.007318433328714388
+        },
+        "replicates": 10000,
+        "seed": 20260714,
+        "semantics": "group bootstrap over cells; diagnostic confidence interval"
+      },
+      "paired_correctness": {
+        "both_correct": 30,
+        "both_wrong": 4,
+        "instrumentation_only_correct": 1,
+        "mcnemar_exact_two_sided_p": 1.0,
+        "sim_outcome_only_correct": 2
+      },
+      "sim_plus_outcome": {
+        "classification": {
+          "accuracy": 0.8648648648648649,
+          "balanced_accuracy": 0.7222222222222222,
+          "brier": 0.10613344425735322,
+          "confusion": {
+            "false_negative": 0,
+            "false_positive": 5,
+            "true_negative": 4,
+            "true_positive": 28
+          },
+          "log_loss": 0.3404203142465075
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 32,
+          "correctly_saved_h20_hours": 0.21727432337249997,
+          "decision_coverage": 0.13513513513513514,
+          "early_accept": 5,
+          "early_reject": 0,
+          "false_accept": 0,
+          "false_accept_examples": [],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.0,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.21727432337249997,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": 0.20363877229302757,
+          "valid_zero_error_policy": true
+        }
+      },
+      "sim_plus_outcome_plus_instrumentation": {
+        "classification": {
+          "accuracy": 0.8378378378378378,
+          "balanced_accuracy": 0.7420634920634921,
+          "brier": 0.09881501092863883,
+          "confusion": {
+            "false_negative": 2,
+            "false_positive": 4,
+            "true_negative": 5,
+            "true_positive": 26
+          },
+          "log_loss": 0.312914193285738
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 30,
+          "correctly_saved_h20_hours": 0.2384080185036111,
+          "decision_coverage": 0.1891891891891892,
+          "early_accept": 6,
+          "early_reject": 1,
+          "false_accept": 0,
+          "false_accept_examples": [],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.0,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.2384080185036111,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": 0.22344617366339725,
+          "valid_zero_error_policy": true
+        }
+      }
+    }
+  },
+  "sanity": {
+    "examples": {
+      "distinct_n": 1,
+      "max": 1.0,
+      "min": 1.0,
+      "n": 37
+    },
+    "frozen_simulator_runs": 92,
+    "invariants": {
+      "all_examples_matched_once": true,
+      "labels_not_identical": true,
+      "per_config_results_not_all_identical": true,
+      "same_nested_folds": true,
+      "simulator_ratios_bounded": true
+    },
+    "labels": {
+      "distinct_n": 2,
+      "max": 1.0,
+      "min": 0.0,
+      "n": 37,
+      "negative": 9,
+      "positive": 28
+    },
+    "matched_simulator_pass_rate": {
+      "distinct_n": 12,
+      "max": 1.0,
+      "min": 0.06884057971014493,
+      "n": 37
+    },
+    "red_flags": []
+  },
+  "schema": "fidelity-strong-baseline-v1",
+  "scope": "retrospective one-task headroom audit; not contribution evidence",
+  "status": "PASS"
+}

runs/fidelity-headroom/test_analysis.py

@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import importlib.util
+import math
+import sys
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def load_analysis():
+    spec = importlib.util.spec_from_file_location("fidelity_headroom", HERE / "analyze_existing.py")
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def main() -> None:
+    analysis = load_analysis()
+    curve = analysis.topk_curve(
+        {"a": 3.0, "b": 2.0, "c": 1.0},
+        {"a": 1.0, "b": 2.0, "c": 2.0},
+        2e-6,
+    )
+    assert curve["points"][0]["expanded_k"] == 2
+    assert curve["points"][0]["candidates"] == ["b", "c"]
+    assert math.isclose(curve["points"][0]["real_regret"], 1.0 / 3.0)
+    assert curve["points"][2]["real_regret"] == 0.0
+    assert curve["minimum_k"]["five_percent"] == {"nominal_k": 3, "expanded_k": 3}
+    assert analysis._mcnemar_exact_p(0, 1) == 1.0
+    assert analysis._mcnemar_exact_p(0, 5) == 0.0625
+    print("fidelity headroom analysis: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_pilot_tools.py

@@ -0,0 +1,136 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import json
+import math
+import sys
+import tempfile
+from dataclasses import dataclass
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+
+import pilot_controller as controller  # noqa: E402
+import prepare_pilot as prepare  # noqa: E402
+
+
+@dataclass
+class Request:
+    row_id: str
+    sampling_u: float
+    arrival_s: float = 0.0
+    prompt_tokens_hint: int = 1
+
+
+def main() -> None:
+    requests = [
+        Request("a", 0.1),
+        Request("b", 0.2),
+        Request("c", 0.2),
+        Request("d", 0.9),
+    ]
+    anchor, selected = prepare.attainable_anchor(requests, target_count=2)
+    assert anchor == 0.2
+    assert [request.row_id for request in selected] == ["a", "b", "c"]
+
+    with tempfile.TemporaryDirectory() as temporary:
+        root = Path(temporary)
+        source = root / "source.jsonl"
+        rows = []
+        for index, role in enumerate(prepare.ROLES):
+            rows.append(
+                {
+                    "request_id": role,
+                    "timestamp": float(index),
+                    "sampling_u": (index + 0.5) / len(prepare.ROLES),
+                    "input_length": 16 + index,
+                    "messages": [{"role": "user", "content": role}],
+                }
+            )
+        source.write_text(
+            "".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8"
+        )
+        windows, stats = prepare.materialize_bands(
+            source,
+            {
+                "window_id": "source",
+                "trace_type": "chat",
+                "window_start": 0.0,
+                "window_end": 600.0,
+            },
+            root / "private",
+        )
+        assert windows.is_file()
+        assert all(stats[role]["rows"] == 1 for role in prepare.ROLES)
+        for role in prepare.ROLES:
+            row = json.loads((root / "private" / "traces" / f"{role}.jsonl").read_text())
+            assert row["fidelity_pilot_band"] == role
+            assert abs(float(row["sampling_u"]) - 0.5) < 1e-12
+
+    assert len(controller.ORDER) == 6
+    assert set(controller.ORDER) == set(prepare.CELLS)
+    assert math.isclose(
+        sum(
+            controller.CELL_ESTIMATE_H20_HOURS[int(config["tp"])]
+            for config in prepare.CELLS.values()
+        ) + controller.SAFETY_H20_HOURS,
+        3.0,
+    )
+    selection = {
+        "selected_count": 122,
+        "request_id_order_sha256": "request-hash",
+        "arrival_order_sha256": "arrival-hash",
+        "input_length_order_sha256": "length-hash",
+    }
+    warmup = {
+        "kind": "warmup",
+        "selection": {"count": 16},
+        "invariants": {
+            "warmup_16": True,
+            "warmup_exact_16": True,
+            "warmup_long": True,
+        },
+    }
+    controller.validate_result_selection(
+        result=warmup,
+        selection=selection,
+        cell="tp1_mns8",
+        role="burnin",
+        warmup=True,
+    )
+    measured = {
+        "kind": "anchor",
+        "selection": {
+            "count": 122,
+            "request_id_order_sha256": "request-hash",
+            "arrival_order_sha256": "arrival-hash",
+            "raw_length_order_sha256": "length-hash",
+        },
+        "invariants": {},
+    }
+    controller.validate_result_selection(
+        result=measured,
+        selection=selection,
+        cell="tp1_mns8",
+        role="low1",
+        warmup=False,
+    )
+    try:
+        controller.validate_result_selection(
+            result=warmup,
+            selection=selection,
+            cell="tp1_mns8",
+            role="low1",
+            warmup=False,
+        )
+    except RuntimeError as error:
+        assert "selection count mismatch" in str(error)
+    else:
+        raise AssertionError("measured selection accepted a warmup subset")
+    print("fidelity pilot tools: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_prefix_analysis.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import math
+import sys
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+
+import analyze_prefixes as analysis  # noqa: E402
+
+
+def main() -> None:
+    exact, exact_source = analysis.completion_elapsed_s(
+        {"completed_elapsed_s": 7.25}
+    )
+    assert exact == 7.25 and exact_source == "exact_monotonic"
+
+    reconstructed, reconstructed_source = analysis.completion_elapsed_s(
+        {
+            "success": True,
+            "arrival_s": 2.0,
+            "ttft_ms": 100.0,
+            "tpot_ms": 10.0,
+            "completion_tokens": 11,
+        }
+    )
+    assert math.isclose(reconstructed or 0.0, 2.2)
+    assert reconstructed_source == "reconstructed_from_latency"
+    missing, missing_source = analysis.completion_elapsed_s({"success": False})
+    assert missing is None and missing_source == "unobserved_failure"
+
+    examples = [
+        analysis.PrefixExample(
+            cell=f"c{index}",
+            anchor=float(index),
+            cutoff_s=5.0,
+            tp=1,
+            full_elapsed_s=65.0,
+            feasible=label,
+            primary_feasible=label,
+            outcome=(float(index),),
+            instrumentation=(float(index % 2),),
+            completion_time_source="exact_monotonic",
+        )
+        for index, label in enumerate((0, 1, 1))
+    ]
+    labels = analysis.np.asarray([0, 1, 1])
+    probabilities = analysis.np.asarray([0.01, 0.99, 0.60])
+    policy = analysis.policy_metrics(examples, labels, probabilities, 0.95)
+    assert policy["early_accept"] == 1
+    assert policy["early_reject"] == 1
+    assert policy["abstain_continue_full"] == 1
+    assert policy["false_accept"] == 0 and policy["false_reject"] == 0
+    assert policy["valid_zero_error_policy"]
+    assert policy["valid_cost_reduction_fraction"] is not None
+    model = analysis.fit_frozen_model(
+        examples,
+        instrumentation_aware=True,
+        regularization=1.0,
+    )
+    frozen_probability = analysis.predict_frozen_model(model, examples)
+    assert len(frozen_probability) == len(examples)
+    assert analysis.np.all(frozen_probability >= 0.0)
+    assert analysis.np.all(frozen_probability <= 1.0)
+    print("fidelity prefix analysis: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_strong_baseline.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from analyze_strong_baseline import analyze
+
+
+ROOT = Path(__file__).resolve().parents[2]
+REPLAYSERVE = ROOT.parent / "replayserve"
+
+
+def main() -> None:
+    result = analyze(
+        ROOT / "runs/opprof-phase6/phase6/metrics.json",
+        ROOT / "runs/opprof-phase6/phase6/solo-authoritative/cells",
+        REPLAYSERVE / "runs/simfid_s2rb/results/raw",
+        REPLAYSERVE / "runs/simfid_s2rb/results/metrics.json",
+    )
+    assert result["status"] == "PASS", json.dumps(result["sanity"], indent=2)
+    assert result["sanity"]["frozen_simulator_runs"] == 92
+    assert result["sanity"]["labels"]["n"] == 37
+    headline = result["headline"]
+    assert headline["sim_plus_outcome"]["policy_0p95"]["false_accept"] == 0
+    assert headline["sim_plus_outcome"]["policy_0p95"]["false_reject"] == 0
+    assert (
+        headline["sim_plus_outcome_plus_instrumentation"]["policy_0p95"][
+            "false_accept"
+        ]
+        == 0
+    )
+    print("fidelity strong baseline: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_strong_pilot.py

@@ -0,0 +1,75 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import json
+import tempfile
+from pathlib import Path
+
+import numpy as np
+
+from analyze_prefixes import PrefixExample
+from analyze_strong_pilot import (
+    fit_model,
+    load_pilot_simulator,
+    predict_model,
+)
+
+
+def example(index: int) -> PrefixExample:
+    label = int(index >= 4)
+    return PrefixExample(
+        cell=f"cell-{index // 2}",
+        anchor=float(index),
+        cutoff_s=5.0,
+        tp=1,
+        full_elapsed_s=10.0,
+        feasible=label,
+        primary_feasible=label,
+        outcome=tuple(float(index + offset) for offset in range(13)),
+        instrumentation=tuple(float(index * offset + 1) for offset in range(17)),
+        completion_time_source="exact_monotonic",
+    )
+
+
+def main() -> None:
+    examples = [example(index) for index in range(8)]
+    simulator = [(float(index), index / 10.0, float(index >= 4)) for index in range(8)]
+    for instrumentation_aware in (False, True):
+        model = fit_model(
+            examples,
+            simulator,
+            instrumentation_aware=instrumentation_aware,
+            regularization=1.0,
+        )
+        probability = predict_model(model, examples, simulator)
+        assert probability.shape == (8,)
+        assert np.all((probability >= 0.0) & (probability <= 1.0))
+
+    payload = {
+        "status": "PASS",
+        "results": [
+            {
+                "cell": f"cell-{index // 2}",
+                "role": "low1" if index % 2 == 0 else "high1",
+                "scorer": {
+                    "throughput_requests_per_second_per_gpu": 1.0 + index,
+                    "slo": {
+                        "pass_rate": index / 12.0,
+                        "feasible": index % 2 == 0,
+                    },
+                },
+            }
+            for index in range(12)
+        ],
+    }
+    with tempfile.TemporaryDirectory() as temporary:
+        path = Path(temporary) / "metrics.json"
+        path.write_text(json.dumps(payload), encoding="utf-8")
+        features, red_flags = load_pilot_simulator(path)
+    assert len(features) == 12
+    assert red_flags == []
+    print("fidelity strong pilot: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/opprof-oracle-gap/launch-echo.log

@@ -0,0 +1,13 @@
+RUN_ECHO stage=primary-C11 host=dash0 gpu=0 cpus=0-19 config=C11 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C00 host=dash0 gpu=0 cpus=0-19 config=C00 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C00 host=dash0 gpu=0 cpus=0-19 config=C00 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C01 host=dash0 gpu=0 cpus=0-19 config=C01 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C01 host=dash0 gpu=0 cpus=0-19 config=C01 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C10 host=dash0 gpu=0 cpus=0-19 config=C10 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=primary-C10 host=dash0 gpu=0 cpus=0-19 config=C10 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=confirm-C10 host=dash0 gpu=0 cpus=0-19 config=C10 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=confirm-C10 host=dash0 gpu=0 cpus=0-19 config=C10 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=confirm-C01 host=dash0 gpu=0 cpus=0-19 config=C01 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=confirm-C00 host=dash0 gpu=0 cpus=0-19 config=C00 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=confirm-C11 host=dash0 gpu=0 cpus=0-19 config=C11 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h
+RUN_ECHO stage=closure-r1-C00 host=dash0 gpu=0 cpus=0-19 config=C00 model=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01,P06.jsonl output=/home/admin/cpfs/wjh/oracle-gap-20260713/runs expected_server_plus_trials=20-45min budget_cap=6.0H20h

runs/opprof-oracle-gap/trials.csv

@@ -0,0 +1,105 @@
+trial_path,phase,config,target_rate_rps,repetition,role,clean_seconds,achieved_offered_rps,cohort_n,pass_n,pass_rate,slo_goodput_rps,offered_rate_valid,schedule_valid,raw_slo_feasible,feasible,exact_output_n,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,schedule_lag_max_ms,schedule_lag_p95_ms,schedule_lag_p99_ms,failure_reasons
+trials/P01-C00/rate-26/rep-0/score.json,P01,C00,26.0,0,primary,60.0,26.0,1560,1560,1.0,26.0,True,True,True,True,1560,110.87747401325032,143.8727189670317,159.6376050147228,41.272228730598556,43.98393507913819,44.77031942870882,2.3002910893410444,1.6921606729738414,2.0629908540286124,{}
+trials/P01-C00/rate-28/rep-0/score.json,P01,C00,28.0,0,primary,60.0,28.0,1680,1601,0.9529761904761904,26.683333333333334,True,True,True,True,1680,124.47624997003004,159.8520129919052,169.08409597817808,46.3585068253682,49.967933936776326,50.40311877752492,5.177478247787803,1.6857225564308465,2.054154989309609,"{""tpot_slo"":79}"
+trials/P01-C00/rate-28/rep-1/score.json,P01,C00,28.0,1,boundary-confirmation,60.0,28.0,1680,1659,0.9875,27.65,True,True,True,True,1680,123.72062000213191,159.21212401008233,168.65225398214534,46.01303692112514,49.41609907906414,50.04231119066447,2.9356907471083105,1.665140618570149,2.045694272965193,"{""tpot_slo"":21}"
+trials/P01-C00/rate-28/rep-2/score.json,P01,C00,28.0,2,boundary-confirmation,60.0,28.0,1680,1645,0.9791666666666666,27.416666666666668,True,True,True,True,1680,123.91230202047154,158.21082500042394,168.29883796162903,46.164931619488115,49.66844036521035,50.222261873487795,2.195626962929964,1.626886718440801,2.0519818062894046,"{""tpot_slo"":35}"
+trials/P01-C00/rate-30/rep-0/score.json,P01,C00,30.0,0,primary,60.0,30.0,1800,307,0.17055555555555554,5.116666666666666,True,True,False,False,1800,138.99163604946807,175.20886095007882,186.90624000737444,51.83528874638594,54.82435365122492,55.49848122182228,3.6000446416437626,1.7311419360339642,2.0488349837251008,"{""tpot_slo"":1493}"
+trials/P01-C00/rate-30/rep-1/score.json,P01,C00,30.0,1,boundary-confirmation,60.0,30.0,1800,479,0.26611111111111113,7.983333333333333,True,True,False,False,1800,137.74755800841376,174.40702999010682,187.4343649833463,51.48036580898666,54.92729398362812,55.44106577733709,4.635824996512383,1.635887660086155,2.0532096386887133,"{""tpot_slo"":1321}"
+trials/P01-C00/rate-30/rep-2/score.json,P01,C00,30.0,2,boundary-confirmation,60.0,30.0,1800,414,0.23,6.9,True,True,False,False,1800,138.20564799243584,174.3248249986209,184.11366897635162,51.62530752371938,55.15781777729798,55.70552152367161,2.614296681713313,1.703035377431661,2.073209674563259,"{""tpot_slo"":1386}"
+trials/P01-C00/rate-32/rep-0/score.json,P01,C00,32.0,0,primary,60.0,32.0,1920,87,0.0453125,1.45,True,True,False,False,1920,155.5941189872101,193.93265299731866,211.04579704115167,57.9167679839191,62.15287795260785,62.6541800161321,3.307300037704408,1.723894034512341,2.1944420295767486,"{""tpot_slo"":1833}"
+trials/P01-C00/rate-34/rep-0/score.json,P01,C00,34.0,0,primary,60.0,34.0,2040,88,0.043137254901960784,1.4666666666666666,True,True,False,False,2040,176.93204001989216,221.90839203540236,237.5196199864149,64.94585336518607,69.83023355548669,71.09916014301162,3.92269337316975,1.7322570784017444,2.3150334018282592,"{""tpot_slo"":1952}"
+trials/P01-C00/rate-36/rep-0/score.json,P01,C00,36.0,0,primary,60.0,36.0,2160,81,0.0375,1.35,True,True,False,False,2160,201.2797609786503,249.29071601945907,267.4241259810515,74.56418415919568,79.37573587303864,80.72255477715757,4.510630737058818,2.05585197545588,2.8801406733691692,"{""tpot_slo"":2079}"
+trials/P01-C01/rate-26/rep-0/score.json,P01,C01,26.0,0,primary,60.0,26.0,1560,1560,1.0,26.0,True,True,True,True,1560,109.35463797068223,141.4266600040719,156.78152500186116,40.75073330138352,43.68351201530516,44.807886191096806,2.4665576056577265,1.7485098796896636,2.0897933281958103,{}
+trials/P01-C01/rate-28/rep-0/score.json,P01,C01,28.0,0,primary,60.0,28.0,1680,1632,0.9714285714285714,27.2,True,True,True,True,1680,124.94011997478083,159.36688397778198,166.59722500480711,46.1645503168083,49.64453539726073,50.473119238472826,2.1891174255870283,1.6197741497308016,2.002480032388121,"{""tpot_slo"":48}"
+trials/P01-C01/rate-28/rep-1/score.json,P01,C01,28.0,1,boundary-confirmation,60.0,28.0,1680,1588,0.9452380952380952,26.466666666666665,True,True,False,False,1680,123.78435401478782,159.4017290044576,168.0457249749452,46.30296269804978,50.03878576237531,50.66540757090681,2.3081725812517107,1.547503750771284,2.0317512680776417,"{""tpot_slo"":92}"
+trials/P01-C01/rate-28/rep-2/score.json,P01,C01,28.0,2,boundary-confirmation,60.0,28.0,1680,1669,0.993452380952381,27.816666666666666,True,True,True,True,1680,123.87456797296181,159.58094800589606,168.1017209775746,46.167882237886445,49.62505357125626,49.97083171370572,2.81829503364861,1.630675047636032,2.110089873895049,"{""tpot_slo"":11}"
+trials/P01-C01/rate-30/rep-0/score.json,P01,C01,30.0,0,primary,60.0,30.0,1800,406,0.22555555555555556,6.766666666666667,True,True,False,False,1800,138.7023749994114,174.91436598356813,186.39138899743557,51.529372444502,55.032949429005384,55.51141852305995,3.2389176194556057,1.7149029881693423,2.075109339784831,"{""tpot_slo"":1394}"
+trials/P01-C01/rate-30/rep-1/score.json,P01,C01,30.0,1,boundary-confirmation,60.0,30.0,1800,383,0.2127777777777778,6.383333333333334,True,True,False,False,1800,139.0131320222281,174.85803802264854,185.4456909932196,51.60883849205833,55.771121270178504,56.22731407989733,2.5585776893422008,1.7057106597349048,2.1022360306233168,"{""tpot_slo"":1417}"
+trials/P01-C01/rate-30/rep-2/score.json,P01,C01,30.0,2,boundary-confirmation,60.0,30.0,1800,364,0.20222222222222222,6.066666666666666,True,True,False,False,1800,139.08331200946122,174.3557599838823,184.92947099730372,51.89378812746514,55.43139969767441,56.029993317289545,3.436596365645528,1.6893130377866328,2.101277350448072,"{""tpot_slo"":1436}"
+trials/P01-C01/rate-32/rep-0/score.json,P01,C01,32.0,0,primary,60.0,32.0,1920,89,0.04635416666666667,1.4833333333333334,True,True,False,False,1920,154.02632602490485,190.3878870070912,210.70248901378363,57.73119722208422,62.29816603156487,62.87742125347168,2.908719005063176,1.639701018575579,2.0793279982171953,"{""tpot_slo"":1831}"
+trials/P01-C01/rate-34/rep-0/score.json,P01,C01,34.0,0,primary,60.0,34.0,2040,90,0.04411764705882353,1.5,True,True,False,False,2040,176.2790570501238,220.11423198273405,235.4159569949843,64.42572111134521,69.2640547769972,70.48677477712137,4.586904076859355,1.7256339197047055,2.46569694718346,"{""tpot_slo"":1950}"
+trials/P01-C01/rate-36/rep-0/score.json,P01,C01,36.0,0,primary,60.0,36.0,2160,82,0.03796296296296296,1.3666666666666667,True,True,False,False,2160,200.13104798272252,246.79395399289206,266.32684899959713,74.13756723852,78.38002976218593,79.73950822183508,6.9154552184045315,2.069939684588462,2.8538682381622493,"{""tpot_slo"":2078}"
+trials/P01-C10/rate-24/rep-0/score.json,P01,C10,24.0,0,primary-extension,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,97.52241603564471,116.05227499967441,122.57484695874155,37.891577190113445,40.75966766639982,41.13237590469893,3.6939186975359917,1.7255899729207158,2.0753800054080784,{}
+trials/P01-C10/rate-24/rep-1/score.json,P01,C10,24.0,1,boundary-confirmation,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,98.17885496886447,118.82886500097811,125.91037503443658,38.32998042852278,40.7927984765376,41.027600936118574,2.9292047256603837,1.768625690601766,2.1016597165726125,{}
+trials/P01-C10/rate-24/rep-2/score.json,P01,C10,24.0,2,boundary-confirmation,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,97.96303004259244,126.96650595171377,137.19069201033562,38.67084376277432,41.26724865036233,41.506455126514155,2.5533746811561286,1.722060958854854,2.077668672427535,{}
+trials/P01-C10/rate-26/rep-0/score.json,P01,C10,26.0,0,primary,60.0,26.0,1560,0,0.0,0.0,True,True,False,False,1560,3864.615537051577,4926.1638119933195,5062.709584017284,39.198912619169626,40.58124177760282,41.15692004820125,2.5109824491664767,1.7559383413754404,2.064570668153465,"{""ttft_slo"":1560}"
+trials/P01-C10/rate-26/rep-1/score.json,P01,C10,26.0,1,boundary-confirmation,60.0,26.0,1560,1560,1.0,26.0,True,True,True,True,1560,252.71200499264523,497.90483101969585,632.6556759886444,37.49335380936308,39.14127271393284,39.90859952330264,3.0049763154238462,1.7557168612256646,2.0546196028590202,{}
+trials/P01-C10/rate-26/rep-2/score.json,P01,C10,26.0,2,boundary-confirmation,60.0,26.0,1560,0,0.0,0.0,True,True,False,False,1560,3994.878589990549,4745.065895025618,4836.570041952655,38.89002274659033,40.17117711168433,40.39965068278391,2.420338918454945,1.6692215576767921,2.05352931516245,"{""ttft_slo"":1560}"
+trials/P01-C10/rate-28/rep-0/score.json,P01,C10,28.0,0,primary,60.0,26.083333333333332,1565,0,0.0,0.0,False,False,False,False,1565,7570.850218005944,7743.26098500751,7767.141760967206,38.62161282542354,39.79506171382372,40.20180085688711,4137.868300836999,3841.5666787186638,4090.4739925754257,"{""ttft_slo"":1565}"
+trials/P01-C10/rate-30/rep-0/score.json,P01,C10,30.0,0,primary,60.0,26.066666666666666,1564,0,0.0,0.0,False,False,False,False,1564,7471.911213011481,7596.271771006286,7660.170332994312,37.75816839761175,38.96897019068193,39.542113761565396,9581.972879357636,9184.90837601712,9528.559824393597,"{""ttft_slo"":1564}"
+trials/P01-C10/rate-32/rep-0/score.json,P01,C10,32.0,0,primary,60.0,27.733333333333334,1664,0,0.0,0.0,False,False,False,False,1664,6994.684477976989,7170.971911051311,7199.6434789616615,34.7310509680519,36.133673142767435,36.555938174142426,9920.33471504692,9432.675201038364,9748.585979046766,"{""ttft_slo"":1664}"
+trials/P01-C10/rate-34/rep-0/score.json,P01,C10,34.0,0,primary,60.0,26.666666666666668,1600,0,0.0,0.0,False,False,False,False,1600,7292.200801020954,7465.953706996515,7510.689996997826,36.594293714087044,37.8891855555897,38.275920475522675,19949.719168245792,19317.845753917936,19914.817562676035,"{""ttft_slo"":1600}"
+trials/P01-C10/rate-36/rep-0/score.json,P01,C10,36.0,0,primary,60.0,26.966666666666665,1618,0,0.0,0.0,False,False,False,False,1618,7226.758364005946,7377.549752010964,7406.953382014763,36.146196222583214,37.36288985699445,37.69975279398736,24686.17197702406,24000.297317747027,24624.147864291444,"{""ttft_slo"":1618}"
+trials/P01-C11/rate-24/rep-0/score.json,P01,C11,24.0,0,primary-extension,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,102.60815400397405,147.0231090206653,167.74030699161813,39.74406866704128,41.34076804805931,42.04290312629873,2.937526674941182,1.6711382777430117,2.0573336514644325,{}
+trials/P01-C11/rate-24/rep-1/score.json,P01,C11,24.0,1,boundary-confirmation,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,97.34877798473462,117.04645899590105,125.5875930073671,37.771026253707646,40.67583511110454,41.213303270365394,5.921298987232149,1.7458907095715404,2.079935686197132,{}
+trials/P01-C11/rate-24/rep-2/score.json,P01,C11,24.0,2,boundary-confirmation,60.0,24.0,1440,1440,1.0,24.0,True,True,True,True,1440,96.28819499630481,115.87436601985246,122.07344203488901,37.68616379423451,40.112987238292895,40.710791253433044,3.3418433158658445,1.6138353385031223,2.0559499971568584,{}
+trials/P01-C11/rate-26/rep-0/score.json,P01,C11,26.0,0,primary,60.0,25.783333333333335,1547,0,0.0,0.0,True,True,False,False,1547,6533.619961992372,7947.257625986822,7982.082434988115,40.18144979370788,41.07262126986854,42.167470412557975,573.5937684075907,386.3291516317986,548.7673947936855,"{""ttft_slo"":1547}"
+trials/P01-C11/rate-26/rep-1/score.json,P01,C11,26.0,1,boundary-confirmation,60.0,26.0,1560,1560,1.0,26.0,True,True,True,True,1560,593.4362710104324,827.8850860078819,934.9126719753258,37.31267711162449,38.505050603167284,38.729826237873304,2.241449663415551,1.7251850222237408,2.0647295750677586,{}
+trials/P01-C11/rate-26/rep-2/score.json,P01,C11,26.0,2,boundary-confirmation,60.0,26.0,1560,0,0.0,0.0,True,True,False,False,1560,5812.711999984458,6669.004526047502,6752.215686021373,39.62677446035077,41.18544706291268,41.53003749190017,2.1458613919094205,1.6790343215689063,2.039063081610948,"{""ttft_slo"":1560}"
+trials/P01-C11/rate-28/rep-0/score.json,P01,C11,28.0,0,primary,60.0,26.233333333333334,1574,0,0.0,0.0,False,False,False,False,1574,7565.811808046419,7749.4256109930575,7759.166635980364,38.48338009519798,39.82913404765968,40.06808598355819,3845.5762091325596,3568.3553874259815,3815.4344389913604,"{""ttft_slo"":1574}"
+trials/P01-C11/rate-30/rep-0/score.json,P01,C11,30.0,0,primary,60.0,26.1,1566,0,0.0,0.0,False,False,False,False,1566,7462.299443024676,7605.860833020415,7636.226687987801,37.638444031241335,38.706122920663645,39.0103649207583,9453.580788627733,9022.533191659022,9404.877326625865,"{""ttft_slo"":1566}"
+trials/P01-C11/rate-32/rep-0/score.json,P01,C11,32.0,0,primary,60.0,28.25,1695,0,0.0,0.0,False,False,False,False,1695,6933.729738055263,7114.611251046881,7164.560960023664,34.169790237986795,35.679708412360576,36.14399266717512,9283.821924007498,8650.737509015016,9040.9968290478,"{""ttft_slo"":1695}"
+trials/P01-C11/rate-34/rep-0/score.json,P01,C11,34.0,0,primary,60.0,26.6,1596,0,0.0,0.0,False,False,False,False,1596,7322.663314000238,7463.087519980036,7493.740423000418,36.76072112654173,38.06862412648837,39.04326098401927,20215.17994365422,19248.93292755587,19825.970116246026,"{""ttft_slo"":1596}"
+trials/P01-C11/rate-36/rep-0/score.json,P01,C11,36.0,0,primary,60.0,27.15,1629,0,0.0,0.0,False,False,False,False,1629,7185.944458993617,7345.619179017376,7373.497121967375,35.93770284131761,37.092669237782026,37.386540825190465,23960.0648356718,23284.954287286382,23910.954526276328,"{""ttft_slo"":1629}"
+trials/P06-C00/rate-1.4/rep-0/score.json,P06,C00,1.4,0,primary,120.0,1.4,168,168,1.0,1.4,True,True,True,True,168,999.2175319930539,1855.332705017645,2123.494433995802,14.184352559645326,17.348588929546235,18.162435246563156,7.325362705159932,6.399379402864724,7.314503251109272,{}
+trials/P06-C00/rate-1.5/rep-0/score.json,P06,C00,1.5,0,primary,120.0,1.4666666666666666,176,176,1.0,1.4666666666666666,True,True,True,True,176,993.2098449789919,1804.5693339663558,2139.430449984502,14.179927917742289,18.19816446378317,20.127085575295354,7.586089021060616,6.320645683445036,7.3406120063737035,{}
+trials/P06-C00/rate-1.6/rep-0/score.json,P06,C00,1.6,0,primary,120.0,1.6,192,192,1.0,1.6,True,True,True,True,192,997.0329970237799,1733.372846036218,2134.9421920021996,16.06630710572817,21.62564592766293,22.13207139725179,10.16304298536852,7.14888097718358,9.413381980266422,{}
+trials/P06-C00/rate-1.7/rep-0/score.json,P06,C00,1.7,0,primary,120.0,1.7333333333333334,208,208,1.0,1.7333333333333334,True,True,True,True,208,999.191242037341,1732.8481450094841,2074.4936399860308,18.194802745597148,22.597885336599326,24.458052634116655,7.205226575024426,6.399967067409307,6.824508309364319,{}
+trials/P06-C00/rate-1.8/rep-0/score.json,P06,C00,1.8,0,primary,120.0,1.8,216,215,0.9953703703703703,1.7916666666666667,True,True,True,True,216,1004.904159985017,1762.3687379527837,2076.484896009788,20.300264271963098,23.445108162404136,23.846686213302796,7.712147431448102,6.571699457708746,7.433776569087058,"{""ttft_slo"":1}"
+trials/P06-C00/rate-1.9/rep-0/score.json,P06,C00,1.9,0,primary,120.0,1.8666666666666667,224,223,0.9955357142857143,1.8583333333333334,True,True,True,True,224,1008.6866070050746,1725.8416549884714,2094.005164981354,22.55910544618855,27.32792746370436,28.425333408958693,7.935012050438672,6.445528415497392,7.354569446761161,"{""ttft_slo"":1}"
+trials/P06-C00/rate-2/rep-0/score.json,P06,C00,2.0,0,primary,120.0,2.0,240,237,0.9875,1.975,True,True,True,True,240,1008.1276059499942,1844.6353260078467,2217.431389959529,26.691506409000628,33.19407374552067,34.29328832297391,8.592377009335905,6.834403029642999,8.295246050693095,"{""ttft_slo"":3}"
+trials/P06-C00/rate-2.1/rep-0/score.json,P06,C00,2.1,0,primary-extension,120.0,2.1333333333333333,256,253,0.98828125,2.1083333333333334,True,True,True,True,256,1021.1376319639385,1874.3867019657046,2236.3475980237126,30.77564142275795,34.25728149906643,35.95762665954431,9.336929477285594,7.481765991542488,8.548939367756248,"{""ttft_slo"":3}"
+trials/P06-C00/rate-2.2/rep-0/score.json,P06,C00,2.2,0,primary-extension,120.0,2.2,264,260,0.9848484848484849,2.1666666666666665,True,True,True,True,264,1016.4398739580065,1892.5629690056667,2254.457817005459,33.431356234900626,38.75392900199817,40.532795230971026,8.67525755893439,6.830454338341951,8.405035536270589,"{""ttft_slo"":4}"
+trials/P06-C00/rate-2.3/rep-0/score.json,P06,C00,2.3,0,primary-extension,120.0,2.2666666666666666,272,268,0.9852941176470589,2.2333333333333334,True,True,True,True,272,1028.8451670203358,1908.7893930263817,2253.68108501425,37.45989758911344,42.8498864755738,44.0359943385287,8.370409661438316,6.645695248153061,8.179216703865677,"{""ttft_slo"":4}"
+trials/P06-C00/rate-2.3/rep-1/score.json,P06,C00,2.3,1,boundary-closure,120.0,2.2666666666666666,272,268,0.9852941176470589,2.2333333333333334,True,True,True,True,272,1056.8546410067938,1936.134911957197,2271.4714580215514,38.52701387477496,43.82504283363062,44.65749040118431,9.594396688044071,6.700532685499638,7.922522665467113,"{""ttft_slo"":4}"
+trials/P06-C00/rate-2.3/rep-2/score.json,P06,C00,2.3,2,boundary-closure,120.0,2.2666666666666666,272,268,0.9852941176470589,2.2333333333333334,True,True,True,True,272,1023.681657970883,2005.0558719667606,2281.9403469911776,37.73080680230924,44.07464489231429,44.84600109586295,9.688543679658324,6.723627564497292,9.51991870533675,"{""ttft_slo"":4}"
+trials/P06-C00/rate-2.4/rep-0/score.json,P06,C00,2.4,0,primary-extension,120.0,2.4,288,274,0.9513888888888888,2.283333333333333,True,True,True,True,288,1047.7718930342235,1939.0789800090715,2299.159614020027,44.111866635974835,49.369580082211705,51.03415431313074,8.399292710237205,6.617546721827239,8.075028716120869,"{""tpot_slo"":10,""ttft_slo"":4}"
+trials/P06-C00/rate-2.4/rep-1/score.json,P06,C00,2.4,1,boundary-confirmation,120.0,2.4,288,271,0.9409722222222222,2.2583333333333333,True,True,False,False,288,1038.3890920202248,1913.3997529861517,2282.3066530399956,44.10149966145149,49.90086417213272,52.977919430584045,8.468122687190771,6.662778672762215,7.901694276370108,"{""tpot_slo"":13,""ttft_slo"":4}"
+trials/P06-C00/rate-2.4/rep-2/score.json,P06,C00,2.4,2,boundary-confirmation,120.0,2.4,288,273,0.9479166666666666,2.275,True,True,False,False,288,1038.4550529997796,1935.180893051438,2284.888348018285,44.084652043025166,49.2430650137015,53.145144203447224,10.83466998534277,7.184286660049111,9.758414002135396,"{""tpot_slo"":11,""ttft_slo"":4}"
+trials/P06-C00/rate-2.5/rep-0/score.json,P06,C00,2.5,0,primary-extension,120.0,2.533333333333333,304,171,0.5625,1.425,True,True,False,False,304,1042.2218869789504,2067.6334840245545,2326.744472957216,48.563286011679544,54.620607958950735,57.85391967308046,8.668092952575535,6.844838964752853,7.631328015122563,"{""tpot_slo"":132,""ttft_slo"":5}"
+trials/P06-C00/rate-2.5/rep-1/score.json,P06,C00,2.5,1,boundary-confirmation,120.0,2.533333333333333,304,163,0.5361842105263158,1.3583333333333334,True,True,False,False,304,1058.9527579722926,2087.2758819605224,2304.234509996604,48.46152181804559,55.69985224658062,58.651440686896194,9.696525987237692,6.925722060259432,8.803103992249817,"{""tpot_slo"":139,""ttft_slo"":5}"
+trials/P06-C00/rate-2.5/rep-2/score.json,P06,C00,2.5,2,boundary-confirmation,120.0,2.533333333333333,304,155,0.5098684210526315,1.2916666666666667,True,True,False,False,304,1049.7184470295906,2060.312128975056,2316.979594004806,49.96833974953901,55.53648186496271,58.368578595004486,8.793669985607266,7.0653450093232095,8.415801043156534,"{""tpot_slo"":148,""ttft_slo"":5}"
+trials/P06-C01/rate-1.4/rep-0/score.json,P06,C01,1.4,0,primary,120.0,1.4,168,166,0.9880952380952381,1.3833333333333333,True,True,True,True,168,870.9071970079094,2094.233661016915,2450.609703955706,15.16279330330859,18.223098260277624,18.65344091000513,7.756370992865413,6.575752224307507,7.233251410070807,"{""ttft_slo"":2}"
+trials/P06-C01/rate-1.5/rep-0/score.json,P06,C01,1.5,0,primary,120.0,1.4666666666666666,176,174,0.9886363636363636,1.45,True,True,True,True,176,856.920883001294,2077.852713991888,2404.4516870053485,15.114829923712453,21.891871407088583,23.431140612450896,7.3857519892044365,6.386720982845873,7.349602004978806,"{""ttft_slo"":2}"
+trials/P06-C01/rate-1.6/rep-0/score.json,P06,C01,1.6,0,primary,120.0,1.6,192,190,0.9895833333333334,1.5833333333333333,True,True,True,True,192,877.8399180155247,1984.3015700462274,2398.247136035934,17.95897510373389,23.491408933458008,24.43991283751266,7.940459006931633,6.813998974394053,7.770671974867582,"{""ttft_slo"":2}"
+trials/P06-C01/rate-1.7/rep-0/score.json,P06,C01,1.7,0,primary,120.0,1.7333333333333334,208,206,0.9903846153846154,1.7166666666666666,True,True,True,True,208,865.2636900078505,1968.6543480493128,2176.717725000344,20.55894938951865,24.509686565572494,25.151764009755702,7.413975603412837,6.295370752923191,7.269357622135431,"{""ttft_slo"":2}"
+trials/P06-C01/rate-1.8/rep-0/score.json,P06,C01,1.8,0,primary,120.0,1.8,216,212,0.9814814814814815,1.7666666666666666,True,True,True,True,216,884.8230779985897,2010.2625639992766,2270.7025869749486,22.31720594712943,29.147994661486823,30.879378019546845,10.790552652906626,6.586181407328695,7.714397390373051,"{""ttft_slo"":4}"
+trials/P06-C01/rate-1.9/rep-0/score.json,P06,C01,1.9,0,primary,120.0,1.8666666666666667,224,220,0.9821428571428571,1.8333333333333333,True,True,True,True,224,871.5889130253345,2014.9384770193137,2264.04358696891,26.832327434502417,30.454946645747857,31.631232761352372,7.387189427390695,6.136250507552177,7.152303704060614,"{""ttft_slo"":4}"
+trials/P06-C01/rate-2/rep-0/score.json,P06,C01,2.0,0,primary,120.0,2.0,240,233,0.9708333333333333,1.9416666666666667,True,True,True,True,240,872.5066980114207,2134.159804030787,2552.078590961173,28.788431911978055,35.65545248732503,36.88229199800979,7.818170997779816,6.43993797712028,7.529255992267281,"{""ttft_slo"":7}"
+trials/P06-C01/rate-2.1/rep-0/score.json,P06,C01,2.1,0,primary-extension,120.0,2.1333333333333333,256,245,0.95703125,2.0416666666666665,True,True,True,True,256,917.0198910287581,2213.4800929925404,2593.613261007704,34.30701432673993,40.38027477887625,41.43307809392173,10.985759610775858,7.624904450494796,9.49472957290709,"{""ttft_slo"":11}"
+trials/P06-C01/rate-2.2/rep-0/score.json,P06,C01,2.2,0,primary-extension,120.0,2.2,264,252,0.9545454545454546,2.1,True,True,True,True,264,945.8785869646817,2318.5488900053315,2714.1117680002935,40.473415896204465,47.044261179954624,48.30041582584053,12.640100321732461,7.256975513882935,8.479340351186693,"{""ttft_slo"":12}"
+trials/P06-C01/rate-2.2/rep-1/score.json,P06,C01,2.2,1,boundary-confirmation,120.0,2.2,264,254,0.9621212121212122,2.1166666666666667,True,True,True,True,264,937.7531060017645,2288.793044979684,2689.1637159860693,39.88727447547205,44.97671687867084,46.09535447171016,8.372949319891632,6.535524502396584,7.7698222594335675,"{""ttft_slo"":10}"
+trials/P06-C01/rate-2.2/rep-2/score.json,P06,C01,2.2,2,boundary-confirmation,120.0,2.2,264,251,0.9507575757575758,2.091666666666667,True,True,True,True,264,943.6753669870086,2305.475205008406,2666.3335349876434,40.217721972609056,46.534055287730126,47.48751569861444,9.360338270198554,6.640697305556387,8.254165295511484,"{""ttft_slo"":13}"
+trials/P06-C01/rate-2.3/rep-0/score.json,P06,C01,2.3,0,primary-extension,120.0,2.2666666666666666,272,222,0.8161764705882353,1.85,True,True,False,False,272,971.3688230258413,2346.123350027483,2757.595370989293,45.034927745675944,51.62063973977225,55.295872516609336,8.836901630274951,7.390526297967881,8.478775678668171,"{""tpot_slo"":35,""ttft_slo"":16}"
+trials/P06-C01/rate-2.3/rep-1/score.json,P06,C01,2.3,1,boundary-confirmation,120.0,2.2666666666666666,272,232,0.8529411764705882,1.9333333333333333,True,True,False,False,272,963.1062899716198,2335.012650990393,2770.2474140096456,45.059407203469185,50.939032240677804,53.42882929345659,10.456104762852192,7.382042706012726,8.998437726404518,"{""tpot_slo"":25,""ttft_slo"":16}"
+trials/P06-C01/rate-2.3/rep-2/score.json,P06,C01,2.3,2,boundary-confirmation,120.0,2.2666666666666666,272,219,0.8051470588235294,1.825,True,True,False,False,272,968.3994479710236,2338.2130070240237,2772.5177410175093,45.12062682583381,51.60208501754631,55.46992149902508,9.490319702308625,6.954058830160648,7.915747701190412,"{""tpot_slo"":39,""ttft_slo"":15}"
+trials/P06-C10/rate-1.4/rep-0/score.json,P06,C10,1.4,0,primary,120.0,1.4,168,167,0.9940476190476191,1.3916666666666666,True,True,True,True,168,989.1352539998479,1854.9544009729289,2126.261135970708,14.310991064619834,17.33015237377005,18.133633745558114,8.03026300854981,6.68573915027082,7.717043161392212,"{""ttft_slo"":1}"
+trials/P06-C10/rate-1.5/rep-0/score.json,P06,C10,1.5,0,primary,120.0,1.4666666666666666,176,176,1.0,1.4666666666666666,True,True,True,True,176,989.0364960301667,1836.014078988228,2159.4515729811974,14.38420469275313,18.27264551281539,20.49906990411573,7.232828997075558,6.385942979250103,7.173349673394114,{}
+trials/P06-C10/rate-1.6/rep-0/score.json,P06,C10,1.6,0,primary,120.0,1.6,192,192,1.0,1.6,True,True,True,True,192,1005.6778370053507,1733.1810360192321,2129.3352029751986,16.17475541492144,21.547753428548223,22.38600313688087,11.067131999880075,6.535599008202553,10.750612011179328,{}
+trials/P06-C10/rate-1.7/rep-0/score.json,P06,C10,1.7,0,primary,120.0,1.7333333333333334,208,208,1.0,1.7333333333333334,True,True,True,True,208,1005.7410600129515,1793.3206979651004,2149.9399559688754,18.121018011802754,22.653281632191163,23.578543808204603,8.620336942840368,6.635099765844643,7.54233862971887,{}
+trials/P06-C10/rate-1.8/rep-0/score.json,P06,C10,1.8,0,primary,120.0,1.8,216,215,0.9953703703703703,1.7916666666666667,True,True,True,True,216,1003.2750739483163,1837.6042150193825,2153.3647059695795,20.33729544431135,23.54741711743584,24.420280583155453,7.7063715434633195,6.210717547219247,7.363768992945552,"{""ttft_slo"":1}"
+trials/P06-C10/rate-1.9/rep-0/score.json,P06,C10,1.9,0,primary,120.0,1.8666666666666667,224,223,0.9955357142857143,1.8583333333333334,True,True,True,True,224,1028.2902300241403,1734.957567998208,2139.568875019904,23.068141569465457,28.3249747396516,29.331570328873948,10.697210382204503,6.612500932533294,8.539305708836764,"{""ttft_slo"":1}"
+trials/P06-C10/rate-2/rep-0/score.json,P06,C10,2.0,0,primary,120.0,2.0,240,237,0.9875,1.975,True,True,True,True,240,1015.3626359533519,1876.5030350186862,2216.4827240048908,26.61050810754744,32.98075636598207,34.85392479450622,14.75413900334388,7.143530005123466,12.002518982626498,"{""ttft_slo"":3}"
+trials/P06-C10/rate-2.1/rep-0/score.json,P06,C10,2.1,0,primary-extension,120.0,2.1333333333333333,256,252,0.984375,2.1,True,True,True,True,256,1017.7665610099211,1897.7975359885022,2233.356000040658,29.193307035247233,34.04904124857821,35.716844309251684,8.122143452055752,7.0329742738977075,8.025550458114594,"{""ttft_slo"":4}"
+trials/P06-C10/rate-2.2/rep-0/score.json,P06,C10,2.2,0,primary-extension,120.0,2.2,264,260,0.9848484848484849,2.1666666666666665,True,True,True,True,264,1028.838190017268,1924.7021020273678,2267.4863139982335,34.656678156589145,39.05688540715225,40.56380550300068,8.019849366974086,6.283827999141067,7.548786816187203,"{""ttft_slo"":4}"
+trials/P06-C10/rate-2.3/rep-0/score.json,P06,C10,2.3,0,primary-extension,120.0,2.2666666666666666,272,268,0.9852941176470589,2.2333333333333334,True,True,True,True,272,1030.823885987047,2007.8507559956051,2252.5435159914196,37.69898847546957,43.23842918587786,44.374794215250965,8.239199698437005,6.477339018601924,7.5856699841097,"{""ttft_slo"":4}"
+trials/P06-C10/rate-2.4/rep-0/score.json,P06,C10,2.4,0,primary-extension,120.0,2.4,288,274,0.9513888888888888,2.283333333333333,True,True,True,True,288,1034.7871870035306,1914.081139024347,2285.277316987049,43.703152659488566,49.364211770989286,50.49469348336178,8.994451723992825,6.490265019237995,8.65559169324115,"{""tpot_slo"":10,""ttft_slo"":4}"
+trials/P06-C10/rate-2.4/rep-1/score.json,P06,C10,2.4,1,boundary-confirmation,120.0,2.4,288,278,0.9652777777777778,2.316666666666667,True,True,True,True,288,1042.35012200661,1931.6510139615275,2304.684264003299,45.058667197629426,49.32953689043244,50.9193064501407,8.361130661796778,6.718640972394496,8.037615334615111,"{""tpot_slo"":6,""ttft_slo"":4}"
+trials/P06-C10/rate-2.4/rep-2/score.json,P06,C10,2.4,2,boundary-confirmation,120.0,2.4,288,279,0.96875,2.325,True,True,True,True,288,1022.7319020195864,1937.5082700280473,2259.4223510241136,43.80097978273607,48.74295069858406,50.16730903327071,9.690148697700351,7.029823667835444,7.890908687841147,"{""tpot_slo"":5,""ttft_slo"":4}"
+trials/P06-C10/rate-2.5/rep-0/score.json,P06,C10,2.5,0,primary-extension,120.0,2.533333333333333,304,95,0.3125,0.7916666666666666,True,True,False,False,304,2603.841262985952,4117.584208026528,4780.076910043135,48.75482211150083,52.61768415659564,53.111906254408,9.685411059763283,6.941412982996553,8.32790503045544,"{""tpot_slo"":93,""ttft_slo"":129}"
+trials/P06-C10/rate-2.5/rep-1/score.json,P06,C10,2.5,1,boundary-confirmation,120.0,2.533333333333333,304,103,0.33881578947368424,0.8583333333333333,True,True,False,False,304,2862.704548984766,4183.153211022727,4453.049428993836,48.63343907633173,52.460244035260814,52.98061303326445,8.852682018186897,6.924911984242499,8.437061042059213,"{""tpot_slo"":78,""ttft_slo"":138}"
+trials/P06-C10/rate-2.5/rep-2/score.json,P06,C10,2.5,2,boundary-confirmation,120.0,2.533333333333333,304,65,0.2138157894736842,0.5416666666666666,True,True,False,False,304,3408.5888129775412,4655.079743999522,4986.388145014644,48.67060376908047,52.2114794285341,52.96427048145886,10.025584022514522,7.519927050452679,9.58072004141286,"{""tpot_slo"":82,""ttft_slo"":197}"
+trials/P06-C11/rate-1.4/rep-0/score.json,P06,C11,1.4,0,primary,120.0,1.4,168,166,0.9880952380952381,1.3833333333333333,True,True,True,True,168,878.8207969628274,2104.5628290157765,2456.5626359544694,15.149171608582197,18.309408667248558,18.975726387453484,7.393979234620929,6.200018804520369,7.383851974736899,"{""ttft_slo"":2}"
+trials/P06-C11/rate-1.5/rep-0/score.json,P06,C11,1.5,0,primary,120.0,1.4666666666666666,176,174,0.9886363636363636,1.45,True,True,True,True,176,868.387623981107,2078.5744480090216,2430.6161800050177,15.37368986883188,21.409829477497073,22.96634120153584,9.10428969655186,6.963004008866847,8.968600712250918,"{""ttft_slo"":2}"
+trials/P06-C11/rate-1.6/rep-0/score.json,P06,C11,1.6,0,primary,120.0,1.6,192,190,0.9895833333333334,1.5833333333333333,True,True,True,True,192,880.2933589904569,1996.0739740054123,2426.3907080166973,19.148470602775586,23.556329107637392,24.3911435792301,12.246161000803113,6.944428023416549,11.784875998273492,"{""ttft_slo"":2}"
+trials/P06-C11/rate-1.7/rep-0/score.json,P06,C11,1.7,0,primary,120.0,1.7333333333333334,208,206,0.9903846153846154,1.7166666666666666,True,True,True,True,208,868.3885979698971,1974.1372629650868,2170.4040309996344,20.554707737693313,24.361233739774683,25.208697178052034,8.172314323019236,6.486376631073654,7.166995259467512,"{""ttft_slo"":2}"
+trials/P06-C11/rate-1.8/rep-0/score.json,P06,C11,1.8,0,primary,120.0,1.8,216,212,0.9814814814814815,1.7666666666666666,True,True,True,True,216,860.0498370360583,2015.8445590059273,2296.1058749933727,22.033081184015582,28.35712986695261,29.818611892387672,8.214946719817817,6.435671588405967,7.979147601872683,"{""ttft_slo"":4}"
+trials/P06-C11/rate-1.9/rep-0/score.json,P06,C11,1.9,0,primary,120.0,1.8666666666666667,224,220,0.9821428571428571,1.8333333333333333,True,True,True,True,224,873.2819089782424,2015.0140480254777,2233.055104035884,26.420655446181733,30.60916253037973,31.97331095106149,7.478082727175206,6.193810375407338,7.1653242921456695,"{""ttft_slo"":4}"
+trials/P06-C11/rate-2/rep-0/score.json,P06,C11,2.0,0,primary,120.0,2.0,240,233,0.9708333333333333,1.9416666666666667,True,True,True,True,240,885.916200990323,2135.891862970311,2545.2551289927214,29.176486246565762,36.21475781799784,37.257001457900834,8.14571394585073,6.473987945355475,7.991001999471337,"{""ttft_slo"":7}"
+trials/P06-C11/rate-2.1/rep-0/score.json,P06,C11,2.1,0,primary-extension,120.0,2.1333333333333333,256,244,0.953125,2.033333333333333,True,True,True,True,256,910.2991609834135,2218.1496039847843,2635.4756229557097,34.070244894377566,40.17389689240907,40.98456636594732,9.201513486914337,6.535229331348091,8.932627330068499,"{""ttft_slo"":12}"
+trials/P06-C11/rate-2.2/rep-0/score.json,P06,C11,2.2,0,primary-extension,120.0,2.2,264,252,0.9545454545454546,2.1,True,True,True,True,264,939.8728320375085,2325.3240960184485,2723.1857150327414,40.232674399146596,46.644165585115516,47.97150479452393,8.847062883432955,7.093118503689766,8.64546129014343,"{""ttft_slo"":12}"
+trials/P06-C11/rate-2.2/rep-1/score.json,P06,C11,2.2,1,boundary-confirmation,120.0,2.2,264,255,0.9659090909090909,2.125,True,True,True,True,264,937.060812022537,2310.2555879740976,2691.6481269872747,39.889533819955155,45.028635876735315,46.05559236791492,8.61785898450762,6.791460502427071,8.322948298882693,"{""ttft_slo"":9}"
+trials/P06-C11/rate-2.2/rep-2/score.json,P06,C11,2.2,2,boundary-confirmation,120.0,2.2,264,252,0.9545454545454546,2.1,True,True,True,True,264,938.2475850288756,2306.774304015562,2683.795469987672,40.23246330137541,46.400342516622956,47.64727340899991,8.344491885509342,6.439735821913928,7.930240361019969,"{""ttft_slo"":12}"
+trials/P06-C11/rate-2.3/rep-0/score.json,P06,C11,2.3,0,primary-extension,120.0,2.2666666666666666,272,203,0.7463235294117647,1.6916666666666667,True,True,False,False,272,1074.6694429544732,2435.8343050116673,2784.457146015484,44.76343049704682,52.034396264262845,54.29683911359874,8.639712177682668,6.902627006638795,8.276157022919506,"{""tpot_slo"":51,""ttft_slo"":19}"
+trials/P06-C11/rate-2.3/rep-1/score.json,P06,C11,2.3,1,boundary-confirmation,120.0,2.2666666666666666,272,180,0.6617647058823529,1.5,True,True,False,False,272,1291.9953179662116,2771.1534849950112,3167.952421004884,47.08591616236825,52.35070534633225,54.310699465772075,13.530849944800138,7.158174004871398,9.898048709146678,"{""tpot_slo"":74,""ttft_slo"":23}"
+trials/P06-C11/rate-2.3/rep-2/score.json,P06,C11,2.3,2,boundary-confirmation,120.0,2.2666666666666666,272,177,0.6507352941176471,1.475,True,True,False,False,272,1149.452324025333,2552.789599983953,2787.1654250193387,45.428387461906325,52.92104801173102,55.13124520753268,11.129230260848999,7.671434723306447,10.892333288211375,"{""tpot_slo"":75,""ttft_slo"":25}"

runs/opprof-phase3-ea/provenance/opprof_phase3_client.py

@@ -0,0 +1,784 @@
+#!/usr/bin/env python3
+"""Token-exact fixed-duration client for the OpProf Phase-3 protocol."""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import gzip
+import hashlib
+import json
+import math
+import os
+import time
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import aiohttp
+
+SCHEMA = 1
+TOKEN_BASE = 1000
+TOKEN_SPAN = 100000
+
+
+class ManifestExhausted(RuntimeError):
+    pass
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as f:
+        for chunk in iter(lambda: f.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def atomic_json(path: Path, value: Any, mode: int = 0o640) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_name(path.name + f".tmp.{os.getpid()}")
+    fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode)
+    with os.fdopen(fd, "w", encoding="utf-8") as f:
+        json.dump(value, f, sort_keys=True, indent=2)
+        f.write("\n")
+        f.flush()
+        os.fsync(f.fileno())
+    os.replace(tmp, path)
+
+
+def atomic_jsonl(path: Path, rows: list[dict[str, Any]], mode: int = 0o640) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_name(path.name + f".tmp.{os.getpid()}")
+    fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode)
+    with os.fdopen(fd, "w", encoding="utf-8") as f:
+        for row in rows:
+            f.write(json.dumps(row, sort_keys=True, separators=(",", ":")) + "\n")
+        f.flush()
+        os.fsync(f.fileno())
+    os.replace(tmp, path)
+
+
+def parse_range(value: str) -> tuple[int, int]:
+    lo_text, hi_text = value.split(":", 1)
+    lo, hi = int(lo_text), int(hi_text)
+    if lo <= 0 or hi < lo:
+        raise argparse.ArgumentTypeError(f"invalid positive range: {value}")
+    return lo, hi
+
+
+def _integer_counts(weights: list[float], total: int) -> list[int]:
+    raw = [w * total for w in weights]
+    counts = [math.floor(x) for x in raw]
+    order = sorted(
+        range(len(raw)), key=lambda i: raw[i] - counts[i], reverse=True
+    )
+    for idx in order[: total - sum(counts)]:
+        counts[idx] += 1
+    return counts
+
+
+def numeric_sanity(values: list[float | int]) -> dict[str, Any]:
+    finite = [float(x) for x in values if math.isfinite(float(x))]
+    return {
+        "n": len(values),
+        "finite_n": len(finite),
+        "missing_n": len(values) - len(finite),
+        "min": min(finite) if finite else None,
+        "max": max(finite) if finite else None,
+        "distinct_n": len(set(finite)),
+    }
+
+
+def manifest_summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
+    return {
+        "schema": SCHEMA,
+        "rows": len(rows),
+        "input_tokens": numeric_sanity([int(r["input_tokens"]) for r in rows]),
+        "output_tokens": numeric_sanity([int(r["output_tokens"]) for r in rows]),
+        "arrival_values": sorted({str(r["arrival"]) for r in rows}),
+        "pattern_values": sorted({str(r["pattern_id"]) for r in rows}),
+    }
+
+
+def materialize(args: argparse.Namespace) -> dict[str, Any]:
+    import numpy as np
+
+    rng = np.random.default_rng(args.workload_seed)
+    n = args.num_requests
+    if args.kind == "prefix-pool":
+        if args.num_prefixes <= 0 or args.prefix_len <= 0 or args.suffix_fixed <= 0:
+            raise ValueError("prefix-pool requires positive pool/prefix/suffix")
+        lengths = np.full(n, args.prefix_len + args.suffix_fixed, dtype=np.int64)
+        prefix_ids = np.arange(n, dtype=np.int64) % args.num_prefixes
+        rng.shuffle(prefix_ids)
+    else:
+        prefix_ids = np.full(n, -1, dtype=np.int64)
+        if args.input_uniform:
+            lo, hi = parse_range(args.input_uniform)
+            lengths = rng.integers(lo, hi + 1, n, dtype=np.int64)
+        elif args.input_fixed:
+            lengths = np.full(n, args.input_fixed, dtype=np.int64)
+        elif args.input_mixture:
+            spec = json.loads(args.input_mixture)
+            if not isinstance(spec, dict) or not spec:
+                raise ValueError("input mixture must be a non-empty JSON object")
+            keys = list(spec)
+            weights = [float(spec[key]) for key in keys]
+            if any(w < 0 for w in weights) or not math.isclose(sum(weights), 1.0):
+                raise ValueError("mixture weights must be non-negative and sum to 1")
+            pieces = []
+            for key, count in zip(
+                keys, _integer_counts(weights, n), strict=True
+            ):
+                kind, lo_text, hi_text = key.split(":")
+                if kind != "uniform":
+                    raise ValueError(f"unsupported mixture component: {key}")
+                pieces.append(
+                    rng.integers(
+                        int(lo_text), int(hi_text) + 1, count, dtype=np.int64
+                    )
+                )
+            lengths = np.concatenate(pieces)
+            rng.shuffle(lengths)
+        else:
+            raise ValueError("exactly one input distribution is required")
+    if args.output_fixed <= 0 or args.arrival not in {"steady", "burst:8"}:
+        raise ValueError("invalid output length or arrival class")
+
+    rows = []
+    for i in range(n):
+        row = {
+            "schema": SCHEMA,
+            "request_id": f"{args.id}-{i:05d}",
+            "pattern_id": args.id,
+            "kind": args.kind,
+            "input_tokens": int(lengths[i]),
+            "output_tokens": args.output_fixed,
+            "arrival": args.arrival,
+            "token_seed": int(args.workload_seed * 1000003 + i),
+        }
+        if args.kind == "prefix-pool":
+            row.update(
+                {
+                    "prefix_id": int(prefix_ids[i]),
+                    "num_prefixes": args.num_prefixes,
+                    "prefix_tokens": args.prefix_len,
+                }
+            )
+        rows.append(row)
+
+    out = Path(args.out)
+    atomic_jsonl(out, rows, mode=0o600)
+    summary = manifest_summary(rows)
+    summary.update({"sha256": sha256_file(out), "path": str(out)})
+    atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600)
+    print(json.dumps(summary, sort_keys=True))
+    return summary
+
+
+def materialize_private(args: argparse.Namespace) -> dict[str, Any]:
+    from transformers import AutoTokenizer
+
+    source = Path(args.source)
+    selected: list[dict[str, Any]] = []
+    with source.open(encoding="utf-8") as f:
+        for source_index, line in enumerate(f):
+            row = json.loads(line)
+            if (
+                float(row["sampling_u"]) <= args.sampling_u_max
+                and int(row["input_length"]) <= args.max_input_tokens
+            ):
+                selected.append(
+                    {
+                        "schema": SCHEMA,
+                        "request_id": f"{args.id}-{len(selected):05d}",
+                        "pattern_id": args.id,
+                        "kind": "private-trace",
+                        "input_tokens": int(row["input_length"]),
+                        "output_tokens": min(
+                            int(row["output_length"]), args.output_cap
+                        ),
+                        "arrival": args.arrival,
+                        "source_index": source_index,
+                        "prompt": row["prompt"],
+                    }
+                )
+
+    tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
+    diffs = [
+        len(tokenizer.encode(row["prompt"], add_special_tokens=False))
+        - row["input_tokens"]
+        for row in selected
+    ]
+    exact = sum(diff == 0 for diff in diffs)
+    exact_fraction = exact / len(diffs) if diffs else 0.0
+    max_abs = max((abs(diff) for diff in diffs), default=-1)
+    if exact_fraction < 0.99 or max_abs > 1:
+        raise RuntimeError(
+            "tokenizer parity gate failed: "
+            f"exact_fraction={exact_fraction:.6f} max_abs_error={max_abs}"
+        )
+
+    out = Path(args.out)
+    atomic_jsonl(out, selected, mode=0o600)
+    summary = manifest_summary(selected)
+    summary.update(
+        {
+            "sha256": sha256_file(out),
+            "source_sha256": sha256_file(source),
+            "tokenizer_exact_n": exact,
+            "tokenizer_exact_fraction": exact_fraction,
+            "tokenizer_max_abs_error": max_abs,
+            "path": str(out),
+        }
+    )
+    atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600)
+    print(json.dumps(summary, sort_keys=True))
+    return summary
+
+
+def load_manifest(path: Path) -> list[dict[str, Any]]:
+    rows = [json.loads(line) for line in path.read_text().splitlines() if line]
+    required = {
+        "request_id",
+        "pattern_id",
+        "input_tokens",
+        "output_tokens",
+        "arrival",
+    }
+    if not rows:
+        raise ValueError("empty manifest")
+    for row in rows:
+        if not required.issubset(row):
+            raise ValueError(f"manifest row lacks {sorted(required - set(row))}")
+    if len({row["request_id"] for row in rows}) != len(rows):
+        raise ValueError("duplicate request_id")
+    return rows
+
+
+def _token_stream(seed: int, count: int) -> list[int]:
+    state = seed & 0xFFFFFFFF
+    out = []
+    for _ in range(count):
+        state = (1664525 * state + 1013904223) & 0xFFFFFFFF
+        out.append(TOKEN_BASE + state % TOKEN_SPAN)
+    return out
+
+
+def synthetic_prompt(row: dict[str, Any]) -> list[int]:
+    length = int(row["input_tokens"])
+    seed = int(row.get("token_seed", 0))
+    if row.get("kind") == "prefix-pool":
+        prefix_n = int(row["prefix_tokens"])
+        tokens = _token_stream(0xA5A50000 + int(row["prefix_id"]), prefix_n)
+        tokens += _token_stream(seed, length - prefix_n)
+        offset = prefix_n
+    else:
+        tokens = _token_stream(seed, length)
+        offset = 0
+    if length - offset >= 3:
+        index = int(row["request_id"].rsplit("-", 1)[1])
+        tokens[offset : offset + 3] = [
+            TOKEN_BASE + index % 100,
+            TOKEN_BASE + (index // 100) % 100,
+            TOKEN_BASE + (index // 10000) % 100,
+        ]
+    return tokens
+
+
+@dataclass
+class RunContext:
+    args: argparse.Namespace
+    rows: list[dict[str, Any]]
+    t0: float
+    clean_end: float
+    stop_event: asyncio.Event
+    lock: asyncio.Lock
+    next_index: int = 0
+    in_flight: int = 0
+    max_in_flight: int = 0
+    exhausted: bool = False
+    admission_stop_s: float | None = None
+
+    async def next_row(self) -> dict[str, Any]:
+        async with self.lock:
+            if self.next_index >= len(self.rows):
+                self.exhausted = True
+                raise ManifestExhausted(
+                    f"manifest exhausted after {self.next_index} admissions"
+                )
+            row = self.rows[self.next_index]
+            self.next_index += 1
+            return row
+
+
+async def request_one(
+    ctx: RunContext,
+    session: aiohttp.ClientSession,
+    row: dict[str, Any],
+    scheduled: float,
+) -> dict[str, Any]:
+    loop = asyncio.get_running_loop()
+    admitted = loop.time()
+    ctx.in_flight += 1
+    ctx.max_in_flight = max(ctx.max_in_flight, ctx.in_flight)
+    status = 0
+    actual_output: int | None = None
+    first_token: float | None = None
+    error_kind: str | None = None
+    try:
+        prompt: str | list[int] = (
+            row["prompt"]
+            if row.get("kind") == "private-trace"
+            else synthetic_prompt(row)
+        )
+        if not isinstance(prompt, str) and len(prompt) != int(row["input_tokens"]):
+            raise AssertionError("synthetic prompt length drift")
+        payload = {
+            "model": ctx.args.model,
+            "prompt": prompt,
+            "max_tokens": int(row["output_tokens"]),
+            "temperature": ctx.args.temperature,
+            "ignore_eos": ctx.args.ignore_eos,
+            "stream": True,
+            "stream_options": {"include_usage": True},
+            "add_special_tokens": False,
+            "seed": ctx.args.server_seed,
+        }
+        headers = {
+            "Content-Type": "application/json",
+            "x-request-id": str(row["request_id"]),
+        }
+        async with session.post(
+            ctx.args.base_url.rstrip("/") + "/v1/completions",
+            json=payload,
+            headers=headers,
+        ) as response:
+            status = response.status
+            if status != 200:
+                error_kind = f"http_{status}"
+            else:
+                buf = b""
+                async for chunk in response.content.iter_any():
+                    buf += chunk
+                    while b"\n" in buf:
+                        line, buf = buf.split(b"\n", 1)
+                        line = line.strip()
+                        if not line.startswith(b"data:"):
+                            continue
+                        data = line[5:].strip()
+                        if data == b"[DONE]":
+                            continue
+                        event = json.loads(data)
+                        if event.get("choices") and first_token is None:
+                            first_token = loop.time()
+                        if event.get("usage") is not None:
+                            actual_output = int(event["usage"]["completion_tokens"])
+                if actual_output is None:
+                    error_kind = "missing_usage"
+    except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
+        error_kind = type(exc).__name__
+    except Exception as exc:
+        error_kind = type(exc).__name__
+    finally:
+        completed = loop.time()
+        ctx.in_flight -= 1
+    success = (
+        status == 200
+        and error_kind is None
+        and actual_output == int(row["output_tokens"])
+    )
+    if status == 200 and actual_output is not None and not success:
+        error_kind = "output_token_mismatch"
+    return {
+        "schema": SCHEMA,
+        "request_id": row["request_id"],
+        "scheduled_s": scheduled - ctx.t0,
+        "admitted_s": admitted - ctx.t0,
+        "first_token_s": None if first_token is None else first_token - ctx.t0,
+        "completed_s": completed - ctx.t0,
+        "input_tokens": int(row["input_tokens"]),
+        "requested_output_tokens": int(row["output_tokens"]),
+        "actual_output_tokens": actual_output,
+        "http_status": status,
+        "success": success,
+        "error_kind": error_kind,
+    }
+
+
+async def saturation_load(
+    ctx: RunContext, session: aiohttp.ClientSession
+) -> list[dict[str, Any]]:
+    results: list[dict[str, Any]] = []
+
+    async def worker() -> None:
+        while not ctx.stop_event.is_set():
+            try:
+                row = await ctx.next_row()
+            except ManifestExhausted:
+                ctx.stop_event.set()
+                return
+            results.append(
+                await request_one(ctx, session, row, asyncio.get_running_loop().time())
+            )
+
+    tasks = [
+        asyncio.create_task(worker()) for _ in range(ctx.args.max_concurrency)
+    ]
+    await asyncio.gather(*tasks)
+    return results
+
+
+async def finite_load(
+    ctx: RunContext, session: aiohttp.ClientSession, rate: float
+) -> list[dict[str, Any]]:
+    sem = asyncio.Semaphore(ctx.args.max_concurrency)
+    tasks: list[asyncio.Task[dict[str, Any]]] = []
+    batch = 8 if str(ctx.rows[0]["arrival"]) == "burst:8" else 1
+    period = batch / rate
+    event_index = 0
+
+    async def limited(row: dict[str, Any], scheduled: float) -> dict[str, Any]:
+        async with sem:
+            return await request_one(ctx, session, row, scheduled)
+
+    while not ctx.stop_event.is_set():
+        scheduled = ctx.t0 + event_index * period
+        delay = scheduled - asyncio.get_running_loop().time()
+        if delay > 0:
+            try:
+                await asyncio.wait_for(ctx.stop_event.wait(), timeout=delay)
+                break
+            except asyncio.TimeoutError:
+                pass
+        if ctx.stop_event.is_set():
+            break
+        try:
+            for _ in range(batch):
+                tasks.append(
+                    asyncio.create_task(limited(await ctx.next_row(), scheduled))
+                )
+        except ManifestExhausted:
+            ctx.stop_event.set()
+            break
+        event_index += 1
+    return await asyncio.gather(*tasks) if tasks else []
+
+
+async def post_profile(
+    session: aiohttp.ClientSession, base_url: str, endpoint: str
+) -> tuple[float, float, int]:
+    loop = asyncio.get_running_loop()
+    before = loop.time()
+    async with session.post(base_url.rstrip("/") + endpoint) as response:
+        status = response.status
+        await response.read()
+    return before, loop.time(), status
+
+
+def _trace_loadable(path: Path) -> bool:
+    try:
+        opener = gzip.open if path.suffix == ".gz" else open
+        with opener(path, "rt", encoding="utf-8") as f:
+            parsed = json.load(f)
+        return isinstance(parsed, dict) and isinstance(parsed.get("traceEvents"), list)
+    except (OSError, EOFError, json.JSONDecodeError):
+        return False
+
+
+async def wait_new_trace(
+    trace_dir: Path, before: set[Path], timeout: float
+) -> Path:
+    deadline = asyncio.get_running_loop().time() + timeout
+    while asyncio.get_running_loop().time() < deadline:
+        for path in sorted(set(trace_dir.glob("*.pt.trace.json*")) - before):
+            if _trace_loadable(path):
+                return path
+        await asyncio.sleep(0.25)
+    raise TimeoutError(f"no new loadable trace within {timeout}s")
+
+
+async def timeline(
+    ctx: RunContext, session: aiohttp.ClientSession
+) -> list[dict[str, Any]]:
+    args = ctx.args
+    profiles: list[dict[str, Any]] = []
+    await asyncio.sleep(max(0, ctx.clean_end - asyncio.get_running_loop().time()))
+    if args.profile_after_clean:
+        trace_dir = Path(args.profile_trace_dir)
+        for window in range(args.num_profile_windows):
+            prior = set(trace_dir.glob("*.pt.trace.json*"))
+            start_before, start_after, start_status = await post_profile(
+                session, args.base_url, "/start_profile"
+            )
+            trace = await wait_new_trace(
+                trace_dir, prior, args.profile_timeout_seconds
+            )
+            trace_ready = asyncio.get_running_loop().time()
+            stop_before, stop_after, stop_status = await post_profile(
+                session, args.base_url, "/stop_profile"
+            )
+            profiles.append(
+                {
+                    "window": window + 1,
+                    "start_call_s": start_before - ctx.t0,
+                    "start_return_s": start_after - ctx.t0,
+                    "trace_ready_s": trace_ready - ctx.t0,
+                    "stop_call_s": stop_before - ctx.t0,
+                    "stop_return_s": stop_after - ctx.t0,
+                    "start_status": start_status,
+                    "stop_status": stop_status,
+                    "trace_file": trace.name,
+                    "trace_sha256": sha256_file(trace),
+                }
+            )
+            if start_status != 200 or stop_status != 200:
+                raise RuntimeError("profile endpoint returned non-200")
+            await asyncio.sleep(args.recovery_seconds)
+    else:
+        await asyncio.sleep(args.post_clean_seconds)
+    ctx.admission_stop_s = asyncio.get_running_loop().time() - ctx.t0
+    ctx.stop_event.set()
+    return profiles
+
+
+def segment_summary(
+    records: list[dict[str, Any]], start: float, end: float
+) -> dict[str, Any]:
+    admitted = [r for r in records if start <= r["admitted_s"] < end]
+    completed = [r for r in records if start <= r["completed_s"] < end]
+    successes = [r for r in completed if r["success"]]
+    duration = end - start
+    return {
+        "start_s": start,
+        "end_s": end,
+        "duration_s": duration,
+        "admitted": len(admitted),
+        "completed": len(successes),
+        "failed": len(completed) - len(successes),
+        "offered_rps": len(admitted) / duration,
+        "completed_throughput_rps": len(successes) / duration,
+        "input_tokens": sum(r["input_tokens"] for r in successes),
+        "output_tokens": sum(r["actual_output_tokens"] or 0 for r in successes),
+    }
+
+
+async def run_load(args: argparse.Namespace) -> dict[str, Any]:
+    manifest = Path(args.manifest)
+    rows = load_manifest(manifest)
+    arrivals = {row["arrival"] for row in rows}
+    if len(arrivals) != 1:
+        raise ValueError("a manifest must have one arrival class")
+    if args.load_point == "saturation":
+        if args.request_rate != "inf":
+            raise ValueError("saturation requires --request-rate inf")
+        rate = math.inf
+    else:
+        if not args.saturation_result:
+            raise ValueError("moderate requires --saturation-result")
+        sat = json.loads(Path(args.saturation_result).read_text())
+        rate = args.rate_fraction * float(sat["clean"]["completed_throughput_rps"])
+        if not math.isfinite(rate) or rate <= 0:
+            raise ValueError("derived moderate rate must be positive and finite")
+
+    loop = asyncio.get_running_loop()
+    t0 = loop.time()
+    clean_seconds = args.clean_segment_seconds * args.num_clean_segments
+    ctx = RunContext(
+        args=args,
+        rows=rows,
+        t0=t0,
+        clean_end=t0 + args.warmup_seconds + clean_seconds,
+        stop_event=asyncio.Event(),
+        lock=asyncio.Lock(),
+    )
+    timeout = aiohttp.ClientTimeout(total=None, connect=30, sock_read=600)
+    connector = aiohttp.TCPConnector(limit=args.max_concurrency)
+    async with aiohttp.ClientSession(timeout=timeout, connector=connector) as session:
+        profile_task = asyncio.create_task(timeline(ctx, session))
+        load_task = asyncio.create_task(
+            saturation_load(ctx, session)
+            if math.isinf(rate)
+            else finite_load(ctx, session, rate)
+        )
+        try:
+            profiles = await profile_task
+        except Exception:
+            ctx.stop_event.set()
+            await load_task
+            raise
+        records = await load_task
+
+    clean_start = args.warmup_seconds
+    clean_end = clean_start + clean_seconds
+    clean = segment_summary(records, clean_start, clean_end)
+    segments = []
+    for i in range(args.num_clean_segments):
+        start = clean_start + i * args.clean_segment_seconds
+        segments.append(
+            {
+                "name": chr(ord("A") + i),
+                **segment_summary(
+                    records, start, start + args.clean_segment_seconds
+                ),
+            }
+        )
+    successful = [r for r in records if r["success"]]
+    elapsed_seconds = loop.time() - t0
+    if ctx.admission_stop_s is None:
+        raise RuntimeError("admission stop timestamp was not recorded")
+    drain_seconds = elapsed_seconds - ctx.admission_stop_s
+    result = {
+        "schema": SCHEMA,
+        "manifest_sha256": sha256_file(manifest),
+        "manifest_rows": len(rows),
+        "manifest_admitted": ctx.next_index,
+        "manifest_wrapped": False,
+        "manifest_exhausted": ctx.exhausted,
+        "load_point": args.load_point,
+        "request_rate": "inf" if math.isinf(rate) else rate,
+        "rate_fraction": None if math.isinf(rate) else args.rate_fraction,
+        "arrival": next(iter(arrivals)),
+        "warmup_seconds": args.warmup_seconds,
+        "clean_segment_seconds": args.clean_segment_seconds,
+        "num_clean_segments": args.num_clean_segments,
+        "elapsed_seconds": elapsed_seconds,
+        "admission_stop_s": ctx.admission_stop_s,
+        "drain_seconds": drain_seconds,
+        "max_in_flight": ctx.max_in_flight,
+        "records": len(records),
+        "successful_records": len(successful),
+        "failed_records": len(records) - len(successful),
+        "clean": clean,
+        "segments": segments,
+        "profiles": profiles,
+    }
+    sanity = {
+        "schema": SCHEMA,
+        "numeric": {
+            "input_tokens": numeric_sanity([r["input_tokens"] for r in records]),
+            "requested_output_tokens": numeric_sanity(
+                [r["requested_output_tokens"] for r in records]
+            ),
+            "actual_output_tokens": numeric_sanity(
+                [
+                    r["actual_output_tokens"]
+                    for r in records
+                    if r["actual_output_tokens"] is not None
+                ]
+            ),
+            "scheduled_s": numeric_sanity([r["scheduled_s"] for r in records]),
+            "admitted_s": numeric_sanity([r["admitted_s"] for r in records]),
+            "completed_s": numeric_sanity([r["completed_s"] for r in records]),
+        },
+        "invariants": {
+            "clean_duration_exact": math.isclose(clean["duration_s"], clean_seconds),
+            "segment_count_exact": len(segments) == args.num_clean_segments,
+            "manifest_no_wrap": ctx.next_index <= len(rows),
+            "manifest_not_exhausted": not ctx.exhausted,
+            "concurrency_bounded": ctx.max_in_flight <= args.max_concurrency,
+            "drain_within_timeout": drain_seconds <= args.drain_timeout_seconds,
+            "output_tokens_exact": all(
+                r["actual_output_tokens"] == r["requested_output_tokens"]
+                for r in successful
+            ),
+            "clean_failures_zero": clean["failed"] == 0,
+            "profile_count_exact": len(profiles)
+            == (args.num_profile_windows if args.profile_after_clean else 0),
+            "profile_status_ok": all(
+                p["start_status"] == 200 and p["stop_status"] == 200
+                for p in profiles
+            ),
+        },
+    }
+    if not math.isinf(rate):
+        sanity["invariants"]["moderate_offered_within_5pct"] = (
+            abs(clean["offered_rps"] / rate - 1) <= 0.05
+        )
+    out = Path(args.result_dir)
+    out.mkdir(parents=True, exist_ok=True)
+    atomic_jsonl(out / "requests.jsonl", sorted(records, key=lambda r: r["admitted_s"]))
+    atomic_jsonl(out / "segments.jsonl", segments)
+    atomic_json(out / "result.json", result)
+    atomic_json(out / "sanity.json", sanity)
+    if ctx.exhausted:
+        raise ManifestExhausted("manifest exhausted; result retained for diagnosis")
+    failed = [name for name, ok in sanity["invariants"].items() if not ok]
+    if failed:
+        raise RuntimeError(f"client sanity failure: {failed}")
+    return result
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser()
+    sub = parser.add_subparsers(dest="command", required=True)
+    mat = sub.add_parser("materialize")
+    mat.add_argument("--id", required=True)
+    mat.add_argument("--kind", choices=("synthetic", "prefix-pool"), required=True)
+    group = mat.add_mutually_exclusive_group()
+    group.add_argument("--input-uniform")
+    group.add_argument("--input-fixed", type=int)
+    group.add_argument("--input-mixture")
+    mat.add_argument("--output-fixed", type=int, required=True)
+    mat.add_argument("--prefix", default="none")
+    mat.add_argument("--arrival", required=True)
+    mat.add_argument("--num-requests", type=int, required=True)
+    mat.add_argument("--workload-seed", type=int, required=True)
+    mat.add_argument("--num-prefixes", type=int, default=0)
+    mat.add_argument("--prefix-len", type=int, default=0)
+    mat.add_argument("--suffix-fixed", type=int, default=0)
+    mat.add_argument("--out", required=True)
+    private = sub.add_parser("materialize-private")
+    private.add_argument("--id", required=True)
+    private.add_argument("--source", required=True)
+    private.add_argument("--sampling-u-max", type=float, required=True)
+    private.add_argument("--max-input-tokens", type=int, required=True)
+    private.add_argument("--output-cap", type=int, required=True)
+    private.add_argument("--preserve-prompts", action="store_true", required=True)
+    private.add_argument("--disable-shuffle", action="store_true", required=True)
+    private.add_argument("--arrival", required=True)
+    private.add_argument("--model", required=True)
+    private.add_argument("--out", required=True)
+    run = sub.add_parser("run")
+    run.add_argument("--manifest", required=True)
+    run.add_argument("--base-url", required=True)
+    run.add_argument("--model", required=True)
+    run.add_argument("--load-point", choices=("saturation", "moderate"), required=True)
+    run.add_argument("--request-rate")
+    run.add_argument("--saturation-result")
+    run.add_argument("--rate-fraction", type=float, default=0.60)
+    run.add_argument("--max-concurrency", type=int, default=256)
+    run.add_argument("--ignore-eos", action="store_true")
+    run.add_argument("--temperature", type=float, default=0.0)
+    run.add_argument("--warmup-seconds", type=float, default=60)
+    run.add_argument("--clean-segment-seconds", type=float, default=80)
+    run.add_argument("--num-clean-segments", type=int, default=3)
+    run.add_argument("--profile-after-clean", action="store_true")
+    run.add_argument("--num-profile-windows", type=int, default=0)
+    run.add_argument("--profile-warmup-iterations", type=int, default=2)
+    run.add_argument("--profile-active-iterations", type=int, default=8)
+    run.add_argument("--profile-trace-dir")
+    run.add_argument("--profile-timeout-seconds", type=float, default=120)
+    run.add_argument("--recovery-seconds", type=float, default=30)
+    run.add_argument("--post-clean-seconds", type=float, default=0)
+    run.add_argument("--drain-timeout-seconds", type=float, default=120)
+    run.add_argument("--workload-seed", type=int, default=20260712)
+    run.add_argument("--server-seed", type=int, default=20260712)
+    run.add_argument("--result-dir", required=True)
+    return parser
+
+
+def main() -> None:
+    args = build_parser().parse_args()
+    if args.command == "materialize":
+        materialize(args)
+    elif args.command == "materialize-private":
+        materialize_private(args)
+    else:
+        if args.profile_after_clean and not args.profile_trace_dir:
+            raise ValueError("--profile-after-clean requires --profile-trace-dir")
+        print(json.dumps(asyncio.run(run_load(args)), sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/opprof-phase3-ea/provenance/test_phase3_tools.py

@@ -0,0 +1,353 @@
+#!/usr/bin/env python3
+
+from __future__ import annotations
+
+import asyncio
+import importlib.util
+import json
+import sys
+import tempfile
+import unittest
+from unittest import mock
+from pathlib import Path
+from types import SimpleNamespace
+
+from aiohttp import web
+
+
+HERE = Path(__file__).parent
+
+
+def load_module(name: str, filename: str):
+    spec = importlib.util.spec_from_file_location(name, HERE / filename)
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[name] = module
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+client = load_module("phase3_client", "opprof_phase3_client.py")
+controller = load_module("phase3_controller", "opprof_phase3_controller.py")
+
+
+def rows(n: int, arrival: str = "steady") -> list[dict]:
+    return [
+        {
+            "schema": 1,
+            "request_id": f"T-{i:05d}",
+            "pattern_id": "T",
+            "kind": "synthetic",
+            "input_tokens": 8,
+            "output_tokens": 2,
+            "arrival": arrival,
+            "token_seed": i + 1,
+        }
+        for i in range(n)
+    ]
+
+
+class MockServer:
+    def __init__(self) -> None:
+        self.active = 0
+        self.max_active = 0
+        self.payloads = []
+        self.runner = None
+        self.port = None
+
+    async def completion(self, request):
+        payload = await request.json()
+        self.payloads.append(payload)
+        self.active += 1
+        self.max_active = max(self.max_active, self.active)
+        await asyncio.sleep(0.01)
+        response = web.StreamResponse(
+            status=200, headers={"Content-Type": "text/event-stream"}
+        )
+        await response.prepare(request)
+        await response.write(
+            b'data: {"choices":[{"text":"x"}],"usage":null}\n\n'
+        )
+        usage = json.dumps(
+            {
+                "choices": [],
+                "usage": {
+                    "prompt_tokens": len(payload["prompt"]),
+                    "completion_tokens": payload["max_tokens"],
+                },
+            }
+        ).encode()
+        await response.write(b"data: " + usage + b"\n\n")
+        await response.write(b"data: [DONE]\n\n")
+        await response.write_eof()
+        self.active -= 1
+        return response
+
+    async def start(self):
+        app = web.Application()
+        app.router.add_post("/v1/completions", self.completion)
+        self.runner = web.AppRunner(app)
+        await self.runner.setup()
+        site = web.TCPSite(self.runner, "127.0.0.1", 0)
+        await site.start()
+        self.port = site._server.sockets[0].getsockname()[1]
+
+    async def stop(self):
+        await self.runner.cleanup()
+
+
+def run_args(
+    manifest: Path,
+    result_dir: Path,
+    port: int,
+    load_point: str = "saturation",
+    saturation_result: Path | None = None,
+):
+    return SimpleNamespace(
+        manifest=str(manifest),
+        base_url=f"http://127.0.0.1:{port}",
+        model="mock",
+        load_point=load_point,
+        request_rate="inf" if load_point == "saturation" else None,
+        saturation_result=None if saturation_result is None else str(saturation_result),
+        rate_fraction=0.60,
+        max_concurrency=3,
+        ignore_eos=True,
+        temperature=0,
+        warmup_seconds=0.04,
+        clean_segment_seconds=0.04,
+        num_clean_segments=3,
+        profile_after_clean=False,
+        num_profile_windows=0,
+        profile_warmup_iterations=2,
+        profile_active_iterations=8,
+        profile_trace_dir=None,
+        profile_timeout_seconds=1,
+        recovery_seconds=0,
+        post_clean_seconds=0,
+        drain_timeout_seconds=1,
+        workload_seed=20260712,
+        server_seed=20260712,
+        result_dir=str(result_dir),
+    )
+
+
+class Phase3ToolTests(unittest.TestCase):
+    def test_synthetic_prompt_exact_and_unique_early_prefix(self):
+        generated = [client.synthetic_prompt(row) for row in rows(200)]
+        self.assertTrue(all(len(value) == 8 for value in generated))
+        self.assertEqual(len({tuple(value[:3]) for value in generated}), 200)
+
+    def test_p05_manifest_has_exact_half_modes(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            out = Path(tmp) / "P05.jsonl"
+            args = SimpleNamespace(
+                id="P05",
+                kind="synthetic",
+                input_uniform=None,
+                input_fixed=None,
+                input_mixture='{"uniform:128:512":0.5,"uniform:4096:8192":0.5}',
+                output_fixed=64,
+                prefix="none",
+                arrival="steady",
+                num_requests=100,
+                workload_seed=20260712,
+                num_prefixes=0,
+                prefix_len=0,
+                suffix_fixed=0,
+                out=str(out),
+            )
+            client.materialize(args)
+            manifest = client.load_manifest(out)
+            self.assertEqual(sum(r["input_tokens"] <= 512 for r in manifest), 50)
+            self.assertEqual(sum(r["input_tokens"] >= 4096 for r in manifest), 50)
+
+    def test_prefix_pool_balanced(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            out = Path(tmp) / "P08.jsonl"
+            args = SimpleNamespace(
+                id="P08",
+                kind="prefix-pool",
+                input_uniform=None,
+                input_fixed=None,
+                input_mixture=None,
+                output_fixed=512,
+                prefix="none",
+                arrival="burst:8",
+                num_requests=80,
+                workload_seed=20260712,
+                num_prefixes=8,
+                prefix_len=1024,
+                suffix_fixed=256,
+                out=str(out),
+            )
+            client.materialize(args)
+            manifest = client.load_manifest(out)
+            counts = {i: 0 for i in range(8)}
+            for row in manifest:
+                counts[row["prefix_id"]] += 1
+                self.assertEqual(row["input_tokens"], 1280)
+            self.assertEqual(set(counts.values()), {10})
+
+    def test_fixed_duration_saturation_and_redaction(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(1000))
+                mock = MockServer()
+                await mock.start()
+                try:
+                    result = await client.run_load(
+                        run_args(manifest, root / "result", mock.port)
+                    )
+                finally:
+                    await mock.stop()
+                self.assertEqual(result["max_in_flight"], 3)
+                self.assertEqual(mock.max_active, 3)
+                self.assertAlmostEqual(result["clean"]["duration_s"], 0.12, places=9)
+                self.assertEqual(len(result["segments"]), 3)
+                self.assertLessEqual(result["drain_seconds"], 1)
+                self.assertTrue(
+                    all(p["max_tokens"] == 2 and p["ignore_eos"] for p in mock.payloads)
+                )
+                text = (root / "result/requests.jsonl").read_text()
+                self.assertNotIn('"prompt":', text)
+                self.assertNotIn('"text":', text)
+
+        asyncio.run(case())
+
+    def test_drain_timeout_is_a_hard_sanity_failure(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(1000))
+                mock = MockServer()
+                await mock.start()
+                try:
+                    args = run_args(manifest, root / "result", mock.port)
+                    args.drain_timeout_seconds = 0
+                    with self.assertRaisesRegex(RuntimeError, "drain_within_timeout"):
+                        await client.run_load(args)
+                finally:
+                    await mock.stop()
+                sanity = json.loads((root / "result/sanity.json").read_text())
+                self.assertFalse(sanity["invariants"]["drain_within_timeout"])
+
+        asyncio.run(case())
+
+    def test_burst_schedule_is_eight_at_once(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(1000, "burst:8"))
+                sat = root / "sat.json"
+                client.atomic_json(
+                    sat, {"clean": {"completed_throughput_rps": 100.0}}
+                )
+                mock = MockServer()
+                await mock.start()
+                try:
+                    args = run_args(
+                        manifest, root / "result", mock.port, "moderate", sat
+                    )
+                    args.warmup_seconds = 0
+                    args.clean_segment_seconds = 0.4 / 3
+                    result = await client.run_load(args)
+                finally:
+                    await mock.stop()
+                records = [
+                    json.loads(line)
+                    for line in (root / "result/requests.jsonl").read_text().splitlines()
+                ]
+                groups = {}
+                for record in records:
+                    groups.setdefault(round(record["scheduled_s"], 6), 0)
+                    groups[round(record["scheduled_s"], 6)] += 1
+                self.assertTrue(all(value == 8 for value in groups.values()))
+                starts = sorted(groups)
+                if len(starts) > 1:
+                    self.assertAlmostEqual(starts[1] - starts[0], 8 / 60, places=5)
+                self.assertAlmostEqual(result["request_rate"], 60.0)
+
+        asyncio.run(case())
+
+    def test_manifest_exhaustion_stops_instead_of_wrapping(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(2))
+                mock = MockServer()
+                await mock.start()
+                try:
+                    with self.assertRaises(client.ManifestExhausted):
+                        await client.run_load(
+                            run_args(manifest, root / "result", mock.port)
+                        )
+                finally:
+                    await mock.stop()
+
+        asyncio.run(case())
+
+    def test_cpu_affinity_sets_are_disjoint_and_cover_host(self):
+        values = []
+        for spec in controller.CPU_MAP.values():
+            lo, hi = [int(x) for x in spec.split("-")]
+            values.extend(range(lo, hi + 1))
+        self.assertEqual(sorted(values), list(range(160)))
+        self.assertEqual(len(values), len(set(values)))
+
+    def test_kernel_mapping_priority(self):
+        cases = {
+            "void vllm::moe::topkGating": "moe_router",
+            "ncclDevKernel_AllReduce": "collective",
+            "flash_fwd_kernel": "attention",
+            "nvjet_sm90_tst": "moe_gemm",
+            "argmax_kernel": "sampler",
+            "cutlass_gemm": "dense_gemm",
+            "triton_red_fused_add_rms_norm": "norm_elementwise",
+            "cache_swap_kernel": "kv_memory",
+            "unknown": "other",
+        }
+        self.assertEqual(
+            {name: controller.classify_kernel(name) for name in cases}, cases
+        )
+
+    def test_atomic_state_replacement(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "state.json"
+            controller.atomic_json(path, {"schema": 1, "value": 1})
+            controller.atomic_json(path, {"schema": 1, "value": 2})
+            self.assertEqual(json.loads(path.read_text())["value"], 2)
+            self.assertFalse(list(path.parent.glob("*.tmp.*")))
+
+    def test_fingerprint_survives_json_roundtrip(self):
+        original_run_text = controller.run_text
+        original_hash = controller.sha256_file
+        try:
+            controller.run_text = lambda *args, **kwargs: "deadbeef\n"
+            controller.sha256_file = lambda path: "a" * 64
+            fingerprint = controller.make_fingerprint()
+        finally:
+            controller.run_text = original_run_text
+            controller.sha256_file = original_hash
+        self.assertEqual(json.loads(json.dumps(fingerprint)), fingerprint)
+        self.assertEqual(set(fingerprint["cpu_map"]), {str(i) for i in range(8)})
+
+    def test_server_shutdown_signals_parent_before_group(self):
+        process = SimpleNamespace(pid=12345, poll=lambda: None)
+        with (
+            mock.patch.object(controller.os, "kill") as kill,
+            mock.patch.object(controller.os, "killpg") as killpg,
+            mock.patch.object(controller, "_process_group_alive", return_value=False),
+        ):
+            controller.stop_servers([process])
+        kill.assert_called_once_with(12345, controller.signal.SIGINT)
+        killpg.assert_not_called()
+
+
+if __name__ == "__main__":
+    unittest.main(verbosity=2)

runs/opprof-phase3/phase3/access-blocker-20260712.json

@@ -0,0 +1,47 @@
+{
+  "schema": 1,
+  "status": "external_access_blocked",
+  "host": "dash0",
+  "last_reachability_probe_local": "2026-07-12T16:16:39+08:00",
+  "failure": "SSH connection timed out during banner exchange before any remote command executed",
+  "last_durable_remote_state": {
+    "observed_before_outage": true,
+    "controller_pid": 2237019,
+    "controller_status": "running",
+    "active_stage": "primary-02-saturation",
+    "active_stage_phase": "starting_servers",
+    "completed_measured_runs": 8,
+    "drain_quarantined_runs": 0,
+    "clean_window_failures": 0,
+    "missing_trace_files": 8,
+    "gpu_hours_total": 6.8156048206885655
+  },
+  "resume": {
+    "state": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/controller-state.json",
+    "log": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/phase3-matrix-controller.log",
+    "command": "cd /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712 && /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python scripts/opprof_phase3_matrix.py run --resume",
+    "rule": "inspect state/PID/GPU ownership first; do not relaunch a live controller"
+  },
+  "profile_control_repair": {
+    "reason": "P03/C01 saturated all data-plane connector slots, so /start_profile timed out before reaching the server",
+    "change": "dedicated aiohttp TCPConnector(limit=2) and ClientSession for profile control calls",
+    "old_client_sha256": "a87d92efecd5a8765b51067800b6382f9b174a2ede65f8933fcc9f846ff03d84",
+    "new_client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0",
+    "remote_record": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/repairs/repair-004-profile-control-connector.json",
+    "tests": "13/13 unittest PASS; ruff PASS; py_compile PASS"
+  },
+  "analysis_ready": {
+    "script_sha256": "205d9012d9462d84403a0f1435c8a449389aa4bef448d8d6dbced707a11559b0",
+    "test_sha256": "eac6f55042865c9a24811eef9c1a13c23a26dd1504b54348513d8e6a7ba2b939",
+    "tests": "4/4 unittest PASS; ruff PASS; py_compile PASS",
+    "executed_on_matrix": false
+  },
+  "not_claimed": [
+    "matrix completion",
+    "final quarantine count",
+    "H1a verdict",
+    "H1b verdict",
+    "final GPU hours",
+    "final GPU cleanup"
+  ]
+}

runs/opprof-phase3/phase3/analysis/launch-ap37.log

@@ -0,0 +1 @@
+CPU_ANALYSIS A-P3-7: accepted_runs=40, run_root=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3, private_manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests, bootstrap=100000 seed=20260714, output=runs/phase3/analysis/metrics-ap37.json, expected=10-30m CPU-only, GPU_cost=0

runs/opprof-phase3/phase3/controller-state-stop-ap36.json

@@ -0,0 +1,737 @@
+{
+  "clean_window_failures": 0,
+  "completed_burnins": 5,
+  "completed_measured_runs": 40,
+  "controller_pid": 2438791,
+  "created_at": 1783833885.960389,
+  "drain_quarantined_runs": 0,
+  "fingerprint": {
+    "cells": [
+      "P08-C00",
+      "P03-C10",
+      "P07-C00",
+      "P10-C01",
+      "P01-C10",
+      "P01-C01",
+      "P10-C10",
+      "P03-C01",
+      "P09-C00",
+      "P06-C01",
+      "P10-C11",
+      "P01-C00",
+      "P01-C11",
+      "P10-C00",
+      "P04-C00",
+      "P03-C00",
+      "P06-C10",
+      "P02-C00",
+      "P06-C00",
+      "P06-C11",
+      "P11-C00",
+      "P10-C00-TP2",
+      "P03-C11",
+      "P05-C00"
+    ],
+    "client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0",
+    "common_controller_sha256": "95f7169a1771e385aab40fcaecd967dc5cff0c21ea67bdd139382774ba43f01f",
+    "controller_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f",
+    "cpu_map": {
+      "0": "0-19",
+      "1": "20-39",
+      "2": "40-59",
+      "3": "60-79",
+      "4": "80-99",
+      "5": "100-119",
+      "6": "120-139",
+      "7": "140-159"
+    },
+    "manifests": {
+      "P01": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl",
+        "rows": 32768,
+        "sha256": "13ffb226c83373f54c4a7afea6c78cb7cd29720f1858d56728826fc1367b31a4"
+      },
+      "P02": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P02.jsonl",
+        "rows": 32768,
+        "sha256": "0138ada3fccc98298daee66c26bd1952c987cb42ed8d5341d66b698a597417f9"
+      },
+      "P03": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P03.jsonl",
+        "rows": 32768,
+        "sha256": "432cfbc26d36f105c179c83f3bb0f3b24b8b3f205788263b4171797f0a4d6fa1"
+      },
+      "P04": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P04.jsonl",
+        "rows": 32768,
+        "sha256": "caf8d0941b093956a81e1413adc4a4ea9d92460f1b2ed0f6a9b118d0119d6247"
+      },
+      "P05": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P05.jsonl",
+        "rows": 32768,
+        "sha256": "192e213109f8cb429b99d9eb0f227bb4390fc03f63b02d5456569369bff5a3d7"
+      },
+      "P06": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P06.jsonl",
+        "rows": 32768,
+        "sha256": "65954bc6e47de9e7be07b8975f97f0bc4639979ef6d33e8c664557ded34b9f96"
+      },
+      "P07": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P07.jsonl",
+        "rows": 32768,
+        "sha256": "74df66e21a705cd583493199a875e226e93d2da7cfede9bca81dfd9bcb8c9cc6"
+      },
+      "P08": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P08.jsonl",
+        "rows": 32768,
+        "sha256": "0c4f835aae099265c3eb596d06c6c2fa7070dd280558eb289c602e8c3434dfe9"
+      },
+      "P09": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl",
+        "rows": 32768,
+        "sha256": "7af92ee3c27dc7d2cf895d6ff3a6e737ec4b6da13d6841ca59e1166f28a0ae1e"
+      },
+      "P10": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl",
+        "rows": 4011,
+        "sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33"
+      },
+      "P11": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P11.jsonl",
+        "rows": 32768,
+        "sha256": "7d196df38963528ff181cf72ce39c8ad913c8f61d40b1425410d3c6c30b6be18"
+      }
+    },
+    "model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B",
+    "source_commit": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05",
+    "source_tree": "a3d536b287a724e60abbec68b45eed7e088a15d1"
+  },
+  "gpu_hours_this_stage": 0.7173924536837472,
+  "gpu_hours_total": 14.025875418755744,
+  "missing_trace_files": 8,
+  "repairs": [
+    {
+      "change": "dedicated aiohttp TCPConnector(limit=2) for profile endpoint session; request stream unchanged",
+      "evidence": "/start_profile connection acquisition timed out before server receipt while 256 data-plane connections were occupied",
+      "failed_run": "P03-C01-saturation",
+      "failed_stage": "primary-02-saturation",
+      "new_client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0",
+      "old_client_sha256": "a87d92efecd5a8765b51067800b6382f9b174a2ede65f8933fcc9f846ff03d84",
+      "repair": "profile-control-connector-isolation",
+      "retry": "one exact whole-wave retry; failed directories retained as .interrupted-*",
+      "schema": 1,
+      "tests": {
+        "py_compile": "PASS",
+        "ruff": "PASS",
+        "unittest": "13/13 PASS"
+      },
+      "timestamp": 1783837763.431071
+    },
+    {
+      "budget": {
+        "hard_cap": 16.0,
+        "projected_remaining_h20_hours": 6.3857560787267165,
+        "projected_total_h20_hours": 14.513826778670154,
+        "projected_total_with_15pct_remaining_contingency": 15.471690190479162,
+        "used_h20_hours": 8.128070699943438
+      },
+      "cause": "controller required failed_records==0 across excluded profile/recovery time although the registered hard gate is zero clean-window failures",
+      "change": "replace all_failures_zero with clean-window boundary validation plus failed-record accounting; excluded failures remain reported",
+      "evidence": {
+        "clean_completed": 5828,
+        "clean_failed": 0,
+        "error_kind": "ServerDisconnectedError",
+        "excluded_window_failures": 2,
+        "failure_completion_s": [
+          373.1505718010012,
+          373.9740898209857
+        ],
+        "run": "P01-C01-moderate"
+      },
+      "failed_stage": "primary-02-moderate",
+      "gpu_rerun": false,
+      "new_controller_sha256": "becfe00889274b51023016b1e7edb866d10e477249504f2032859a4d621f295f",
+      "old_controller_sha256": "167e48f98f307e16ee44b321068a82a13813b3bf9a4d76882f774f15fe85e595",
+      "repair": "clean-window-failure-scope",
+      "schema": 1,
+      "tests": {
+        "py_compile": "PASS",
+        "ruff": "PASS",
+        "unittest": "14/14 PASS"
+      },
+      "timestamp": 1783847092.6920242
+    },
+    {
+      "cause": "single preflight NVML sample observed 4 MiB on GPU0-3 with 0% utilization and no compute apps immediately after a verified cleanup",
+      "change": "bounded 60s preflight requiring three consecutive samples with zero memory, zero utilization, and no compute apps; no nonzero-memory tolerance",
+      "failed_stage": "primary-06-saturation",
+      "gpu_hours_total_unchanged": 11.920448313262728,
+      "gpu_launch_before_failure": false,
+      "new_common_controller_sha256": "95f7169a1771e385aab40fcaecd967dc5cff0c21ea67bdd139382774ba43f01f",
+      "old_common_controller_sha256": "15ad254298a38c4a9318468db89ab32707b6196bb38d5e2a007f2267529397a5",
+      "repair": "stable-zero-preflight",
+      "schema": 1,
+      "tests": {
+        "py_compile": "PASS",
+        "ruff": "PASS",
+        "unittest": "15/15 PASS"
+      },
+      "timestamp": 1783850898.1496143
+    },
+    {
+      "code_change": false,
+      "gpu_hours_used": 12.606695837948058,
+      "hard_cap": 16.0,
+      "projected_remaining_h20_hours": 2.1455133807990285,
+      "projected_total_with_15pct_contingency": 15.07403622586694,
+      "protocol_action": "single exact whole-wave infrastructure retry under identical 4-way placement; gate unchanged",
+      "reason": "first TP2 P10 inference-shape cold start/autotune after readiness yielded 15 successful warmup completions against required 32; clean window and artifacts otherwise valid",
+      "repair": "tp2-warmup-exact-retry",
+      "run": "P10-C00-TP2-saturation",
+      "schema": 1,
+      "stage": "primary-06-saturation",
+      "timestamp": 1783851710.3100822,
+      "warmup_required": 32,
+      "warmup_success": 15
+    },
+    {
+      "accepted_runs_preserved": 40,
+      "amendment": "A-P3-6",
+      "changed_fingerprint_keys": [
+        "controller_sha256"
+      ],
+      "created_at": 1783853258.3098204,
+      "failed_stage_preserved": "primary-06-saturation",
+      "new_controller_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f",
+      "old_controller_sha256": "becfe00889274b51023016b1e7edb866d10e477249504f2032859a4d621f295f",
+      "projected_remaining_h20_hours": 2.1,
+      "projected_total_h20_hours": 15.408482965071997,
+      "repair_id": "repair-008-ap36",
+      "retained_attempt_re_adjudicated": false,
+      "retained_attempt_reason": "21 completions but A-P3-6 normalized drift 2.3385345997286295 and bin step counts 13/9/44",
+      "schema": 1
+    }
+  ],
+  "schema": 1,
+  "stages": {
+    "burnin-01": {
+      "assignments": [
+        {
+          "cell": "P06-C00",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P06-C10",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P06-C01",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P06-C11",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": true,
+      "clients": {},
+      "completed_at": 1783834288.1238313,
+      "confirmation": false,
+      "gpu_hours": 0.30808498481909435,
+      "load_point": "saturation",
+      "profile": false,
+      "servers": {},
+      "started_at": 1783833886.4448946,
+      "status": "complete",
+      "validation_attempt1_failure": "RuntimeError(\"server config/backend failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/burnins/C01: {'triton_moe': True, 'chunked_mbt': False, 'tp_effective': True, 'drain_shutdown': True, 'mns_effective': True}\")"
+    },
+    "burnin-02": {
+      "assignments": [
+        {
+          "cell": "P06-C00-TP2",
+          "gpus": [
+            0,
+            1
+          ]
+        }
+      ],
+      "burnin": true,
+      "clients": {},
+      "completed_at": 1783834671.921921,
+      "confirmation": false,
+      "gpu_hours": 0.19903395679261948,
+      "load_point": "saturation",
+      "profile": false,
+      "servers": {},
+      "started_at": 1783834301.7813473,
+      "status": "complete"
+    },
+    "primary-01-moderate": {
+      "assignments": [
+        {
+          "cell": "P08-C00",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P03-C10",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P07-C00",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P10-C01",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {},
+      "completed_at": 1783836723.3639715,
+      "confirmation": false,
+      "gpu_hours": 0.5195140059126748,
+      "load_point": "moderate",
+      "profile": true,
+      "servers": {},
+      "started_at": 1783836150.2169476,
+      "status": "complete",
+      "validation_attempt1_failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P10-C01/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'all_failures_zero': True, 'manifest_no_wrap': True, 'warmup_completions': False, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]\")",
+      "warmup_feasibility": "P10 target=min(32,floor(0.445*60))=26; observed=27"
+    },
+    "primary-01-saturation": {
+      "assignments": [
+        {
+          "cell": "P08-C00",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P03-C10",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P07-C00",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P10-C01",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {},
+      "completed_at": 1783836130.8498049,
+      "confirmation": false,
+      "drain_readjudication": "P10/C01 288.619107924s <= 600s",
+      "gpu_hours": 0.8836704309119119,
+      "load_point": "saturation",
+      "missing_trace_files": 8,
+      "profile": true,
+      "servers": {},
+      "started_at": 1783834671.9563339,
+      "status": "complete",
+      "validation_attempt1_failure": "RuntimeError(\"client failures: {'P10-C01-saturation': 1}\")"
+    },
+    "primary-02-moderate": {
+      "assignments": [
+        {
+          "cell": "P01-C10",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P01-C01",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P10-C10",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P03-C01",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {},
+      "completed_at": 1783847092.7229948,
+      "confirmation": false,
+      "failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P01-C01/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'all_failures_zero': False, 'manifest_no_wrap': True, 'warmup_completions': True, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]\")",
+      "gpu_hours": 0.5151156750652525,
+      "load_point": "moderate",
+      "profile": true,
+      "readjudicated_without_gpu": true,
+      "servers": {},
+      "started_at": 1783838513.4472992,
+      "status": "complete",
+      "validation_attempt1_failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P01-C01/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'all_failures_zero': False, 'manifest_no_wrap': True, 'warmup_completions': True, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]\")"
+    },
+    "primary-02-saturation": {
+      "assignments": [
+        {
+          "cell": "P01-C10",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P01-C01",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P10-C10",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P03-C01",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {},
+      "completed_at": 1783838513.4027848,
+      "confirmation": false,
+      "gpu_hours": 0.7973502041896184,
+      "load_point": "saturation",
+      "profile": true,
+      "servers": {},
+      "started_at": 1783837773.6088765,
+      "status": "complete"
+    },
+    "primary-03-moderate": {
+      "assignments": [
+        {
+          "cell": "P09-C00",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P06-C01",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P10-C11",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P01-C00",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {},
+      "completed_at": 1783848369.3507695,
+      "confirmation": false,
+      "gpu_hours": 0.5205460974905226,
+      "load_point": "moderate",
+      "profile": true,
+      "servers": {},
+      "started_at": 1783847881.06146,
+      "status": "complete"
+    },
+    "primary-03-saturation": {
+      "assignments": [
+        {
+          "cell": "P09-C00",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P06-C01",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P10-C11",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P01-C00",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {},
+      "completed_at": 1783847881.0197804,
+      "confirmation": false,
+      "gpu_hours": 0.8273645816908942,
+      "load_point": "saturation",
+      "profile": true,
+      "servers": {},
+      "started_at": 1783847109.5683346,
+      "status": "complete"
+    },
+    "primary-04-moderate": {
+      "assignments": [
+        {
+          "cell": "P01-C11",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P10-C00",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P04-C00",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P03-C00",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {},
+      "completed_at": 1783849583.5228572,
+      "confirmation": false,
+      "gpu_hours": 0.5205216948853598,
+      "load_point": "moderate",
+      "profile": true,
+      "servers": {},
+      "started_at": 1783849096.4363403,
+      "status": "complete"
+    },
+    "primary-04-saturation": {
+      "assignments": [
+        {
+          "cell": "P01-C11",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P10-C00",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P04-C00",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P03-C00",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {},
+      "completed_at": 1783849096.3948114,
+      "confirmation": false,
+      "gpu_hours": 0.7872003297011058,
+      "load_point": "saturation",
+      "profile": true,
+      "servers": {},
+      "started_at": 1783848369.390857,
+      "status": "complete"
+    },
+    "primary-05-moderate": {
+      "assignments": [
+        {
+          "cell": "P06-C10",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P02-C00",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P06-C00",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P06-C11",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {},
+      "completed_at": 1783850641.6781306,
+      "confirmation": false,
+      "gpu_hours": 0.5138198028008143,
+      "load_point": "moderate",
+      "profile": true,
+      "servers": {},
+      "started_at": 1783850163.1211865,
+      "status": "complete"
+    },
+    "primary-05-saturation": {
+      "assignments": [
+        {
+          "cell": "P06-C10",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P02-C00",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P06-C00",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P06-C11",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {},
+      "completed_at": 1783850163.0472832,
+      "confirmation": false,
+      "gpu_hours": 0.6229251067505942,
+      "load_point": "saturation",
+      "profile": true,
+      "servers": {},
+      "started_at": 1783849583.5652869,
+      "status": "complete"
+    },
+    "primary-06-saturation": {
+      "assignments": [
+        {
+          "cell": "P11-C00",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P10-C00-TP2",
+          "gpus": [
+            1,
+            2
+          ]
+        },
+        {
+          "cell": "P03-C11",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {
+        "P03-C11-saturation": {
+          "pgid": 2440965,
+          "pid": 2440965
+        },
+        "P10-C00-TP2-saturation": {
+          "pgid": 2440964,
+          "pid": 2440964
+        },
+        "P11-C00-saturation": {
+          "pgid": 2440962,
+          "pid": 2440962
+        }
+      },
+      "confirmation": false,
+      "failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P10-C00-TP2/saturation: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'failed_records_accounted': True, 'manifest_no_wrap': True, 'warmup_completions': False, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]; warmup_completions=17; warmup_gate_branch=failed; warmup_stability={'passed': False, 'reason': 'A-P3-6 stabilization criterion not met', 'window_seconds': [45.0, 60.0], 'bin_seconds': 5.0, 'step_counts': [11, 10, 16], 'scheduled_tokens': [90112, 81920, 70313], 'scheduled_token_throughput': [18022.4, 16384.0, 14062.6], 'mean_scheduled_token_throughput': 16156.333333333334, 'slope_tokens_per_second_squared': -395.98000000000013, 'normalized_drift': 0.367639109533929, 'normalized_drift_limit': 0.1, 'step_indices_continuous': True}\")",
+      "gpu_hours": 0.7173924536837472,
+      "load_point": "saturation",
+      "profile": true,
+      "servers": {
+        "P03-C11-saturation": {
+          "gpus": [
+            3
+          ],
+          "pgid": 2438868,
+          "pid": 2438868
+        },
+        "P10-C00-TP2-saturation": {
+          "gpus": [
+            1,
+            2
+          ],
+          "pgid": 2438867,
+          "pid": 2438867
+        },
+        "P11-C00-saturation": {
+          "gpus": [
+            0
+          ],
+          "pgid": 2438866,
+          "pid": 2438866
+        }
+      },
+      "started_at": 1783853279.9695158,
+      "status": "failed"
+    }
+  },
+  "status": "failed",
+  "updated_at": 1783853946.066117
+}

runs/opprof-phase3/phase3/remote-evidence/P03-C10/result.json

@@ -0,0 +1,103 @@
+{
+  "admission_stop_s": 417.3262625900097,
+  "arrival": "steady",
+  "clean": {
+    "admitted": 559,
+    "completed": 559,
+    "completed_throughput_rps": 2.3291666666666666,
+    "duration_s": 240.0,
+    "end_s": 300.0,
+    "failed": 0,
+    "input_tokens": 3429894,
+    "offered_rps": 2.3291666666666666,
+    "output_tokens": 35776,
+    "start_s": 60.0
+  },
+  "clean_segment_seconds": 80.0,
+  "drain_seconds": 91.29178713698639,
+  "elapsed_seconds": 508.6180497269961,
+  "failed_records": 0,
+  "load_point": "saturation",
+  "manifest_admitted": 1094,
+  "manifest_exhausted": false,
+  "manifest_rows": 32768,
+  "manifest_sha256": "432cfbc26d36f105c179c83f3bb0f3b24b8b3f205788263b4171797f0a4d6fa1",
+  "manifest_wrapped": false,
+  "max_in_flight": 256,
+  "num_clean_segments": 3,
+  "profiles": [
+    {
+      "start_call_s": 300.0043962339987,
+      "start_return_s": 301.3819164079905,
+      "start_status": 200,
+      "stop_call_s": 321.8697677869932,
+      "stop_return_s": 326.31530767300865,
+      "stop_status": 200,
+      "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835059474707213.pt.trace.json.gz",
+      "trace_ready_s": 321.86976291501196,
+      "trace_sha256": "527ddb81b1d54a12a463508bc163d85a2c443d94c8a897959a39735e030b90a2",
+      "window": 1
+    },
+    {
+      "start_call_s": 356.32926020701416,
+      "start_return_s": 361.34978177401354,
+      "start_status": 200,
+      "stop_call_s": 381.577829301008,
+      "stop_return_s": 387.31277802900877,
+      "stop_status": 200,
+      "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835119198018909.pt.trace.json.gz",
+      "trace_ready_s": 381.57782580400817,
+      "trace_sha256": "0882d9705f1208580d5e693f94ba64d89ab12ef0aae2731a82d460a46a3e4cb1",
+      "window": 2
+    }
+  ],
+  "rate_fraction": null,
+  "records": 1094,
+  "request_rate": "inf",
+  "schema": 1,
+  "segments": [
+    {
+      "admitted": 191,
+      "completed": 191,
+      "completed_throughput_rps": 2.3875,
+      "duration_s": 80.0,
+      "end_s": 140.0,
+      "failed": 0,
+      "input_tokens": 1154628,
+      "name": "A",
+      "offered_rps": 2.3875,
+      "output_tokens": 12224,
+      "start_s": 60.0
+    },
+    {
+      "admitted": 184,
+      "completed": 184,
+      "completed_throughput_rps": 2.3,
+      "duration_s": 80.0,
+      "end_s": 220.0,
+      "failed": 0,
+      "input_tokens": 1135217,
+      "name": "B",
+      "offered_rps": 2.3,
+      "output_tokens": 11776,
+      "start_s": 140.0
+    },
+    {
+      "admitted": 184,
+      "completed": 184,
+      "completed_throughput_rps": 2.3,
+      "duration_s": 80.0,
+      "end_s": 300.0,
+      "failed": 0,
+      "input_tokens": 1140049,
+      "name": "C",
+      "offered_rps": 2.3,
+      "output_tokens": 11776,
+      "start_s": 220.0
+    }
+  ],
+  "successful_records": 1094,
+  "t0_mono_ns": 166323993366724,
+  "t0_wall_ns": 1783834751119796663,
+  "warmup_seconds": 60.0
+}

runs/opprof-phase3/phase3/remote-evidence/P03-C10/sanity.json

@@ -0,0 +1,65 @@
+{
+  "invariants": {
+    "clean_duration_exact": true,
+    "clean_failures_zero": true,
+    "concurrency_bounded": true,
+    "drain_within_timeout": true,
+    "manifest_no_wrap": true,
+    "manifest_not_exhausted": true,
+    "output_tokens_exact": true,
+    "profile_count_exact": true,
+    "profile_status_ok": true,
+    "segment_count_exact": true
+  },
+  "numeric": {
+    "actual_output_tokens": {
+      "distinct_n": 1,
+      "finite_n": 1094,
+      "max": 64.0,
+      "min": 64.0,
+      "missing_n": 0,
+      "n": 1094
+    },
+    "admitted_s": {
+      "distinct_n": 1094,
+      "finite_n": 1094,
+      "max": 416.7779763180006,
+      "min": 0.0008771540015004575,
+      "missing_n": 0,
+      "n": 1094
+    },
+    "completed_s": {
+      "distinct_n": 1094,
+      "finite_n": 1094,
+      "max": 508.61568894199445,
+      "min": 22.18080371801625,
+      "missing_n": 0,
+      "n": 1094
+    },
+    "input_tokens": {
+      "distinct_n": 964,
+      "finite_n": 1094,
+      "max": 8190.0,
+      "min": 4097.0,
+      "missing_n": 0,
+      "n": 1094
+    },
+    "requested_output_tokens": {
+      "distinct_n": 1,
+      "finite_n": 1094,
+      "max": 64.0,
+      "min": 64.0,
+      "missing_n": 0,
+      "n": 1094
+    },
+    "scheduled_s": {
+      "distinct_n": 1094,
+      "finite_n": 1094,
+      "max": 416.7779758319957,
+      "min": 0.0008756940078455955,
+      "missing_n": 0,
+      "n": 1094
+    }
+  },
+  "schema": 1
+}

runs/opprof-phase3/phase3/remote-evidence/P07-C00/result.json

@@ -0,0 +1,103 @@
+{
+  "admission_stop_s": 417.66287675101194,
+  "arrival": "burst:8",
+  "clean": {
+    "admitted": 1226,
+    "completed": 1226,
+    "completed_throughput_rps": 5.108333333333333,
+    "duration_s": 240.0,
+    "end_s": 300.0,
+    "failed": 0,
+    "input_tokens": 1569280,
+    "offered_rps": 5.108333333333333,
+    "output_tokens": 627712,
+    "start_s": 60.0
+  },
+  "clean_segment_seconds": 80.0,
+  "drain_seconds": 26.098238860984566,
+  "elapsed_seconds": 443.7611156119965,
+  "failed_records": 0,
+  "load_point": "saturation",
+  "manifest_admitted": 2078,
+  "manifest_exhausted": false,
+  "manifest_rows": 32768,
+  "manifest_sha256": "74df66e21a705cd583493199a875e226e93d2da7cfede9bca81dfd9bcb8c9cc6",
+  "manifest_wrapped": false,
+  "max_in_flight": 256,
+  "num_clean_segments": 3,
+  "profiles": [
+    {
+      "start_call_s": 300.0039349270228,
+      "start_return_s": 300.9924320260179,
+      "start_status": 200,
+      "stop_call_s": 323.90002606701455,
+      "stop_return_s": 328.078528626007,
+      "stop_status": 200,
+      "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835058771060551.pt.trace.json.gz",
+      "trace_ready_s": 323.9000201699964,
+      "trace_sha256": "2d12038b03c6f9c28d1a488abd880ff2763e1f3c0ab502cfdd2507b649de2af7",
+      "window": 1
+    },
+    {
+      "start_call_s": 358.0940292410087,
+      "start_return_s": 358.8905565890018,
+      "start_status": 200,
+      "stop_call_s": 382.04741394601297,
+      "stop_return_s": 387.64796055300394,
+      "stop_status": 200,
+      "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835116964699401.pt.trace.json.gz",
+      "trace_ready_s": 382.04741024502437,
+      "trace_sha256": "2184d4f83ea22c0e5455dd3914922548ce8d3d5eba34f89562df69326e3832b5",
+      "window": 2
+    }
+  ],
+  "rate_fraction": null,
+  "records": 2078,
+  "request_rate": "inf",
+  "schema": 1,
+  "segments": [
+    {
+      "admitted": 394,
+      "completed": 394,
+      "completed_throughput_rps": 4.925,
+      "duration_s": 80.0,
+      "end_s": 140.0,
+      "failed": 0,
+      "input_tokens": 504320,
+      "name": "A",
+      "offered_rps": 4.925,
+      "output_tokens": 201728,
+      "start_s": 60.0
+    },
+    {
+      "admitted": 397,
+      "completed": 397,
+      "completed_throughput_rps": 4.9625,
+      "duration_s": 80.0,
+      "end_s": 220.0,
+      "failed": 0,
+      "input_tokens": 508160,
+      "name": "B",
+      "offered_rps": 4.9625,
+      "output_tokens": 203264,
+      "start_s": 140.0
+    },
+    {
+      "admitted": 435,
+      "completed": 435,
+      "completed_throughput_rps": 5.4375,
+      "duration_s": 80.0,
+      "end_s": 300.0,
+      "failed": 0,
+      "input_tokens": 556800,
+      "name": "C",
+      "offered_rps": 5.4375,
+      "output_tokens": 222720,
+      "start_s": 220.0
+    }
+  ],
+  "successful_records": 2078,
+  "t0_mono_ns": 166323999932716,
+  "t0_wall_ns": 1783834751126361482,
+  "warmup_seconds": 60.0
+}

runs/opprof-phase3/phase3/remote-evidence/P07-C00/sanity.json

@@ -0,0 +1,65 @@
+{
+  "invariants": {
+    "clean_duration_exact": true,
+    "clean_failures_zero": true,
+    "concurrency_bounded": true,
+    "drain_within_timeout": true,
+    "manifest_no_wrap": true,
+    "manifest_not_exhausted": true,
+    "output_tokens_exact": true,
+    "profile_count_exact": true,
+    "profile_status_ok": true,
+    "segment_count_exact": true
+  },
+  "numeric": {
+    "actual_output_tokens": {
+      "distinct_n": 1,
+      "finite_n": 2078,
+      "max": 512.0,
+      "min": 512.0,
+      "missing_n": 0,
+      "n": 2078
+    },
+    "admitted_s": {
+      "distinct_n": 2078,
+      "finite_n": 2078,
+      "max": 417.40899016702315,
+      "min": 0.0008179180149454623,
+      "missing_n": 0,
+      "n": 2078
+    },
+    "completed_s": {
+      "distinct_n": 2078,
+      "finite_n": 2078,
+      "max": 443.75592052700813,
+      "min": 35.15151289102505,
+      "missing_n": 0,
+      "n": 2078
+    },
+    "input_tokens": {
+      "distinct_n": 1,
+      "finite_n": 2078,
+      "max": 1280.0,
+      "min": 1280.0,
+      "missing_n": 0,
+      "n": 2078
+    },
+    "requested_output_tokens": {
+      "distinct_n": 1,
+      "finite_n": 2078,
+      "max": 512.0,
+      "min": 512.0,
+      "missing_n": 0,
+      "n": 2078
+    },
+    "scheduled_s": {
+      "distinct_n": 2078,
+      "finite_n": 2078,
+      "max": 417.4089898300008,
+      "min": 0.0008165129984263331,
+      "missing_n": 0,
+      "n": 2078
+    }
+  },
+  "schema": 1
+}

runs/opprof-phase3/phase3/remote-evidence/P08-C00/result.json

@@ -0,0 +1,103 @@
+{
+  "admission_stop_s": 493.2576180040196,
+  "arrival": "burst:8",
+  "clean": {
+    "admitted": 2233,
+    "completed": 2233,
+    "completed_throughput_rps": 9.304166666666667,
+    "duration_s": 240.0,
+    "end_s": 300.0,
+    "failed": 0,
+    "input_tokens": 2858240,
+    "offered_rps": 9.304166666666667,
+    "output_tokens": 1143296,
+    "start_s": 60.0
+  },
+  "clean_segment_seconds": 80.0,
+  "drain_seconds": 14.923095890000695,
+  "elapsed_seconds": 508.1807138940203,
+  "failed_records": 0,
+  "load_point": "saturation",
+  "manifest_admitted": 3941,
+  "manifest_exhausted": false,
+  "manifest_rows": 32768,
+  "manifest_sha256": "0c4f835aae099265c3eb596d06c6c2fa7070dd280558eb289c602e8c3434dfe9",
+  "manifest_wrapped": false,
+  "max_in_flight": 256,
+  "num_clean_segments": 3,
+  "profiles": [
+    {
+      "start_call_s": 300.0054247789958,
+      "start_return_s": 323.45228312400286,
+      "start_status": 200,
+      "stop_call_s": 354.6066670610162,
+      "stop_return_s": 381.28106322701205,
+      "stop_status": 200,
+      "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835082061801278.pt.trace.json.gz",
+      "trace_ready_s": 354.6066622030048,
+      "trace_sha256": "53d36238e82e5d031d8ed7841094c56d715b7cd5009368b64e380bf89a0f0dec",
+      "window": 1
+    },
+    {
+      "start_call_s": 411.30330635100836,
+      "start_return_s": 411.4401313569979,
+      "start_status": 200,
+      "stop_call_s": 436.0742275560042,
+      "stop_return_s": 463.23921855099616,
+      "stop_status": 200,
+      "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835169024306574.pt.trace.json.gz",
+      "trace_ready_s": 436.0742245099973,
+      "trace_sha256": "e088bd800a4e43446effbbcd4fce7d9ba1defaceb378a6e0db04d64780e1d09a",
+      "window": 2
+    }
+  ],
+  "rate_fraction": null,
+  "records": 3941,
+  "request_rate": "inf",
+  "schema": 1,
+  "segments": [
+    {
+      "admitted": 697,
+      "completed": 697,
+      "completed_throughput_rps": 8.7125,
+      "duration_s": 80.0,
+      "end_s": 140.0,
+      "failed": 0,
+      "input_tokens": 892160,
+      "name": "A",
+      "offered_rps": 8.7125,
+      "output_tokens": 356864,
+      "start_s": 60.0
+    },
+    {
+      "admitted": 768,
+      "completed": 768,
+      "completed_throughput_rps": 9.6,
+      "duration_s": 80.0,
+      "end_s": 220.0,
+      "failed": 0,
+      "input_tokens": 983040,
+      "name": "B",
+      "offered_rps": 9.6,
+      "output_tokens": 393216,
+      "start_s": 140.0
+    },
+    {
+      "admitted": 768,
+      "completed": 768,
+      "completed_throughput_rps": 9.6,
+      "duration_s": 80.0,
+      "end_s": 300.0,
+      "failed": 0,
+      "input_tokens": 983040,
+      "name": "C",
+      "offered_rps": 9.6,
+      "output_tokens": 393216,
+      "start_s": 220.0
+    }
+  ],
+  "successful_records": 3941,
+  "t0_mono_ns": 166324009670273,
+  "t0_wall_ns": 1783834751136099481,
+  "warmup_seconds": 60.0
+}

runs/opprof-phase3/phase3/remote-evidence/P08-C00/sanity.json

@@ -0,0 +1,65 @@
+{
+  "invariants": {
+    "clean_duration_exact": true,
+    "clean_failures_zero": true,
+    "concurrency_bounded": true,
+    "drain_within_timeout": true,
+    "manifest_no_wrap": true,
+    "manifest_not_exhausted": true,
+    "output_tokens_exact": true,
+    "profile_count_exact": true,
+    "profile_status_ok": true,
+    "segment_count_exact": true
+  },
+  "numeric": {
+    "actual_output_tokens": {
+      "distinct_n": 1,
+      "finite_n": 3941,
+      "max": 512.0,
+      "min": 512.0,
+      "missing_n": 0,
+      "n": 3941
+    },
+    "admitted_s": {
+      "distinct_n": 3941,
+      "finite_n": 3941,
+      "max": 493.25117621201207,
+      "min": 0.0009094980196096003,
+      "missing_n": 0,
+      "n": 3941
+    },
+    "completed_s": {
+      "distinct_n": 3941,
+      "finite_n": 3941,
+      "max": 508.1755032700021,
+      "min": 30.030000179016497,
+      "missing_n": 0,
+      "n": 3941
+    },
+    "input_tokens": {
+      "distinct_n": 1,
+      "finite_n": 3941,
+      "max": 1280.0,
+      "min": 1280.0,
+      "missing_n": 0,
+      "n": 3941
+    },
+    "requested_output_tokens": {
+      "distinct_n": 1,
+      "finite_n": 3941,
+      "max": 512.0,
+      "min": 512.0,
+      "missing_n": 0,
+      "n": 3941
+    },
+    "scheduled_s": {
+      "distinct_n": 3941,
+      "finite_n": 3941,
+      "max": 493.25117589501315,
+      "min": 0.000907983019715175,
+      "missing_n": 0,
+      "n": 3941
+    }
+  },
+  "schema": 1
+}

runs/opprof-phase3/phase3/remote-evidence/P10-C01/result.json

@@ -0,0 +1,103 @@
+{
+  "admission_stop_s": 420.6437278209778,
+  "arrival": "steady",
+  "clean": {
+    "admitted": 178,
+    "completed": 178,
+    "completed_throughput_rps": 0.7416666666666667,
+    "duration_s": 240.0,
+    "end_s": 300.0,
+    "failed": 0,
+    "input_tokens": 1755158,
+    "offered_rps": 0.7416666666666667,
+    "output_tokens": 40658,
+    "start_s": 60.0
+  },
+  "clean_segment_seconds": 80.0,
+  "drain_seconds": 288.6191079240234,
+  "elapsed_seconds": 709.2628357450012,
+  "failed_records": 0,
+  "load_point": "saturation",
+  "manifest_admitted": 558,
+  "manifest_exhausted": false,
+  "manifest_rows": 4011,
+  "manifest_sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33",
+  "manifest_wrapped": false,
+  "max_in_flight": 256,
+  "num_clean_segments": 3,
+  "profiles": [
+    {
+      "start_call_s": 300.0035294489935,
+      "start_return_s": 310.3520467719936,
+      "start_status": 200,
+      "stop_call_s": 325.0655262139917,
+      "stop_return_s": 338.54296391498065,
+      "stop_status": 200,
+      "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835066783986388.pt.trace.json.gz",
+      "trace_ready_s": 325.0655210709956,
+      "trace_sha256": "d457af416fa91f44e617c7259167fca467bea8466a3e4c4b916f811fb5899c73",
+      "window": 1
+    },
+    {
+      "start_call_s": 368.55468072599615,
+      "start_return_s": 372.31818850699347,
+      "start_status": 200,
+      "stop_call_s": 386.0271748309897,
+      "stop_return_s": 390.6334540609969,
+      "stop_status": 200,
+      "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835128341345321.pt.trace.json.gz",
+      "trace_ready_s": 386.02717122397735,
+      "trace_sha256": "41e8b34a0464cfd92d16b21c6242113b23594081697f5d1e88cdc3647542a96a",
+      "window": 2
+    }
+  ],
+  "rate_fraction": null,
+  "records": 558,
+  "request_rate": "inf",
+  "schema": 1,
+  "segments": [
+    {
+      "admitted": 48,
+      "completed": 48,
+      "completed_throughput_rps": 0.6,
+      "duration_s": 80.0,
+      "end_s": 140.0,
+      "failed": 0,
+      "input_tokens": 548329,
+      "name": "A",
+      "offered_rps": 0.6,
+      "output_tokens": 11027,
+      "start_s": 60.0
+    },
+    {
+      "admitted": 70,
+      "completed": 70,
+      "completed_throughput_rps": 0.875,
+      "duration_s": 80.0,
+      "end_s": 220.0,
+      "failed": 0,
+      "input_tokens": 574251,
+      "name": "B",
+      "offered_rps": 0.875,
+      "output_tokens": 16298,
+      "start_s": 140.0
+    },
+    {
+      "admitted": 60,
+      "completed": 60,
+      "completed_throughput_rps": 0.75,
+      "duration_s": 80.0,
+      "end_s": 300.0,
+      "failed": 0,
+      "input_tokens": 632578,
+      "name": "C",
+      "offered_rps": 0.75,
+      "output_tokens": 13333,
+      "start_s": 220.0
+    }
+  ],
+  "successful_records": 558,
+  "t0_mono_ns": 166325149689000,
+  "t0_wall_ns": 1783834752276118250,
+  "warmup_seconds": 60.0
+}

runs/opprof-phase3/phase3/remote-evidence/P10-C01/sanity.json

@@ -0,0 +1,65 @@
+{
+  "invariants": {
+    "clean_duration_exact": true,
+    "clean_failures_zero": true,
+    "concurrency_bounded": true,
+    "drain_within_timeout": false,
+    "manifest_no_wrap": true,
+    "manifest_not_exhausted": true,
+    "output_tokens_exact": true,
+    "profile_count_exact": true,
+    "profile_status_ok": true,
+    "segment_count_exact": true
+  },
+  "numeric": {
+    "actual_output_tokens": {
+      "distinct_n": 92,
+      "finite_n": 558,
+      "max": 256.0,
+      "min": 6.0,
+      "missing_n": 0,
+      "n": 558
+    },
+    "admitted_s": {
+      "distinct_n": 558,
+      "finite_n": 558,
+      "max": 420.08466253298684,
+      "min": 0.000850101001560688,
+      "missing_n": 0,
+      "n": 558
+    },
+    "completed_s": {
+      "distinct_n": 558,
+      "finite_n": 558,
+      "max": 709.2613093179825,
+      "min": 14.001143716974184,
+      "missing_n": 0,
+      "n": 558
+    },
+    "input_tokens": {
+      "distinct_n": 535,
+      "finite_n": 558,
+      "max": 32527.0,
+      "min": 70.0,
+      "missing_n": 0,
+      "n": 558
+    },
+    "requested_output_tokens": {
+      "distinct_n": 92,
+      "finite_n": 558,
+      "max": 256.0,
+      "min": 6.0,
+      "missing_n": 0,
+      "n": 558
+    },
+    "scheduled_s": {
+      "distinct_n": 558,
+      "finite_n": 558,
+      "max": 420.08466206499725,
+      "min": 0.0008485929865855724,
+      "missing_n": 0,
+      "n": 558
+    }
+  },
+  "schema": 1
+}

runs/opprof-phase3/phase3/remote-evidence/controller-state.json

@@ -0,0 +1,258 @@
+{
+  "completed_burnins": 5,
+  "completed_measured_runs": 0,
+  "controller_pid": 2187729,
+  "created_at": 1783833885.960389,
+  "fingerprint": {
+    "cells": [
+      "P08-C00",
+      "P03-C10",
+      "P07-C00",
+      "P10-C01",
+      "P01-C10",
+      "P01-C01",
+      "P10-C10",
+      "P03-C01",
+      "P09-C00",
+      "P06-C01",
+      "P10-C11",
+      "P01-C00",
+      "P01-C11",
+      "P10-C00",
+      "P04-C00",
+      "P03-C00",
+      "P06-C10",
+      "P02-C00",
+      "P06-C00",
+      "P06-C11",
+      "P11-C00",
+      "P10-C00-TP2",
+      "P03-C11",
+      "P05-C00"
+    ],
+    "client_sha256": "a87d92efecd5a8765b51067800b6382f9b174a2ede65f8933fcc9f846ff03d84",
+    "common_controller_sha256": "15ad254298a38c4a9318468db89ab32707b6196bb38d5e2a007f2267529397a5",
+    "controller_sha256": "c2d3232fb99c66cea55e30e6cb1aa6a84d3a816434ed8155a309f41df2837f73",
+    "cpu_map": {
+      "0": "0-19",
+      "1": "20-39",
+      "2": "40-59",
+      "3": "60-79",
+      "4": "80-99",
+      "5": "100-119",
+      "6": "120-139",
+      "7": "140-159"
+    },
+    "manifests": {
+      "P01": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl",
+        "rows": 32768,
+        "sha256": "13ffb226c83373f54c4a7afea6c78cb7cd29720f1858d56728826fc1367b31a4"
+      },
+      "P02": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P02.jsonl",
+        "rows": 32768,
+        "sha256": "0138ada3fccc98298daee66c26bd1952c987cb42ed8d5341d66b698a597417f9"
+      },
+      "P03": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P03.jsonl",
+        "rows": 32768,
+        "sha256": "432cfbc26d36f105c179c83f3bb0f3b24b8b3f205788263b4171797f0a4d6fa1"
+      },
+      "P04": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P04.jsonl",
+        "rows": 32768,
+        "sha256": "caf8d0941b093956a81e1413adc4a4ea9d92460f1b2ed0f6a9b118d0119d6247"
+      },
+      "P05": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P05.jsonl",
+        "rows": 32768,
+        "sha256": "192e213109f8cb429b99d9eb0f227bb4390fc03f63b02d5456569369bff5a3d7"
+      },
+      "P06": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P06.jsonl",
+        "rows": 32768,
+        "sha256": "65954bc6e47de9e7be07b8975f97f0bc4639979ef6d33e8c664557ded34b9f96"
+      },
+      "P07": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P07.jsonl",
+        "rows": 32768,
+        "sha256": "74df66e21a705cd583493199a875e226e93d2da7cfede9bca81dfd9bcb8c9cc6"
+      },
+      "P08": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P08.jsonl",
+        "rows": 32768,
+        "sha256": "0c4f835aae099265c3eb596d06c6c2fa7070dd280558eb289c602e8c3434dfe9"
+      },
+      "P09": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl",
+        "rows": 32768,
+        "sha256": "7af92ee3c27dc7d2cf895d6ff3a6e737ec4b6da13d6841ca59e1166f28a0ae1e"
+      },
+      "P10": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl",
+        "rows": 4011,
+        "sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33"
+      },
+      "P11": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P11.jsonl",
+        "rows": 32768,
+        "sha256": "7d196df38963528ff181cf72ce39c8ad913c8f61d40b1425410d3c6c30b6be18"
+      }
+    },
+    "model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B",
+    "source_commit": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05",
+    "source_tree": "a3d536b287a724e60abbec68b45eed7e088a15d1"
+  },
+  "gpu_hours_this_stage": 0.8836704309119119,
+  "gpu_hours_total": 5.473521912336349,
+  "schema": 1,
+  "stages": {
+    "burnin-01": {
+      "assignments": [
+        {
+          "cell": "P06-C00",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P06-C10",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P06-C01",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P06-C11",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": true,
+      "clients": {},
+      "completed_at": 1783834288.1238313,
+      "confirmation": false,
+      "gpu_hours": 0.30808498481909435,
+      "load_point": "saturation",
+      "profile": false,
+      "servers": {},
+      "started_at": 1783833886.4448946,
+      "status": "complete",
+      "validation_attempt1_failure": "RuntimeError(\"server config/backend failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/burnins/C01: {'triton_moe': True, 'chunked_mbt': False, 'tp_effective': True, 'drain_shutdown': True, 'mns_effective': True}\")"
+    },
+    "burnin-02": {
+      "assignments": [
+        {
+          "cell": "P06-C00-TP2",
+          "gpus": [
+            0,
+            1
+          ]
+        }
+      ],
+      "burnin": true,
+      "clients": {},
+      "completed_at": 1783834671.921921,
+      "confirmation": false,
+      "gpu_hours": 0.19903395679261948,
+      "load_point": "saturation",
+      "profile": false,
+      "servers": {},
+      "started_at": 1783834301.7813473,
+      "status": "complete"
+    },
+    "primary-01-saturation": {
+      "assignments": [
+        {
+          "cell": "P08-C00",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "P03-C10",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "P07-C00",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "P10-C01",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {
+        "P03-C10-saturation": {
+          "pgid": 2195599,
+          "pid": 2195599
+        },
+        "P07-C00-saturation": {
+          "pgid": 2195600,
+          "pid": 2195600
+        },
+        "P08-C00-saturation": {
+          "pgid": 2195597,
+          "pid": 2195597
+        },
+        "P10-C01-saturation": {
+          "pgid": 2195601,
+          "pid": 2195601
+        }
+      },
+      "confirmation": false,
+      "failure": "RuntimeError(\"client failures: {'P10-C01-saturation': 1}\")",
+      "gpu_hours": 0.8836704309119119,
+      "load_point": "saturation",
+      "profile": true,
+      "servers": {
+        "P03-C10-saturation": {
+          "gpus": [
+            1
+          ],
+          "pgid": 2193680,
+          "pid": 2193680
+        },
+        "P07-C00-saturation": {
+          "gpus": [
+            2
+          ],
+          "pgid": 2193681,
+          "pid": 2193681
+        },
+        "P08-C00-saturation": {
+          "gpus": [
+            0
+          ],
+          "pgid": 2193679,
+          "pid": 2193679
+        },
+        "P10-C01-saturation": {
+          "gpus": [
+            3
+          ],
+          "pgid": 2193682,
+          "pid": 2193682
+        }
+      },
+      "started_at": 1783834671.9563339,
+      "status": "failed"
+    }
+  },
+  "status": "failed",
+  "updated_at": 1783835479.1651883
+}

runs/opprof-phase3/phase3/remote-evidence/repair-001-explicit-mbt-log.json

@@ -0,0 +1,16 @@
+{
+  "at": 1783834288.13871,
+  "finding": "Explicit MBT=2048 is logged in non-default args; scheduler.py emits the separate Chunked-prefill line only for default MBT.",
+  "gpu_rerun": false,
+  "kind": "validator-only",
+  "new_controller_sha256": "c2d3232fb99c66cea55e30e6cb1aa6a84d3a816434ed8155a309f41df2837f73",
+  "old_controller_sha256": "74859614a8da3341bf827bb75b667db7d219f059d1d29c1182b9a77c2ee59e92",
+  "protocol_or_workload_changed": false,
+  "revalidated_runs": [
+    "P06-C00-burnin",
+    "P06-C10-burnin",
+    "P06-C01-burnin",
+    "P06-C11-burnin"
+  ],
+  "schema": 1
+}

runs/opprof-phase3/phase3/remote-evidence/sidecar-verification-state.json

@@ -0,0 +1,355 @@
+{
+  "gpu_seconds": 427.43714332580566,
+  "results": {
+    "graceful": {
+      "accounting": {
+        "bytes": 1462585,
+        "checkpoint_age_seconds": -0.382308841,
+        "counters": {
+          "dropped_records": 0,
+          "encoded_records": 1598,
+          "written_records": 1598
+        },
+        "footer_count": 1,
+        "invariants": {
+          "all_schema_1": true,
+          "encoded_balanced": true,
+          "final_sidecar": true,
+          "footer_sidecar_agree": true,
+          "last_step_matches": true,
+          "one_footer_last": true,
+          "steps_contiguous": true,
+          "written_matches_records": true,
+          "zero_drops": true
+        },
+        "last_step_index": 1597,
+        "records": 1598,
+        "sidecar": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/graceful/opprof/opprof-v1-dp0-pid2163417-1783832631507578426.jsonl.footer.json",
+        "stream": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/graceful/opprof/opprof-v1-dp0-pid2163417-1783832631507578426.jsonl"
+      },
+      "clean": {
+        "admitted": 5202,
+        "completed": 5202,
+        "completed_throughput_rps": 43.35,
+        "duration_s": 120.0,
+        "end_s": 140.0,
+        "failed": 0,
+        "input_tokens": 1662118,
+        "offered_rps": 43.35,
+        "output_tokens": 332928,
+        "start_s": 20.0
+      },
+      "failed_records": 0,
+      "gpu_seconds": 219.92383646965027,
+      "gpu_zero_samples": [
+        [
+          {
+            "index": 0,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 1,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 2,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 3,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 4,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 5,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 6,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 7,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          }
+        ],
+        [
+          {
+            "index": 0,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 1,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 2,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 3,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 4,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 5,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 6,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 7,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          }
+        ],
+        [
+          {
+            "index": 0,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 1,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 2,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 3,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 4,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 5,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 6,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 7,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          }
+        ]
+      ],
+      "mode": "graceful",
+      "schema": 1,
+      "server_returncode": 0,
+      "status": "pass",
+      "termination_wall_ns": 1783832775735239240
+    },
+    "hard-kill": {
+      "accounting": {
+        "bytes": 1460860,
+        "checkpoint_age_seconds": 0.02436319,
+        "counters": {
+          "dropped_records": 0,
+          "encoded_records": 1596,
+          "written_records": 1596
+        },
+        "footer_count": 0,
+        "invariants": {
+          "all_schema_1": true,
+          "checkpoint_sidecar": true,
+          "checkpoint_within_bound": true,
+          "encoded_balanced": true,
+          "last_step_matches": true,
+          "no_in_stream_footer": true,
+          "steps_contiguous": true,
+          "written_matches_records": true,
+          "zero_drops": true
+        },
+        "last_step_index": 1595,
+        "records": 1596,
+        "sidecar": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/hard-kill/opprof/opprof-v1-dp0-pid2166360-1783832841962667333.jsonl.footer.json",
+        "stream": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/hard-kill/opprof/opprof-v1-dp0-pid2166360-1783832841962667333.jsonl"
+      },
+      "clean": {
+        "admitted": 5179,
+        "completed": 5179,
+        "completed_throughput_rps": 43.15833333333333,
+        "duration_s": 120.0,
+        "end_s": 140.0,
+        "failed": 0,
+        "input_tokens": 1655384,
+        "offered_rps": 43.15833333333333,
+        "output_tokens": 331456,
+        "start_s": 20.0
+      },
+      "failed_records": 0,
+      "gpu_seconds": 207.5133068561554,
+      "gpu_zero_samples": [
+        [
+          {
+            "index": 0,
+            "memory_mib": 0,
+            "utilization_pct": 1
+          },
+          {
+            "index": 1,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 2,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 3,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 4,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 5,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 6,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 7,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          }
+        ],
+        [
+          {
+            "index": 0,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 1,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 2,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 3,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 4,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 5,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 6,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 7,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          }
+        ],
+        [
+          {
+            "index": 0,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 1,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 2,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 3,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 4,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 5,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 6,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          },
+          {
+            "index": 7,
+            "memory_mib": 0,
+            "utilization_pct": 0
+          }
+        ]
+      ],
+      "mode": "hard-kill",
+      "schema": 1,
+      "server_returncode": -9,
+      "status": "pass",
+      "termination_wall_ns": 1783832986410202706
+    }
+  },
+  "schema": 1,
+  "status": "complete"
+}

runs/opprof-phase3/phase3/repair-008-ap36.json

@@ -0,0 +1,17 @@
+{
+  "accepted_runs_preserved": 40,
+  "amendment": "A-P3-6",
+  "changed_fingerprint_keys": [
+    "controller_sha256"
+  ],
+  "created_at": 1783853258.3098204,
+  "failed_stage_preserved": "primary-06-saturation",
+  "new_controller_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f",
+  "old_controller_sha256": "becfe00889274b51023016b1e7edb866d10e477249504f2032859a4d621f295f",
+  "projected_remaining_h20_hours": 2.1,
+  "projected_total_h20_hours": 15.408482965071997,
+  "repair_id": "repair-008-ap36",
+  "retained_attempt_re_adjudicated": false,
+  "retained_attempt_reason": "21 completions but A-P3-6 normalized drift 2.3385345997286295 and bin step counts 13/9/44",
+  "schema": 1
+}

runs/opprof-phase3/phase4/capture-p09/controller-state.json

@@ -0,0 +1,469 @@
+{
+  "arms": {
+    "OFF": {
+      "client_pid": 2487398,
+      "server_pid": 2486295,
+      "started_at": 1783856664.5207698,
+      "status": "complete",
+      "summary": {
+        "arm": "OFF",
+        "bucket_tokens": 237911,
+        "capture_sizes": "default",
+        "clean_completed": 1184,
+        "clean_completed_throughput_rps": 4.933333333333334,
+        "clean_failed": 0,
+        "clean_offered_rps": 4.920833333333333,
+        "drain_seconds": 0.7853007119847462,
+        "e2e_latency_mean_s": 1.681152764688729,
+        "e2e_latency_p95_s": 3.8763178953449815,
+        "gpu_hours": 0.13751606033907995,
+        "graph_hit_steps": 12252,
+        "graph_miss_rate": 0.045199501246882795,
+        "layer1_invariants": {
+          "cudagraph_identity": true,
+          "footer_balanced": true,
+          "footer_written_matches": true,
+          "schema_1": true,
+          "sidecar_agrees": true,
+          "sidecar_final": true,
+          "steps_unique_contiguous": true,
+          "token_composition": true,
+          "zero_drops": true
+        },
+        "layer1_records": 16491,
+        "model_step_duration_ms": 479367.749483,
+        "model_steps": 12832,
+        "padding_fraction": 0.08545632610514016,
+        "padding_tokens": 20331,
+        "schema": 1,
+        "token_efficiency_per_ms": 4.5540506267984115,
+        "useful_tokens": 2183065
+      }
+    },
+    "ON": {
+      "client_pid": 2479800,
+      "server_pid": 2477652,
+      "started_at": 1783856086.549906,
+      "status": "complete",
+      "summary": {
+        "arm": "ON",
+        "bucket_tokens": 227714,
+        "capture_sizes": [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6,
+          7,
+          8,
+          9,
+          16,
+          24,
+          32,
+          40,
+          48,
+          56,
+          64,
+          72,
+          80,
+          88,
+          96,
+          104,
+          112,
+          120,
+          128,
+          136,
+          144,
+          152,
+          160,
+          168,
+          176,
+          184,
+          192,
+          200,
+          208,
+          216,
+          224,
+          232,
+          240,
+          248,
+          256,
+          272,
+          288,
+          304,
+          320,
+          336,
+          352,
+          368,
+          384,
+          400,
+          416,
+          432,
+          448,
+          464,
+          480,
+          496,
+          512
+        ],
+        "clean_completed": 1190,
+        "clean_completed_throughput_rps": 4.958333333333333,
+        "clean_failed": 0,
+        "clean_offered_rps": 4.920833333333333,
+        "drain_seconds": 0.7876331160077825,
+        "e2e_latency_mean_s": 1.612280680370388,
+        "e2e_latency_p95_s": 3.9930475192406423,
+        "gpu_hours": 0.15887338949574364,
+        "graph_hit_steps": 14253,
+        "graph_miss_rate": 0.0389724226282786,
+        "layer1_invariants": {
+          "cudagraph_identity": true,
+          "footer_balanced": true,
+          "footer_written_matches": true,
+          "schema_1": true,
+          "sidecar_agrees": true,
+          "sidecar_final": true,
+          "steps_unique_contiguous": true,
+          "token_composition": true,
+          "zero_drops": true
+        },
+        "layer1_records": 17579,
+        "model_step_duration_ms": 478655.543996,
+        "model_steps": 14831,
+        "padding_fraction": 0.035658764941988635,
+        "padding_tokens": 8120,
+        "schema": 1,
+        "token_efficiency_per_ms": 4.562222306607711,
+        "useful_tokens": 2183733
+      }
+    }
+  },
+  "controller_pid": 2477456,
+  "created_at": 1783856081.3481774,
+  "gpu_hours_increment": 0.2963894498348236,
+  "plan": {
+    "added_capture_sizes": [
+      3,
+      5,
+      6,
+      7,
+      9
+    ],
+    "clean_seconds": 240,
+    "gpu": 0,
+    "gpu_hour_limit": 16.0,
+    "load": "moderate",
+    "manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl",
+    "measured_padding_recovery_bound": 0.049776380388728225,
+    "on_capture_sizes": [
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      16,
+      24,
+      32,
+      40,
+      48,
+      56,
+      64,
+      72,
+      80,
+      88,
+      96,
+      104,
+      112,
+      120,
+      128,
+      136,
+      144,
+      152,
+      160,
+      168,
+      176,
+      184,
+      192,
+      200,
+      208,
+      216,
+      224,
+      232,
+      240,
+      248,
+      256,
+      272,
+      288,
+      304,
+      320,
+      336,
+      352,
+      368,
+      384,
+      400,
+      416,
+      432,
+      448,
+      464,
+      480,
+      496,
+      512
+    ],
+    "order": [
+      "ON",
+      "OFF"
+    ],
+    "pattern": "P09",
+    "primary_metric": "clean graph-hit padding_fraction",
+    "prior_gpu_hours": 14.025875418755744,
+    "profile": false,
+    "projected_increment_gpu_hours": 0.5,
+    "projected_total_gpu_hours": 14.525875418755744,
+    "rate_fraction": 0.6,
+    "saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json",
+    "schema": 1,
+    "secondary_metrics": [
+      "Layer-1 useful scheduled tokens/model-step millisecond",
+      "clean completed request throughput",
+      "clean request latency"
+    ],
+    "warmup_seconds": 60
+  },
+  "result": {
+    "delta": {
+      "completed_throughput_relative": 0.0050675675675675436,
+      "e2e_mean_latency_relative": -0.04096717785851722,
+      "e2e_p95_latency_relative": 0.03011353223527924,
+      "padding_fraction_points": -0.049797561163151524,
+      "padding_reduction_fraction": 0.5827252753866776,
+      "token_efficiency_relative": 0.0017943761453185214
+    },
+    "gpu_hours_increment": 0.2963894498348236,
+    "gpu_hours_total": 14.322264868590567,
+    "off": {
+      "arm": "OFF",
+      "bucket_tokens": 237911,
+      "capture_sizes": "default",
+      "clean_completed": 1184,
+      "clean_completed_throughput_rps": 4.933333333333334,
+      "clean_failed": 0,
+      "clean_offered_rps": 4.920833333333333,
+      "drain_seconds": 0.7853007119847462,
+      "e2e_latency_mean_s": 1.681152764688729,
+      "e2e_latency_p95_s": 3.8763178953449815,
+      "gpu_hours": 0.13751606033907995,
+      "graph_hit_steps": 12252,
+      "graph_miss_rate": 0.045199501246882795,
+      "layer1_invariants": {
+        "cudagraph_identity": true,
+        "footer_balanced": true,
+        "footer_written_matches": true,
+        "schema_1": true,
+        "sidecar_agrees": true,
+        "sidecar_final": true,
+        "steps_unique_contiguous": true,
+        "token_composition": true,
+        "zero_drops": true
+      },
+      "layer1_records": 16491,
+      "model_step_duration_ms": 479367.749483,
+      "model_steps": 12832,
+      "padding_fraction": 0.08545632610514016,
+      "padding_tokens": 20331,
+      "schema": 1,
+      "token_efficiency_per_ms": 4.5540506267984115,
+      "useful_tokens": 2183065
+    },
+    "on": {
+      "arm": "ON",
+      "bucket_tokens": 227714,
+      "capture_sizes": [
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        16,
+        24,
+        32,
+        40,
+        48,
+        56,
+        64,
+        72,
+        80,
+        88,
+        96,
+        104,
+        112,
+        120,
+        128,
+        136,
+        144,
+        152,
+        160,
+        168,
+        176,
+        184,
+        192,
+        200,
+        208,
+        216,
+        224,
+        232,
+        240,
+        248,
+        256,
+        272,
+        288,
+        304,
+        320,
+        336,
+        352,
+        368,
+        384,
+        400,
+        416,
+        432,
+        448,
+        464,
+        480,
+        496,
+        512
+      ],
+      "clean_completed": 1190,
+      "clean_completed_throughput_rps": 4.958333333333333,
+      "clean_failed": 0,
+      "clean_offered_rps": 4.920833333333333,
+      "drain_seconds": 0.7876331160077825,
+      "e2e_latency_mean_s": 1.612280680370388,
+      "e2e_latency_p95_s": 3.9930475192406423,
+      "gpu_hours": 0.15887338949574364,
+      "graph_hit_steps": 14253,
+      "graph_miss_rate": 0.0389724226282786,
+      "layer1_invariants": {
+        "cudagraph_identity": true,
+        "footer_balanced": true,
+        "footer_written_matches": true,
+        "schema_1": true,
+        "sidecar_agrees": true,
+        "sidecar_final": true,
+        "steps_unique_contiguous": true,
+        "token_composition": true,
+        "zero_drops": true
+      },
+      "layer1_records": 17579,
+      "model_step_duration_ms": 478655.543996,
+      "model_steps": 14831,
+      "padding_fraction": 0.035658764941988635,
+      "padding_tokens": 8120,
+      "schema": 1,
+      "token_efficiency_per_ms": 4.562222306607711,
+      "useful_tokens": 2183733
+    },
+    "plan": {
+      "added_capture_sizes": [
+        3,
+        5,
+        6,
+        7,
+        9
+      ],
+      "clean_seconds": 240,
+      "gpu": 0,
+      "gpu_hour_limit": 16.0,
+      "load": "moderate",
+      "manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl",
+      "measured_padding_recovery_bound": 0.049776380388728225,
+      "on_capture_sizes": [
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        16,
+        24,
+        32,
+        40,
+        48,
+        56,
+        64,
+        72,
+        80,
+        88,
+        96,
+        104,
+        112,
+        120,
+        128,
+        136,
+        144,
+        152,
+        160,
+        168,
+        176,
+        184,
+        192,
+        200,
+        208,
+        216,
+        224,
+        232,
+        240,
+        248,
+        256,
+        272,
+        288,
+        304,
+        320,
+        336,
+        352,
+        368,
+        384,
+        400,
+        416,
+        432,
+        448,
+        464,
+        480,
+        496,
+        512
+      ],
+      "order": [
+        "ON",
+        "OFF"
+      ],
+      "pattern": "P09",
+      "primary_metric": "clean graph-hit padding_fraction",
+      "prior_gpu_hours": 14.025875418755744,
+      "profile": false,
+      "projected_increment_gpu_hours": 0.5,
+      "projected_total_gpu_hours": 14.525875418755744,
+      "rate_fraction": 0.6,
+      "saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json",
+      "schema": 1,
+      "secondary_metrics": [
+        "Layer-1 useful scheduled tokens/model-step millisecond",
+        "clean completed request throughput",
+        "clean request latency"
+      ],
+      "warmup_seconds": 60
+    },
+    "schema": 1
+  },
+  "schema": 1,
+  "status": "complete",
+  "updated_at": 1783857160.3890784
+}

runs/opprof-phase3/phase4/capture-p09/controller.log

@@ -0,0 +1,5 @@
+GPU_COMMAND P09-capture-ON-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120 --cudagraph-capture-sizes 1 2 3 4 5 6 7 8 9 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512; expected=6-9m
+GPU_COMMAND P09-capture-ON-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/on/client; expected=5-7m
+GPU_COMMAND P09-capture-OFF-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120; expected=6-9m
+GPU_COMMAND P09-capture-OFF-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/off/client; expected=5-7m
+{"delta": {"completed_throughput_relative": 0.0050675675675675436, "e2e_mean_latency_relative": -0.04096717785851722, "e2e_p95_latency_relative": 0.03011353223527924, "padding_fraction_points": -0.049797561163151524, "padding_reduction_fraction": 0.5827252753866776, "token_efficiency_relative": 0.0017943761453185214}, "gpu_hours_increment": 0.2963894498348236, "gpu_hours_total": 14.322264868590567, "off": {"arm": "OFF", "bucket_tokens": 237911, "capture_sizes": "default", "clean_completed": 1184, "clean_completed_throughput_rps": 4.933333333333334, "clean_failed": 0, "clean_offered_rps": 4.920833333333333, "drain_seconds": 0.7853007119847462, "e2e_latency_mean_s": 1.681152764688729, "e2e_latency_p95_s": 3.8763178953449815, "gpu_hours": 0.13751606033907995, "graph_hit_steps": 12252, "graph_miss_rate": 0.045199501246882795, "layer1_invariants": {"cudagraph_identity": true, "footer_balanced": true, "footer_written_matches": true, "schema_1": true, "sidecar_agrees": true, "sidecar_final": true, "steps_unique_contiguous": true, "token_composition": true, "zero_drops": true}, "layer1_records": 16491, "model_step_duration_ms": 479367.749483, "model_steps": 12832, "padding_fraction": 0.08545632610514016, "padding_tokens": 20331, "schema": 1, "token_efficiency_per_ms": 4.5540506267984115, "useful_tokens": 2183065}, "on": {"arm": "ON", "bucket_tokens": 227714, "capture_sizes": [1, 2, 3, 4, 5, 6, 7, 8, 9, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], "clean_completed": 1190, "clean_completed_throughput_rps": 4.958333333333333, "clean_failed": 0, "clean_offered_rps": 4.920833333333333, "drain_seconds": 0.7876331160077825, "e2e_latency_mean_s": 1.612280680370388, "e2e_latency_p95_s": 3.9930475192406423, "gpu_hours": 0.15887338949574364, "graph_hit_steps": 14253, "graph_miss_rate": 0.0389724226282786, "layer1_invariants": {"cudagraph_identity": true, "footer_balanced": true, "footer_written_matches": true, "schema_1": true, "sidecar_agrees": true, "sidecar_final": true, "steps_unique_contiguous": true, "token_composition": true, "zero_drops": true}, "layer1_records": 17579, "model_step_duration_ms": 478655.543996, "model_steps": 14831, "padding_fraction": 0.035658764941988635, "padding_tokens": 8120, "schema": 1, "token_efficiency_per_ms": 4.562222306607711, "useful_tokens": 2183733}, "plan": {"added_capture_sizes": [3, 5, 6, 7, 9], "clean_seconds": 240, "gpu": 0, "gpu_hour_limit": 16.0, "load": "moderate", "manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl", "measured_padding_recovery_bound": 0.049776380388728225, "on_capture_sizes": [1, 2, 3, 4, 5, 6, 7, 8, 9, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], "order": ["ON", "OFF"], "pattern": "P09", "primary_metric": "clean graph-hit padding_fraction", "prior_gpu_hours": 14.025875418755744, "profile": false, "projected_increment_gpu_hours": 0.5, "projected_total_gpu_hours": 14.525875418755744, "rate_fraction": 0.6, "saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json", "schema": 1, "secondary_metrics": ["Layer-1 useful scheduled tokens/model-step millisecond", "clean completed request throughput", "clean request latency"], "warmup_seconds": 60}, "schema": 1}

runs/opprof-phase3/phase4/capture-p09/launch.log

@@ -0,0 +1 @@
+GPU_VALIDATION Phase4 P09 capture sizes: order=ON,OFF, GPU0 TP1, manifest=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl, saturation_source=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json, added_sizes=3,5,6,7,9, warmup=60s, clean=240s/arm, projected=0.50 H20-hours, cumulative_projection=14.525875/16, expected_wall=15-22m

runs/opprof-phase3/phase4/capture-p09/off/client-sanity.json

@@ -0,0 +1,66 @@
+{
+  "invariants": {
+    "clean_duration_exact": true,
+    "clean_failures_zero": true,
+    "concurrency_bounded": true,
+    "drain_within_timeout": true,
+    "manifest_no_wrap": true,
+    "manifest_not_exhausted": true,
+    "moderate_offered_within_5pct": true,
+    "output_tokens_exact": true,
+    "profile_count_exact": true,
+    "profile_status_ok": true,
+    "segment_count_exact": true
+  },
+  "numeric": {
+    "actual_output_tokens": {
+      "distinct_n": 1,
+      "finite_n": 1477,
+      "max": 64.0,
+      "min": 64.0,
+      "missing_n": 0,
+      "n": 1477
+    },
+    "admitted_s": {
+      "distinct_n": 1477,
+      "finite_n": 1477,
+      "max": 299.8483776419889,
+      "min": 0.00022695399820804596,
+      "missing_n": 0,
+      "n": 1477
+    },
+    "completed_s": {
+      "distinct_n": 1477,
+      "finite_n": 1477,
+      "max": 300.78334441498737,
+      "min": 0.6682249770092312,
+      "missing_n": 0,
+      "n": 1477
+    },
+    "input_tokens": {
+      "distinct_n": 944,
+      "finite_n": 1477,
+      "max": 8161.0,
+      "min": 128.0,
+      "missing_n": 0,
+      "n": 1477
+    },
+    "requested_output_tokens": {
+      "distinct_n": 1,
+      "finite_n": 1477,
+      "max": 64.0,
+      "min": 64.0,
+      "missing_n": 0,
+      "n": 1477
+    },
+    "scheduled_s": {
+      "distinct_n": 1477,
+      "finite_n": 1477,
+      "max": 299.84763839512016,
+      "min": 0.0,
+      "missing_n": 0,
+      "n": 1477
+    }
+  },
+  "schema": 1
+}

runs/opprof-phase3/phase4/capture-p09/off/commands.log

@@ -0,0 +1,2 @@
+GPU_COMMAND P09-capture-OFF-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120 ; expected=6-9m
+GPU_COMMAND P09-capture-OFF-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/off/client ; expected=5-7m

runs/opprof-phase3/phase4/capture-p09/off/summary.json

@@ -0,0 +1,34 @@
+{
+  "arm": "OFF",
+  "bucket_tokens": 237911,
+  "capture_sizes": "default",
+  "clean_completed": 1184,
+  "clean_completed_throughput_rps": 4.933333333333334,
+  "clean_failed": 0,
+  "clean_offered_rps": 4.920833333333333,
+  "drain_seconds": 0.7853007119847462,
+  "e2e_latency_mean_s": 1.681152764688729,
+  "e2e_latency_p95_s": 3.8763178953449815,
+  "gpu_hours": 0.13751606033907995,
+  "graph_hit_steps": 12252,
+  "graph_miss_rate": 0.045199501246882795,
+  "layer1_invariants": {
+    "cudagraph_identity": true,
+    "footer_balanced": true,
+    "footer_written_matches": true,
+    "schema_1": true,
+    "sidecar_agrees": true,
+    "sidecar_final": true,
+    "steps_unique_contiguous": true,
+    "token_composition": true,
+    "zero_drops": true
+  },
+  "layer1_records": 16491,
+  "model_step_duration_ms": 479367.749483,
+  "model_steps": 12832,
+  "padding_fraction": 0.08545632610514016,
+  "padding_tokens": 20331,
+  "schema": 1,
+  "token_efficiency_per_ms": 4.5540506267984115,
+  "useful_tokens": 2183065
+}

runs/opprof-phase3/phase4/capture-p09/on/client-sanity.json

@@ -0,0 +1,66 @@
+{
+  "invariants": {
+    "clean_duration_exact": true,
+    "clean_failures_zero": true,
+    "concurrency_bounded": true,
+    "drain_within_timeout": true,
+    "manifest_no_wrap": true,
+    "manifest_not_exhausted": true,
+    "moderate_offered_within_5pct": true,
+    "output_tokens_exact": true,
+    "profile_count_exact": true,
+    "profile_status_ok": true,
+    "segment_count_exact": true
+  },
+  "numeric": {
+    "actual_output_tokens": {
+      "distinct_n": 1,
+      "finite_n": 1477,
+      "max": 64.0,
+      "min": 64.0,
+      "missing_n": 0,
+      "n": 1477
+    },
+    "admitted_s": {
+      "distinct_n": 1477,
+      "finite_n": 1477,
+      "max": 299.8487557930057,
+      "min": 0.00024887899053283036,
+      "missing_n": 0,
+      "n": 1477
+    },
+    "completed_s": {
+      "distinct_n": 1477,
+      "finite_n": 1477,
+      "max": 300.78616064199014,
+      "min": 15.648636110010557,
+      "missing_n": 0,
+      "n": 1477
+    },
+    "input_tokens": {
+      "distinct_n": 944,
+      "finite_n": 1477,
+      "max": 8161.0,
+      "min": 128.0,
+      "missing_n": 0,
+      "n": 1477
+    },
+    "requested_output_tokens": {
+      "distinct_n": 1,
+      "finite_n": 1477,
+      "max": 64.0,
+      "min": 64.0,
+      "missing_n": 0,
+      "n": 1477
+    },
+    "scheduled_s": {
+      "distinct_n": 1477,
+      "finite_n": 1477,
+      "max": 299.84763839512016,
+      "min": 0.0,
+      "missing_n": 0,
+      "n": 1477
+    }
+  },
+  "schema": 1
+}

runs/opprof-phase3/phase4/capture-p09/on/commands.log

@@ -0,0 +1,2 @@
+GPU_COMMAND P09-capture-ON-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120 --cudagraph-capture-sizes 1 2 3 4 5 6 7 8 9 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512 ; expected=6-9m
+GPU_COMMAND P09-capture-ON-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/on/client ; expected=5-7m

runs/opprof-phase3/phase4/capture-p09/on/summary.json

@@ -0,0 +1,91 @@
+{
+  "arm": "ON",
+  "bucket_tokens": 227714,
+  "capture_sizes": [
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    16,
+    24,
+    32,
+    40,
+    48,
+    56,
+    64,
+    72,
+    80,
+    88,
+    96,
+    104,
+    112,
+    120,
+    128,
+    136,
+    144,
+    152,
+    160,
+    168,
+    176,
+    184,
+    192,
+    200,
+    208,
+    216,
+    224,
+    232,
+    240,
+    248,
+    256,
+    272,
+    288,
+    304,
+    320,
+    336,
+    352,
+    368,
+    384,
+    400,
+    416,
+    432,
+    448,
+    464,
+    480,
+    496,
+    512
+  ],
+  "clean_completed": 1190,
+  "clean_completed_throughput_rps": 4.958333333333333,
+  "clean_failed": 0,
+  "clean_offered_rps": 4.920833333333333,
+  "drain_seconds": 0.7876331160077825,
+  "e2e_latency_mean_s": 1.612280680370388,
+  "e2e_latency_p95_s": 3.9930475192406423,
+  "gpu_hours": 0.15887338949574364,
+  "graph_hit_steps": 14253,
+  "graph_miss_rate": 0.0389724226282786,
+  "layer1_invariants": {
+    "cudagraph_identity": true,
+    "footer_balanced": true,
+    "footer_written_matches": true,
+    "schema_1": true,
+    "sidecar_agrees": true,
+    "sidecar_final": true,
+    "steps_unique_contiguous": true,
+    "token_composition": true,
+    "zero_drops": true
+  },
+  "layer1_records": 17579,
+  "model_step_duration_ms": 478655.543996,
+  "model_steps": 14831,
+  "padding_fraction": 0.035658764941988635,
+  "padding_tokens": 8120,
+  "schema": 1,
+  "token_efficiency_per_ms": 4.562222306607711,
+  "useful_tokens": 2183733
+}

runs/opprof-phase3/phase4/capture-p09/result.json

@@ -0,0 +1,230 @@
+{
+  "delta": {
+    "completed_throughput_relative": 0.0050675675675675436,
+    "e2e_mean_latency_relative": -0.04096717785851722,
+    "e2e_p95_latency_relative": 0.03011353223527924,
+    "padding_fraction_points": -0.049797561163151524,
+    "padding_reduction_fraction": 0.5827252753866776,
+    "token_efficiency_relative": 0.0017943761453185214
+  },
+  "gpu_hours_increment": 0.2963894498348236,
+  "gpu_hours_total": 14.322264868590567,
+  "off": {
+    "arm": "OFF",
+    "bucket_tokens": 237911,
+    "capture_sizes": "default",
+    "clean_completed": 1184,
+    "clean_completed_throughput_rps": 4.933333333333334,
+    "clean_failed": 0,
+    "clean_offered_rps": 4.920833333333333,
+    "drain_seconds": 0.7853007119847462,
+    "e2e_latency_mean_s": 1.681152764688729,
+    "e2e_latency_p95_s": 3.8763178953449815,
+    "gpu_hours": 0.13751606033907995,
+    "graph_hit_steps": 12252,
+    "graph_miss_rate": 0.045199501246882795,
+    "layer1_invariants": {
+      "cudagraph_identity": true,
+      "footer_balanced": true,
+      "footer_written_matches": true,
+      "schema_1": true,
+      "sidecar_agrees": true,
+      "sidecar_final": true,
+      "steps_unique_contiguous": true,
+      "token_composition": true,
+      "zero_drops": true
+    },
+    "layer1_records": 16491,
+    "model_step_duration_ms": 479367.749483,
+    "model_steps": 12832,
+    "padding_fraction": 0.08545632610514016,
+    "padding_tokens": 20331,
+    "schema": 1,
+    "token_efficiency_per_ms": 4.5540506267984115,
+    "useful_tokens": 2183065
+  },
+  "on": {
+    "arm": "ON",
+    "bucket_tokens": 227714,
+    "capture_sizes": [
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      16,
+      24,
+      32,
+      40,
+      48,
+      56,
+      64,
+      72,
+      80,
+      88,
+      96,
+      104,
+      112,
+      120,
+      128,
+      136,
+      144,
+      152,
+      160,
+      168,
+      176,
+      184,
+      192,
+      200,
+      208,
+      216,
+      224,
+      232,
+      240,
+      248,
+      256,
+      272,
+      288,
+      304,
+      320,
+      336,
+      352,
+      368,
+      384,
+      400,
+      416,
+      432,
+      448,
+      464,
+      480,
+      496,
+      512
+    ],
+    "clean_completed": 1190,
+    "clean_completed_throughput_rps": 4.958333333333333,
+    "clean_failed": 0,
+    "clean_offered_rps": 4.920833333333333,
+    "drain_seconds": 0.7876331160077825,
+    "e2e_latency_mean_s": 1.612280680370388,
+    "e2e_latency_p95_s": 3.9930475192406423,
+    "gpu_hours": 0.15887338949574364,
+    "graph_hit_steps": 14253,
+    "graph_miss_rate": 0.0389724226282786,
+    "layer1_invariants": {
+      "cudagraph_identity": true,
+      "footer_balanced": true,
+      "footer_written_matches": true,
+      "schema_1": true,
+      "sidecar_agrees": true,
+      "sidecar_final": true,
+      "steps_unique_contiguous": true,
+      "token_composition": true,
+      "zero_drops": true
+    },
+    "layer1_records": 17579,
+    "model_step_duration_ms": 478655.543996,
+    "model_steps": 14831,
+    "padding_fraction": 0.035658764941988635,
+    "padding_tokens": 8120,
+    "schema": 1,
+    "token_efficiency_per_ms": 4.562222306607711,
+    "useful_tokens": 2183733
+  },
+  "plan": {
+    "added_capture_sizes": [
+      3,
+      5,
+      6,
+      7,
+      9
+    ],
+    "clean_seconds": 240,
+    "gpu": 0,
+    "gpu_hour_limit": 16.0,
+    "load": "moderate",
+    "manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl",
+    "measured_padding_recovery_bound": 0.049776380388728225,
+    "on_capture_sizes": [
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      16,
+      24,
+      32,
+      40,
+      48,
+      56,
+      64,
+      72,
+      80,
+      88,
+      96,
+      104,
+      112,
+      120,
+      128,
+      136,
+      144,
+      152,
+      160,
+      168,
+      176,
+      184,
+      192,
+      200,
+      208,
+      216,
+      224,
+      232,
+      240,
+      248,
+      256,
+      272,
+      288,
+      304,
+      320,
+      336,
+      352,
+      368,
+      384,
+      400,
+      416,
+      432,
+      448,
+      464,
+      480,
+      496,
+      512
+    ],
+    "order": [
+      "ON",
+      "OFF"
+    ],
+    "pattern": "P09",
+    "primary_metric": "clean graph-hit padding_fraction",
+    "prior_gpu_hours": 14.025875418755744,
+    "profile": false,
+    "projected_increment_gpu_hours": 0.5,
+    "projected_total_gpu_hours": 14.525875418755744,
+    "rate_fraction": 0.6,
+    "saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json",
+    "schema": 1,
+    "secondary_metrics": [
+      "Layer-1 useful scheduled tokens/model-step millisecond",
+      "clean completed request throughput",
+      "clean request latency"
+    ],
+    "warmup_seconds": 60
+  },
+  "schema": 1
+}

runs/opprof-phase3/phase4/capture_validation.py

@@ -0,0 +1,373 @@
+#!/usr/bin/env python3
+"""One-pair P09 CUDAGraph capture-size validation for OpProf Phase 4."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import shlex
+import signal
+import subprocess
+import sys
+import time
+import urllib.request
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+import opprof_phase3_controller as common
+import opprof_phase3_matrix as matrix
+
+
+WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712")
+ROOT = WORKDIR / "runs/phase4-capture-p09"
+PHASE3 = WORKDIR / "runs/phase3"
+PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests")
+MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B")
+SOURCE = Path("/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0")
+VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv")
+CLIENT = WORKDIR / "scripts/opprof_phase3_client.py"
+STATE = ROOT / "controller-state.json"
+GPU = 0
+PORT = 8200
+PRIOR_GPU_HOURS = 14.025875418755744
+GPU_HOUR_LIMIT = 16.0
+EXPECTED_INCREMENT_HOURS = 0.5
+ADDED_CAPTURE_SIZES = (3, 5, 6, 7, 9)
+DEFAULT_CAPTURE_SIZES = (
+    1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104,
+    112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200,
+    208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336,
+    352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512,
+)
+ON_CAPTURE_SIZES = tuple(sorted(set(DEFAULT_CAPTURE_SIZES + ADDED_CAPTURE_SIZES)))
+
+
+def plan() -> dict[str, Any]:
+    return {
+        "schema": 1,
+        "pattern": "P09",
+        "load": "moderate",
+        "order": ["ON", "OFF"],
+        "gpu": GPU,
+        "warmup_seconds": 60,
+        "clean_seconds": 240,
+        "profile": False,
+        "manifest": str(PRIVATE / "P09.jsonl"),
+        "saturation_result": str(
+            PHASE3 / "primary/P09-C00/saturation/client/result.json"
+        ),
+        "rate_fraction": 0.60,
+        "added_capture_sizes": list(ADDED_CAPTURE_SIZES),
+        "on_capture_sizes": list(ON_CAPTURE_SIZES),
+        "measured_padding_recovery_bound": 0.049776380388728225,
+        "primary_metric": "clean graph-hit padding_fraction",
+        "secondary_metrics": [
+            "Layer-1 useful scheduled tokens/model-step millisecond",
+            "clean completed request throughput",
+            "clean request latency",
+        ],
+        "prior_gpu_hours": PRIOR_GPU_HOURS,
+        "projected_increment_gpu_hours": EXPECTED_INCREMENT_HOURS,
+        "projected_total_gpu_hours": PRIOR_GPU_HOURS + EXPECTED_INCREMENT_HOURS,
+        "gpu_hour_limit": GPU_HOUR_LIMIT,
+    }
+
+
+def save_state(state: dict[str, Any]) -> None:
+    state["updated_at"] = time.time()
+    state["controller_pid"] = os.getpid()
+    common.atomic_json(STATE, state)
+
+
+def wait_ready(server: subprocess.Popen[Any]) -> None:
+    deadline = time.monotonic() + 300
+    while time.monotonic() < deadline:
+        if server.poll() is not None:
+            raise RuntimeError("server exited before readiness")
+        try:
+            with urllib.request.urlopen(
+                f"http://127.0.0.1:{PORT}/health", timeout=1
+            ) as response:
+                if response.status == 200:
+                    return
+        except Exception:
+            pass
+        time.sleep(1)
+    raise TimeoutError("server readiness timeout")
+
+
+def server_command(arm: str) -> list[str]:
+    command = [
+        "taskset", "-c", "0-19", str(VENV / "bin/vllm"), "serve", str(MODEL),
+        "--host", "127.0.0.1", "--port", str(PORT),
+        "--tensor-parallel-size", "1", "--enable-chunked-prefill",
+        "--enable-prefix-caching", "--shutdown-timeout", "120",
+    ]
+    if arm == "ON":
+        command.extend(("--cudagraph-capture-sizes", *map(str, ON_CAPTURE_SIZES)))
+    return command
+
+
+def client_command(run_dir: Path) -> list[str]:
+    return [
+        "taskset", "-c", "0-19", str(VENV / "bin/python"), str(CLIENT), "run",
+        "--manifest", str(PRIVATE / "P09.jsonl"),
+        "--base-url", f"http://127.0.0.1:{PORT}", "--model", str(MODEL),
+        "--load-point", "moderate", "--saturation-result",
+        str(PHASE3 / "primary/P09-C00/saturation/client/result.json"),
+        "--rate-fraction", "0.60", "--max-concurrency", "256", "--ignore-eos",
+        "--temperature", "0", "--warmup-seconds", "60",
+        "--clean-segment-seconds", "80", "--num-clean-segments", "3",
+        "--recovery-seconds", "30", "--drain-timeout-seconds", "120",
+        "--workload-seed", "20260712", "--server-seed", "20260712",
+        "--result-dir", str(run_dir / "client"),
+    ]
+
+
+def summarize(run_dir: Path, arm: str) -> dict[str, Any]:
+    result = json.loads((run_dir / "client/result.json").read_text())
+    requests = [
+        json.loads(line)
+        for line in (run_dir / "client/requests.jsonl").read_text().splitlines()
+    ]
+    t0 = int(result["t0_mono_ns"])
+    start, end = t0 + int(60e9), t0 + int(300e9)
+    stream = next((run_dir / "opprof").glob("*.jsonl"))
+    records = []
+    for line in stream.read_text().splitlines():
+        record = json.loads(line)
+        if (
+            "step_index" in record
+            and start <= int(record["submit_mono_ns"]) < end
+        ):
+            records.append(record)
+    model = [record for record in records if record["model_executed"]]
+    hits = [
+        record
+        for record in model
+        if record["cudagraph"]["hit"]
+        and int(record["cudagraph"]["bucket_tokens"]) > 0
+    ]
+    pad = sum(int(record["cudagraph"]["padding_tokens"]) for record in hits)
+    bucket = sum(int(record["cudagraph"]["bucket_tokens"]) for record in hits)
+    useful = sum(
+        int(record["prefill_tokens"]) + int(record["decode_tokens"])
+        for record in records
+    )
+    duration_ms = sum(
+        (int(record["complete_mono_ns"]) - int(record["submit_mono_ns"])) / 1e6
+        for record in records
+    )
+    completed = [
+        request
+        for request in requests
+        if request["success"] and 60 <= float(request["completed_s"]) < 300
+    ]
+    e2e = np.asarray(
+        [float(request["completed_s"] - request["admitted_s"]) for request in completed]
+    )
+    layer1 = matrix.validate_layer1(run_dir)
+    return {
+        "schema": 1,
+        "arm": arm,
+        "capture_sizes": list(ON_CAPTURE_SIZES) if arm == "ON" else "default",
+        "clean_completed": len(completed),
+        "clean_failed": int(result["clean"]["failed"]),
+        "clean_completed_throughput_rps": float(
+            result["clean"]["completed_throughput_rps"]
+        ),
+        "clean_offered_rps": float(result["clean"]["offered_rps"]),
+        "e2e_latency_mean_s": float(e2e.mean()),
+        "e2e_latency_p95_s": float(np.quantile(e2e, 0.95)),
+        "model_steps": len(model),
+        "graph_hit_steps": len(hits),
+        "padding_tokens": pad,
+        "bucket_tokens": bucket,
+        "padding_fraction": pad / bucket,
+        "graph_miss_rate": sum(not record["cudagraph"]["hit"] for record in model)
+        / len(model),
+        "useful_tokens": useful,
+        "model_step_duration_ms": duration_ms,
+        "token_efficiency_per_ms": useful / duration_ms,
+        "layer1_records": layer1["records"],
+        "layer1_invariants": layer1["invariants"],
+        "drain_seconds": float(result["drain_seconds"]),
+    }
+
+
+def run_arm(state: dict[str, Any], arm: str) -> None:
+    if state["arms"].get(arm, {}).get("status") == "complete":
+        return
+    run_dir = ROOT / arm.lower()
+    if run_dir.exists():
+        run_dir.rename(run_dir.with_name(f"{run_dir.name}.interrupted-{int(time.time())}"))
+    run_dir.mkdir(parents=True)
+    common.preflight([GPU], run_dir)
+    server_cmd = server_command(arm)
+    client_cmd = client_command(run_dir)
+    commands_path = run_dir / "commands.log"
+    common.command_log(commands_path, f"P09-capture-{arm}-server", server_cmd, "6-9m")
+    common.command_log(commands_path, f"P09-capture-{arm}-client", client_cmd, "5-7m")
+    print(
+        f"GPU_COMMAND P09-capture-{arm}-server: {shlex.join(server_cmd)}; "
+        "expected=6-9m",
+        flush=True,
+    )
+    print(
+        f"GPU_COMMAND P09-capture-{arm}-client: {shlex.join(client_cmd)}; "
+        "expected=5-7m",
+        flush=True,
+    )
+    environment = os.environ.copy()
+    environment.update(
+        {
+            "CUDA_VISIBLE_DEVICES": str(GPU),
+            "VLLM_OPPROF_DIR": str(run_dir / "opprof"),
+            "HF_HUB_OFFLINE": "1",
+            "TRANSFORMERS_OFFLINE": "1",
+            "PYTHONUNBUFFERED": "1",
+        }
+    )
+    server_log = (run_dir / "server.log").open("ab", buffering=0)
+    server_started = time.time()
+    server = subprocess.Popen(
+        server_cmd,
+        cwd=SOURCE,
+        env=environment,
+        stdout=server_log,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    client = None
+    client_log = None
+    monitor = None
+    owned = {server.pid}
+    state["arms"][arm] = {
+        "status": "starting",
+        "server_pid": server.pid,
+        "started_at": server_started,
+    }
+    save_state(state)
+    failure = None
+    try:
+        wait_ready(server)
+        monitor = common.Monitor(run_dir / "monitor.jsonl", owned)
+        monitor.start()
+        client_log = (run_dir / "client.log").open("ab", buffering=0)
+        client = subprocess.Popen(
+            client_cmd,
+            cwd=WORKDIR,
+            stdout=client_log,
+            stderr=subprocess.STDOUT,
+            start_new_session=True,
+        )
+        owned.add(client.pid)
+        state["arms"][arm].update(status="running", client_pid=client.pid)
+        save_state(state)
+        deadline = time.monotonic() + 900
+        while client.poll() is None and time.monotonic() < deadline:
+            if server.poll() is not None:
+                raise RuntimeError("server exited during client load")
+            if monitor.other_apps:
+                raise RuntimeError(f"other GPU process appeared: {monitor.other_apps}")
+            time.sleep(2)
+        if client.poll() is None:
+            raise TimeoutError("client exceeded 900 seconds")
+        if client.returncode:
+            raise RuntimeError(f"client failed with {client.returncode}")
+    except Exception as error:
+        failure = error
+    finally:
+        if client is not None and client.poll() is None:
+            try:
+                os.killpg(client.pid, signal.SIGKILL)
+            except ProcessLookupError:
+                pass
+        common.stop_servers([server])
+        server_log.close()
+        if client_log is not None:
+            client_log.close()
+        if monitor is not None:
+            monitor.stop()
+        common.verify_idle([GPU], run_dir)
+    gpu_hours = (time.time() - server_started) / 3600
+    state["gpu_hours_increment"] += gpu_hours
+    if PRIOR_GPU_HOURS + state["gpu_hours_increment"] >= GPU_HOUR_LIMIT:
+        failure = failure or RuntimeError("GPU-hour limit reached")
+    if failure is not None:
+        state["arms"][arm].update(status="failed", failure=repr(failure))
+        state["status"] = "failed"
+        save_state(state)
+        raise failure
+    summary = summarize(run_dir, arm)
+    summary["gpu_hours"] = gpu_hours
+    common.atomic_json(run_dir / "summary.json", summary)
+    state["arms"][arm].update(status="complete", summary=summary)
+    save_state(state)
+
+
+def run() -> None:
+    ROOT.mkdir(parents=True, exist_ok=True)
+    if PRIOR_GPU_HOURS + EXPECTED_INCREMENT_HOURS >= GPU_HOUR_LIMIT:
+        raise RuntimeError("projected validation exceeds GPU-hour budget")
+    if STATE.exists():
+        state = json.loads(STATE.read_text())
+    else:
+        state = {
+            "schema": 1,
+            "status": "running",
+            "created_at": time.time(),
+            "plan": plan(),
+            "arms": {},
+            "gpu_hours_increment": 0.0,
+        }
+    for arm in ("ON", "OFF"):
+        run_arm(state, arm)
+    on, off = state["arms"]["ON"]["summary"], state["arms"]["OFF"]["summary"]
+    result = {
+        "schema": 1,
+        "plan": plan(),
+        "on": on,
+        "off": off,
+        "delta": {
+            "padding_fraction_points": on["padding_fraction"] - off["padding_fraction"],
+            "padding_reduction_fraction": 1 - on["padding_fraction"] / off["padding_fraction"],
+            "token_efficiency_relative": on["token_efficiency_per_ms"]
+            / off["token_efficiency_per_ms"]
+            - 1,
+            "completed_throughput_relative": on["clean_completed_throughput_rps"]
+            / off["clean_completed_throughput_rps"]
+            - 1,
+            "e2e_mean_latency_relative": on["e2e_latency_mean_s"]
+            / off["e2e_latency_mean_s"]
+            - 1,
+            "e2e_p95_latency_relative": on["e2e_latency_p95_s"]
+            / off["e2e_latency_p95_s"]
+            - 1,
+        },
+        "gpu_hours_increment": state["gpu_hours_increment"],
+        "gpu_hours_total": PRIOR_GPU_HOURS + state["gpu_hours_increment"],
+    }
+    common.atomic_json(ROOT / "result.json", result)
+    state["status"] = "complete"
+    state["result"] = result
+    save_state(state)
+    print(json.dumps(result, sort_keys=True))
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("command", choices=("plan", "run"))
+    args = parser.parse_args()
+    if args.command == "plan":
+        print(json.dumps(plan(), indent=2, sort_keys=True))
+    else:
+        run()
+
+
+if __name__ == "__main__":
+    main()

runs/opprof-phase3/provenance/opprof_phase3_client.py

@@ -0,0 +1,794 @@
+#!/usr/bin/env python3
+"""Token-exact fixed-duration client for the OpProf Phase-3 protocol."""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import gzip
+import hashlib
+import json
+import math
+import os
+import time
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import aiohttp
+
+SCHEMA = 1
+TOKEN_BASE = 1000
+TOKEN_SPAN = 100000
+
+
+class ManifestExhausted(RuntimeError):
+    pass
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as f:
+        for chunk in iter(lambda: f.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def atomic_json(path: Path, value: Any, mode: int = 0o640) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_name(path.name + f".tmp.{os.getpid()}")
+    fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode)
+    with os.fdopen(fd, "w", encoding="utf-8") as f:
+        json.dump(value, f, sort_keys=True, indent=2)
+        f.write("\n")
+        f.flush()
+        os.fsync(f.fileno())
+    os.replace(tmp, path)
+
+
+def atomic_jsonl(path: Path, rows: list[dict[str, Any]], mode: int = 0o640) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_name(path.name + f".tmp.{os.getpid()}")
+    fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode)
+    with os.fdopen(fd, "w", encoding="utf-8") as f:
+        for row in rows:
+            f.write(json.dumps(row, sort_keys=True, separators=(",", ":")) + "\n")
+        f.flush()
+        os.fsync(f.fileno())
+    os.replace(tmp, path)
+
+
+def parse_range(value: str) -> tuple[int, int]:
+    lo_text, hi_text = value.split(":", 1)
+    lo, hi = int(lo_text), int(hi_text)
+    if lo <= 0 or hi < lo:
+        raise argparse.ArgumentTypeError(f"invalid positive range: {value}")
+    return lo, hi
+
+
+def _integer_counts(weights: list[float], total: int) -> list[int]:
+    raw = [w * total for w in weights]
+    counts = [math.floor(x) for x in raw]
+    order = sorted(
+        range(len(raw)), key=lambda i: raw[i] - counts[i], reverse=True
+    )
+    for idx in order[: total - sum(counts)]:
+        counts[idx] += 1
+    return counts
+
+
+def numeric_sanity(values: list[float | int]) -> dict[str, Any]:
+    finite = [float(x) for x in values if math.isfinite(float(x))]
+    return {
+        "n": len(values),
+        "finite_n": len(finite),
+        "missing_n": len(values) - len(finite),
+        "min": min(finite) if finite else None,
+        "max": max(finite) if finite else None,
+        "distinct_n": len(set(finite)),
+    }
+
+
+def manifest_summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
+    return {
+        "schema": SCHEMA,
+        "rows": len(rows),
+        "input_tokens": numeric_sanity([int(r["input_tokens"]) for r in rows]),
+        "output_tokens": numeric_sanity([int(r["output_tokens"]) for r in rows]),
+        "arrival_values": sorted({str(r["arrival"]) for r in rows}),
+        "pattern_values": sorted({str(r["pattern_id"]) for r in rows}),
+    }
+
+
+def materialize(args: argparse.Namespace) -> dict[str, Any]:
+    import numpy as np
+
+    rng = np.random.default_rng(args.workload_seed)
+    n = args.num_requests
+    if args.kind == "prefix-pool":
+        if args.num_prefixes <= 0 or args.prefix_len <= 0 or args.suffix_fixed <= 0:
+            raise ValueError("prefix-pool requires positive pool/prefix/suffix")
+        lengths = np.full(n, args.prefix_len + args.suffix_fixed, dtype=np.int64)
+        prefix_ids = np.arange(n, dtype=np.int64) % args.num_prefixes
+        rng.shuffle(prefix_ids)
+    else:
+        prefix_ids = np.full(n, -1, dtype=np.int64)
+        if args.input_uniform:
+            lo, hi = parse_range(args.input_uniform)
+            lengths = rng.integers(lo, hi + 1, n, dtype=np.int64)
+        elif args.input_fixed:
+            lengths = np.full(n, args.input_fixed, dtype=np.int64)
+        elif args.input_mixture:
+            spec = json.loads(args.input_mixture)
+            if not isinstance(spec, dict) or not spec:
+                raise ValueError("input mixture must be a non-empty JSON object")
+            keys = list(spec)
+            weights = [float(spec[key]) for key in keys]
+            if any(w < 0 for w in weights) or not math.isclose(sum(weights), 1.0):
+                raise ValueError("mixture weights must be non-negative and sum to 1")
+            pieces = []
+            for key, count in zip(
+                keys, _integer_counts(weights, n), strict=True
+            ):
+                kind, lo_text, hi_text = key.split(":")
+                if kind != "uniform":
+                    raise ValueError(f"unsupported mixture component: {key}")
+                pieces.append(
+                    rng.integers(
+                        int(lo_text), int(hi_text) + 1, count, dtype=np.int64
+                    )
+                )
+            lengths = np.concatenate(pieces)
+            rng.shuffle(lengths)
+        else:
+            raise ValueError("exactly one input distribution is required")
+    if args.output_fixed <= 0 or args.arrival not in {"steady", "burst:8"}:
+        raise ValueError("invalid output length or arrival class")
+
+    rows = []
+    for i in range(n):
+        row = {
+            "schema": SCHEMA,
+            "request_id": f"{args.id}-{i:05d}",
+            "pattern_id": args.id,
+            "kind": args.kind,
+            "input_tokens": int(lengths[i]),
+            "output_tokens": args.output_fixed,
+            "arrival": args.arrival,
+            "token_seed": int(args.workload_seed * 1000003 + i),
+        }
+        if args.kind == "prefix-pool":
+            row.update(
+                {
+                    "prefix_id": int(prefix_ids[i]),
+                    "num_prefixes": args.num_prefixes,
+                    "prefix_tokens": args.prefix_len,
+                }
+            )
+        rows.append(row)
+
+    out = Path(args.out)
+    atomic_jsonl(out, rows, mode=0o600)
+    summary = manifest_summary(rows)
+    summary.update({"sha256": sha256_file(out), "path": str(out)})
+    atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600)
+    print(json.dumps(summary, sort_keys=True))
+    return summary
+
+
+def materialize_private(args: argparse.Namespace) -> dict[str, Any]:
+    from transformers import AutoTokenizer
+
+    source = Path(args.source)
+    selected: list[dict[str, Any]] = []
+    with source.open(encoding="utf-8") as f:
+        for source_index, line in enumerate(f):
+            row = json.loads(line)
+            if (
+                float(row["sampling_u"]) <= args.sampling_u_max
+                and int(row["input_length"]) <= args.max_input_tokens
+            ):
+                selected.append(
+                    {
+                        "schema": SCHEMA,
+                        "request_id": f"{args.id}-{len(selected):05d}",
+                        "pattern_id": args.id,
+                        "kind": "private-trace",
+                        "input_tokens": int(row["input_length"]),
+                        "output_tokens": min(
+                            int(row["output_length"]), args.output_cap
+                        ),
+                        "arrival": args.arrival,
+                        "source_index": source_index,
+                        "prompt": row["prompt"],
+                    }
+                )
+
+    tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
+    diffs = [
+        len(tokenizer.encode(row["prompt"], add_special_tokens=False))
+        - row["input_tokens"]
+        for row in selected
+    ]
+    exact = sum(diff == 0 for diff in diffs)
+    exact_fraction = exact / len(diffs) if diffs else 0.0
+    max_abs = max((abs(diff) for diff in diffs), default=-1)
+    if exact_fraction < 0.99 or max_abs > 1:
+        raise RuntimeError(
+            "tokenizer parity gate failed: "
+            f"exact_fraction={exact_fraction:.6f} max_abs_error={max_abs}"
+        )
+
+    out = Path(args.out)
+    atomic_jsonl(out, selected, mode=0o600)
+    summary = manifest_summary(selected)
+    summary.update(
+        {
+            "sha256": sha256_file(out),
+            "source_sha256": sha256_file(source),
+            "tokenizer_exact_n": exact,
+            "tokenizer_exact_fraction": exact_fraction,
+            "tokenizer_max_abs_error": max_abs,
+            "path": str(out),
+        }
+    )
+    atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600)
+    print(json.dumps(summary, sort_keys=True))
+    return summary
+
+
+def load_manifest(path: Path) -> list[dict[str, Any]]:
+    rows = [json.loads(line) for line in path.read_text().splitlines() if line]
+    required = {
+        "request_id",
+        "pattern_id",
+        "input_tokens",
+        "output_tokens",
+        "arrival",
+    }
+    if not rows:
+        raise ValueError("empty manifest")
+    for row in rows:
+        if not required.issubset(row):
+            raise ValueError(f"manifest row lacks {sorted(required - set(row))}")
+    if len({row["request_id"] for row in rows}) != len(rows):
+        raise ValueError("duplicate request_id")
+    return rows
+
+
+def _token_stream(seed: int, count: int) -> list[int]:
+    state = seed & 0xFFFFFFFF
+    out = []
+    for _ in range(count):
+        state = (1664525 * state + 1013904223) & 0xFFFFFFFF
+        out.append(TOKEN_BASE + state % TOKEN_SPAN)
+    return out
+
+
+def synthetic_prompt(row: dict[str, Any]) -> list[int]:
+    length = int(row["input_tokens"])
+    seed = int(row.get("token_seed", 0))
+    if row.get("kind") == "prefix-pool":
+        prefix_n = int(row["prefix_tokens"])
+        tokens = _token_stream(0xA5A50000 + int(row["prefix_id"]), prefix_n)
+        tokens += _token_stream(seed, length - prefix_n)
+        offset = prefix_n
+    else:
+        tokens = _token_stream(seed, length)
+        offset = 0
+    if length - offset >= 3:
+        index = int(row["request_id"].rsplit("-", 1)[1])
+        tokens[offset : offset + 3] = [
+            TOKEN_BASE + index % 100,
+            TOKEN_BASE + (index // 100) % 100,
+            TOKEN_BASE + (index // 10000) % 100,
+        ]
+    return tokens
+
+
+@dataclass
+class RunContext:
+    args: argparse.Namespace
+    rows: list[dict[str, Any]]
+    t0: float
+    clean_end: float
+    stop_event: asyncio.Event
+    lock: asyncio.Lock
+    next_index: int = 0
+    in_flight: int = 0
+    max_in_flight: int = 0
+    exhausted: bool = False
+    admission_stop_s: float | None = None
+
+    async def next_row(self) -> dict[str, Any]:
+        async with self.lock:
+            if self.next_index >= len(self.rows):
+                self.exhausted = True
+                raise ManifestExhausted(
+                    f"manifest exhausted after {self.next_index} admissions"
+                )
+            row = self.rows[self.next_index]
+            self.next_index += 1
+            return row
+
+
+async def request_one(
+    ctx: RunContext,
+    session: aiohttp.ClientSession,
+    row: dict[str, Any],
+    scheduled: float,
+) -> dict[str, Any]:
+    loop = asyncio.get_running_loop()
+    admitted = loop.time()
+    ctx.in_flight += 1
+    ctx.max_in_flight = max(ctx.max_in_flight, ctx.in_flight)
+    status = 0
+    actual_output: int | None = None
+    first_token: float | None = None
+    error_kind: str | None = None
+    try:
+        prompt: str | list[int] = (
+            row["prompt"]
+            if row.get("kind") == "private-trace"
+            else synthetic_prompt(row)
+        )
+        if not isinstance(prompt, str) and len(prompt) != int(row["input_tokens"]):
+            raise AssertionError("synthetic prompt length drift")
+        payload = {
+            "model": ctx.args.model,
+            "prompt": prompt,
+            "max_tokens": int(row["output_tokens"]),
+            "temperature": ctx.args.temperature,
+            "ignore_eos": ctx.args.ignore_eos,
+            "stream": True,
+            "stream_options": {"include_usage": True},
+            "add_special_tokens": False,
+            "seed": ctx.args.server_seed,
+        }
+        headers = {
+            "Content-Type": "application/json",
+            "x-request-id": str(row["request_id"]),
+        }
+        async with session.post(
+            ctx.args.base_url.rstrip("/") + "/v1/completions",
+            json=payload,
+            headers=headers,
+        ) as response:
+            status = response.status
+            if status != 200:
+                error_kind = f"http_{status}"
+            else:
+                buf = b""
+                async for chunk in response.content.iter_any():
+                    buf += chunk
+                    while b"\n" in buf:
+                        line, buf = buf.split(b"\n", 1)
+                        line = line.strip()
+                        if not line.startswith(b"data:"):
+                            continue
+                        data = line[5:].strip()
+                        if data == b"[DONE]":
+                            continue
+                        event = json.loads(data)
+                        if event.get("choices") and first_token is None:
+                            first_token = loop.time()
+                        if event.get("usage") is not None:
+                            actual_output = int(event["usage"]["completion_tokens"])
+                if actual_output is None:
+                    error_kind = "missing_usage"
+    except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
+        error_kind = type(exc).__name__
+    except Exception as exc:
+        error_kind = type(exc).__name__
+    finally:
+        completed = loop.time()
+        ctx.in_flight -= 1
+    success = (
+        status == 200
+        and error_kind is None
+        and actual_output == int(row["output_tokens"])
+    )
+    if status == 200 and actual_output is not None and not success:
+        error_kind = "output_token_mismatch"
+    return {
+        "schema": SCHEMA,
+        "request_id": row["request_id"],
+        "scheduled_s": scheduled - ctx.t0,
+        "admitted_s": admitted - ctx.t0,
+        "first_token_s": None if first_token is None else first_token - ctx.t0,
+        "completed_s": completed - ctx.t0,
+        "input_tokens": int(row["input_tokens"]),
+        "requested_output_tokens": int(row["output_tokens"]),
+        "actual_output_tokens": actual_output,
+        "http_status": status,
+        "success": success,
+        "error_kind": error_kind,
+    }
+
+
+async def saturation_load(
+    ctx: RunContext, session: aiohttp.ClientSession
+) -> list[dict[str, Any]]:
+    results: list[dict[str, Any]] = []
+
+    async def worker() -> None:
+        while not ctx.stop_event.is_set():
+            try:
+                row = await ctx.next_row()
+            except ManifestExhausted:
+                ctx.stop_event.set()
+                return
+            results.append(
+                await request_one(ctx, session, row, asyncio.get_running_loop().time())
+            )
+
+    tasks = [
+        asyncio.create_task(worker()) for _ in range(ctx.args.max_concurrency)
+    ]
+    await asyncio.gather(*tasks)
+    return results
+
+
+async def finite_load(
+    ctx: RunContext, session: aiohttp.ClientSession, rate: float
+) -> list[dict[str, Any]]:
+    sem = asyncio.Semaphore(ctx.args.max_concurrency)
+    tasks: list[asyncio.Task[dict[str, Any]]] = []
+    batch = 8 if str(ctx.rows[0]["arrival"]) == "burst:8" else 1
+    period = batch / rate
+    event_index = 0
+
+    async def limited(row: dict[str, Any], scheduled: float) -> dict[str, Any]:
+        async with sem:
+            return await request_one(ctx, session, row, scheduled)
+
+    while not ctx.stop_event.is_set():
+        scheduled = ctx.t0 + event_index * period
+        delay = scheduled - asyncio.get_running_loop().time()
+        if delay > 0:
+            try:
+                await asyncio.wait_for(ctx.stop_event.wait(), timeout=delay)
+                break
+            except asyncio.TimeoutError:
+                pass
+        if ctx.stop_event.is_set():
+            break
+        try:
+            for _ in range(batch):
+                tasks.append(
+                    asyncio.create_task(limited(await ctx.next_row(), scheduled))
+                )
+        except ManifestExhausted:
+            ctx.stop_event.set()
+            break
+        event_index += 1
+    return await asyncio.gather(*tasks) if tasks else []
+
+
+async def post_profile(
+    session: aiohttp.ClientSession, base_url: str, endpoint: str
+) -> tuple[float, float, int]:
+    loop = asyncio.get_running_loop()
+    before = loop.time()
+    async with session.post(base_url.rstrip("/") + endpoint) as response:
+        status = response.status
+        await response.read()
+    return before, loop.time(), status
+
+
+def _trace_loadable(path: Path) -> bool:
+    try:
+        opener = gzip.open if path.suffix == ".gz" else open
+        with opener(path, "rt", encoding="utf-8") as f:
+            parsed = json.load(f)
+        return isinstance(parsed, dict) and isinstance(parsed.get("traceEvents"), list)
+    except (OSError, EOFError, json.JSONDecodeError):
+        return False
+
+
+async def wait_new_trace(
+    trace_dir: Path, before: set[Path], timeout: float
+) -> Path:
+    deadline = asyncio.get_running_loop().time() + timeout
+    while asyncio.get_running_loop().time() < deadline:
+        for path in sorted(set(trace_dir.glob("*.pt.trace.json*")) - before):
+            if _trace_loadable(path):
+                return path
+        await asyncio.sleep(0.25)
+    raise TimeoutError(f"no new loadable trace within {timeout}s")
+
+
+async def timeline(
+    ctx: RunContext, session: aiohttp.ClientSession
+) -> list[dict[str, Any]]:
+    args = ctx.args
+    profiles: list[dict[str, Any]] = []
+    await asyncio.sleep(max(0, ctx.clean_end - asyncio.get_running_loop().time()))
+    if args.profile_after_clean:
+        trace_dir = Path(args.profile_trace_dir)
+        for window in range(args.num_profile_windows):
+            prior = set(trace_dir.glob("*.pt.trace.json*"))
+            start_before, start_after, start_status = await post_profile(
+                session, args.base_url, "/start_profile"
+            )
+            trace = await wait_new_trace(
+                trace_dir, prior, args.profile_timeout_seconds
+            )
+            trace_ready = asyncio.get_running_loop().time()
+            stop_before, stop_after, stop_status = await post_profile(
+                session, args.base_url, "/stop_profile"
+            )
+            profiles.append(
+                {
+                    "window": window + 1,
+                    "start_call_s": start_before - ctx.t0,
+                    "start_return_s": start_after - ctx.t0,
+                    "trace_ready_s": trace_ready - ctx.t0,
+                    "stop_call_s": stop_before - ctx.t0,
+                    "stop_return_s": stop_after - ctx.t0,
+                    "start_status": start_status,
+                    "stop_status": stop_status,
+                    "trace_file": trace.name,
+                    "trace_sha256": sha256_file(trace),
+                }
+            )
+            if start_status != 200 or stop_status != 200:
+                raise RuntimeError("profile endpoint returned non-200")
+            await asyncio.sleep(args.recovery_seconds)
+    else:
+        await asyncio.sleep(args.post_clean_seconds)
+    ctx.admission_stop_s = asyncio.get_running_loop().time() - ctx.t0
+    ctx.stop_event.set()
+    return profiles
+
+
+def segment_summary(
+    records: list[dict[str, Any]], start: float, end: float
+) -> dict[str, Any]:
+    admitted = [r for r in records if start <= r["admitted_s"] < end]
+    completed = [r for r in records if start <= r["completed_s"] < end]
+    successes = [r for r in completed if r["success"]]
+    duration = end - start
+    return {
+        "start_s": start,
+        "end_s": end,
+        "duration_s": duration,
+        "admitted": len(admitted),
+        "completed": len(successes),
+        "failed": len(completed) - len(successes),
+        "offered_rps": len(admitted) / duration,
+        "completed_throughput_rps": len(successes) / duration,
+        "input_tokens": sum(r["input_tokens"] for r in successes),
+        "output_tokens": sum(r["actual_output_tokens"] or 0 for r in successes),
+    }
+
+
+async def run_load(args: argparse.Namespace) -> dict[str, Any]:
+    manifest = Path(args.manifest)
+    rows = load_manifest(manifest)
+    arrivals = {row["arrival"] for row in rows}
+    if len(arrivals) != 1:
+        raise ValueError("a manifest must have one arrival class")
+    if args.load_point == "saturation":
+        if args.request_rate != "inf":
+            raise ValueError("saturation requires --request-rate inf")
+        rate = math.inf
+    else:
+        if not args.saturation_result:
+            raise ValueError("moderate requires --saturation-result")
+        sat = json.loads(Path(args.saturation_result).read_text())
+        rate = args.rate_fraction * float(sat["clean"]["completed_throughput_rps"])
+        if not math.isfinite(rate) or rate <= 0:
+            raise ValueError("derived moderate rate must be positive and finite")
+
+    loop = asyncio.get_running_loop()
+    t0 = loop.time()
+    t0_mono_ns = int(t0 * 1e9)
+    t0_wall_ns = time.time_ns()
+    clean_seconds = args.clean_segment_seconds * args.num_clean_segments
+    ctx = RunContext(
+        args=args,
+        rows=rows,
+        t0=t0,
+        clean_end=t0 + args.warmup_seconds + clean_seconds,
+        stop_event=asyncio.Event(),
+        lock=asyncio.Lock(),
+    )
+    timeout = aiohttp.ClientTimeout(total=None, connect=30, sock_read=600)
+    connector = aiohttp.TCPConnector(limit=args.max_concurrency)
+    control_connector = aiohttp.TCPConnector(limit=2)
+    async with (
+        aiohttp.ClientSession(timeout=timeout, connector=connector) as session,
+        aiohttp.ClientSession(
+            timeout=timeout, connector=control_connector
+        ) as control_session,
+    ):
+        profile_task = asyncio.create_task(timeline(ctx, control_session))
+        load_task = asyncio.create_task(
+            saturation_load(ctx, session)
+            if math.isinf(rate)
+            else finite_load(ctx, session, rate)
+        )
+        try:
+            profiles = await profile_task
+        except Exception:
+            ctx.stop_event.set()
+            await load_task
+            raise
+        records = await load_task
+
+    clean_start = args.warmup_seconds
+    clean_end = clean_start + clean_seconds
+    clean = segment_summary(records, clean_start, clean_end)
+    segments = []
+    for i in range(args.num_clean_segments):
+        start = clean_start + i * args.clean_segment_seconds
+        segments.append(
+            {
+                "name": chr(ord("A") + i),
+                **segment_summary(
+                    records, start, start + args.clean_segment_seconds
+                ),
+            }
+        )
+    successful = [r for r in records if r["success"]]
+    elapsed_seconds = loop.time() - t0
+    if ctx.admission_stop_s is None:
+        raise RuntimeError("admission stop timestamp was not recorded")
+    drain_seconds = elapsed_seconds - ctx.admission_stop_s
+    result = {
+        "schema": SCHEMA,
+        "manifest_sha256": sha256_file(manifest),
+        "manifest_rows": len(rows),
+        "manifest_admitted": ctx.next_index,
+        "manifest_wrapped": False,
+        "manifest_exhausted": ctx.exhausted,
+        "load_point": args.load_point,
+        "t0_mono_ns": t0_mono_ns,
+        "t0_wall_ns": t0_wall_ns,
+        "request_rate": "inf" if math.isinf(rate) else rate,
+        "rate_fraction": None if math.isinf(rate) else args.rate_fraction,
+        "arrival": next(iter(arrivals)),
+        "warmup_seconds": args.warmup_seconds,
+        "clean_segment_seconds": args.clean_segment_seconds,
+        "num_clean_segments": args.num_clean_segments,
+        "elapsed_seconds": elapsed_seconds,
+        "admission_stop_s": ctx.admission_stop_s,
+        "drain_seconds": drain_seconds,
+        "max_in_flight": ctx.max_in_flight,
+        "records": len(records),
+        "successful_records": len(successful),
+        "failed_records": len(records) - len(successful),
+        "clean": clean,
+        "segments": segments,
+        "profiles": profiles,
+    }
+    sanity = {
+        "schema": SCHEMA,
+        "numeric": {
+            "input_tokens": numeric_sanity([r["input_tokens"] for r in records]),
+            "requested_output_tokens": numeric_sanity(
+                [r["requested_output_tokens"] for r in records]
+            ),
+            "actual_output_tokens": numeric_sanity(
+                [
+                    r["actual_output_tokens"]
+                    for r in records
+                    if r["actual_output_tokens"] is not None
+                ]
+            ),
+            "scheduled_s": numeric_sanity([r["scheduled_s"] for r in records]),
+            "admitted_s": numeric_sanity([r["admitted_s"] for r in records]),
+            "completed_s": numeric_sanity([r["completed_s"] for r in records]),
+        },
+        "invariants": {
+            "clean_duration_exact": math.isclose(clean["duration_s"], clean_seconds),
+            "segment_count_exact": len(segments) == args.num_clean_segments,
+            "manifest_no_wrap": ctx.next_index <= len(rows),
+            "manifest_not_exhausted": not ctx.exhausted,
+            "concurrency_bounded": ctx.max_in_flight <= args.max_concurrency,
+            "drain_within_timeout": drain_seconds <= args.drain_timeout_seconds,
+            "output_tokens_exact": all(
+                r["actual_output_tokens"] == r["requested_output_tokens"]
+                for r in successful
+            ),
+            "clean_failures_zero": clean["failed"] == 0,
+            "profile_count_exact": len(profiles)
+            == (args.num_profile_windows if args.profile_after_clean else 0),
+            "profile_status_ok": all(
+                p["start_status"] == 200 and p["stop_status"] == 200
+                for p in profiles
+            ),
+        },
+    }
+    if not math.isinf(rate):
+        sanity["invariants"]["moderate_offered_within_5pct"] = (
+            abs(clean["offered_rps"] / rate - 1) <= 0.05
+        )
+    out = Path(args.result_dir)
+    out.mkdir(parents=True, exist_ok=True)
+    atomic_jsonl(out / "requests.jsonl", sorted(records, key=lambda r: r["admitted_s"]))
+    atomic_jsonl(out / "segments.jsonl", segments)
+    atomic_json(out / "result.json", result)
+    atomic_json(out / "sanity.json", sanity)
+    if ctx.exhausted:
+        raise ManifestExhausted("manifest exhausted; result retained for diagnosis")
+    failed = [name for name, ok in sanity["invariants"].items() if not ok]
+    if failed:
+        raise RuntimeError(f"client sanity failure: {failed}")
+    return result
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser()
+    sub = parser.add_subparsers(dest="command", required=True)
+    mat = sub.add_parser("materialize")
+    mat.add_argument("--id", required=True)
+    mat.add_argument("--kind", choices=("synthetic", "prefix-pool"), required=True)
+    group = mat.add_mutually_exclusive_group()
+    group.add_argument("--input-uniform")
+    group.add_argument("--input-fixed", type=int)
+    group.add_argument("--input-mixture")
+    mat.add_argument("--output-fixed", type=int, required=True)
+    mat.add_argument("--prefix", default="none")
+    mat.add_argument("--arrival", required=True)
+    mat.add_argument("--num-requests", type=int, required=True)
+    mat.add_argument("--workload-seed", type=int, required=True)
+    mat.add_argument("--num-prefixes", type=int, default=0)
+    mat.add_argument("--prefix-len", type=int, default=0)
+    mat.add_argument("--suffix-fixed", type=int, default=0)
+    mat.add_argument("--out", required=True)
+    private = sub.add_parser("materialize-private")
+    private.add_argument("--id", required=True)
+    private.add_argument("--source", required=True)
+    private.add_argument("--sampling-u-max", type=float, required=True)
+    private.add_argument("--max-input-tokens", type=int, required=True)
+    private.add_argument("--output-cap", type=int, required=True)
+    private.add_argument("--preserve-prompts", action="store_true", required=True)
+    private.add_argument("--disable-shuffle", action="store_true", required=True)
+    private.add_argument("--arrival", required=True)
+    private.add_argument("--model", required=True)
+    private.add_argument("--out", required=True)
+    run = sub.add_parser("run")
+    run.add_argument("--manifest", required=True)
+    run.add_argument("--base-url", required=True)
+    run.add_argument("--model", required=True)
+    run.add_argument("--load-point", choices=("saturation", "moderate"), required=True)
+    run.add_argument("--request-rate")
+    run.add_argument("--saturation-result")
+    run.add_argument("--rate-fraction", type=float, default=0.60)
+    run.add_argument("--max-concurrency", type=int, default=256)
+    run.add_argument("--ignore-eos", action="store_true")
+    run.add_argument("--temperature", type=float, default=0.0)
+    run.add_argument("--warmup-seconds", type=float, default=60)
+    run.add_argument("--clean-segment-seconds", type=float, default=80)
+    run.add_argument("--num-clean-segments", type=int, default=3)
+    run.add_argument("--profile-after-clean", action="store_true")
+    run.add_argument("--num-profile-windows", type=int, default=0)
+    run.add_argument("--profile-warmup-iterations", type=int, default=2)
+    run.add_argument("--profile-active-iterations", type=int, default=8)
+    run.add_argument("--profile-trace-dir")
+    run.add_argument("--profile-timeout-seconds", type=float, default=120)
+    run.add_argument("--recovery-seconds", type=float, default=30)
+    run.add_argument("--post-clean-seconds", type=float, default=0)
+    run.add_argument("--drain-timeout-seconds", type=float, default=120)
+    run.add_argument("--workload-seed", type=int, default=20260712)
+    run.add_argument("--server-seed", type=int, default=20260712)
+    run.add_argument("--result-dir", required=True)
+    return parser
+
+
+def main() -> None:
+    args = build_parser().parse_args()
+    if args.command == "materialize":
+        materialize(args)
+    elif args.command == "materialize-private":
+        materialize_private(args)
+    else:
+        if args.profile_after_clean and not args.profile_trace_dir:
+            raise ValueError("--profile-after-clean requires --profile-trace-dir")
+        print(json.dumps(asyncio.run(run_load(args)), sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/opprof-phase3/provenance/test_phase3_analysis.py

@@ -0,0 +1,109 @@
+from __future__ import annotations
+
+import json
+import tempfile
+import unittest
+from pathlib import Path
+
+import analyze_phase3 as analysis
+
+
+class Phase3AnalysisTests(unittest.TestCase):
+    def test_ap36_stability_formula(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            root = Path(tmp)
+            (root / "client").mkdir()
+            (root / "opprof").mkdir()
+            (root / "client/result.json").write_text(
+                json.dumps({"t0_mono_ns": 0, "warmup_seconds": 60})
+            )
+            (root / "client/requests.jsonl").write_text(
+                "".join(
+                    json.dumps({"success": True, "completed_s": index + 1}) + "\n"
+                    for index in range(16)
+                )
+            )
+            records = []
+            for bin_index in range(3):
+                for step in range(16):
+                    records.append(
+                        {
+                            "step_index": len(records),
+                            "model_executed": True,
+                            "submit_mono_ns": int(
+                                (45 + 5 * bin_index + (step + 0.5) / 16 * 5)
+                                * 1e9
+                            ),
+                            "prefill_tokens": 100,
+                            "decode_tokens": 0,
+                        }
+                    )
+            (root / "opprof/test.jsonl").write_text(
+                "".join(json.dumps(item) + "\n" for item in records)
+            )
+            result = analysis.ap36_warmup_stability(root)
+            self.assertTrue(result["passes"])
+            self.assertEqual(result["normalized_drift"], 0)
+
+    def test_ap37_partial_verdict_can_confirm_but_not_refute(self):
+        self.assertEqual(analysis.partial_verdict(True), "PASS")
+        self.assertEqual(analysis.partial_verdict(False), "INCONCLUSIVE")
+
+    def test_accepted_markers_come_only_from_complete_stages(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            root = Path(tmp)
+            complete = root / "stages/primary-01-saturation"
+            complete.mkdir(parents=True)
+            (complete / "stage-complete.json").write_text(
+                json.dumps({"runs": ["P01-C00-saturation"]})
+            )
+            accepted = root / "primary/P01-C00/saturation"
+            accepted.mkdir(parents=True)
+            (accepted / "run-complete.json").write_text(
+                json.dumps({"run_id": "P01-C00-saturation"})
+            )
+            unaccepted = root / "primary/P05-C00/saturation"
+            unaccepted.mkdir(parents=True)
+            (unaccepted / "run-complete.json").write_text(
+                json.dumps({"run_id": "P05-C00-saturation"})
+            )
+            primary, confirmations, stages, excluded = analysis.accepted_marker_paths(
+                root
+            )
+            self.assertEqual(primary, [accepted / "run-complete.json"])
+            self.assertEqual(confirmations, [])
+            self.assertEqual(stages, [complete])
+            self.assertEqual(excluded, ["P05-C00-saturation"])
+
+    def test_r64_is_ratio_of_cohort_sums(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "manifest.jsonl"
+            rows = [{"input_tokens": value} for value in (1, 3, 2, 2)]
+            path.write_text("".join(json.dumps(row) + "\n" for row in rows))
+            value, pieces = analysis.manifest_raggedness(path, 2)
+            self.assertEqual(pieces, [(2.0, 6.0), (0.0, 4.0)])
+            self.assertAlmostEqual(value, 0.2)
+
+    def test_one_percentage_point_ranking_ties(self):
+        shares = dict.fromkeys(analysis.FAMILIES, 0.0)
+        shares.update(attention=0.40, moe_gemm=0.395, moe_router=0.20)
+        ranked = {item["family"]: item["rank"] for item in analysis.ranked_families(shares)}
+        self.assertEqual(ranked["attention"], ranked["moe_gemm"])
+        self.assertGreater(ranked["moe_router"], ranked["attention"])
+
+    def test_holm_uses_declared_total_test_family(self):
+        values = [{"p": 0.001}, {"p": 0.01}]
+        analysis.holm(values, total_tests=10)
+        self.assertAlmostEqual(values[0]["p_holm"], 0.01)
+        self.assertAlmostEqual(values[1]["p_holm"], 0.09)
+
+    def test_robust_spline_prediction_is_nonnegative(self):
+        rows = [(float(x), float(n), float(2 * x + n)) for x in range(1, 20) for n in (1, 4)]
+        predict, hull = analysis.fit_nonnegative_robust(rows)
+        self.assertGreaterEqual(predict(3, 2), 0)
+        self.assertTrue(analysis.inside_convex(hull, (3, 2)))
+        self.assertFalse(analysis.inside_convex(hull, (100, 2)))
+
+
+if __name__ == "__main__":
+    unittest.main()

runs/opprof-phase3/provenance/test_phase3_tools.py

@@ -0,0 +1,486 @@
+#!/usr/bin/env python3
+
+from __future__ import annotations
+
+import asyncio
+import importlib.util
+import json
+import sys
+import tempfile
+import unittest
+from unittest import mock
+from pathlib import Path
+from types import SimpleNamespace
+
+from aiohttp import web
+
+
+HERE = Path(__file__).parent
+
+
+def load_module(name: str, filename: str):
+    spec = importlib.util.spec_from_file_location(name, HERE / filename)
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[name] = module
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+client = load_module("phase3_client", "opprof_phase3_client.py")
+controller = load_module("phase3_controller", "opprof_phase3_controller.py")
+sys.modules["opprof_phase3_controller"] = controller
+matrix = load_module("phase3_matrix", "opprof_phase3_matrix.py")
+
+
+def rows(n: int, arrival: str = "steady") -> list[dict]:
+    return [
+        {
+            "schema": 1,
+            "request_id": f"T-{i:05d}",
+            "pattern_id": "T",
+            "kind": "synthetic",
+            "input_tokens": 8,
+            "output_tokens": 2,
+            "arrival": arrival,
+            "token_seed": i + 1,
+        }
+        for i in range(n)
+    ]
+
+
+class MockServer:
+    def __init__(self, delay: float = 0.01, trace_dir: Path | None = None) -> None:
+        self.active = 0
+        self.max_active = 0
+        self.payloads = []
+        self.runner = None
+        self.port = None
+        self.delay = delay
+        self.trace_dir = trace_dir
+        self.profile_count = 0
+
+    async def completion(self, request):
+        payload = await request.json()
+        self.payloads.append(payload)
+        self.active += 1
+        self.max_active = max(self.max_active, self.active)
+        await asyncio.sleep(self.delay)
+        response = web.StreamResponse(
+            status=200, headers={"Content-Type": "text/event-stream"}
+        )
+        await response.prepare(request)
+        await response.write(
+            b'data: {"choices":[{"text":"x"}],"usage":null}\n\n'
+        )
+        usage = json.dumps(
+            {
+                "choices": [],
+                "usage": {
+                    "prompt_tokens": len(payload["prompt"]),
+                    "completion_tokens": payload["max_tokens"],
+                },
+            }
+        ).encode()
+        await response.write(b"data: " + usage + b"\n\n")
+        await response.write(b"data: [DONE]\n\n")
+        await response.write_eof()
+        self.active -= 1
+        return response
+
+    async def start_profile(self, request):
+        self.profile_count += 1
+        path = self.trace_dir / f"window-{self.profile_count}.pt.trace.json"
+        path.write_text('{"traceEvents": []}')
+        return web.Response(status=200)
+
+    async def stop_profile(self, request):
+        return web.Response(status=200)
+
+    async def start(self):
+        app = web.Application()
+        app.router.add_post("/v1/completions", self.completion)
+        if self.trace_dir is not None:
+            app.router.add_post("/start_profile", self.start_profile)
+            app.router.add_post("/stop_profile", self.stop_profile)
+        self.runner = web.AppRunner(app)
+        await self.runner.setup()
+        site = web.TCPSite(self.runner, "127.0.0.1", 0)
+        await site.start()
+        self.port = site._server.sockets[0].getsockname()[1]
+
+    async def stop(self):
+        await self.runner.cleanup()
+
+
+def run_args(
+    manifest: Path,
+    result_dir: Path,
+    port: int,
+    load_point: str = "saturation",
+    saturation_result: Path | None = None,
+):
+    return SimpleNamespace(
+        manifest=str(manifest),
+        base_url=f"http://127.0.0.1:{port}",
+        model="mock",
+        load_point=load_point,
+        request_rate="inf" if load_point == "saturation" else None,
+        saturation_result=None if saturation_result is None else str(saturation_result),
+        rate_fraction=0.60,
+        max_concurrency=3,
+        ignore_eos=True,
+        temperature=0,
+        warmup_seconds=0.04,
+        clean_segment_seconds=0.04,
+        num_clean_segments=3,
+        profile_after_clean=False,
+        num_profile_windows=0,
+        profile_warmup_iterations=2,
+        profile_active_iterations=8,
+        profile_trace_dir=None,
+        profile_timeout_seconds=1,
+        recovery_seconds=0,
+        post_clean_seconds=0,
+        drain_timeout_seconds=1,
+        workload_seed=20260712,
+        server_seed=20260712,
+        result_dir=str(result_dir),
+    )
+
+
+class Phase3ToolTests(unittest.TestCase):
+    def write_warmup_stream(self, root: Path, tokens: list[int]) -> None:
+        stream_dir = root / "opprof"
+        stream_dir.mkdir()
+        records = []
+        step = 0
+        for bin_index, token_count in enumerate(tokens):
+            per_step, remainder = divmod(token_count, 16)
+            for in_bin in range(16):
+                records.append(
+                    {
+                        "step_index": step,
+                        "model_executed": True,
+                        "submit_mono_ns": int(
+                            1e9
+                            + (45 + bin_index * 5 + (in_bin + 0.5) / 16 * 5)
+                            * 1e9
+                        ),
+                        "prefill_tokens": per_step + (in_bin < remainder),
+                        "decode_tokens": 0,
+                    }
+                )
+                step += 1
+        (stream_dir / "test.jsonl").write_text(
+            "".join(json.dumps(record) + "\n" for record in records)
+        )
+
+    def test_p10_warmup_stability_accepts_flat_trailing_quartile(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            root = Path(tmp)
+            self.write_warmup_stream(root, [1600, 1600, 1600])
+            result = matrix.p10_warmup_stability(root, int(1e9))
+            self.assertTrue(result["passed"])
+            self.assertEqual(result["step_counts"], [16, 16, 16])
+            self.assertEqual(result["scheduled_token_throughput"], [320, 320, 320])
+            self.assertEqual(result["normalized_drift"], 0)
+
+    def test_p10_warmup_stability_rejects_large_drift(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            root = Path(tmp)
+            self.write_warmup_stream(root, [1600, 3200, 4800])
+            result = matrix.p10_warmup_stability(root, int(1e9))
+            self.assertFalse(result["passed"])
+            self.assertGreater(result["normalized_drift"], 0.10)
+
+    def test_only_clean_window_request_failures_are_hard_failures(self):
+        requests = [
+            {"success": False, "completed_s": 59.9, "error_kind": "warmup"},
+            {"success": False, "completed_s": 60.0, "error_kind": "clean"},
+            {"success": False, "completed_s": 299.9, "error_kind": "clean"},
+            {"success": False, "completed_s": 300.0, "error_kind": "recovery"},
+            {"success": True, "completed_s": 100.0, "error_kind": None},
+        ]
+        summary = matrix.summarize_request_failures(requests, 60.0, 300.0)
+        self.assertEqual(summary["failed"], 4)
+        self.assertEqual(summary["clean_failed"], 2)
+        self.assertEqual(summary["excluded"], 2)
+        self.assertEqual(summary["excluded_kinds"], {"warmup": 1, "recovery": 1})
+
+    def test_synthetic_prompt_exact_and_unique_early_prefix(self):
+        generated = [client.synthetic_prompt(row) for row in rows(200)]
+        self.assertTrue(all(len(value) == 8 for value in generated))
+        self.assertEqual(len({tuple(value[:3]) for value in generated}), 200)
+
+    def test_p05_manifest_has_exact_half_modes(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            out = Path(tmp) / "P05.jsonl"
+            args = SimpleNamespace(
+                id="P05",
+                kind="synthetic",
+                input_uniform=None,
+                input_fixed=None,
+                input_mixture='{"uniform:128:512":0.5,"uniform:4096:8192":0.5}',
+                output_fixed=64,
+                prefix="none",
+                arrival="steady",
+                num_requests=100,
+                workload_seed=20260712,
+                num_prefixes=0,
+                prefix_len=0,
+                suffix_fixed=0,
+                out=str(out),
+            )
+            client.materialize(args)
+            manifest = client.load_manifest(out)
+            self.assertEqual(sum(r["input_tokens"] <= 512 for r in manifest), 50)
+            self.assertEqual(sum(r["input_tokens"] >= 4096 for r in manifest), 50)
+
+    def test_prefix_pool_balanced(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            out = Path(tmp) / "P08.jsonl"
+            args = SimpleNamespace(
+                id="P08",
+                kind="prefix-pool",
+                input_uniform=None,
+                input_fixed=None,
+                input_mixture=None,
+                output_fixed=512,
+                prefix="none",
+                arrival="burst:8",
+                num_requests=80,
+                workload_seed=20260712,
+                num_prefixes=8,
+                prefix_len=1024,
+                suffix_fixed=256,
+                out=str(out),
+            )
+            client.materialize(args)
+            manifest = client.load_manifest(out)
+            counts = {i: 0 for i in range(8)}
+            for row in manifest:
+                counts[row["prefix_id"]] += 1
+                self.assertEqual(row["input_tokens"], 1280)
+            self.assertEqual(set(counts.values()), {10})
+
+    def test_fixed_duration_saturation_and_redaction(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(1000))
+                mock = MockServer()
+                await mock.start()
+                try:
+                    result = await client.run_load(
+                        run_args(manifest, root / "result", mock.port)
+                    )
+                finally:
+                    await mock.stop()
+                self.assertEqual(result["max_in_flight"], 3)
+                self.assertEqual(mock.max_active, 3)
+                self.assertAlmostEqual(result["clean"]["duration_s"], 0.12, places=9)
+                self.assertEqual(len(result["segments"]), 3)
+                self.assertLessEqual(result["drain_seconds"], 1)
+                self.assertTrue(
+                    all(p["max_tokens"] == 2 and p["ignore_eos"] for p in mock.payloads)
+                )
+                text = (root / "result/requests.jsonl").read_text()
+                self.assertNotIn('"prompt":', text)
+                self.assertNotIn('"text":', text)
+
+        asyncio.run(case())
+
+    def test_profile_control_plane_is_not_starved_by_data_connector(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                trace_dir = root / "traces"
+                trace_dir.mkdir()
+                client.atomic_jsonl(manifest, rows(1000))
+                server = MockServer(delay=0.5, trace_dir=trace_dir)
+                await server.start()
+                args = run_args(manifest, root / "result", server.port)
+                args.max_concurrency = 1
+                args.warmup_seconds = 0.01
+                args.clean_segment_seconds = 0.01
+                args.num_clean_segments = 1
+                args.profile_after_clean = True
+                args.num_profile_windows = 1
+                args.profile_trace_dir = str(trace_dir)
+                args.recovery_seconds = 0
+                try:
+                    result = await client.run_load(args)
+                finally:
+                    await server.stop()
+                self.assertEqual(server.max_active, 1)
+                self.assertEqual(server.profile_count, 1)
+                self.assertLess(result["profiles"][0]["start_return_s"], 0.25)
+
+        asyncio.run(case())
+
+    def test_drain_timeout_is_a_hard_sanity_failure(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(1000))
+                mock = MockServer()
+                await mock.start()
+                try:
+                    args = run_args(manifest, root / "result", mock.port)
+                    args.drain_timeout_seconds = 0
+                    with self.assertRaisesRegex(RuntimeError, "drain_within_timeout"):
+                        await client.run_load(args)
+                finally:
+                    await mock.stop()
+                sanity = json.loads((root / "result/sanity.json").read_text())
+                self.assertFalse(sanity["invariants"]["drain_within_timeout"])
+
+        asyncio.run(case())
+
+    def test_burst_schedule_is_eight_at_once(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(1000, "burst:8"))
+                sat = root / "sat.json"
+                client.atomic_json(
+                    sat, {"clean": {"completed_throughput_rps": 100.0}}
+                )
+                mock = MockServer()
+                await mock.start()
+                try:
+                    args = run_args(
+                        manifest, root / "result", mock.port, "moderate", sat
+                    )
+                    args.warmup_seconds = 0
+                    args.clean_segment_seconds = 0.4 / 3
+                    result = await client.run_load(args)
+                finally:
+                    await mock.stop()
+                records = [
+                    json.loads(line)
+                    for line in (root / "result/requests.jsonl").read_text().splitlines()
+                ]
+                groups = {}
+                for record in records:
+                    groups.setdefault(round(record["scheduled_s"], 6), 0)
+                    groups[round(record["scheduled_s"], 6)] += 1
+                self.assertTrue(all(value == 8 for value in groups.values()))
+                starts = sorted(groups)
+                if len(starts) > 1:
+                    self.assertAlmostEqual(starts[1] - starts[0], 8 / 60, places=5)
+                self.assertAlmostEqual(result["request_rate"], 60.0)
+
+        asyncio.run(case())
+
+    def test_manifest_exhaustion_stops_instead_of_wrapping(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(2))
+                mock = MockServer()
+                await mock.start()
+                try:
+                    with self.assertRaises(client.ManifestExhausted):
+                        await client.run_load(
+                            run_args(manifest, root / "result", mock.port)
+                        )
+                finally:
+                    await mock.stop()
+
+        asyncio.run(case())
+
+    def test_cpu_affinity_sets_are_disjoint_and_cover_host(self):
+        values = []
+        for spec in controller.CPU_MAP.values():
+            lo, hi = [int(x) for x in spec.split("-")]
+            values.extend(range(lo, hi + 1))
+        self.assertEqual(sorted(values), list(range(160)))
+        self.assertEqual(len(values), len(set(values)))
+
+    def test_preflight_waits_for_three_stable_zero_samples(self):
+        def sample(memory):
+            return [
+                {
+                    "index": 0,
+                    "uuid": "GPU-test",
+                    "memory_used_mib": memory,
+                    "utilization_pct": 0,
+                }
+            ]
+
+        with tempfile.TemporaryDirectory() as tmp:
+            with (
+                mock.patch.object(
+                    controller,
+                    "gpu_query",
+                    side_effect=[sample(4), sample(0), sample(0), sample(0)],
+                ),
+                mock.patch.object(controller, "compute_apps", return_value=[]),
+                mock.patch.object(controller, "run_text", return_value=""),
+                mock.patch.object(controller.time, "sleep"),
+            ):
+                root = Path(tmp)
+                controller.preflight([0], root)
+                samples = json.loads((root / "gpu-before-samples.json").read_text())
+                self.assertEqual(len(samples), 4)
+                self.assertEqual(samples[0]["gpus"][0]["memory_used_mib"], 4)
+                self.assertEqual(samples[-1]["gpus"][0]["memory_used_mib"], 0)
+
+    def test_kernel_mapping_priority(self):
+        cases = {
+            "void vllm::moe::topkGating": "moe_router",
+            "ncclDevKernel_AllReduce": "collective",
+            "flash_fwd_kernel": "attention",
+            "nvjet_sm90_tst": "moe_gemm",
+            "argmax_kernel": "sampler",
+            "cutlass_gemm": "dense_gemm",
+            "triton_red_fused_add_rms_norm": "norm_elementwise",
+            "cache_swap_kernel": "kv_memory",
+            "unknown": "other",
+        }
+        self.assertEqual(
+            {name: controller.classify_kernel(name) for name in cases}, cases
+        )
+
+    def test_atomic_state_replacement(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "state.json"
+            controller.atomic_json(path, {"schema": 1, "value": 1})
+            controller.atomic_json(path, {"schema": 1, "value": 2})
+            self.assertEqual(json.loads(path.read_text())["value"], 2)
+            self.assertFalse(list(path.parent.glob("*.tmp.*")))
+
+    def test_fingerprint_survives_json_roundtrip(self):
+        original_run_text = controller.run_text
+        original_hash = controller.sha256_file
+        try:
+            controller.run_text = lambda *args, **kwargs: "deadbeef\n"
+            controller.sha256_file = lambda path: "a" * 64
+            fingerprint = controller.make_fingerprint()
+        finally:
+            controller.run_text = original_run_text
+            controller.sha256_file = original_hash
+        self.assertEqual(json.loads(json.dumps(fingerprint)), fingerprint)
+        self.assertEqual(set(fingerprint["cpu_map"]), {str(i) for i in range(8)})
+
+    def test_server_shutdown_signals_parent_before_group(self):
+        process = SimpleNamespace(pid=12345, poll=lambda: None)
+        with (
+            mock.patch.object(controller.os, "kill") as kill,
+            mock.patch.object(controller.os, "killpg") as killpg,
+            mock.patch.object(controller, "_process_group_alive", return_value=False),
+        ):
+            controller.stop_servers([process])
+        kill.assert_called_once_with(12345, controller.signal.SIGINT)
+        killpg.assert_not_called()
+
+
+if __name__ == "__main__":
+    unittest.main(verbosity=2)

runs/opprof-phase3/provenance/test_phase3_tools_ea.py

@@ -0,0 +1,353 @@
+#!/usr/bin/env python3
+
+from __future__ import annotations
+
+import asyncio
+import importlib.util
+import json
+import sys
+import tempfile
+import unittest
+from unittest import mock
+from pathlib import Path
+from types import SimpleNamespace
+
+from aiohttp import web
+
+
+HERE = Path(__file__).parent
+
+
+def load_module(name: str, filename: str):
+    spec = importlib.util.spec_from_file_location(name, HERE / filename)
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[name] = module
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+client = load_module("phase3_client", "opprof_phase3_client.py")
+controller = load_module("phase3_controller", "opprof_phase3_controller.py")
+
+
+def rows(n: int, arrival: str = "steady") -> list[dict]:
+    return [
+        {
+            "schema": 1,
+            "request_id": f"T-{i:05d}",
+            "pattern_id": "T",
+            "kind": "synthetic",
+            "input_tokens": 8,
+            "output_tokens": 2,
+            "arrival": arrival,
+            "token_seed": i + 1,
+        }
+        for i in range(n)
+    ]
+
+
+class MockServer:
+    def __init__(self) -> None:
+        self.active = 0
+        self.max_active = 0
+        self.payloads = []
+        self.runner = None
+        self.port = None
+
+    async def completion(self, request):
+        payload = await request.json()
+        self.payloads.append(payload)
+        self.active += 1
+        self.max_active = max(self.max_active, self.active)
+        await asyncio.sleep(0.01)
+        response = web.StreamResponse(
+            status=200, headers={"Content-Type": "text/event-stream"}
+        )
+        await response.prepare(request)
+        await response.write(
+            b'data: {"choices":[{"text":"x"}],"usage":null}\n\n'
+        )
+        usage = json.dumps(
+            {
+                "choices": [],
+                "usage": {
+                    "prompt_tokens": len(payload["prompt"]),
+                    "completion_tokens": payload["max_tokens"],
+                },
+            }
+        ).encode()
+        await response.write(b"data: " + usage + b"\n\n")
+        await response.write(b"data: [DONE]\n\n")
+        await response.write_eof()
+        self.active -= 1
+        return response
+
+    async def start(self):
+        app = web.Application()
+        app.router.add_post("/v1/completions", self.completion)
+        self.runner = web.AppRunner(app)
+        await self.runner.setup()
+        site = web.TCPSite(self.runner, "127.0.0.1", 0)
+        await site.start()
+        self.port = site._server.sockets[0].getsockname()[1]
+
+    async def stop(self):
+        await self.runner.cleanup()
+
+
+def run_args(
+    manifest: Path,
+    result_dir: Path,
+    port: int,
+    load_point: str = "saturation",
+    saturation_result: Path | None = None,
+):
+    return SimpleNamespace(
+        manifest=str(manifest),
+        base_url=f"http://127.0.0.1:{port}",
+        model="mock",
+        load_point=load_point,
+        request_rate="inf" if load_point == "saturation" else None,
+        saturation_result=None if saturation_result is None else str(saturation_result),
+        rate_fraction=0.60,
+        max_concurrency=3,
+        ignore_eos=True,
+        temperature=0,
+        warmup_seconds=0.04,
+        clean_segment_seconds=0.04,
+        num_clean_segments=3,
+        profile_after_clean=False,
+        num_profile_windows=0,
+        profile_warmup_iterations=2,
+        profile_active_iterations=8,
+        profile_trace_dir=None,
+        profile_timeout_seconds=1,
+        recovery_seconds=0,
+        post_clean_seconds=0,
+        drain_timeout_seconds=1,
+        workload_seed=20260712,
+        server_seed=20260712,
+        result_dir=str(result_dir),
+    )
+
+
+class Phase3ToolTests(unittest.TestCase):
+    def test_synthetic_prompt_exact_and_unique_early_prefix(self):
+        generated = [client.synthetic_prompt(row) for row in rows(200)]
+        self.assertTrue(all(len(value) == 8 for value in generated))
+        self.assertEqual(len({tuple(value[:3]) for value in generated}), 200)
+
+    def test_p05_manifest_has_exact_half_modes(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            out = Path(tmp) / "P05.jsonl"
+            args = SimpleNamespace(
+                id="P05",
+                kind="synthetic",
+                input_uniform=None,
+                input_fixed=None,
+                input_mixture='{"uniform:128:512":0.5,"uniform:4096:8192":0.5}',
+                output_fixed=64,
+                prefix="none",
+                arrival="steady",
+                num_requests=100,
+                workload_seed=20260712,
+                num_prefixes=0,
+                prefix_len=0,
+                suffix_fixed=0,
+                out=str(out),
+            )
+            client.materialize(args)
+            manifest = client.load_manifest(out)
+            self.assertEqual(sum(r["input_tokens"] <= 512 for r in manifest), 50)
+            self.assertEqual(sum(r["input_tokens"] >= 4096 for r in manifest), 50)
+
+    def test_prefix_pool_balanced(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            out = Path(tmp) / "P08.jsonl"
+            args = SimpleNamespace(
+                id="P08",
+                kind="prefix-pool",
+                input_uniform=None,
+                input_fixed=None,
+                input_mixture=None,
+                output_fixed=512,
+                prefix="none",
+                arrival="burst:8",
+                num_requests=80,
+                workload_seed=20260712,
+                num_prefixes=8,
+                prefix_len=1024,
+                suffix_fixed=256,
+                out=str(out),
+            )
+            client.materialize(args)
+            manifest = client.load_manifest(out)
+            counts = {i: 0 for i in range(8)}
+            for row in manifest:
+                counts[row["prefix_id"]] += 1
+                self.assertEqual(row["input_tokens"], 1280)
+            self.assertEqual(set(counts.values()), {10})
+
+    def test_fixed_duration_saturation_and_redaction(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(1000))
+                mock = MockServer()
+                await mock.start()
+                try:
+                    result = await client.run_load(
+                        run_args(manifest, root / "result", mock.port)
+                    )
+                finally:
+                    await mock.stop()
+                self.assertEqual(result["max_in_flight"], 3)
+                self.assertEqual(mock.max_active, 3)
+                self.assertAlmostEqual(result["clean"]["duration_s"], 0.12, places=9)
+                self.assertEqual(len(result["segments"]), 3)
+                self.assertLessEqual(result["drain_seconds"], 1)
+                self.assertTrue(
+                    all(p["max_tokens"] == 2 and p["ignore_eos"] for p in mock.payloads)
+                )
+                text = (root / "result/requests.jsonl").read_text()
+                self.assertNotIn('"prompt":', text)
+                self.assertNotIn('"text":', text)
+
+        asyncio.run(case())
+
+    def test_drain_timeout_is_a_hard_sanity_failure(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(1000))
+                mock = MockServer()
+                await mock.start()
+                try:
+                    args = run_args(manifest, root / "result", mock.port)
+                    args.drain_timeout_seconds = 0
+                    with self.assertRaisesRegex(RuntimeError, "drain_within_timeout"):
+                        await client.run_load(args)
+                finally:
+                    await mock.stop()
+                sanity = json.loads((root / "result/sanity.json").read_text())
+                self.assertFalse(sanity["invariants"]["drain_within_timeout"])
+
+        asyncio.run(case())
+
+    def test_burst_schedule_is_eight_at_once(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(1000, "burst:8"))
+                sat = root / "sat.json"
+                client.atomic_json(
+                    sat, {"clean": {"completed_throughput_rps": 100.0}}
+                )
+                mock = MockServer()
+                await mock.start()
+                try:
+                    args = run_args(
+                        manifest, root / "result", mock.port, "moderate", sat
+                    )
+                    args.warmup_seconds = 0
+                    args.clean_segment_seconds = 0.4 / 3
+                    result = await client.run_load(args)
+                finally:
+                    await mock.stop()
+                records = [
+                    json.loads(line)
+                    for line in (root / "result/requests.jsonl").read_text().splitlines()
+                ]
+                groups = {}
+                for record in records:
+                    groups.setdefault(round(record["scheduled_s"], 6), 0)
+                    groups[round(record["scheduled_s"], 6)] += 1
+                self.assertTrue(all(value == 8 for value in groups.values()))
+                starts = sorted(groups)
+                if len(starts) > 1:
+                    self.assertAlmostEqual(starts[1] - starts[0], 8 / 60, places=5)
+                self.assertAlmostEqual(result["request_rate"], 60.0)
+
+        asyncio.run(case())
+
+    def test_manifest_exhaustion_stops_instead_of_wrapping(self):
+        async def case():
+            with tempfile.TemporaryDirectory() as tmp:
+                root = Path(tmp)
+                manifest = root / "m.jsonl"
+                client.atomic_jsonl(manifest, rows(2))
+                mock = MockServer()
+                await mock.start()
+                try:
+                    with self.assertRaises(client.ManifestExhausted):
+                        await client.run_load(
+                            run_args(manifest, root / "result", mock.port)
+                        )
+                finally:
+                    await mock.stop()
+
+        asyncio.run(case())
+
+    def test_cpu_affinity_sets_are_disjoint_and_cover_host(self):
+        values = []
+        for spec in controller.CPU_MAP.values():
+            lo, hi = [int(x) for x in spec.split("-")]
+            values.extend(range(lo, hi + 1))
+        self.assertEqual(sorted(values), list(range(160)))
+        self.assertEqual(len(values), len(set(values)))
+
+    def test_kernel_mapping_priority(self):
+        cases = {
+            "void vllm::moe::topkGating": "moe_router",
+            "ncclDevKernel_AllReduce": "collective",
+            "flash_fwd_kernel": "attention",
+            "nvjet_sm90_tst": "moe_gemm",
+            "argmax_kernel": "sampler",
+            "cutlass_gemm": "dense_gemm",
+            "triton_red_fused_add_rms_norm": "norm_elementwise",
+            "cache_swap_kernel": "kv_memory",
+            "unknown": "other",
+        }
+        self.assertEqual(
+            {name: controller.classify_kernel(name) for name in cases}, cases
+        )
+
+    def test_atomic_state_replacement(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "state.json"
+            controller.atomic_json(path, {"schema": 1, "value": 1})
+            controller.atomic_json(path, {"schema": 1, "value": 2})
+            self.assertEqual(json.loads(path.read_text())["value"], 2)
+            self.assertFalse(list(path.parent.glob("*.tmp.*")))
+
+    def test_fingerprint_survives_json_roundtrip(self):
+        original_run_text = controller.run_text
+        original_hash = controller.sha256_file
+        try:
+            controller.run_text = lambda *args, **kwargs: "deadbeef\n"
+            controller.sha256_file = lambda path: "a" * 64
+            fingerprint = controller.make_fingerprint()
+        finally:
+            controller.run_text = original_run_text
+            controller.sha256_file = original_hash
+        self.assertEqual(json.loads(json.dumps(fingerprint)), fingerprint)
+        self.assertEqual(set(fingerprint["cpu_map"]), {str(i) for i in range(8)})
+
+    def test_server_shutdown_signals_parent_before_group(self):
+        process = SimpleNamespace(pid=12345, poll=lambda: None)
+        with (
+            mock.patch.object(controller.os, "kill") as kill,
+            mock.patch.object(controller.os, "killpg") as killpg,
+            mock.patch.object(controller, "_process_group_alive", return_value=False),
+        ):
+            controller.stop_servers([process])
+        kill.assert_called_once_with(12345, controller.signal.SIGINT)
+        killpg.assert_not_called()
+
+
+if __name__ == "__main__":
+    unittest.main(verbosity=2)

runs/opprof-phase3/provenance/verify_shutdown_footer.py

@@ -0,0 +1,353 @@
+#!/usr/bin/env python3
+"""One-GPU verification of API-parent-first OpProf shutdown."""
+
+from __future__ import annotations
+
+import json
+import os
+import shlex
+import shutil
+import signal
+import subprocess
+import time
+import urllib.request
+from pathlib import Path
+from typing import Any
+
+WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712")
+RUN_DIR = WORKDIR / "runs/e-b-shutdown-verification"
+SOURCE = Path(
+    "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0"
+)
+VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv")
+MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B")
+CLIENT = WORKDIR / "scripts/opprof_phase3_client.py"
+MANIFEST = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl")
+PORT = 8010
+
+
+def atomic_json(path: Path, value: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_name(path.name + f".tmp.{os.getpid()}")
+    with tmp.open("w", encoding="utf-8") as f:
+        json.dump(value, f, sort_keys=True, indent=2)
+        f.write("\n")
+        f.flush()
+        os.fsync(f.fileno())
+    os.replace(tmp, path)
+
+
+def run_text(command: list[str], check: bool = True) -> str:
+    result = subprocess.run(
+        command, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
+    )
+    if check and result.returncode:
+        raise RuntimeError(
+            f"command failed ({result.returncode}): {shlex.join(command)}\n"
+            f"{result.stdout}"
+        )
+    return result.stdout
+
+
+def gpu_rows() -> list[dict[str, int]]:
+    output = run_text(
+        [
+            "nvidia-smi",
+            "--query-gpu=index,memory.used,utilization.gpu",
+            "--format=csv,noheader,nounits",
+        ]
+    )
+    rows = []
+    for line in output.strip().splitlines():
+        index, memory, utilization = [int(part.strip()) for part in line.split(",")]
+        rows.append(
+            {"index": index, "memory_mib": memory, "utilization_pct": utilization}
+        )
+    return rows
+
+
+def compute_apps() -> str:
+    return run_text(
+        [
+            "nvidia-smi",
+            "--query-compute-apps=gpu_uuid,pid,process_name,used_memory",
+            "--format=csv,noheader,nounits",
+        ],
+        check=False,
+    ).strip()
+
+
+def group_alive(pgid: int, process: subprocess.Popen[Any] | None = None) -> bool:
+    if process is not None:
+        process.poll()  # reap an exited API parent before probing its group
+    try:
+        os.killpg(pgid, 0)
+        return True
+    except ProcessLookupError:
+        return False
+
+
+def wait_ready(process: subprocess.Popen[Any], timeout: float = 300) -> None:
+    deadline = time.monotonic() + timeout
+    while time.monotonic() < deadline:
+        if process.poll() is not None:
+            raise RuntimeError("server exited before readiness")
+        try:
+            with urllib.request.urlopen(
+                f"http://127.0.0.1:{PORT}/health", timeout=1
+            ) as response:
+                if response.status == 200:
+                    return
+        except Exception:
+            pass
+        time.sleep(1)
+    raise TimeoutError("server readiness timeout")
+
+
+def shutdown_parent_first(process: subprocess.Popen[Any]) -> dict[str, Any]:
+    called_at = time.time()
+    os.kill(process.pid, signal.SIGINT)
+    deadline = time.monotonic() + 120
+    while time.monotonic() < deadline and group_alive(process.pid, process):
+        time.sleep(1)
+    graceful = not group_alive(process.pid, process)
+    if not graceful:
+        for sig in (signal.SIGTERM, signal.SIGKILL):
+            try:
+                os.killpg(process.pid, sig)
+            except ProcessLookupError:
+                break
+            time.sleep(5)
+    return {
+        "api_parent_pid": process.pid,
+        "sigint_wall_time": called_at,
+        "group_gone_wall_time": time.time(),
+        "graceful": graceful,
+        "returncode": process.poll(),
+    }
+
+
+def wait_stable_zero() -> list[list[dict[str, int]]]:
+    samples = []
+    consecutive = 0
+    deadline = time.monotonic() + 120
+    while time.monotonic() < deadline and consecutive < 3:
+        rows = gpu_rows()
+        samples.append(rows)
+        zero = rows[0]["memory_mib"] == 0 and not compute_apps()
+        consecutive = consecutive + 1 if zero else 0
+        if consecutive < 3:
+            time.sleep(5)
+    if consecutive < 3:
+        raise RuntimeError("GPU0 did not reach three stable zero samples")
+    return samples
+
+
+def validate_footer() -> dict[str, Any]:
+    files = sorted((RUN_DIR / "opprof").glob("*.jsonl"))
+    if len(files) != 1:
+        raise RuntimeError(f"expected one OpProf JSONL, found {len(files)}")
+    lines = files[0].read_text().splitlines()
+    decoded = [json.loads(line) for line in lines]
+    if not decoded or decoded[-1].get("record_type") != "footer":
+        raise RuntimeError("Layer-1 footer missing")
+    records, footer = decoded[:-1], decoded[-1]
+    indices = [record["step_index"] for record in records]
+    invariants = {
+        "all_schema_1": all(item.get("schema") == 1 for item in decoded),
+        "one_footer_last": sum(
+            item.get("record_type") == "footer" for item in decoded
+        )
+        == 1,
+        "steps_contiguous": indices == list(range(len(indices))),
+        "written_matches_records": footer["written_records"] == len(records),
+        "encoded_balanced": footer["encoded_records"]
+        == footer["written_records"] + footer["dropped_records"],
+        "zero_drops": footer["dropped_records"] == 0
+        and all(record["dropped_records_before"] == 0 for record in records),
+    }
+    if not all(invariants.values()):
+        raise RuntimeError(f"footer accounting invalid: {invariants}, {footer}")
+    return {
+        "path": str(files[0]),
+        "bytes": files[0].stat().st_size,
+        "records": len(records),
+        "footer": footer,
+        "invariants": invariants,
+    }
+
+
+def main() -> None:
+    if RUN_DIR.exists():
+        raise RuntimeError(f"refusing to overwrite {RUN_DIR}")
+    RUN_DIR.mkdir(parents=True)
+    state: dict[str, Any] = {
+        "schema": 1,
+        "status": "preflight",
+        "started_at": time.time(),
+        "gpu": 0,
+    }
+    atomic_json(RUN_DIR / "state.json", state)
+    before = gpu_rows()
+    if before[0]["memory_mib"] != 0 or before[0]["utilization_pct"] != 0:
+        raise RuntimeError(f"GPU0 is not idle: {before[0]}")
+    if compute_apps():
+        raise RuntimeError("a compute process exists before verification")
+    (RUN_DIR / "gpu-before.json").write_text(json.dumps(before, indent=2) + "\n")
+    (RUN_DIR / "clocks-before.txt").write_text(
+        run_text(["nvidia-smi", "-q", "-d", "CLOCK"])
+    )
+    profile_config = {
+        "profiler": "torch",
+        "torch_profiler_dir": "/tmp/wjh-opprof-p3-footer-verify",
+        "ignore_frontend": True,
+        "wait_iterations": 0,
+        "warmup_iterations": 2,
+        "active_iterations": 8,
+    }
+    trace_dir = Path(profile_config["torch_profiler_dir"])
+    if trace_dir.exists():
+        shutil.rmtree(trace_dir)
+    trace_dir.mkdir()
+    server_command = [
+        "taskset",
+        "-c",
+        "0-19",
+        str(VENV / "bin/vllm"),
+        "serve",
+        str(MODEL),
+        "--host",
+        "127.0.0.1",
+        "--port",
+        str(PORT),
+        "--tensor-parallel-size",
+        "1",
+        "--enable-chunked-prefill",
+        "--enable-prefix-caching",
+        "--profiler-config",
+        json.dumps(profile_config, separators=(",", ":")),
+    ]
+    client_command = [
+        "taskset",
+        "-c",
+        "0-19",
+        str(VENV / "bin/python"),
+        str(CLIENT),
+        "run",
+        "--manifest",
+        str(MANIFEST),
+        "--base-url",
+        f"http://127.0.0.1:{PORT}",
+        "--model",
+        str(MODEL),
+        "--load-point",
+        "saturation",
+        "--request-rate",
+        "inf",
+        "--max-concurrency",
+        "256",
+        "--ignore-eos",
+        "--temperature",
+        "0",
+        "--warmup-seconds",
+        "30",
+        "--clean-segment-seconds",
+        "40",
+        "--num-clean-segments",
+        "3",
+        "--drain-timeout-seconds",
+        "120",
+        "--workload-seed",
+        "20260712",
+        "--result-dir",
+        str(RUN_DIR / "client"),
+    ]
+    with (RUN_DIR / "commands.log").open("w") as f:
+        f.write(
+            "GPU_COMMAND shutdown footer server: "
+            + shlex.join(server_command)
+            + " ; expected=60-180s startup + 150-180s load\n"
+        )
+        f.write(
+            "GPU_COMMAND shutdown footer client: "
+            + shlex.join(client_command)
+            + " ; expected=30s warmup + 120s clean + drain\n"
+        )
+    server_log = (RUN_DIR / "server.log").open("ab", buffering=0)
+    env = os.environ.copy()
+    env.update(
+        {
+            "CUDA_VISIBLE_DEVICES": "0",
+            "VLLM_OPPROF_DIR": str(RUN_DIR / "opprof"),
+            "HF_HUB_OFFLINE": "1",
+            "TRANSFORMERS_OFFLINE": "1",
+            "PYTHONUNBUFFERED": "1",
+        }
+    )
+    server: subprocess.Popen[Any] | None = None
+    try:
+        print(
+            "GPU_COMMAND shutdown-fix verification: P01/C00 GPU0, "
+            "30s warmup + 120s clean, expected 4-6 wall-min",
+            flush=True,
+        )
+        server = subprocess.Popen(
+            server_command,
+            cwd=SOURCE,
+            env=env,
+            stdout=server_log,
+            stderr=subprocess.STDOUT,
+            start_new_session=True,
+        )
+        state.update({"status": "server_starting", "server_pid": server.pid})
+        atomic_json(RUN_DIR / "state.json", state)
+        wait_ready(server)
+        state.update({"status": "client_running", "server_ready_at": time.time()})
+        atomic_json(RUN_DIR / "state.json", state)
+        with (RUN_DIR / "client.log").open("ab", buffering=0) as client_log:
+            result = subprocess.run(
+                client_command,
+                cwd=WORKDIR,
+                stdout=client_log,
+                stderr=subprocess.STDOUT,
+            )
+        if result.returncode:
+            raise RuntimeError(f"client exited {result.returncode}")
+        state["status"] = "parent_first_shutdown"
+        atomic_json(RUN_DIR / "state.json", state)
+        shutdown = shutdown_parent_first(server)
+        if not shutdown["graceful"]:
+            raise RuntimeError(f"server required escalation: {shutdown}")
+        footer = validate_footer()
+        zero_samples = wait_stable_zero()
+        client_result = json.loads((RUN_DIR / "client/result.json").read_text())
+        result_json = {
+            "schema": 1,
+            "status": "pass",
+            "gpu": 0,
+            "shutdown": shutdown,
+            "footer": footer,
+            "clean": client_result["clean"],
+            "failed_records": client_result["failed_records"],
+            "gpu_zero_samples": zero_samples,
+            "gpu_seconds": zero_samples and time.time() - state["started_at"],
+        }
+        atomic_json(RUN_DIR / "result.json", result_json)
+        state.update({"status": "complete", "completed_at": time.time()})
+        atomic_json(RUN_DIR / "state.json", state)
+        print(json.dumps(result_json, sort_keys=True), flush=True)
+    except Exception as error:
+        state.update({"status": "failed", "failure": repr(error)})
+        atomic_json(RUN_DIR / "state.json", state)
+        if server is not None and group_alive(server.pid, server):
+            try:
+                os.killpg(server.pid, signal.SIGKILL)
+            except ProcessLookupError:
+                pass
+        raise
+    finally:
+        server_log.close()
+
+
+if __name__ == "__main__":
+    main()

runs/opprof-phase3/provenance/verify_sidecar_shutdown.py

@@ -0,0 +1,392 @@
+#!/usr/bin/env python3
+"""GPU verification of graceful and hard-kill OpProf accounting."""
+
+from __future__ import annotations
+
+import json
+import os
+import shlex
+import signal
+import subprocess
+import time
+import urllib.request
+from pathlib import Path
+from typing import Any
+
+WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712")
+RUN_ROOT = WORKDIR / "runs/e-b-sidecar-verification"
+SOURCE = Path(
+    "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0"
+)
+VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv")
+MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B")
+CLIENT = WORKDIR / "scripts/opprof_phase3_client.py"
+MANIFEST = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl")
+FLUSH_INTERVAL_SECONDS = 1.0
+CHECKPOINT_TOLERANCE_SECONDS = 0.1
+
+
+def atomic_json(path: Path, value: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_name(f"{path.name}.tmp-{os.getpid()}")
+    with temporary.open("w", encoding="utf-8") as output:
+        json.dump(value, output, indent=2, sort_keys=True)
+        output.write("\n")
+        output.flush()
+        os.fsync(output.fileno())
+    os.replace(temporary, path)
+
+
+def run_text(command: list[str], check: bool = True) -> str:
+    result = subprocess.run(
+        command,
+        text=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+    )
+    if check and result.returncode:
+        raise RuntimeError(
+            f"command failed ({result.returncode}): {shlex.join(command)}\n"
+            f"{result.stdout}"
+        )
+    return result.stdout
+
+
+def gpu_rows() -> list[dict[str, int]]:
+    output = run_text(
+        [
+            "nvidia-smi",
+            "--query-gpu=index,memory.used,utilization.gpu",
+            "--format=csv,noheader,nounits",
+        ]
+    )
+    rows = []
+    for line in output.strip().splitlines():
+        index, memory, utilization = (int(value.strip()) for value in line.split(","))
+        rows.append(
+            {
+                "index": index,
+                "memory_mib": memory,
+                "utilization_pct": utilization,
+            }
+        )
+    return rows
+
+
+def compute_apps() -> str:
+    return run_text(
+        [
+            "nvidia-smi",
+            "--query-compute-apps=gpu_uuid,pid,process_name,used_memory",
+            "--format=csv,noheader,nounits",
+        ],
+        check=False,
+    ).strip()
+
+
+def assert_idle() -> list[dict[str, int]]:
+    rows = gpu_rows()
+    if any(row["memory_mib"] or row["utilization_pct"] for row in rows):
+        raise RuntimeError(f"dash0 is not GPU-idle: {rows}")
+    applications = compute_apps()
+    if applications:
+        raise RuntimeError(f"compute applications present: {applications}")
+    return rows
+
+
+def wait_ready(process: subprocess.Popen[Any], port: int) -> None:
+    deadline = time.monotonic() + 300
+    while time.monotonic() < deadline:
+        if process.poll() is not None:
+            raise RuntimeError("server exited before readiness")
+        try:
+            with urllib.request.urlopen(
+                f"http://127.0.0.1:{port}/health", timeout=1
+            ) as response:
+                if response.status == 200:
+                    return
+        except Exception:
+            pass
+        time.sleep(1)
+    raise TimeoutError("server readiness timeout")
+
+
+def wait_zero() -> tuple[list[list[dict[str, int]]], float]:
+    samples: list[list[dict[str, int]]] = []
+    consecutive = 0
+    deadline = time.monotonic() + 180
+    while time.monotonic() < deadline and consecutive < 3:
+        rows = gpu_rows()
+        samples.append(rows)
+        zero = all(row["memory_mib"] == 0 for row in rows) and not compute_apps()
+        consecutive = consecutive + 1 if zero else 0
+        if consecutive < 3:
+            time.sleep(2)
+    if consecutive < 3:
+        raise RuntimeError("GPUs did not reach three stable zero samples")
+    return samples, time.time()
+
+
+def wait_fresh_sidecar(run_dir: Path) -> dict[str, Any]:
+    deadline = time.monotonic() + 10
+    while time.monotonic() < deadline:
+        files = sorted((run_dir / "opprof").glob("*.jsonl.footer.json"))
+        if len(files) == 1:
+            sidecar = json.loads(files[0].read_text())
+            age = time.time_ns() - sidecar["checkpoint_wall_ns"]
+            if 0 <= age <= 250_000_000:
+                return sidecar
+        time.sleep(0.02)
+    raise TimeoutError("could not observe a fresh OpProf sidecar")
+
+
+def validate_accounting(
+    run_dir: Path, mode: str, termination_wall_ns: int
+) -> dict[str, Any]:
+    streams = sorted((run_dir / "opprof").glob("*.jsonl"))
+    sidecars = sorted((run_dir / "opprof").glob("*.jsonl.footer.json"))
+    if len(streams) != 1 or len(sidecars) != 1:
+        raise RuntimeError(
+            f"expected one stream/sidecar, got {len(streams)}/{len(sidecars)}"
+        )
+    raw = streams[0].read_bytes()
+    if not raw.endswith(b"\n"):
+        raise RuntimeError("partial final JSONL line")
+    decoded = [json.loads(line) for line in raw.splitlines()]
+    footers = [row for row in decoded if row.get("record_type") == "footer"]
+    records = [row for row in decoded if row.get("record_type") != "footer"]
+    sidecar = json.loads(sidecars[0].read_text())
+    indices = [row["step_index"] for row in records]
+    checkpoint_age = (
+        termination_wall_ns - sidecar["checkpoint_wall_ns"]
+    ) / 1e9
+    common = {
+        "all_schema_1": all(row.get("schema") == 1 for row in decoded)
+        and sidecar.get("schema") == 1,
+        "steps_contiguous": indices == list(range(len(indices))),
+        "written_matches_records": sidecar["written_records"] == len(records),
+        "encoded_balanced": sidecar["encoded_records"]
+        == sidecar["written_records"] + sidecar["dropped_records"],
+        "last_step_matches": bool(records)
+        and sidecar["last_step_index"] == records[-1]["step_index"],
+        "zero_drops": sidecar["dropped_records"] == 0
+        and all(row["dropped_records_before"] == 0 for row in records),
+    }
+    if mode == "graceful":
+        footer_ok = len(footers) == 1 and decoded[-1] is footers[0]
+        agreement = footer_ok and all(
+            footers[0][counter] == sidecar[counter]
+            for counter in ("encoded_records", "written_records", "dropped_records")
+        )
+        specific = {
+            "one_footer_last": footer_ok,
+            "final_sidecar": sidecar["final"] is True,
+            "footer_sidecar_agree": agreement,
+        }
+    else:
+        specific = {
+            "no_in_stream_footer": not footers,
+            "checkpoint_sidecar": sidecar["final"] is False,
+            "checkpoint_within_bound": checkpoint_age
+            <= FLUSH_INTERVAL_SECONDS + CHECKPOINT_TOLERANCE_SECONDS,
+        }
+    invariants = {**common, **specific}
+    if not all(invariants.values()):
+        raise RuntimeError(
+            f"{mode} accounting invalid: {invariants}; sidecar={sidecar}"
+        )
+    return {
+        "stream": str(streams[0]),
+        "sidecar": str(sidecars[0]),
+        "bytes": len(raw),
+        "records": len(records),
+        "footer_count": len(footers),
+        "checkpoint_age_seconds": checkpoint_age,
+        "counters": {
+            key: sidecar[key]
+            for key in ("encoded_records", "written_records", "dropped_records")
+        },
+        "last_step_index": sidecar["last_step_index"],
+        "invariants": invariants,
+    }
+
+
+def run_trial(mode: str, port: int) -> dict[str, Any]:
+    run_dir = RUN_ROOT / mode
+    if run_dir.exists():
+        raise RuntimeError(f"refusing to overwrite {run_dir}")
+    run_dir.mkdir(parents=True)
+    before = assert_idle()
+    atomic_json(run_dir / "gpu-before.json", before)
+    (run_dir / "clocks-before.txt").write_text(
+        run_text(["nvidia-smi", "-q", "-d", "CLOCK"])
+    )
+    server_command = [
+        "taskset",
+        "-c",
+        "0-19",
+        str(VENV / "bin/vllm"),
+        "serve",
+        str(MODEL),
+        "--host",
+        "127.0.0.1",
+        "--port",
+        str(port),
+        "--tensor-parallel-size",
+        "1",
+        "--enable-chunked-prefill",
+        "--enable-prefix-caching",
+        "--shutdown-timeout",
+        "120",
+    ]
+    client_command = [
+        "taskset",
+        "-c",
+        "0-19",
+        str(VENV / "bin/python"),
+        str(CLIENT),
+        "run",
+        "--manifest",
+        str(MANIFEST),
+        "--base-url",
+        f"http://127.0.0.1:{port}",
+        "--model",
+        str(MODEL),
+        "--load-point",
+        "saturation",
+        "--request-rate",
+        "inf",
+        "--max-concurrency",
+        "256",
+        "--ignore-eos",
+        "--temperature",
+        "0",
+        "--warmup-seconds",
+        "20",
+        "--clean-segment-seconds",
+        "40",
+        "--num-clean-segments",
+        "3",
+        "--drain-timeout-seconds",
+        "120",
+        "--workload-seed",
+        "20260712",
+        "--result-dir",
+        str(run_dir / "client"),
+    ]
+    with (run_dir / "commands.log").open("w", encoding="utf-8") as output:
+        output.write(
+            f"GPU_COMMAND {mode} server: {shlex.join(server_command)}; "
+            "expected=60-180s startup + 140-180s load\n"
+        )
+        output.write(
+            f"GPU_COMMAND {mode} client: {shlex.join(client_command)}; "
+            "expected=20s warmup + 120s clean + drain\n"
+        )
+    environment = os.environ.copy()
+    environment.update(
+        {
+            "CUDA_VISIBLE_DEVICES": "0",
+            "VLLM_OPPROF_DIR": str(run_dir / "opprof"),
+            "HF_HUB_OFFLINE": "1",
+            "TRANSFORMERS_OFFLINE": "1",
+            "PYTHONUNBUFFERED": "1",
+        }
+    )
+    started = time.time()
+    server_log = (run_dir / "server.log").open("ab", buffering=0)
+    server: subprocess.Popen[Any] | None = None
+    try:
+        print(
+            f"GPU_COMMAND sidecar-{mode}: P01/C00 GPU0, 20s warmup + "
+            "120s clean, expected 4-6 wall-min",
+            flush=True,
+        )
+        server = subprocess.Popen(
+            server_command,
+            cwd=SOURCE,
+            env=environment,
+            stdout=server_log,
+            stderr=subprocess.STDOUT,
+            start_new_session=True,
+        )
+        atomic_json(
+            run_dir / "state.json",
+            {"status": "server_starting", "server_pid": server.pid},
+        )
+        wait_ready(server, port)
+        with (run_dir / "client.log").open("ab", buffering=0) as client_log:
+            client = subprocess.run(
+                client_command,
+                cwd=WORKDIR,
+                stdout=client_log,
+                stderr=subprocess.STDOUT,
+            )
+        if client.returncode:
+            raise RuntimeError(f"client exited {client.returncode}")
+        if mode == "graceful":
+            termination_wall_ns = time.time_ns()
+            os.kill(server.pid, signal.SIGINT)
+            server.wait(timeout=150)
+            log_text = (run_dir / "server.log").read_text(errors="replace")
+            if "mode=drain timeout=120s" not in log_text:
+                raise RuntimeError("official drain-mode log not observed")
+        else:
+            wait_fresh_sidecar(run_dir)
+            termination_wall_ns = time.time_ns()
+            os.killpg(server.pid, signal.SIGKILL)
+            server.wait(timeout=30)
+        zero_samples, zero_at = wait_zero()
+        accounting = validate_accounting(run_dir, mode, termination_wall_ns)
+        client_result = json.loads((run_dir / "client/result.json").read_text())
+        result = {
+            "schema": 1,
+            "status": "pass",
+            "mode": mode,
+            "server_returncode": server.returncode,
+            "termination_wall_ns": termination_wall_ns,
+            "accounting": accounting,
+            "clean": client_result["clean"],
+            "failed_records": client_result["failed_records"],
+            "gpu_zero_samples": zero_samples,
+            "gpu_seconds": zero_at - started,
+        }
+        atomic_json(run_dir / "result.json", result)
+        atomic_json(run_dir / "state.json", {"status": "complete"})
+        return result
+    except Exception as error:
+        atomic_json(
+            run_dir / "state.json",
+            {"status": "failed", "failure": repr(error)},
+        )
+        if server is not None and server.poll() is None:
+            try:
+                os.killpg(server.pid, signal.SIGKILL)
+            except ProcessLookupError:
+                pass
+        wait_zero()
+        raise
+    finally:
+        server_log.close()
+
+
+def main() -> None:
+    if RUN_ROOT.exists():
+        raise RuntimeError(f"refusing to overwrite {RUN_ROOT}")
+    RUN_ROOT.mkdir(parents=True)
+    state: dict[str, Any] = {"schema": 1, "status": "running", "results": {}}
+    atomic_json(RUN_ROOT / "state.json", state)
+    for offset, mode in enumerate(("graceful", "hard-kill")):
+        result = run_trial(mode, 8010 + offset)
+        state["results"][mode] = result
+        atomic_json(RUN_ROOT / "state.json", state)
+    state["status"] = "complete"
+    state["gpu_seconds"] = sum(
+        result["gpu_seconds"] for result in state["results"].values()
+    )
+    atomic_json(RUN_ROOT / "state.json", state)
+    print(json.dumps(state, sort_keys=True), flush=True)
+
+
+if __name__ == "__main__":
+    main()

runs/opprof-phase5/analyze_phase5.py

@@ -0,0 +1,537 @@
+#!/usr/bin/env python3
+"""Frozen Phase-5 bridge/control ledger analysis."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+
+SEED = 20260716
+RESAMPLES = 100_000
+RATE = 0.4725
+ARMS = ("base", "A1", "A2", "A3", "A4")
+MECHANISMS = ("A1", "A2", "A3", "A4")
+CAPTURE_SIZES = {
+    1, 2, 3, 4, 5, 6, 7, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88,
+    96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192,
+    200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336,
+    352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512,
+}
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def numeric(values: list[float | int | None]) -> dict[str, Any]:
+    finite = [float(value) for value in values if value is not None and math.isfinite(float(value))]
+    return {
+        "n": len(values),
+        "finite_n": len(finite),
+        "missing_n": len(values) - len(finite),
+        "min": min(finite) if finite else None,
+        "max": max(finite) if finite else None,
+        "distinct_n": len(set(finite)),
+    }
+
+
+def ci(draws: np.ndarray) -> list[float]:
+    low, high = np.quantile(draws, [0.025, 0.975])
+    return [float(low), float(high)]
+
+
+def cv(values: list[float]) -> float:
+    array = np.asarray(values, dtype=np.float64)
+    mean = float(array.mean())
+    if mean == 0:
+        return 0.0 if np.all(array == 0) else math.inf
+    return float(array.std(ddof=0) / mean)
+
+
+def parse_run(run_dir: Path) -> dict[str, Any]:
+    result = json.loads((run_dir / "client/result.json").read_text())
+    complete = json.loads((run_dir / "run-complete.json").read_text())
+    t0 = int(result["t0_mono_ns"])
+    clean_start = float(result["clean"]["start_s"])
+    clean_end = float(result["clean"]["end_s"])
+    stream = next((run_dir / "opprof").glob("*.jsonl"))
+    records = []
+    for line in stream.read_text().splitlines():
+        item = json.loads(line)
+        if "step_index" not in item:
+            continue
+        relative = (int(item["submit_mono_ns"]) - t0) / 1e9
+        if clean_start <= relative < clean_end:
+            item["relative_s"] = relative
+            records.append(item)
+    blocks = []
+    decode_means = []
+    waiting_means = []
+    for block_index in range(48):
+        start = clean_start + 5 * block_index
+        selected = [item for item in records if start <= item["relative_s"] < start + 5]
+        if not selected:
+            # Fixed-time blocks are the resampling unit.  A rate-following trace
+            # can legitimately have an idle block, which contributes no tokens
+            # and no model-step time but must remain in the arrival-variance
+            # distribution.
+            blocks.append([0, 0.0])
+            decode_means.append(0.0)
+            waiting_means.append(0.0)
+            continue
+        tokens = sum(int(item["prefill_tokens"]) + int(item["decode_tokens"]) for item in selected)
+        duration = sum(
+            (int(item["complete_mono_ns"]) - int(item["submit_mono_ns"])) / 1e6
+            for item in selected
+        )
+        blocks.append([tokens, duration])
+        decode_means.append(float(np.mean([int(item["decode_batch_size"]) for item in selected])))
+        waiting_means.append(float(np.mean([int(item["queues"]["waiting"]) for item in selected])))
+    pure_decode = [
+        item for item in records
+        if int(item["prefill_tokens"]) == 0 and int(item["decode_batch_size"]) > 0
+    ]
+    pure_bucket = sum(int(item["cudagraph"]["bucket_tokens"]) for item in pure_decode)
+    pure_padding = sum(int(item["cudagraph"]["padding_tokens"]) for item in pure_decode)
+    support = sorted({int(item["decode_batch_size"]) for item in pure_decode})
+    covered = sum(int(item["decode_batch_size"]) in CAPTURE_SIZES for item in pure_decode)
+    prefix_queries = 0
+    prefix_hits = 0
+    prefix_present = 0
+    for item in records:
+        local = item.get("prefix", {}).get("local")
+        if local is None:
+            continue
+        prefix_present += 1
+        prefix_queries += int(local.get("queries", 0))
+        prefix_hits += int(local.get("hits", 0))
+    block_array = np.asarray(blocks, dtype=np.float64)
+    return {
+        "run_id": complete["run_id"],
+        "run_dir": str(run_dir),
+        "stream_sha256": sha256_file(stream),
+        "blocks": block_array,
+        "token_efficiency_per_ms": float(block_array[:, 0].sum() / block_array[:, 1].sum()),
+        "clean_steps": len(records),
+        "clean_duration_s": clean_end - clean_start,
+        "clean_failed": int(result["clean"]["failed"]),
+        "offered_rps": float(result["clean"]["offered_rps"]),
+        "drain_seconds": float(result["drain_seconds"]),
+        "warmup_completions": int(complete["client"]["warmup_completions"]),
+        "warmup_gate_branch": complete["client"].get("warmup_gate_branch", "P3-pre-A-P5-1"),
+        "warmup_stability": complete["client"].get("warmup_stability"),
+        "cold_start_gate": complete["client"].get("cold_start_gate"),
+        "decode_B_block_cv": cv(decode_means),
+        "waiting_block_cv": cv(waiting_means),
+        "pure_decode_steps": len(pure_decode),
+        "pure_decode_support": support,
+        "pure_decode_support_coverage": covered / len(pure_decode),
+        "pure_decode_padding_tokens": pure_padding,
+        "pure_decode_bucket_tokens": pure_bucket,
+        "pure_decode_padding_fraction": pure_padding / pure_bucket,
+        "prefix_present_steps": prefix_present,
+        "prefix_queries": prefix_queries,
+        "prefix_hits": prefix_hits,
+        "prefix_hit_ratio": prefix_hits / prefix_queries if prefix_queries else 0.0,
+        "layer1_invariants": complete["layer1"]["invariants"],
+        "client_invariants": complete["client"]["invariants"],
+        "server_invariants": complete["server_invariants"],
+        "drain_quarantined": bool(complete["drain_quarantined"]),
+    }
+
+
+def hierarchical_draws(runs: list[dict[str, Any]], rng: np.random.Generator) -> np.ndarray:
+    count = len(runs)
+    output = np.empty(RESAMPLES, dtype=np.float64)
+    for start in range(0, RESAMPLES, 5000):
+        size = min(5000, RESAMPLES - start)
+        token_sum = np.zeros(size, dtype=np.float64)
+        duration_sum = np.zeros(size, dtype=np.float64)
+        selected_runs = rng.integers(0, count, size=(size, count))
+        for position in range(count):
+            block_indices = rng.integers(0, 48, size=(size, 48))
+            for run_index, run in enumerate(runs):
+                mask = selected_runs[:, position] == run_index
+                if not np.any(mask):
+                    continue
+                blocks = run["blocks"]
+                choices = block_indices[mask]
+                token_sum[mask] += blocks[choices, 0].sum(axis=1)
+                duration_sum[mask] += blocks[choices, 1].sum(axis=1)
+        output[start : start + size] = token_sum / duration_sum
+    return output
+
+
+def point_efficiency(runs: list[dict[str, Any]]) -> float:
+    tokens = sum(float(run["blocks"][:, 0].sum()) for run in runs)
+    duration = sum(float(run["blocks"][:, 1].sum()) for run in runs)
+    return tokens / duration
+
+
+def two_sided_p(draws: np.ndarray) -> float:
+    return float(min(1.0, 2 * min(np.mean(draws <= 0), np.mean(draws >= 0))))
+
+
+def holm(pvalues: dict[str, float]) -> dict[str, float]:
+    ordered = sorted(pvalues, key=pvalues.get)
+    adjusted: dict[str, float] = {}
+    running = 0.0
+    total = len(ordered)
+    for rank, key in enumerate(ordered):
+        running = max(running, (total - rank) * pvalues[key])
+        adjusted[key] = min(1.0, running)
+    return adjusted
+
+
+def discover_primary(root: Path) -> dict[str, list[dict[str, Any]]]:
+    result = {arm: [] for arm in ARMS}
+    for marker in sorted((root / "primary").glob("*/moderate/run-complete.json")):
+        if "background" in str(marker):
+            continue
+        parsed = parse_run(marker.parent)
+        arm = parsed["run_id"].split("-r", 1)[0]
+        if arm in result:
+            result[arm].append(parsed)
+    if any(len(result[arm]) != 3 for arm in ARMS):
+        raise RuntimeError(f"primary replication mismatch: { {key:len(value) for key,value in result.items()} }")
+    return result
+
+
+def control_runs(root: Path, p3root: Path, pattern: str) -> tuple[list[dict[str, Any]], str]:
+    fresh = sorted((root / "primary").glob(f"control-{pattern}-r*-C00/moderate/run-complete.json"))
+    if fresh:
+        if len(fresh) != 3:
+            raise RuntimeError(f"partial fresh control set: {pattern}: {len(fresh)}")
+        return [parse_run(path.parent) for path in fresh], "P5-rerun"
+    return [parse_run(p3root / f"primary/{pattern}-C00/moderate")], "P3-reused"
+
+
+def aggregate_aux(runs: list[dict[str, Any]], key: str) -> float:
+    return float(np.mean([float(run[key]) for run in runs]))
+
+
+def analyze(root: Path, p3root: Path, private: Path) -> dict[str, Any]:
+    primary = discover_primary(root)
+    p3_base = [parse_run(p3root / "primary/P10-C00/moderate")]
+    controls = {}
+    control_source = {}
+    for pattern in ("P03", "P04"):
+        controls[pattern], control_source[pattern] = control_runs(root, p3root, pattern)
+    rng = np.random.default_rng(SEED)
+    draws = {arm: hierarchical_draws(primary[arm], rng) for arm in ARMS}
+    p3_base_draws = hierarchical_draws(p3_base, rng)
+    control_draws = {pattern: hierarchical_draws(controls[pattern], rng) for pattern in controls}
+    points = {arm: point_efficiency(primary[arm]) for arm in ARMS}
+    p3_base_point = point_efficiency(p3_base)
+    control_points = {pattern: point_efficiency(controls[pattern]) for pattern in controls}
+
+    bridge_draws = draws["A3"] - p3_base_draws
+    bridge_point = points["A3"] - p3_base_point
+    bridge_ci = ci(bridge_draws)
+    bridge = {
+        "point_delta": bridge_point,
+        "relative_abs_delta": abs(bridge_point) / p3_base_point,
+        "ci95": bridge_ci,
+        "within_3pct": abs(bridge_point) / p3_base_point <= 0.03,
+        "ci_contains_zero": bridge_ci[0] <= 0 <= bridge_ci[1],
+    }
+    bridge["reuse_passed"] = bridge["within_3pct"] and bridge["ci_contains_zero"]
+
+    deltas = {arm: draws[arm] - draws["base"] for arm in MECHANISMS}
+    raw_p = {arm: two_sided_p(deltas[arm]) for arm in MECHANISMS}
+    holm_p = holm(raw_p)
+    manifests = {
+        name: json.loads((private / f"P10-{name}.jsonl.summary.json").read_text())
+        for name in ("base", "A1", "A3")
+    }
+    base_prefix = aggregate_aux(primary["base"], "prefix_hit_ratio")
+    a1_prefix = aggregate_aux(primary["A1"], "prefix_hit_ratio")
+    base_padding = aggregate_aux(primary["base"], "pure_decode_padding_fraction")
+    a2_padding = aggregate_aux(primary["A2"], "pure_decode_padding_fraction")
+    padding_reduction = (base_padding - a2_padding) / base_padding if base_padding > 0 else 0.0
+    manipulations = {
+        "A1": {
+            "passed": (
+                manifests["base"]["r16"] - manifests["A1"]["r16"] >= 0.15
+                and (manifests["base"]["r16"] - manifests["A1"]["r16"]) / manifests["base"]["r16"] >= 0.20
+                and manifests["A1"]["max_added_delay_seconds"] <= 64
+                and abs(a1_prefix - base_prefix) <= 0.01
+            ),
+            "base_R16": manifests["base"]["r16"],
+            "ablated_R16": manifests["A1"]["r16"],
+            "prefix_hit_ratio_delta": a1_prefix - base_prefix,
+            "max_added_delay_seconds": manifests["A1"]["max_added_delay_seconds"],
+        },
+        "A2": {
+            "passed": aggregate_aux(primary["A2"], "pure_decode_support_coverage") >= 0.99 and padding_reduction >= 0.90,
+            "support_coverage": aggregate_aux(primary["A2"], "pure_decode_support_coverage"),
+            "base_padding_fraction": base_padding,
+            "ablated_padding_fraction": a2_padding,
+            "padding_reduction": padding_reduction,
+            "observed_support": sorted({value for run in primary["A2"] for value in run["pure_decode_support"]}),
+        },
+        "A3": {
+            "passed": (
+                aggregate_aux(primary["A3"], "decode_B_block_cv") < aggregate_aux(primary["base"], "decode_B_block_cv")
+                and aggregate_aux(primary["A3"], "waiting_block_cv") < aggregate_aux(primary["base"], "waiting_block_cv")
+            ),
+            "base_decode_B_cv": aggregate_aux(primary["base"], "decode_B_block_cv"),
+            "ablated_decode_B_cv": aggregate_aux(primary["A3"], "decode_B_block_cv"),
+            "base_waiting_cv": aggregate_aux(primary["base"], "waiting_block_cv"),
+            "ablated_waiting_cv": aggregate_aux(primary["A3"], "waiting_block_cv"),
+        },
+        "A4": {
+            "passed": sum(run["prefix_queries"] for run in primary["A4"]) == 0 and sum(run["prefix_hits"] for run in primary["A4"]) == 0,
+            "prefix_queries": sum(run["prefix_queries"] for run in primary["A4"]),
+            "prefix_hits": sum(run["prefix_hits"] for run in primary["A4"]),
+        },
+    }
+
+    ledgers = {}
+    for control in ("P03", "P04"):
+        denominator = control_draws[control] - draws["base"]
+        point_denominator = control_points[control] - points["base"]
+        denominator_ci = ci(denominator)
+        stable_denominator = bool(
+            point_denominator > 0
+            and denominator_ci[0] > 0
+            and np.mean(denominator <= 0) <= 0.05
+        )
+        rows = {}
+        share_draws = {}
+        for arm in MECHANISMS:
+            share_draws[arm] = deltas[arm] / denominator
+            point = (points[arm] - points["base"]) / point_denominator
+            interval = ci(share_draws[arm])
+            reportable = stable_denominator and manipulations[arm]["passed"]
+            rows[arm] = {
+                "delta_E": points[arm] - points["base"],
+                "delta_E_ci95": ci(deltas[arm]),
+                "share": point if reportable else None,
+                "share_ci95": interval if reportable else None,
+                "diagnostic_share": point,
+                "diagnostic_share_ci95": interval,
+                "share_status": (
+                    "REPORTABLE"
+                    if reportable
+                    else (
+                        "N/A—unstable control denominator"
+                        if not stable_denominator
+                        else "N/A—manipulation failed"
+                    )
+                ),
+                "p_two_sided": raw_p[arm],
+                "p_holm": holm_p[arm],
+                "manipulation_passed": manipulations[arm]["passed"],
+            }
+        residual_draws = 1.0 - sum(share_draws.values())
+        residual_point = 1.0 - sum(row["diagnostic_share"] for row in rows.values())
+        ledgers[control] = {
+            "status": "EVALUABLE" if stable_denominator else "INCONCLUSIVE—unstable denominator",
+            "control_source": control_source[control],
+            "control_E": control_points[control],
+            "base_E": points["base"],
+            "gap_E": point_denominator,
+            "gap_ci95": denominator_ci,
+            "denominator_nonpositive_fraction": float(np.mean(denominator <= 0)),
+            "stable_denominator": stable_denominator,
+            "mechanisms": rows,
+            "diagnostic_share_sum": sum(row["diagnostic_share"] for row in rows.values()),
+            "residual_interaction": residual_point,
+            "residual_interaction_ci95": ci(residual_draws),
+            "residual_status": (
+                "REPORTABLE"
+                if stable_denominator and all(item["passed"] for item in manipulations.values())
+                else "DIAGNOSTIC_ONLY—incomplete official share ledger"
+            ),
+        }
+
+    dominance = {}
+    for arm in MECHANISMS:
+        per_control = {}
+        for control in ("P03", "P04"):
+            row = ledgers[control]["mechanisms"][arm]
+            if not ledgers[control]["stable_denominator"] or not row["manipulation_passed"]:
+                per_control[control] = "NOT_EVALUABLE"
+                continue
+            direction_ok = row["delta_E"] > 0 if arm != "A4" else True
+            per_control[control] = (
+                row["share"] >= 0.30
+                and row["share_ci95"][0] > 0.15
+                and row["p_holm"] < 0.05
+                and direction_ok
+                and row["manipulation_passed"]
+            )
+        dominance[arm] = {
+            "per_control": per_control,
+            "verdict": (
+                "NOT EVALUABLE"
+                if any(value == "NOT_EVALUABLE" for value in per_control.values())
+                else (
+                    "DOMINANT"
+                    if all(per_control.values())
+                    else ("CONTROL-SENSITIVE" if any(per_control.values()) else "NOT DOMINANT")
+                )
+            ),
+        }
+
+    all_runs = [run for arm in ARMS for run in primary[arm]]
+    share_widths = [
+        ledgers[control]["mechanisms"][arm]["diagnostic_share_ci95"][1]
+        - ledgers[control]["mechanisms"][arm]["diagnostic_share_ci95"][0]
+        for control in ("P03", "P04") for arm in MECHANISMS
+    ]
+    state = json.loads((root / "controller-state.json").read_text())
+    amendment_evidence_path = root / "a-p5-1-retained-audit.jsonl"
+    amendment_evidence = []
+    if amendment_evidence_path.exists():
+        amendment_evidence = [
+            json.loads(line) for line in amendment_evidence_path.read_text().splitlines()
+        ]
+    invariants = {
+        "primary_runs_15": len(all_runs) == 15,
+        "three_replicates_per_arm": all(len(primary[arm]) == 3 for arm in ARMS),
+        "clean_duration_240": all(math.isclose(run["clean_duration_s"], 240.0) for run in all_runs),
+        "clean_failures_zero": all(run["clean_failed"] == 0 for run in all_runs),
+        "offered_rate_within_5pct": all(abs(run["offered_rps"] / RATE - 1) <= 0.05 for run in all_runs),
+        "layer1_accounting": all(all(run["layer1_invariants"].values()) for run in all_runs),
+        "client_invariants": all(all(value for key, value in run["client_invariants"].items() if key != "drain_re_adjudicated") for run in all_runs),
+        "server_invariants": all(all(run["server_invariants"].values()) for run in all_runs),
+        "a_p5_1_cold_start_gates": all(
+            run["cold_start_gate"] is not None
+            and run["cold_start_gate"]["passed"]
+            for run in all_runs
+        ),
+        "drain_quarantine_under_20pct": sum(run["drain_quarantined"] for run in all_runs) / 15 <= 0.20,
+        "gpu_budget_below_6": float(state["gpu_hours_total"]) < 6.0,
+        "manifests_same_ids": len({manifests[name]["request_id_set_sha256"] for name in manifests}) == 1,
+        "manifests_same_token_sums": len({manifests[name]["input_tokens"]["sum"] for name in manifests}) == 1 and len({manifests[name]["output_tokens"]["sum"] for name in manifests}) == 1,
+        "control_denominators_stable": all(ledgers[control]["stable_denominator"] for control in ledgers),
+        "bridge_decision_resolved": bridge["reuse_passed"] or all(source == "P5-rerun" for source in control_source.values()),
+        "ratios_finite": all(math.isfinite(ledgers[c]["mechanisms"][a]["diagnostic_share"]) for c in ledgers for a in MECHANISMS),
+        "per_arm_results_not_all_identical": len({round(points[arm], 12) for arm in ARMS}) > 1,
+    }
+    red_flags = [key for key, value in invariants.items() if not value]
+    publishable = (
+        not red_flags
+        and all(item["passed"] for item in manipulations.values())
+        and sum(width > 0.50 for width in share_widths) < 2
+    )
+    return {
+        "schema": 1,
+        "status": "PUBLISHABLE" if publishable else "INCONCLUSIVE_OR_PARTIAL",
+        "limitation": "Recorded-arrival P5 bridge ledger anchored to P3 controls; not a literal decomposition of P3's already-uniform P10 gap.",
+        "analysis_seed": SEED,
+        "bootstrap_resamples": RESAMPLES,
+        "efficiency": {
+            arm: {
+                "point": points[arm],
+                "ci95": ci(draws[arm]),
+                "runs": [
+                    {key: value for key, value in run.items() if key not in {"blocks", "warmup_stability"}}
+                    for run in primary[arm]
+                ],
+            }
+            for arm in ARMS
+        },
+        "p3_base_E": p3_base_point,
+        "bridge": bridge,
+        "control_sources": control_source,
+        "manipulations": manipulations,
+        "holm": {"family": list(MECHANISMS), "raw_p": raw_p, "adjusted_p": holm_p},
+        "ledgers": ledgers,
+        "dominance": dominance,
+        "config_tier_A2": {
+            "delta_E": points["A2"] - points["base"],
+            "relative_E_delta": points["A2"] / points["base"] - 1,
+            "delta_E_ci95": ci(deltas["A2"]),
+            "base_padding_fraction": base_padding,
+            "A2_padding_fraction": a2_padding,
+            "padding_reduction": padding_reduction,
+        },
+        "amendment_A_P5_1": {
+            "reason": "Rate-following throughput drift tracks arrival shape and is not a cold-start stationarity test.",
+            "retained_failed_run_evidence": amendment_evidence,
+            "recorded_drift_range": (
+                [
+                    min(
+                        item["superseded_drift_evidence"]["normalized_drift"]
+                        for item in amendment_evidence
+                        if item["run"] != "A3-r1-C00"
+                    ),
+                    max(
+                        item["superseded_drift_evidence"]["normalized_drift"]
+                        for item in amendment_evidence
+                        if item["run"] != "A3-r1-C00"
+                    ),
+                ]
+                if amendment_evidence else None
+            ),
+            "uniform_A3_drift": (
+                next(
+                    item["superseded_drift_evidence"]["normalized_drift"]
+                    for item in amendment_evidence
+                    if item["run"] == "A3-r1-C00"
+                )
+                if amendment_evidence else None
+            ),
+        },
+        "gpu": {
+            "new_h20_hours": float(state["gpu_hours_total"]),
+            "hard_cap": 6.0,
+            "controller_status": state["status"],
+            "completed_measured_runs_including_background": state["completed_measured_runs"],
+            "completed_burnins": state["completed_burnins"],
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "invariants": invariants,
+            "numeric": {
+                "primary_E": numeric([run["token_efficiency_per_ms"] for run in all_runs]),
+                "clean_steps": numeric([run["clean_steps"] for run in all_runs]),
+                "offered_rps": numeric([run["offered_rps"] for run in all_runs]),
+                "drain_seconds": numeric([run["drain_seconds"] for run in all_runs]),
+                "diagnostic_share": numeric([ledgers[c]["mechanisms"][a]["diagnostic_share"] for c in ledgers for a in MECHANISMS]),
+                "residual_interaction": numeric([ledgers[c]["residual_interaction"] for c in ledgers]),
+                "share_ci_width": numeric(share_widths),
+            },
+            "declared": {
+                "manipulation_failures": [arm for arm, item in manipulations.items() if not item["passed"]],
+                "control_sources": control_source,
+                "bridge_reuse_passed": bridge["reuse_passed"],
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--root", type=Path, required=True)
+    parser.add_argument("--p3-root", type=Path, required=True)
+    parser.add_argument("--private", type=Path, required=True)
+    parser.add_argument("--out", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(args.root, args.p3_root, args.private)
+    args.out.parent.mkdir(parents=True, exist_ok=True)
+    args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n")
+    print(json.dumps({
+        "status": result["status"],
+        "bridge": result["bridge"],
+        "red_flags": result["sanity"]["red_flags"],
+        "gpu": result["gpu"],
+    }, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/opprof-phase5/opprof_phase5_client.py

@@ -0,0 +1,242 @@
+#!/usr/bin/env python3
+"""Phase-5 private-manifest transforms and timestamp-scheduled P3 client wrapper."""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import math
+import sys
+from pathlib import Path
+from typing import Any
+
+import aiohttp
+
+import opprof_phase3_client as p3
+
+
+def _numeric(values: list[float | int]) -> dict[str, Any]:
+    finite = [float(value) for value in values if math.isfinite(float(value))]
+    return {
+        "n": len(values),
+        "finite_n": len(finite),
+        "missing_n": len(values) - len(finite),
+        "min": min(finite) if finite else None,
+        "max": max(finite) if finite else None,
+        "distinct_n": len(set(finite)),
+        "sum": sum(finite),
+    }
+
+
+def _r16(rows: list[dict[str, Any]]) -> float:
+    groups = [rows[index : index + 16] for index in range(0, len(rows) - 15, 16)]
+    useful = sum(sum(int(row["input_tokens"]) for row in group) for group in groups)
+    rectangular = sum(16 * max(int(row["input_tokens"]) for row in group) for group in groups)
+    return 1.0 - useful / rectangular
+
+
+def _source_timestamps(path: Path, indices: set[int], field: str) -> dict[int, float]:
+    result: dict[int, float] = {}
+    maximum = max(indices)
+    with path.open(encoding="utf-8") as source:
+        for index, line in enumerate(source):
+            if index in indices:
+                value = float(json.loads(line)[field])
+                if not math.isfinite(value):
+                    raise ValueError(f"non-finite source timestamp at {index}")
+                result[index] = value
+            if index >= maximum:
+                break
+    if set(result) != indices:
+        raise ValueError("timestamp source did not cover all source_index values")
+    return result
+
+
+def transform(args: argparse.Namespace) -> dict[str, Any]:
+    rows = p3.load_manifest(Path(args.input))[: args.take_first]
+    if len(rows) != args.take_first:
+        raise ValueError(f"requested {args.take_first} rows, found {len(rows)}")
+    indices = [int(row[args.join_key]) for row in rows]
+    timestamps = _source_timestamps(
+        Path(args.timestamp_source), set(indices), args.timestamp_field
+    )
+    source_times = [timestamps[index] for index in indices]
+    if any(right < left for left, right in zip(source_times, source_times[1:])):
+        raise ValueError("selected timestamps are not nondecreasing")
+    if source_times[-1] <= source_times[0]:
+        raise ValueError("selected timestamp span is not positive")
+    end_s = (len(rows) - 1) / args.target_rate
+    scale = end_s / (source_times[-1] - source_times[0])
+    recorded_slots = [(value - source_times[0]) * scale for value in source_times]
+    uniform_slots = [index / args.target_rate for index in range(len(rows))]
+    slots = recorded_slots if args.arrival == "recorded-scaled" else uniform_slots
+
+    for index, row in enumerate(rows):
+        row["arrival"] = args.arrival
+        row["arrival_s"] = slots[index]
+        row["original_index"] = index
+        row["source_timestamp"] = source_times[index]
+
+    original = list(rows)
+    max_added_delay = 0.0
+    if args.service_order == "length-binned":
+        edges = [int(item) for item in args.length_bin_edges.split(",")]
+
+        def bin_id(row: dict[str, Any]) -> int:
+            length = int(row["input_tokens"])
+            for index, edge in enumerate(edges):
+                if length <= edge:
+                    return index
+            raise ValueError(f"input length {length} exceeds final edge")
+
+        reordered: list[dict[str, Any]] = []
+        for offset in range(0, len(rows), args.reorder_block_size):
+            block = rows[offset : offset + args.reorder_block_size]
+            ordered = sorted(
+                block,
+                key=lambda row: (
+                    bin_id(row),
+                    int(row["input_tokens"]),
+                    int(row["original_index"]),
+                ),
+            )
+            block_slots = slots[offset : offset + len(block)]
+            for position, row in enumerate(ordered):
+                added = max(0.0, block_slots[position] - float(row["arrival_s"]))
+                max_added_delay = max(max_added_delay, added)
+                row["arrival_s"] = block_slots[position]
+            reordered.extend(ordered)
+        rows = reordered
+        if max_added_delay > args.max_added_delay_seconds + 1e-9:
+            raise ValueError(
+                f"fairness cap exceeded: {max_added_delay} > {args.max_added_delay_seconds}"
+            )
+    elif args.service_order != "original":
+        raise ValueError(f"unsupported service order: {args.service_order}")
+
+    if sorted(row["request_id"] for row in rows) != sorted(
+        row["request_id"] for row in original
+    ):
+        raise AssertionError("request identity changed")
+    for key in ("input_tokens", "output_tokens"):
+        if sum(int(row[key]) for row in rows) != sum(int(row[key]) for row in original):
+            raise AssertionError(f"{key} total changed")
+    arrival_values = [float(row["arrival_s"]) for row in rows]
+    if any(right < left for left, right in zip(arrival_values, arrival_values[1:])):
+        raise AssertionError("assigned arrival slots are not nondecreasing")
+
+    output = Path(args.out)
+    p3.atomic_jsonl(output, rows, mode=0o600)
+    summary = {
+        "schema": 1,
+        "path": str(output),
+        "sha256": p3.sha256_file(output),
+        "rows": len(rows),
+        "arrival": args.arrival,
+        "service_order": args.service_order,
+        "target_rate": args.target_rate,
+        "input_tokens": _numeric([int(row["input_tokens"]) for row in rows]),
+        "output_tokens": _numeric([int(row["output_tokens"]) for row in rows]),
+        "arrival_s": _numeric(arrival_values),
+        "source_timestamp": _numeric(source_times),
+        "r16": _r16(rows),
+        "max_added_delay_seconds": max_added_delay,
+        "request_id_set_sha256": p3.hashlib.sha256(
+            "\n".join(sorted(str(row["request_id"]) for row in rows)).encode()
+        ).hexdigest(),
+        "invariants": {
+            "same_request_ids": True,
+            "same_input_tokens": True,
+            "same_output_tokens": True,
+            "arrival_nondecreasing": True,
+            "fairness_cap": max_added_delay <= args.max_added_delay_seconds + 1e-9,
+            "no_prompt_in_summary": True,
+        },
+    }
+    p3.atomic_json(output.with_suffix(output.suffix + ".summary.json"), summary, mode=0o600)
+    print(json.dumps(summary, sort_keys=True))
+    return summary
+
+
+async def finite_timestamp_load(
+    ctx: p3.RunContext, session: aiohttp.ClientSession, rate: float
+) -> list[dict[str, Any]]:
+    if "arrival_s" not in ctx.rows[0]:
+        return await _ORIGINAL_FINITE_LOAD(ctx, session, rate)
+    sem = asyncio.Semaphore(ctx.args.max_concurrency)
+    tasks: list[asyncio.Task[dict[str, Any]]] = []
+
+    async def limited(row: dict[str, Any], scheduled: float) -> dict[str, Any]:
+        async with sem:
+            return await p3.request_one(ctx, session, row, scheduled)
+
+    for expected in ctx.rows:
+        scheduled = ctx.t0 + float(expected["arrival_s"])
+        delay = scheduled - asyncio.get_running_loop().time()
+        if delay > 0:
+            try:
+                await asyncio.wait_for(ctx.stop_event.wait(), timeout=delay)
+                break
+            except asyncio.TimeoutError:
+                pass
+        if ctx.stop_event.is_set():
+            break
+        row = await ctx.next_row()
+        if row["request_id"] != expected["request_id"]:
+            raise AssertionError("timestamp scheduler row drift")
+        tasks.append(asyncio.create_task(limited(row, scheduled)))
+    return await asyncio.gather(*tasks) if tasks else []
+
+
+_ORIGINAL_FINITE_LOAD = p3.finite_load
+p3.finite_load = finite_timestamp_load
+
+
+def build_transform_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--in", dest="input", required=True)
+    parser.add_argument("--take-first", type=int, required=True)
+    parser.add_argument("--timestamp-source", required=True)
+    parser.add_argument("--join-key", default="source_index")
+    parser.add_argument("--timestamp-field", default="timestamp")
+    parser.add_argument("--arrival", choices=("recorded-scaled", "uniform"), required=True)
+    parser.add_argument("--target-rate", type=float, required=True)
+    parser.add_argument("--service-order", choices=("original", "length-binned"), required=True)
+    parser.add_argument("--reorder-block-size", type=int, default=32)
+    parser.add_argument("--analysis-cohort-size", type=int, default=16)
+    parser.add_argument("--length-bin-edges", default="512,1024,2048,4096,8192,16384,32768")
+    parser.add_argument("--max-added-delay-seconds", type=float, default=64)
+    parser.add_argument("--out", required=True)
+    return parser
+
+
+def main() -> None:
+    if len(sys.argv) > 1 and sys.argv[1] == "transform":
+        transform(build_transform_parser().parse_args(sys.argv[2:]))
+        return
+    fixed_rate = None
+    if "--fixed-request-rate" in sys.argv:
+        index = sys.argv.index("--fixed-request-rate")
+        fixed_rate = float(sys.argv[index + 1])
+        del sys.argv[index : index + 2]
+    args = p3.build_parser().parse_args()
+    if args.command != "run":
+        p3.main()
+        return
+    if fixed_rate is not None:
+        if args.load_point != "moderate" or fixed_rate <= 0 or not math.isfinite(fixed_rate):
+            raise ValueError("--fixed-request-rate requires positive finite moderate rate")
+        result_dir = Path(args.result_dir)
+        result_dir.mkdir(parents=True, exist_ok=True)
+        source = result_dir / "fixed-rate-source.json"
+        p3.atomic_json(source, {"clean": {"completed_throughput_rps": fixed_rate}})
+        args.saturation_result = str(source)
+        args.rate_fraction = 1.0
+    if args.profile_after_clean and not args.profile_trace_dir:
+        raise ValueError("--profile-after-clean requires --profile-trace-dir")
+    print(json.dumps(asyncio.run(p3.run_load(args)), sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/opprof-phase5/opprof_phase5_controller.py

@@ -0,0 +1,670 @@
+#!/usr/bin/env python3
+"""Detached, resumable Phase-5 four-way primary/control controller."""
+
+from __future__ import annotations
+
+import argparse
+import datetime as dt
+import hashlib
+import json
+import math
+import os
+import re
+import shlex
+import shutil
+import subprocess
+import time
+from pathlib import Path
+from typing import Any
+
+import opprof_phase3_matrix as m
+
+
+WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712")
+RUN_ROOT = WORKDIR / "runs/phase5"
+PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase5-private/manifests")
+P3_PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests")
+MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B")
+SOURCE = Path("/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0")
+VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv")
+CLIENT = WORKDIR / "scripts/opprof_phase5_client.py"
+P3_CLIENT = WORKDIR / "scripts/opprof_phase3_client.py"
+STATE = RUN_ROOT / "controller-state.json"
+RATE = 0.4725
+CAPTURE_SIZES = (
+    1, 2, 3, 4, 5, 6, 7, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88,
+    96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192,
+    200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336,
+    352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512,
+)
+
+
+m.WORKDIR = WORKDIR
+m.RUN_ROOT = RUN_ROOT
+m.PRIVATE = PRIVATE
+m.MODEL = MODEL
+m.SOURCE = SOURCE
+m.VENV = VENV
+m.CLIENT = CLIENT
+m.STATE = STATE
+m.GPU_HOUR_LIMIT = 6.0
+m.PRIOR_GPU_HOURS = 0.0
+m.CONFIGS = {
+    "C00": {"tp": 1, "mns": 1024, "mbt": 8192, "flags": []},
+    "A2": {"tp": 1, "mns": 1024, "mbt": 8192, "flags": []},
+    "A4": {"tp": 1, "mns": 1024, "mbt": 8192, "flags": []},
+}
+
+
+def sha256_file(path: Path) -> str:
+    return m.sha256_file(path)
+
+
+def arm_name(pattern: str) -> str:
+    return pattern.split("-r", 1)[0]
+
+
+def manifest_for(pattern: str, burnin: bool) -> Path:
+    if burnin or pattern.startswith("background"):
+        return P3_PRIVATE / "P06.jsonl"
+    if pattern.startswith("control-P03"):
+        return P3_PRIVATE / "P03.jsonl"
+    if pattern.startswith("control-P04"):
+        return P3_PRIVATE / "P04.jsonl"
+    arm = arm_name(pattern)
+    return PRIVATE / ("P10-base.jsonl" if arm in {"base", "A2", "A4"} else f"P10-{arm}.jsonl")
+
+
+def saturation_result_for(pattern: str) -> Path:
+    if pattern.startswith("control-P03"):
+        cell = "P03-C00"
+    elif pattern.startswith("control-P04"):
+        cell = "P04-C00"
+    else:
+        cell = "P10-C00"
+    return WORKDIR / f"runs/phase3/primary/{cell}/saturation/client/result.json"
+
+
+def drain_budget(_pattern: str) -> int:
+    return 600
+
+
+m.drain_budget = drain_budget
+
+
+def server_command(assignment: m.Assignment, port: int, _trace_dir: Path) -> list[str]:
+    arm = arm_name(assignment.cell.pattern)
+    command = [
+        "taskset", "-c", m.cpu_mask(assignment.gpus), str(VENV / "bin/vllm"),
+        "serve", str(MODEL), "--host", "127.0.0.1", "--port", str(port),
+        "--tensor-parallel-size", "1", "--enable-chunked-prefill",
+    ]
+    if arm != "A4" and assignment.cell.config != "A4":
+        command.append("--enable-prefix-caching")
+    command.extend(("--shutdown-timeout", "600"))
+    if arm == "A2" or assignment.cell.config == "A2":
+        command.extend(("--cudagraph-capture-sizes", *map(str, CAPTURE_SIZES)))
+    return command
+
+
+def client_command(
+    assignment: m.Assignment,
+    port: int,
+    run_dir: Path,
+    _load_point: str,
+    _profile: bool,
+    burnin: bool,
+    _saturation_result: Path | None,
+) -> list[str]:
+    pattern = assignment.cell.pattern
+    common = [
+        "taskset", "-c", m.cpu_mask(assignment.gpus), str(VENV / "bin/python"),
+        str(CLIENT), "run", "--manifest", str(manifest_for(pattern, burnin)),
+        "--base-url", f"http://127.0.0.1:{port}", "--model", str(MODEL),
+        "--max-concurrency", "256", "--ignore-eos", "--temperature", "0",
+        "--workload-seed", "20260712", "--server-seed", "20260712",
+        "--result-dir", str(run_dir / "client"),
+    ]
+    if burnin:
+        return common + [
+            "--load-point", "saturation", "--request-rate", "inf",
+            "--warmup-seconds", "0", "--clean-segment-seconds", "20",
+            "--num-clean-segments", "3", "--post-clean-seconds", "0",
+            "--drain-timeout-seconds", "600",
+        ]
+    if pattern.startswith("background"):
+        return common + [
+            "--load-point", "saturation", "--request-rate", "inf",
+            "--warmup-seconds", "60", "--clean-segment-seconds", "80",
+            "--num-clean-segments", "3", "--post-clean-seconds", "0",
+            "--drain-timeout-seconds", "600",
+        ]
+    if pattern.startswith("control-"):
+        return common + [
+            "--load-point", "moderate", "--saturation-result",
+            str(saturation_result_for(pattern)), "--rate-fraction", "0.60",
+            "--warmup-seconds", "60", "--clean-segment-seconds", "80",
+            "--num-clean-segments", "3", "--post-clean-seconds", "0",
+            "--drain-timeout-seconds", "600",
+        ]
+    return common + [
+        "--load-point", "moderate", "--fixed-request-rate", str(RATE),
+        "--warmup-seconds", "60", "--clean-segment-seconds", "80",
+        "--num-clean-segments", "3", "--post-clean-seconds", "0",
+        "--drain-timeout-seconds", "600",
+    ]
+
+
+m.server_command = server_command
+m.client_command = client_command
+
+
+_ORIGINAL_VALIDATE_CLIENT = m.validate_client
+
+
+def _log_event_time(line: str, t0_wall_ns: int) -> float | None:
+    match = re.search(r"INFO\s+(\d{2})-(\d{2})\s+(\d{2}):(\d{2}):(\d{2})", line)
+    if match is None:
+        return None
+    month, day, hour, minute, second = map(int, match.groups())
+    year = dt.datetime.fromtimestamp(t0_wall_ns / 1e9, tz=dt.timezone.utc).year
+    stamp = dt.datetime(year, month, day, hour, minute, second, tzinfo=dt.timezone.utc)
+    return stamp.timestamp()
+
+
+def cold_start_gate(run_dir: Path) -> dict[str, Any]:
+    result = json.loads((run_dir / "client/result.json").read_text())
+    requests = [
+        json.loads(line)
+        for line in (run_dir / "client/requests.jsonl").read_text().splitlines()
+    ]
+    warm = [
+        request for request in requests
+        if request["success"] and 0 <= float(request["completed_s"]) < 60
+    ]
+    warm_long = [request for request in warm if int(request["input_tokens"]) >= 8192]
+    t0_mono_ns = int(result["t0_mono_ns"])
+    stream = next((run_dir / "opprof").glob("*.jsonl"))
+    warm_descriptors: set[tuple[str, int]] = set()
+    clean_descriptors: set[tuple[str, int]] = set()
+    for line in stream.read_text().splitlines():
+        item = json.loads(line)
+        if "step_index" not in item or not item.get("model_executed"):
+            continue
+        graph = item["cudagraph"]
+        if not graph.get("hit"):
+            continue
+        relative_s = (int(item["submit_mono_ns"]) - t0_mono_ns) / 1e9
+        descriptor = (str(graph["runtime_mode"]), int(graph["bucket_tokens"]))
+        if 0 <= relative_s < 60:
+            warm_descriptors.add(descriptor)
+        elif 60 <= relative_s < 300:
+            clean_descriptors.add(descriptor)
+
+    log_lines = (run_dir / "server.log").read_text(errors="replace").splitlines()
+    ready_indices = [
+        index for index, line in enumerate(log_lines)
+        if "Application startup complete" in line
+    ]
+    if len(ready_indices) != 1:
+        raise RuntimeError(f"expected one server ready marker: {run_dir}: {ready_indices}")
+    ready_index = ready_indices[0]
+    event_pattern = re.compile(
+        r"torch\.compile took|Directly load AOT compilation|\bCompiling\b|Capturing CUDA graphs",
+        re.IGNORECASE,
+    )
+    events = []
+    clean_boundary_wall_s = int(result["t0_wall_ns"]) / 1e9 + 60
+    clean_end_wall_s = int(result["t0_wall_ns"]) / 1e9 + 300
+    event_gate = True
+    clean_events = 0
+    for index, line in enumerate(log_lines):
+        if not event_pattern.search(line):
+            continue
+        timestamp = _log_event_time(line, int(result["t0_wall_ns"]))
+        phase = "pre-ready" if index < ready_index else "post-ready"
+        if index >= ready_index:
+            if timestamp is None:
+                event_gate = False
+                phase = "post-ready-unparseable"
+            elif timestamp >= clean_boundary_wall_s:
+                event_gate = False
+                phase = "clean" if timestamp < clean_end_wall_s else "post-clean"
+                if timestamp < clean_end_wall_s:
+                    clean_events += 1
+            else:
+                phase = "warmup"
+        events.append(
+            {
+                "line": index + 1,
+                "phase": phase,
+                "timestamp_s": timestamp,
+                "message_prefix": line[:200],
+            }
+        )
+
+    config_match = re.search(
+        r"cudagraph_capture_sizes['\"]?:\s*\[([^\]]+)\]",
+        "\n".join(log_lines[:ready_index]),
+    )
+    startup_sizes = (
+        {int(value.strip()) for value in config_match.group(1).split(",")}
+        if config_match else set()
+    )
+    startup_modes = set()
+    for line in log_lines[:ready_index]:
+        if "Capturing CUDA graphs" not in line:
+            continue
+        if "PIECEWISE" in line:
+            startup_modes.add("PIECEWISE")
+        if "FULL" in line:
+            startup_modes.add("FULL")
+    uncovered = sorted(
+        descriptor for descriptor in clean_descriptors
+        if descriptor[0] not in startup_modes or descriptor[1] not in startup_sizes
+    )
+    passed = (
+        event_gate
+        and len(warm) >= 16
+        and len(warm_long) >= 1
+        and startup_modes == {"FULL", "PIECEWISE"}
+        and bool(startup_sizes)
+        and not uncovered
+        and clean_events == 0
+    )
+    return {
+        "amendment": "A-P5-1",
+        "passed": passed,
+        "warmup_completions": len(warm),
+        "warmup_long_completions": len(warm_long),
+        "warmup_long_min_tokens": min(
+            (int(request["input_tokens"]) for request in warm_long), default=None
+        ),
+        "server_ready_line": ready_index + 1,
+        "compile_capture_events": events,
+        "event_gate_passed": event_gate,
+        "clean_capture_events": clean_events,
+        "startup_capture_sizes": sorted(startup_sizes),
+        "startup_capture_modes": sorted(startup_modes),
+        "warmup_descriptors": [list(item) for item in sorted(warm_descriptors)],
+        "clean_descriptors": [list(item) for item in sorted(clean_descriptors)],
+        "uncovered_clean_descriptors": [list(item) for item in uncovered],
+        "invariants": {
+            "events_preclean": event_gate,
+            "warmup_completions_ge_16": len(warm) >= 16,
+            "warmup_long_completion": len(warm_long) >= 1,
+            "startup_modes_complete": startup_modes == {"FULL", "PIECEWISE"},
+            "startup_sizes_present": bool(startup_sizes),
+            "clean_descriptors_covered": not uncovered,
+            "zero_clean_capture_events": clean_events == 0,
+        },
+    }
+
+
+def validate_rate_client(run_dir: Path) -> dict[str, Any]:
+    result = json.loads((run_dir / "client/result.json").read_text())
+    sanity = json.loads((run_dir / "client/sanity.json").read_text())
+    requests = [
+        json.loads(line)
+        for line in (run_dir / "client/requests.jsonl").read_text().splitlines()
+    ]
+    failed_sanity = [
+        key for key, value in sanity["invariants"].items()
+        if not value and key != "drain_within_timeout"
+    ]
+    failure_summary = m.summarize_request_failures(
+        requests, float(result["clean"]["start_s"]), float(result["clean"]["end_s"])
+    )
+    cold = cold_start_gate(run_dir)
+    quarantined = float(result["drain_seconds"]) > 600
+    invariants = {
+        "client_sanity": not failed_sanity,
+        "clean_duration": math.isclose(float(result["clean"]["duration_s"]), 240.0),
+        "clean_failures_zero": result["clean"]["failed"] == 0
+        and failure_summary["clean_failed"] == 0,
+        "failed_records_accounted": result["failed_records"] == failure_summary["failed"],
+        "manifest_no_wrap": not result["manifest_wrapped"]
+        and not result["manifest_exhausted"],
+        "warmup_cold_start_gate": cold["passed"],
+        "profile_count": len(result["profiles"]) == 0,
+        "drain_re_adjudicated": not quarantined,
+    }
+    non_drain = {key: value for key, value in invariants.items() if key != "drain_re_adjudicated"}
+    if not all(non_drain.values()):
+        raise RuntimeError(
+            f"A-P5-1 rate-client invariant failure: {run_dir}: "
+            f"invariants={invariants}; failed_sanity={failed_sanity}; cold={cold}"
+        )
+    return {
+        "result": result,
+        "sanity": sanity,
+        "request_count": len(requests),
+        "warmup_completions": cold["warmup_completions"],
+        "warmup_required": 16,
+        "warmup_gate_branch": "A-P5-1-cold-start",
+        "warmup_stability": None,
+        "cold_start_gate": cold,
+        "drain_budget_seconds": 600,
+        "drain_quarantined": quarantined,
+        "excluded_window_failures": failure_summary["excluded"],
+        "excluded_window_failure_kinds": failure_summary["excluded_kinds"],
+        "invariants": invariants,
+    }
+
+
+def validate_run(
+    entry: dict[str, Any], profile: bool, burnin: bool, allow_missing_traces: bool = False
+) -> dict[str, Any]:
+    del profile, allow_missing_traces
+    pattern = entry["assignment"].cell.pattern
+    if burnin or pattern.startswith("background"):
+        validation_pattern = "P06"
+    elif pattern.startswith("control-P03"):
+        validation_pattern = "P03"
+    elif pattern.startswith("control-P04"):
+        validation_pattern = "P04"
+    else:
+        validation_pattern = "P10"
+    if burnin or pattern.startswith("background"):
+        client = _ORIGINAL_VALIDATE_CLIENT(
+            entry["run_dir"], validation_pattern, False, burnin
+        )
+    else:
+        client = validate_rate_client(entry["run_dir"])
+    layer1 = m.validate_layer1(entry["run_dir"])
+    log = (entry["run_dir"] / "server.log").read_text(errors="replace")
+    invariants = {
+        "triton_moe": "Using TRITON Unquantized MoE backend" in log,
+        "chunked_mbt": "Chunked prefill is enabled with max_num_batched_tokens=8192" in log,
+        "tp1": "tensor_parallel_size=1" in log,
+        "drain_shutdown": "mode=drain timeout=600s" in log,
+        "a2_sizes": entry["assignment"].cell.config != "A2"
+        or all(str(size) in log for size in (3, 5, 6, 7)),
+    }
+    if not all(invariants.values()):
+        raise RuntimeError(f"server invariant failure: {entry['run_id']}: {invariants}")
+    forbidden = re.compile(r'"(?:prompt|messages|content|text)"\s*:')
+    for path in (
+        entry["run_dir"] / "client/requests.jsonl",
+        entry["run_dir"] / "client/result.json",
+        Path(layer1["stream"]),
+    ):
+        if forbidden.search(path.read_text(errors="replace")):
+            raise RuntimeError(f"private text leaked: {path}")
+    summary = {
+        "schema": 1,
+        "run_id": entry["run_id"],
+        "pattern": pattern,
+        "config": entry["assignment"].cell.config,
+        "gpus": entry["assignment"].gpus,
+        "client": client,
+        "layer1": layer1,
+        "traces": [],
+        "missing_trace_files": 0,
+        "layer2_missing_after_controller_cleanup": False,
+        "drain_quarantined": client["drain_quarantined"],
+        "server_invariants": invariants,
+    }
+    m.atomic_json(entry["run_dir"] / "run-complete.json", summary)
+    return summary
+
+
+m.validate_run = validate_run
+
+
+def manifests() -> dict[str, Any]:
+    result = {}
+    for name in ("base", "A1", "A3"):
+        path = PRIVATE / f"P10-{name}.jsonl"
+        summary = json.loads(path.with_suffix(path.suffix + ".summary.json").read_text())
+        if summary["rows"] != 142 or summary["sha256"] != sha256_file(path):
+            raise RuntimeError(f"manifest verification failed: {name}")
+        result[name] = {"path": str(path), "sha256": summary["sha256"]}
+    return result
+
+
+def fingerprint() -> dict[str, Any]:
+    return {
+        "source_commit": subprocess.check_output(
+            ["git", "-C", str(SOURCE), "rev-parse", "HEAD"], text=True
+        ).strip(),
+        "source_tree": subprocess.check_output(
+            ["git", "-C", str(SOURCE), "rev-parse", "HEAD^{tree}"], text=True
+        ).strip(),
+        "client_sha256": sha256_file(CLIENT),
+        "controller_sha256": sha256_file(Path(__file__)),
+        "p3_client_sha256": sha256_file(P3_CLIENT),
+        "p3_matrix_sha256": sha256_file(Path(m.__file__).resolve()),
+        "manifests": manifests(),
+        "capture_sizes": list(CAPTURE_SIZES),
+        "rate": RATE,
+    }
+
+
+def load_state(resume: bool) -> dict[str, Any]:
+    if STATE.exists():
+        if not resume:
+            raise RuntimeError("controller state exists; use --resume")
+        return json.loads(STATE.read_text())
+    return {
+        "schema": 1,
+        "status": "created",
+        "created_at": time.time(),
+        "controller_pid": os.getpid(),
+        "gpu_hours_total": 0.0,
+        "gpu_hours_this_stage": 0.0,
+        "completed_measured_runs": 0,
+        "completed_burnins": 0,
+        "drain_quarantined_runs": 0,
+        "clean_window_failures": 0,
+        "missing_trace_files": 0,
+        "stages": {},
+        "fingerprint": {},
+    }
+
+
+def save_state(state: dict[str, Any]) -> None:
+    state["controller_pid"] = os.getpid()
+    state["updated_at"] = time.time()
+    m.atomic_json(STATE, state)
+
+
+m.save_state = save_state
+
+
+def ensure_provenance() -> None:
+    destination = RUN_ROOT / "provenance"
+    destination.mkdir(parents=True, exist_ok=True)
+    sources = [CLIENT, Path(__file__).resolve(), Path(m.__file__).resolve(), Path(m.common.__file__).resolve()]
+    hashes = {}
+    for source in sources:
+        target = destination / source.name
+        digest = sha256_file(source)
+        if target.exists() and sha256_file(target) != digest:
+            target = destination / f"{source.stem}.{digest[:12]}{source.suffix}"
+        if target.exists() and sha256_file(target) != digest:
+            raise RuntimeError(f"content-addressed provenance mismatch: {target}")
+        if not target.exists():
+            shutil.copy2(source, target)
+        hashes[target.name] = digest
+    m.atomic_json(destination / "sha256.json", hashes)
+
+
+def primary_cells() -> list[m.Cell]:
+    config = {"base": "C00", "A1": "C00", "A2": "A2", "A3": "C00", "A4": "A4"}
+    items = [m.Cell(f"{arm}-r{replicate}", config[arm]) for arm in config for replicate in range(1, 4)]
+    return sorted(
+        items,
+        key=lambda cell: hashlib.sha256(
+            f"20260715:{cell.pattern.rsplit('-r',1)[1]}:{arm_name(cell.pattern)}".encode()
+        ).hexdigest(),
+    )
+
+
+def pack_unique(items: list[m.Cell], prefix: str) -> list[list[m.Assignment]]:
+    remaining = list(items)
+    waves: list[list[m.Assignment]] = []
+    wave_index = 0
+    while remaining:
+        selected: list[m.Cell] = []
+        used: set[str] = set()
+        for cell in list(remaining):
+            key = arm_name(cell.pattern)
+            if key in used:
+                continue
+            selected.append(cell)
+            used.add(key)
+            remaining.remove(cell)
+            if len(selected) == 4:
+                break
+        while len(selected) < 4:
+            selected.append(m.Cell(f"background-{prefix}-{wave_index}-{len(selected)}", "C00"))
+        assignments = []
+        for slot, cell in enumerate(selected):
+            gpu = (slot + wave_index) % 4
+            assignments.append(m.Assignment(cell, (gpu,)))
+        waves.append(assignments)
+        wave_index += 1
+    return waves
+
+
+def pack_primary(items: list[m.Cell]) -> list[list[m.Assignment]]:
+    """Pack SHA-ordered cells as 4/4/4/3 without duplicate arms per wave."""
+    capacities = (4, 4, 4, 3)
+    waves: list[list[m.Cell]] = [[] for _ in capacities]
+
+    def place(index: int) -> bool:
+        if index == len(items):
+            return all(len(wave) == capacity for wave, capacity in zip(waves, capacities, strict=True))
+        cell = items[index]
+        arm = arm_name(cell.pattern)
+        for wave_index, capacity in enumerate(capacities):
+            if len(waves[wave_index]) >= capacity:
+                continue
+            if any(arm_name(existing.pattern) == arm for existing in waves[wave_index]):
+                continue
+            waves[wave_index].append(cell)
+            if place(index + 1):
+                return True
+            waves[wave_index].pop()
+        return False
+
+    if not place(0):
+        raise RuntimeError("cannot pack frozen primary assignments into 4/4/4/3")
+    result: list[list[m.Assignment]] = []
+    for wave_index, cells in enumerate(waves):
+        if len(cells) == 3:
+            cells.append(m.Cell("background-primary-final", "C00"))
+        result.append(
+            [
+                m.Assignment(cell, ((slot + wave_index) % 4,))
+                for slot, cell in enumerate(cells)
+            ]
+        )
+    return result
+
+
+def execute_primary(resume: bool, amendment_a_p5_1: bool = False) -> None:
+    RUN_ROOT.mkdir(parents=True, exist_ok=True)
+    state = load_state(resume)
+    if resume:
+        m.cleanup_recorded(state)
+    current = fingerprint()
+    if state["fingerprint"] and state["fingerprint"] != current:
+        failure = str(state.get("stages", {}).get("primary-01", {}).get("failure", ""))
+        if not (
+            amendment_a_p5_1
+            and state.get("status") == "failed"
+            and "warmup" in failure.lower()
+        ):
+            raise RuntimeError("resume fingerprint differs from frozen Phase-5 plan")
+        state.setdefault("amendments", {})["A-P5-1"] = {
+            "approved": True,
+            "applied_at": time.time(),
+            "reason": "replace rate-following drift gate with cold-start gates",
+            "prior_fingerprint": state["fingerprint"],
+            "replacement_fingerprint": current,
+            "retained_gpu_hours": state["gpu_hours_total"],
+            "burnins_reused": state["completed_burnins"],
+        }
+        state["fingerprint"] = current
+        state["status"] = "amended_resume_A-P5-1"
+        save_state(state)
+    state["fingerprint"] = current
+    state["status"] = "running_primary"
+    save_state(state)
+    ensure_provenance()
+    burnins = [
+        m.Assignment(m.Cell("burnin-C00", "C00"), (0,)),
+        m.Assignment(m.Cell("burnin-A2", "A2"), (1,)),
+        m.Assignment(m.Cell("burnin-A4", "A4"), (2,)),
+    ]
+    m.run_stage(state, "burnins", burnins, "saturation", profile=False, burnin=True)
+    waves = pack_primary(primary_cells())
+    for index, wave in enumerate(waves, 1):
+        m.run_stage(state, f"primary-{index:02d}", wave, "moderate", profile=False)
+    primary = [
+        path for path in (RUN_ROOT / "primary").glob("*-r*-*/moderate/run-complete.json")
+        if "background" not in str(path)
+    ]
+    if len(primary) != 15:
+        raise RuntimeError(f"primary completion mismatch: {len(primary)} != 15")
+    state["primary_runs"] = 15
+    state["background_runs"] = state["completed_measured_runs"] - 15
+    state["status"] = "primary_complete"
+    state["completed_at"] = time.time()
+    save_state(state)
+
+
+def execute_controls(resume: bool) -> None:
+    state = load_state(resume)
+    m.cleanup_recorded(state)
+    if state.get("fingerprint") != fingerprint():
+        raise RuntimeError("control resume fingerprint mismatch")
+    cells = [
+        m.Cell(f"control-{pattern}-r{replicate}", "C00")
+        for pattern in ("P03", "P04") for replicate in range(1, 4)
+    ]
+    for index, wave in enumerate(pack_unique(cells, "controls"), 1):
+        m.run_stage(state, f"controls-{index:02d}", wave, "moderate", profile=False)
+    state["status"] = "controls_complete"
+    state["controls_completed_at"] = time.time()
+    save_state(state)
+
+
+def plan() -> dict[str, Any]:
+    return {
+        "schema": 1,
+        "primary_runs": 15,
+        "burnins": 3,
+        "waves": [[{"cell": a.cell.cell_id, "gpus": a.gpus} for a in wave] for wave in pack_primary(primary_cells())],
+        "rate": RATE,
+        "clean_seconds": 240,
+        "drain_seconds": 600,
+        "gpu_hour_limit": 6.0,
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    sub = parser.add_subparsers(dest="command", required=True)
+    for name in ("primary", "controls"):
+        item = sub.add_parser(name)
+        item.add_argument("--resume", action="store_true")
+        if name == "primary":
+            item.add_argument("--amendment-a-p5-1", action="store_true")
+    sub.add_parser("plan")
+    sub.add_parser("status")
+    args = parser.parse_args()
+    if args.command == "primary":
+        execute_primary(args.resume, args.amendment_a_p5_1)
+    elif args.command == "controls":
+        execute_controls(args.resume)
+    elif args.command == "plan":
+        print(json.dumps(plan(), sort_keys=True, indent=2))
+    else:
+        print(STATE.read_text() if STATE.exists() else '{"status":"absent"}')
+
+
+if __name__ == "__main__":
+    main()

runs/opprof-phase5/phase5/controller-state.json

@@ -0,0 +1,212 @@
+{
+  "clean_window_failures": 0,
+  "completed_burnins": 3,
+  "completed_measured_runs": 0,
+  "controller_pid": 2505724,
+  "created_at": 1783858074.2830184,
+  "drain_quarantined_runs": 0,
+  "fingerprint": {
+    "capture_sizes": [
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      16,
+      24,
+      32,
+      40,
+      48,
+      56,
+      64,
+      72,
+      80,
+      88,
+      96,
+      104,
+      112,
+      120,
+      128,
+      136,
+      144,
+      152,
+      160,
+      168,
+      176,
+      184,
+      192,
+      200,
+      208,
+      216,
+      224,
+      232,
+      240,
+      248,
+      256,
+      272,
+      288,
+      304,
+      320,
+      336,
+      352,
+      368,
+      384,
+      400,
+      416,
+      432,
+      448,
+      464,
+      480,
+      496,
+      512
+    ],
+    "client_sha256": "f935486ab2588c1fca514be29b59361f53118eb000ea037863de48ce4fc76b16",
+    "controller_sha256": "ea2e2066a8b6d9427c59fe80db44cf87a86776570ca9066bfdff0c0b0e7f4a46",
+    "manifests": {
+      "A1": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase5-private/manifests/P10-A1.jsonl",
+        "sha256": "cab8468983fb7397ec88eb5e88a44c8b1b53d8021b7867e7bec6f58d3d903806"
+      },
+      "A3": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase5-private/manifests/P10-A3.jsonl",
+        "sha256": "ec5eaa29fcd3ec421e4f68a902c7e7fea9c0bc6b16f1013cef30ccc1f97bae24"
+      },
+      "base": {
+        "path": "/home/admin/cpfs/wjh/opprof-phase5-private/manifests/P10-base.jsonl",
+        "sha256": "d3b4f540ddd629dd0e34fff55c3b3837f359bdd6e5947309a1ea2a358f811ffc"
+      }
+    },
+    "p3_client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0",
+    "p3_matrix_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f",
+    "rate": 0.4725,
+    "source_commit": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05",
+    "source_tree": "a3d536b287a724e60abbec68b45eed7e088a15d1"
+  },
+  "gpu_hours_this_stage": 0.4224752351972792,
+  "gpu_hours_total": 0.6476566231913037,
+  "missing_trace_files": 0,
+  "schema": 1,
+  "stages": {
+    "burnins": {
+      "assignments": [
+        {
+          "cell": "burnin-C00-C00",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "burnin-A2-A2",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "burnin-A4-A4",
+          "gpus": [
+            2
+          ]
+        }
+      ],
+      "burnin": true,
+      "clients": {},
+      "completed_at": 1783858361.2828288,
+      "confirmation": false,
+      "gpu_hours": 0.22518138799402448,
+      "load_point": "saturation",
+      "profile": false,
+      "servers": {},
+      "started_at": 1783858074.4911764,
+      "status": "complete"
+    },
+    "primary-01": {
+      "assignments": [
+        {
+          "cell": "base-r2-C00",
+          "gpus": [
+            0
+          ]
+        },
+        {
+          "cell": "A3-r1-C00",
+          "gpus": [
+            1
+          ]
+        },
+        {
+          "cell": "A1-r2-C00",
+          "gpus": [
+            2
+          ]
+        },
+        {
+          "cell": "A4-r1-A4",
+          "gpus": [
+            3
+          ]
+        }
+      ],
+      "burnin": false,
+      "clients": {
+        "A1-r2-C00-moderate": {
+          "pgid": 2512122,
+          "pid": 2512122
+        },
+        "A3-r1-C00-moderate": {
+          "pgid": 2512121,
+          "pid": 2512121
+        },
+        "A4-r1-A4-moderate": {
+          "pgid": 2512123,
+          "pid": 2512123
+        },
+        "base-r2-C00-moderate": {
+          "pgid": 2512119,
+          "pid": 2512119
+        }
+      },
+      "confirmation": false,
+      "failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5/primary/base-r2-C00/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'failed_records_accounted': True, 'manifest_no_wrap': True, 'warmup_completions': False, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]; warmup_completions=25; warmup_gate_branch=failed; warmup_stability={'passed': False, 'reason': 'A-P3-6 stabilization criterion not met', 'window_seconds': [45.0, 60.0], 'bin_seconds': 5.0, 'step_counts': [380, 201, 187], 'scheduled_tokens': [26927, 27616, 463], 'scheduled_token_throughput': [5385.4, 5523.2, 92.6], 'mean_scheduled_token_throughput': 3667.066666666666, 'slope_tokens_per_second_squared': -529.28, 'normalized_drift': 2.1650001817983497, 'normalized_drift_limit': 0.1, 'step_indices_continuous': True}\")",
+      "gpu_hours": 0.4224752351972792,
+      "load_point": "moderate",
+      "profile": false,
+      "servers": {
+        "A1-r2-C00-moderate": {
+          "gpus": [
+            2
+          ],
+          "pgid": 2510424,
+          "pid": 2510424
+        },
+        "A3-r1-C00-moderate": {
+          "gpus": [
+            1
+          ],
+          "pgid": 2510423,
+          "pid": 2510423
+        },
+        "A4-r1-A4-moderate": {
+          "gpus": [
+            3
+          ],
+          "pgid": 2510425,
+          "pid": 2510425
+        },
+        "base-r2-C00-moderate": {
+          "gpus": [
+            0
+          ],
+          "pgid": 2510422,
+          "pid": 2510422
+        }
+      },
+      "started_at": 1783858361.3184204,
+      "status": "failed"
+    }
+  },
+  "status": "failed",
+  "updated_at": 1783858758.067235
+}

runs/opprof-phase5/phase5/launch-echo.log

@@ -0,0 +1 @@
+LAUNCH_ECHO utc=2026-07-12T12:07:54Z host=dash0 gpus=0-3 cpus=0-79 source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0@4b253fd manifests=/home/admin/cpfs/wjh/opprof-phase5-private/manifests outputs=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5 runs=3burnin+15primary+1background rate=0.4725 warmup=60s clean=240s drain=600s est_wall=35-60min est_gpu=1.7-2.1_H20h hard_cap=6.0_H20h conditional_controls=6_if_bridge_fails