From 0d16838097c4b9467d8df58ce449fbbebee2f78b Mon Sep 17 00:00:00 2001
From: Gahow Wang <gahow.wang@gmail.com>
Date: Wed, 15 Jul 2026 01:41:25 +0800
Subject: [PATCH] Audit tuning cost and core challenges

---
 ...ing-core-challenges-cost-audit-20260715.md | 391 ++++++++++
 runs/tuning-cost/analyze.py                   | 507 ++++++++++++
 runs/tuning-cost/manifest.json                |  42 +
 runs/tuning-cost/metrics.json                 | 736 ++++++++++++++++++
 runs/tuning-cost/test_analysis.py             |  57 ++
 5 files changed, 1733 insertions(+)
 create mode 100644 docs/tuning-core-challenges-cost-audit-20260715.md
 create mode 100644 runs/tuning-cost/analyze.py
 create mode 100644 runs/tuning-cost/manifest.json
 create mode 100644 runs/tuning-cost/metrics.json
 create mode 100644 runs/tuning-cost/test_analysis.py

diff --git a/docs/tuning-core-challenges-cost-audit-20260715.md b/docs/tuning-core-challenges-cost-audit-20260715.md
new file mode 100644
index 0000000..200bdae
--- /dev/null
+++ b/docs/tuning-core-challenges-cost-audit-20260715.md
@@ -0,0 +1,391 @@
+# AITuner tuning：核心挑战、统一成本口径与研究路线
+
+日期：2026-07-15（Asia/Singapore）
+
+状态：**问题定义与历史成本审计完成；新的 tuner 贡献尚未建立。**
+
+## 结论先行
+
+我们不应该把 tuning 定义成“根据当前 telemetry 判断哪个 cap 满了，再调对应 knob”。这个定义同时遗漏了 knob interaction、反事实识别、实验成本和跨任务失配。更准确的问题是：
+
+> 给定模型、engine version、hardware、workload、SLO 和一个声明好的合法配置空间，tuner 如何用最少的真实 GPU 成本，依次选择可能包含多个 knob 的 intervention，找到 SLO-goodput regret 不超过 `epsilon` 的配置？
+
+AITuner 可以形成的系统贡献应当是：
+
+> **一个 intervention-calibrated、action-conditioned、cost-aware 的 tuner：它从真实 engine trajectory 和已测 intervention 中学习联合 config action 的反事实收益分布，并以 cost-to-oracle 而非规则命中率作为目标。Harness 只负责实验语义、合法性、配对、记账和可复现性，不负责用人工 bottleneck rule 决定 action。**
+
+现有结果支持这个问题值得做，但不支持宣称它已经解决：
+
+- 在真实 `TP x MNS` surface 上，one-knob-at-a-time 会停在比 oracle 低 **25.6%** 的 coordinate-wise local optimum。
+- 在 action-aware pilot 中，增加 MBBT 在“几乎从未独占打满 MBBT cap”的情况下仍把 source goodput 提高 **48.0%--77.1%**；因此 `cap -> knob` 不是完整模型。
+- 同一 dash0 任务上，当前 guided harness 到 5% empirical regret 只比纯 LLM 少 **5.85%** H20-hours；到 2% regret 则少 **61.09%**。这说明必须比较完整 cost--regret curve，不能只比较最终最好值。
+- Frontier 的 decision-bearing throughput top-1 在 12-cell surface 上有 **30.46%** real regret。Simulator 本身的边际 GPU cost 是 0，但通过 real-final 恢复 oracle 需要 tie-expanded 4 个真实 cell，即 **0.7828 reconstructed H20-hours**。
+
+## 1. Tuning 问题和成功标准
+
+固定 task context：
+
+```text
+T = {model, engine build, hardware, workload/trace, SLO, legal config space C}
+```
+
+每个完整配置 `c in C` 的目标为：
+
+```text
+f_T(c) = max request_rate_per_gpu
+         subject to request SLO pass rate >= target
+```
+
+有限空间 oracle 为：
+
+```text
+f*_T = max_{c in C} f_T(c)
+regret(c) = 1 - f_T(c) / f*_T
+```
+
+顺序 tuner 在第 `t` 步基于历史 `D_t` 选择一个完整 config intervention：
+
+```text
+a_t = c_t -> c_{t+1}
+```
+
+成功不是“最后找到一个不错的值”，而是同时满足：
+
+1. `regret(best_t) <= epsilon`；
+2. 达到该点之前的 all-in H20-hours 最小；
+3. launch、correctness、SLO 和失败率约束不退化；
+4. 结论在 held-out task 上成立，而不是在用于设计规则的 task 上成立。
+
+### 1.1 GPU cost 的统一定义
+
+未来实验的 task-marginal cost 应定义为：
+
+```text
+C_task = sum_j allocated_GPU_count_j
+                 * (GPU_idle_or_release_time_j - allocation_start_time_j)
+```
+
+它包括 method 实际触发的 startup、warm-up、prefix/full replay、confirmation、failure、cleanup；如果 LLM 思考期间 GPU 仍被占用，也计入。Simulator/模型的一次性 onboarding 成本单独报告：
+
+```text
+C_e2e(N tasks) = C_profile_or_training / N + C_task
+```
+
+另外报告 CPU-hours、LLM API latency/cost，但不把它们伪装成 GPU-hours。构建 benchmark oracle 的 exhaustive annotation cost 是公共评测成本，单独报告，不计入任何方法；同时可给一个将其等量加回所有方法的 conservative view。
+
+历史记录没有 allocation start/release timestamp。本次只能从每个 `engine.log` 的首末时间戳重建：
+
+```text
+C_engine_lower_bound = parallel_size * engine_log_span / 3600
+```
+
+因此下面所有历史 H20-hour 数字都是 **engine-lifetime lower bound**，不是 all-in cost。尤其 simulator 的一次性 H20 operator profiling 成本没有记录，不能称为完全免费。
+
+### 1.2 两种 oracle 必须分开
+
+- **Exact finite-surface oracle**：声明好的 12-cell `TP x MNS` 空间全部真实测量，oracle 是 `TP2/MNS32 = 3.2833 req/s/GPU`。
+- **Broader empirical reference**：dash0 两个 sequential run 中观察到的最好值 `3.35 req/s/GPU`。它包含 surface 外的 MBBT/chunk/GMU action，但只是 best observed，不是全局 oracle。
+
+不能把 empirical best 写成 global oracle，也不能让每个方法使用不同的 oracle 定义。
+
+## 2. 现有方案的 cost-to-oracle 审计
+
+可复算输入和完整结果在：
+
+- `runs/tuning-cost/manifest.json`
+- `runs/tuning-cost/analyze.py`
+- `runs/tuning-cost/metrics.json`
+
+### 2.1 严格同任务对照：纯 LLM vs 当前 guided harness
+
+两组均为 dash0、Qwen3-30B-A3B、community-vLLM 0.20.0、8xH20 可见、`chat_w20260311_1000`、input 0--8k、output 128、replay scale 0.1、TTFT 2/4/6s、TPOT 50ms、pass rate 0.95。除 tuner method 和服务端口外，固定 task spec 相同。
+
+Reference 是两组中 best observed `3.35 req/s/GPU`：
+
+| Method | 到 <=5% regret | 到 <=2% regret | 到 <=1% regret | 完整 run 成本 | 最终 best |
+|---|---:|---:|---:|---:|---:|
+| Pure LLM, no harness | 0.2847 H20h，trial 2，regret 2.736% | 1.1458，trial 6，regret 1.493% | 1.3719，trial 7，regret 0% | 2.2825 | 3.35 |
+| Guided harness v2 | 0.2681 H20h，trial 2，regret 2.736% | 0.4458，trial 3，regret 1.990% | 未达到 | 0.6231 | 3.30，regret 1.493% |
+
+直接结论：
+
+- 5% endpoint：guided 比 pure LLM 少 **5.85%**，不是 material contribution。
+- 2% endpoint：guided 比 pure LLM 少 **61.09%**，有明显 headroom signal，但只有一个 task，不能外推。
+- Pure LLM 在 trial 7 已找到 best observed，之后又花了 `2.2825 - 1.3719 = 0.9106 H20h` 而没有改进，说明 trustworthy stopping 本身就是成本来源。
+- Pure LLM 的 trial 3 使用当前 binary 不支持的 `--expert-parallel-size` 并在 launch 前失败。当前 harness 的 legality/version contract 有实际价值，但它仍不是性能 action-ranking 贡献。
+
+### 2.2 Simulator：零边际 GPU cost 不等于零 tuning cost
+
+Frontier fidelity suite 在 CPU 上执行 184 个 simulation，耗时 **2.055 CPU-hours**，simulation 本身为 0 marginal H20-hours。其对应的 exact dash1 12-cell real surface annotation lower bound 为 **3.5953 H20-hours**。
+
+Decision-bearing `frozen-calibrated/throughput-proxy`：
+
+| Policy | Real cells evaluated | Real-final H20h lower bound | Selected real regret |
+|---|---:|---:|---:|
+| Simulator-only top-1 | 0 | 0 | **30.46%**，选 TP1/MNS64 |
+| Throughput top-1 + real final | 1 | 0.1353 | **30.46%** |
+| Throughput top-2 + real final | 2 | 0.2672 | **30.46%** |
+| Throughput nominal top-3 + real final | tie-expanded 4 | 0.7828 | 0%，找到 TP2/MNS32 |
+
+Post-hoc `SLO-gated` reading 把 `{TP2/MNS32, TP2/MNS64}` 放在 top tie bucket；测两个 cell 需 **0.5156 H20h** 并能找到 oracle。但它不是 preregistered decision-bearing policy，而且 anchor verdict 中有 21 个 false-feasible、7 个 false-infeasible，只能作为诊断上界，不能反写成 prospective simulator 结果。
+
+Pure LLM/harness 数据来自 dash0，simulator exact surface 来自 dash1。模型、engine、trace、GPU type 匹配，但 host 和 campaign 不同。因此两块内部可以直接比较，跨块只能做 development-level 指示；paper 结论必须在同 host、同 task execution protocol 下重跑。
+
+### 2.3 我们要达到的成本目标
+
+在当前 reconstructed lower-bound 口径下，一个有意义的单任务 development bar 是：
+
+| Endpoint | 当前最强同任务 baseline | 20% reduction bar | 兼顾 post-hoc sim+real 的 30% bar | 暂定目标 |
+|---|---:|---:|---:|---:|
+| <=5% empirical regret | guided 0.2681 | 0.2144 | 0.3609 | **<=0.2144 H20h** |
+| <=2% empirical regret | guided 0.4458 | 0.3567 | 0.3609 | **<=0.3567 H20h** |
+
+这两个数字不是 paper result，只用于检查 proposed method 是否有足够 headroom：
+
+- 5% endpoint 已经由 baseline + TP2 两个完整 trial 达到。任何必须先跑 source 再跑 target 的 telemetry tuner 都不能靠减少 trial count 获得 20% 优势；它必须能够 one-shot warm-start、跳过 baseline，或安全地缩短其中一次测量。
+- 2% endpoint 有更合理的结构性空间：从一个 source 直接选择 joint `TP2 + MBBT/chunk` target，可能跳过当前中间 trial；如果仍按当前三次完整 trial 顺序执行，就不会达到 bar。
+
+Paper-facing gate 不使用这些跨 campaign 绝对数，而使用 prospective same-host all-in cost：在每个 held-out task 上 regret <=5%，相对最强 safe outcome-only/current harness 至少省 20%，相对 frozen simulator+real 至少省 30%，并报告 task-level paired confidence interval。
+
+## 3. 四个最核心的 tuning challenge
+
+### Challenge 1：响应面是联合、条件化且 regime-dependent 的
+
+#### 问题本质
+
+一般情况下：
+
+```text
+f(c) != base + sum_k effect_k(c_k)
+```
+
+一个 knob 的 effect 是当前完整 context 的函数：
+
+```text
+Delta_x(c, workload, engine state)
+```
+
+它可能随 topology、另一个 runtime knob、load、SLO 或 engine version 改变大小甚至改变符号。因此不能先分别求每个 knob 的最优值再 merge，也不能固定一个低质量 context 去判断另一个 knob。
+
+#### 已有真实证据
+
+在 C1 12-cell real surface：
+
+- `MNS 8 -> 32` 在 TP1/TP2/TP4 下分别提升约 **8.7% / 44.3% / 90.3%**。
+- 从同一 `TP1/MNS8` 起点，先 tune MNS 再 TP 会停在 `TP4/MNS16 = 2.4417`；该点沿任一单维都没有 strictly improving move，但 joint/global surface oracle `TP2/MNS32 = 3.2833` 高 **34.5% relative to the local point**，即 local point 对 oracle 有 **25.6% regret**。
+- C3 中 `MBT 256 -> 384` 的 effect 根据 topology/MNS 从 0 到约 -9.2%；`MNS 64 -> 128` 从 0 到约 +10.1%。
+- Action-aware Regime A 中 MBBT 几乎从不作为 exclusive cap，但 MBBT action 仍把 source goodput 提高 48.0%--77.1%。它通过 chunk size、prefill packing 和 scarce MNS slot residency 的联合变化获得收益。
+
+这直接否定两类通用策略：OAT/coordinate greedy，以及 `which cap is full -> tune that knob`。
+
+#### Tuner 必须具备的能力
+
+- Action 的基本单位是完整 `config delta`，允许 sparse joint action，而不是孤立 knob/value。
+- 对 topology/runtime family 使用 crossed anchors 或信息增益设计，主动测 interaction；不是默认所有 interaction 都强。
+- 能从数据判断 task 是 topology-dominant、runtime-interaction-dominant 还是 flat/noisy，并据此分配实验，而不是把固定 search order 写进规则。
+
+### Challenge 2：当前状态是 observational signal，tuning 需要 counterfactual identification
+
+#### 问题本质
+
+一次 telemetry trace 只能告诉我们：
+
+```text
+P(engine trajectory | current config, workload)
+```
+
+Tuning 真正需要的是：
+
+```text
+P(Delta SLO-goodput, failure, cost
+  | source trajectory, proposed full-config action)
+```
+
+Queue、KV、padding、split prefill 等状态既可能是原因，也可能是 workload/config 的结果。看见某种状态，不等于知道哪个 action 能修复它。一个 action 也可能同时改变多条机制；例如 MBBT 同时改变总 token budget、per-request chunk 和 multi-request packing，现有 telemetry 的解释是 mechanism-consistent，不是已完成的 causal decomposition。
+
+#### 已有真实证据
+
+- 5/10 秒 telemetry 确实太短；300 秒 phase-aware experiment 中，MNS action 的 queue/padding 机制直到 replay 75%--100% 才稳定出现。
+- 但 external TTFT outcome 在 25% 已完美区分该 action 是否修复 SLO。Telemetry 解释了 why，却没有比 outcome 更早或更可靠地指导 tuning。
+- 3.125 req/s/GPU 的 source 无法在 timeout 内 drain；另一组 source 已达 offered ceiling 的 99.1%--100%，数学上不可能通过 10% improvement gate。没有 exposure/headroom 和 censoring control，模型学到的不是 action response。
+- Same-config repeats 与 matched intervention 的波动不可忽略；只比较两个未经配对的 run 会混入 arrival/order/warm-state noise。
+
+#### Tuner 必须具备的能力
+
+- 训练样本必须是 exact-workload paired intervention：`(source trajectory, action) -> target delta`，保留失败和 censoring。
+- 使用 phase-binned continuous trajectory，而不是人工 bottleneck label 或 threshold rule。
+- 输出 response distribution 和 uncertainty；证据不足时 abstain，而不是强行给 diagnosis。
+- Telemetry 的价值必须通过同 cutoff、同 model capacity 的 outcome-only ablation 证明。若不能降低 end-to-end H20-hours，instrumentation 只保留为 debugging/解释工具。
+
+### Challenge 3：这是异构成本下的 sequential experimental design，不是静态 ranking
+
+#### 问题本质
+
+每个 trial 的成本不同：TP4 是 TP1 的四倍 GPU multiplier，startup/warm-up 可能主导短 probe，失败也有成本；同时 tuner 不知道 oracle，只能在 exploitation、information gain 和 cost 之间权衡。选对 top-1 的 accuracy 不能代表 tuning 效果。
+
+必须回答三个连续问题：
+
+1. 下一次测哪个联合 action？
+2. 测多久，何时 continuation/confirmation？
+3. 什么证据允许停止，并声称 best 已在 `epsilon` 内？
+
+#### 已有真实证据
+
+- Pure LLM 达到 best observed 后仍浪费 0.9106 reconstructed H20h。
+- Simulator top-1 虽然 0 marginal GPUh，却因 rank error 损失 30.46%；real-final 的 k 增大又迅速增加 H20h。
+- 5% endpoint 上两个方法都只需两个 trial，selection-count headroom 很小；2% endpoint 才暴露 action quality 和 stopping 的巨大差异。
+- Prefix 不是天然便宜：如果 startup、warm-up 和稳定状态形成占主要成本，缩短 replay window 未必带来等比例 H20h reduction。
+
+#### Tuner 必须具备的能力
+
+- Acquisition 直接优化 expected regret reduction / predicted H20 cost，并把 failure probability 纳入约束。
+- 在 run 前做与 tuning policy 分离的 workload admissibility check：避免 outcome ceiling、无法 drain、无请求或 measurement cap。
+- 使用 uncertainty-aware continuation 和 stop；stop criterion 针对声明的 candidate set 中“仍存在 >epsilon improvement 的概率”，而不是连续几次没提升。
+- 主结果报告 H20-hours-to-5%/2%/1%、fixed-budget regret 和 cost-normalized regret AUC，不 metric shopping。
+
+### Challenge 4：任何 mechanism model 都有 fidelity 和 transfer boundary
+
+#### 问题本质
+
+Simulator、learned surrogate、LLM prior 都是近似。Workload、SLO、model、hardware、engine version 改变后，operator cost、scheduler state transition、合法 flag 和 response surface 都可能变化。模型在 calibration task 上解释得好，不表示能在 held-out task 上排序正确。
+
+#### 已有真实证据
+
+- Frontier throughput reading 在完全匹配的 12-cell task 上仍把 real oracle 排错，top-1 regret 30.46%。这说明预测绝对 throughput 还不够，局部 rank fidelity 才是 tuning 关键。
+- Post-hoc SLO reading 的 top bucket 正确，但有大量 anchor feasibility error，也没有 prospective policy status。
+- Pure LLM 提出了当前 community-vLLM binary 不支持的 flag；engine/API version knowledge 本身会漂移。
+- 已有 cross-version experiment 中 vLLM 0.20 的强配置在 0.24 上出现大幅退化，说明 response prior 不能无条件迁移。
+
+#### Tuner 必须具备的能力
+
+- Simulator 只能作为 prior mean 或 candidate prior；真实 outcome 是 authoritative update。
+- 学习 simulator residual：把 `sim prediction + source state + action` 映射到 real response，而不是用 telemetry 重新实现另一个无校准 simulator。
+- 对 task-level OOD 显式提高 uncertainty/abstain；train/test 按完整 task 分割，不能按 request、anchor 或同一 surface cell 随机分割。
+- 分开报告 cold-start profile/training cost 与 per-task marginal cost，并在 N=1/10/100 等 amortization horizon 下展示。
+
+## 4. 对应的系统设计
+
+### 4.1 Harness：从 rule-based tuner 收缩成 experimental control plane
+
+Harness 保留以下确定性职责：
+
+- engine-version-aware config schema、合法性和资源约束；
+- 完整 config/action canonicalization，禁止隐式 merge 和重复试验；
+- exact trace/request/arrival/length hash，配对、随机化和 counter-rotation；
+- engine trajectory、external outcome、failure/censoring 的统一时间轴；
+- all-in GPU cost ledger、oracle annotation 分账、budget enforcement；
+- data sanity、coverage、SLO/correctness 和 stop-proof audit。
+
+Harness **不**包含 `queue > N -> increase MNS`、`cap full -> tune knob` 或人工 diagnosis-to-action mapping。这里的规则是实验语义和安全 invariant，不是性能决策 heuristic。
+
+### 4.2 Action-conditioned response model
+
+每条学习记录为：
+
+```text
+x = {source full config,
+     workload/SLO context,
+     source external outcome,
+     phase-binned engine trajectory}
+a = normalized full-config delta
+y = {Delta SLO-goodput, target feasibility/failure, measured H20 cost}
+```
+
+学习：
+
+```text
+p_theta(y | x, a, optional simulator prediction)
+```
+
+第一版应使用适合小数据且有 uncertainty 的 action-conditioned Gaussian-process/bootstrapped surrogate；kernel/feature ablation包括：
+
+1. config + external outcome；
+2. 同样输入 + telemetry trajectory；
+3. simulator + config + outcome；
+4. 同样输入 + telemetry residual features。
+
+Telemetry 保留 continuous phase distributions：queue/running residency、MNS/token slack、prefill/decode composition、partial/split prefill、step duration、KV、graph/padding。模型学习它们与 action 的 interaction；不先压成 bottleneck label。
+
+### 4.3 Cost-aware policy
+
+在合法的 single/joint candidate set 上选择：
+
+```text
+a* = argmax_a
+       expected constrained improvement(a)
+       / expected all-in H20 cost(a)
+```
+
+探索项来自 posterior uncertainty/information gain；launch/SLO failure 有显式 penalty。Simulator 可提供 prior mean，但 simulator 与 real discrepancy 会被 posterior residual 更新。一次 target measurement 后更新 response model，并重新计算下一步 action 或停止概率。
+
+LLM 在这个 tuning core 中不是 telemetry classifier。它最多作为可移除的 candidate/prior source，提出 schema 内的 sparse joint actions 或检索 engine mechanism；每个 proposal 都由同一个 response model、cost acquisition 和 real validator 评分。只有 `with LLM` 相对 `same tuner without LLM` 在 held-out tasks 上继续降低 cost-to-oracle，才能讨论 LLM 必要性。
+
+### 4.4 Stop 条件
+
+对一个预先声明的有限 candidate set，满足以下条件才 stop：
+
+```text
+P(exists c: f(c) > best_observed / (1 - epsilon) | D_t) < alpha
+```
+
+并且 best config 通过独立 confirmation、SLO/correctness gate，remaining candidate 的 cost-aware value of information 低于阈值。停止原因、posterior coverage 和未测区域必须写入 audit。
+
+## 5. 下一阶段如何证明，而不是再次构造 heuristic
+
+### R0：已有数据 retrospective premise check
+
+- 用 C1/C3 response surfaces 检查 joint model 是否能避免 OAT trap。
+- 用 action-aware paired records 比较 outcome-only 与 +telemetry 的 action-delta calibration。
+- 用 SimFid surface 比较 direct model 与 simulator-residual model 的 rank/regret。
+- 所有 feature、kernel、candidate encoding 在 held-out task 结果之前冻结。
+
+R0 只能筛选 model family，不能作为 paper result，因为现有 tasks 已参与路线设计。
+
+### R1：prospective same-host cost-to-oracle pilot
+
+- dash0 8xH20，固定 engine build/model；serialized placement，禁止共置干扰。
+- 至少一个未参与 feature/threshold 选择的新 trace window；选择非 ceiling、可 drain 的 offered load。
+- 声明一个可穷举的小 surface，至少包含 topology/runtime crossed actions，而不是只有一个 MNS ladder。
+- Oracle annotation 与 tuner online actions 分开记账；method 只能看到当时可用的数据。
+- 运行 random/search、OAT、纯 LLM、当前 guided harness、frozen simulator+real、outcome-only response、+telemetry response、sim-residual +telemetry。
+- 比较完整 H20 cost-to-regret curve，而不是 action classification accuracy。
+
+Pilot opening gate：
+
+1. telemetry model 相对相同 response model 去掉 telemetry，确实改变至少一个正确的 prospective action ranking；
+2. 最终 regret <=5%，无 false-safe accept；
+3. all-in H20-hours 相对 strongest safe outcome-only 至少下降 20%；
+4. 如果使用 simulator，需相对 frozen simulator+real 至少下降 30%；
+5. instrumentation overhead <=1%，所有成本和失败均计入。
+
+若 1--5 任一失败，就不能把 telemetry/harness 写成 tuning contribution；保留其 debugging/measurement 价值即可。
+
+### R2：task-held-out replication
+
+至少 3 个 workload window x 2 个 SLO regime，按完整 task 做 leave-one-task-out 或固定 train/test split。报告每个 task 的 regret、安全和成本，以及 task-level paired bootstrap CI。只有 R2 通过，才能把单 task 的 61.09% lower-bound saving 升级为项目贡献。
+
+## 6. 当前能说与不能说的贡献
+
+当前能说：
+
+- 我们有真实反例证明 OAT 和 cap-to-knob mapping 不是通用 tuning strategy。
+- Harness 的 legality、exact replay、failure/cost accounting 有必要的实验基础设施价值。
+- 当前 guided sequence 在一个严格同任务比较中显著减少了达到 2% empirical regret 的 reconstructed engine cost。
+- Simulator 的边际计算便宜，但 rank error 会转化成显著 real regret 或更多 real-final 成本。
+
+当前不能说：
+
+- telemetry 已经对 end-to-end tuning 提供独立增益；现有 direct pilot 对此为 negative。
+- 当前 harness 的 heuristic action ranking 是系统贡献；5% endpoint 只省 5.85%。
+- LLM 是必要组件；尚无同 policy 的 with/without LLM held-out ablation。
+- simulator 总 tuning cost 是 0；profile GPU cost 未审计，real verification 不能忽略。
+- 3.35 是 global oracle，或 dash0 与 dash1 数字是完全 controlled comparison。
+
+## Data sanity
+
+- Dash0 sequential numeric scores：n=9，min/max `1.1042/3.35`，distinct=7；两组 config outcome 不全相同。
+- Exact surface scores：n=12，min/max `1.2833/3.2833`，distinct=8；12 cells 完整且与 simulator metrics 中的 real scores 一致。
+- Reconstructed trial/cell attempts 包括 4 个无 engine timestamp 的失败：n=32，min/max `0/0.49778 H20h`，distinct=26；所有可重建成本均非负。
+- Sequential regret observations：n=16，min/max `0/0.34328`，distinct=6，全部在 `[0,1]`。
+- Checked invariants：dash0 fixed task contexts 相同（除 method/port）；trial counts 与 manifest 相符；engine log timestamps monotonic；surface cell 唯一且 MBT=8192；simulator 无失败且 predictions 不全相同；scores/results 不全相同；cost 非负；regret bounded。
+- Measurement limitation：primary 12-cell campaign 的 4 个 TP4 pre-ready failure 没有 engine timestamp，随后由 companion campaign 完整重跑；其失败成本在 engine-lifetime reconstruction 中为 0。因此 `3.5953 H20h` 是 completed annotation lower bound，不能作为 all-in annotation cost。这个缺口已显式保留，没有在其上建立 total-cost claim。
diff --git a/runs/tuning-cost/analyze.py b/runs/tuning-cost/analyze.py
new file mode 100644
index 0000000..be3003a
--- /dev/null
+++ b/runs/tuning-cost/analyze.py
@@ -0,0 +1,507 @@
+#!/usr/bin/env python3
+"""Reconstruct tuning cost and cost-to-oracle curves from existing runs.
+
+The historical engine logs do not contain controller setup/cleanup timestamps.
+Consequently the reported GPU cost is an engine-lifetime lower bound:
+parallel_size * (last engine timestamp - first engine timestamp).  It must not
+be presented as all-in method cost.  New experiments should record allocation
+start/end directly.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+import re
+import subprocess
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+from urllib.parse import urlsplit
+
+
+SCHEMA = "aituner-tuning-cost-analysis-v1"
+TIMESTAMP = re.compile(r"\b(\d{2}-\d{2} \d{2}:\d{2}:\d{2})(?:\.\d+)?\b")
+
+
+def numeric_summary(values: list[float]) -> dict[str, Any]:
+    values = [float(value) for value in values]
+    return {
+        "n": len(values),
+        "min": min(values) if values else None,
+        "max": max(values) if values else None,
+        "distinct_n": len(set(values)),
+    }
+
+
+def sha256_text(text: str) -> str:
+    return hashlib.sha256(text.encode("utf-8")).hexdigest()
+
+
+class Reader:
+    def __init__(self, repo_root: Path):
+        self.repo_root = repo_root
+
+    def _split(self, locator: str) -> tuple[str | None, str]:
+        if locator.startswith("ssh://"):
+            parsed = urlsplit(locator)
+            if not parsed.hostname or not parsed.path.startswith("/"):
+                raise ValueError(f"invalid SSH locator: {locator}")
+            return parsed.hostname, parsed.path
+        path = Path(locator)
+        if not path.is_absolute():
+            path = self.repo_root / path
+        return None, str(path)
+
+    def read_text(self, locator: str) -> str:
+        host, path = self._split(locator)
+        if host is None:
+            return Path(path).read_text(encoding="utf-8", errors="replace")
+        completed = subprocess.run(
+            ["ssh", host, "cat", "--", path],
+            check=True,
+            capture_output=True,
+            text=True,
+        )
+        return completed.stdout
+
+def join_locator(root: str, *parts: str) -> str:
+    return "/".join([root.rstrip("/"), *(part.strip("/") for part in parts)])
+
+
+def timestamp_span(log_text: str, year: int) -> tuple[float | None, bool, int]:
+    parsed = [
+        datetime.strptime(f"{year}-{match}", "%Y-%m-%d %H:%M:%S")
+        for match in TIMESTAMP.findall(log_text)
+    ]
+    if not parsed:
+        return None, True, 0
+    monotonic = all(right >= left for left, right in zip(parsed, parsed[1:]))
+    return (max(parsed) - min(parsed)).total_seconds(), monotonic, len(parsed)
+
+
+def load_campaign(
+    reader: Reader,
+    root: str,
+    year: int,
+    missing_duration_s: dict[str, float] | None = None,
+    missing_reason: dict[str, str] | None = None,
+) -> dict[str, Any]:
+    missing_duration_s = missing_duration_s or {}
+    missing_reason = missing_reason or {}
+    state_text = reader.read_text(join_locator(root, "state.json"))
+    state = json.loads(state_text)
+    trials = []
+    for trial in state["trials"]:
+        trial_id = str(trial["trial_id"])
+        log_text = reader.read_text(join_locator(root, "trials", trial_id, "engine.log"))
+        duration_s, monotonic, timestamp_n = timestamp_span(log_text, year)
+        duration_source = "engine_log_span"
+        duration_note = ""
+        if duration_s is None:
+            duration_s = float(missing_duration_s.get(trial_id, 0.0))
+            duration_source = (
+                "conservative_fallback" if trial_id in missing_duration_s else "no_engine_timestamp"
+            )
+            duration_note = missing_reason.get(trial_id, "")
+        parallel_size = int(trial["parallel_size"])
+        score = trial.get("best_request_rate_per_gpu")
+        trials.append(
+            {
+                "trial_id": trial_id,
+                "status": trial["status"],
+                "failure_stage": trial.get("failure_stage", ""),
+                "parallel_size": parallel_size,
+                "score_req_s_per_gpu": None if score is None else float(score),
+                "config_patch": trial["config_patch"],
+                "duration_s": duration_s,
+                "duration_source": duration_source,
+                "duration_note": duration_note,
+                "engine_timestamp_n": timestamp_n,
+                "engine_timestamps_monotonic": monotonic,
+                "engine_h20_hours_lower_bound": duration_s * parallel_size / 3600.0,
+                "engine_log_sha256": sha256_text(log_text),
+            }
+        )
+    return {
+        "root": root,
+        "state_sha256": sha256_text(state_text),
+        "trials": trials,
+    }
+
+
+def fixed_task_context(spec: dict[str, Any]) -> dict[str, Any]:
+    flags = dict(spec["engine"]["base_flags"])
+    flags.pop("port", None)
+    return {
+        "model": spec["model"],
+        "hardware": spec["hardware"],
+        "engine_version": spec["engine"]["engine_version"],
+        "launch_args": spec["engine"]["launch_args"],
+        "base_flags_without_port": flags,
+        "search": spec["search"],
+        "slo": spec["slo"],
+        "trace": spec["trace"],
+    }
+
+
+def regret(score: float, reference: float) -> float:
+    if reference <= 0:
+        raise ValueError("reference score must be positive")
+    return max(0.0, 1.0 - float(score) / float(reference))
+
+
+def sequential_curve(
+    trials: list[dict[str, Any]], reference: float, thresholds: list[float]
+) -> dict[str, Any]:
+    cumulative_cost = 0.0
+    best_score: float | None = None
+    points = []
+    for trial in trials:
+        cumulative_cost += float(trial["engine_h20_hours_lower_bound"])
+        score = trial["score_req_s_per_gpu"]
+        if score is not None:
+            best_score = score if best_score is None else max(best_score, score)
+        points.append(
+            {
+                "trial_id": trial["trial_id"],
+                "cumulative_engine_h20_hours_lower_bound": cumulative_cost,
+                "best_score_req_s_per_gpu": best_score,
+                "regret": None if best_score is None else regret(best_score, reference),
+            }
+        )
+    hits = {}
+    for threshold in thresholds:
+        hit = next(
+            (
+                point
+                for point in points
+                if point["regret"] is not None
+                and float(point["regret"]) <= float(threshold) + 1e-12
+            ),
+            None,
+        )
+        hits[f"regret_le_{threshold:g}"] = hit
+    return {
+        "reference_score_req_s_per_gpu": reference,
+        "points": points,
+        "cost_to_threshold": hits,
+        "total_engine_h20_hours_lower_bound": cumulative_cost,
+    }
+
+
+def surface_cell(trial: dict[str, Any]) -> str:
+    flags = trial["config_patch"]["flag_patch"]
+    return f"tp{int(flags['tensor-parallel-size'])}_mns{int(flags['max-num-seqs'])}"
+
+
+def tie_expanded_candidates(scores: dict[str, float], nominal_k: int) -> list[str]:
+    ordered = sorted(scores, key=lambda cell: (-float(scores[cell]), cell))
+    if nominal_k <= 0 or nominal_k > len(ordered):
+        raise ValueError("nominal k outside score surface")
+    cutoff = float(scores[ordered[nominal_k - 1]])
+    tolerance = max(1e-12, abs(cutoff) * 1e-12)
+    return [cell for cell in ordered if float(scores[cell]) >= cutoff - tolerance]
+
+
+def real_final_policy(
+    candidates: list[str], real_scores: dict[str, float], cell_costs: dict[str, float]
+) -> dict[str, Any]:
+    oracle = max(real_scores.values())
+    selected = max(candidates, key=lambda cell: (real_scores[cell], cell))
+    return {
+        "candidate_cells": candidates,
+        "real_evaluations": len(candidates),
+        "selected_cell": selected,
+        "selected_real_score_req_s_per_gpu": real_scores[selected],
+        "real_regret": regret(real_scores[selected], oracle),
+        "engine_h20_hours_lower_bound": sum(cell_costs[cell] for cell in candidates),
+    }
+
+
+def percentage_saving(new: float, old: float) -> float:
+    if old <= 0:
+        raise ValueError("baseline cost must be positive")
+    return 1.0 - new / old
+
+
+def build_analysis(manifest_path: Path) -> dict[str, Any]:
+    repo_root = manifest_path.resolve().parents[2]
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    reader = Reader(repo_root)
+    year = int(manifest["year"])
+
+    sequential = {}
+    task_contexts = {}
+    for name, run in manifest["sequential_runs"].items():
+        sequential[name] = load_campaign(
+            reader,
+            run["root"],
+            year,
+            run.get("missing_log_duration_s"),
+            run.get("missing_log_reason"),
+        )
+        spec_text = reader.read_text(join_locator(run["root"], "study_spec.snapshot.json"))
+        sequential[name]["study_spec_sha256"] = sha256_text(spec_text)
+        task_contexts[name] = fixed_task_context(json.loads(spec_text))
+
+    all_sequential_scores = [
+        trial["score_req_s_per_gpu"]
+        for campaign in sequential.values()
+        for trial in campaign["trials"]
+        if trial["score_req_s_per_gpu"] is not None
+    ]
+    empirical_reference = max(all_sequential_scores)
+    thresholds = [float(value) for value in manifest["threshold_regrets"]]
+    for campaign in sequential.values():
+        campaign["curve"] = sequential_curve(
+            campaign["trials"], empirical_reference, thresholds
+        )
+
+    surface_manifest = manifest["real_surface"]
+    primary = load_campaign(reader, surface_manifest["primary_root"], year)
+    companion = load_campaign(reader, surface_manifest["tp4_companion_root"], year)
+    completed_surface_trials = [
+        trial
+        for trial in primary["trials"] + companion["trials"]
+        if trial["status"] == "completed" and trial["score_req_s_per_gpu"] is not None
+    ]
+    cells: dict[str, dict[str, Any]] = {}
+    for trial in completed_surface_trials:
+        flags = trial["config_patch"]["flag_patch"]
+        if int(flags["max-num-batched-tokens"]) != int(
+            surface_manifest["fixed_max_num_batched_tokens"]
+        ):
+            raise ValueError("surface MBT invariant failed")
+        cell = surface_cell(trial)
+        if cell in cells:
+            raise ValueError(f"duplicate completed surface cell: {cell}")
+        cells[cell] = trial
+    real_scores = {cell: float(trial["score_req_s_per_gpu"]) for cell, trial in cells.items()}
+    cell_costs = {
+        cell: float(trial["engine_h20_hours_lower_bound"]) for cell, trial in cells.items()
+    }
+    surface_oracle_score = max(real_scores.values())
+    surface_oracle_cells = [
+        cell for cell, score in real_scores.items() if math.isclose(score, surface_oracle_score)
+    ]
+
+    simulator_text = reader.read_text(manifest["simulator_metrics"])
+    simulator = json.loads(simulator_text)
+    simulator_real_scores = {
+        cell: float(score) for cell, score in simulator["real_scores"].items()
+    }
+    surface_matches_simulator = set(real_scores) == set(simulator_real_scores) and all(
+        math.isclose(real_scores[cell], simulator_real_scores[cell], abs_tol=1e-12)
+        for cell in real_scores
+    )
+    throughput = simulator["analyses"]["frozen-calibrated/throughput-proxy"]
+    throughput_scores = {
+        cell: float(score) for cell, score in throughput["simulated_scores"].items()
+    }
+    throughput_real_final = {
+        f"nominal_k_{k}": real_final_policy(
+            tie_expanded_candidates(throughput_scores, k), real_scores, cell_costs
+        )
+        for k in (1, 2, 3)
+    }
+    throughput_top1 = tie_expanded_candidates(throughput_scores, 1)
+    simulator_only_cell = throughput_top1[0]
+    slo = simulator["analyses"]["frozen-calibrated/SLO-gated"]
+    slo_top_bucket = list(slo["metrics"]["top1"]["candidate_cells"])
+    slo_diagnostic = real_final_policy(slo_top_bucket, real_scores, cell_costs)
+
+    pure_hits = sequential["pure_llm"]["curve"]["cost_to_threshold"]
+    guided_hits = sequential["guided_harness"]["curve"]["cost_to_threshold"]
+    direct_comparison = {}
+    for threshold in (0.05, 0.02):
+        key = f"regret_le_{threshold:g}"
+        pure_hit = pure_hits[key]
+        guided_hit = guided_hits[key]
+        if pure_hit is None or guided_hit is None:
+            continue
+        pure_cost = float(pure_hit["cumulative_engine_h20_hours_lower_bound"])
+        guided_cost = float(guided_hit["cumulative_engine_h20_hours_lower_bound"])
+        direct_comparison[key] = {
+            "pure_llm_h20_hours_lower_bound": pure_cost,
+            "guided_harness_h20_hours_lower_bound": guided_cost,
+            "guided_saving_vs_pure_llm": percentage_saving(guided_cost, pure_cost),
+        }
+
+    five_cost = direct_comparison["regret_le_0.05"][
+        "guided_harness_h20_hours_lower_bound"
+    ]
+    two_cost = direct_comparison["regret_le_0.02"][
+        "guided_harness_h20_hours_lower_bound"
+    ]
+    sim_real_cost = float(slo_diagnostic["engine_h20_hours_lower_bound"])
+    target_bars = {
+        "five_percent_regret": {
+            "twenty_percent_below_current_guided": 0.8 * five_cost,
+            "thirty_percent_below_posthoc_sim_slo_real_final": 0.7 * sim_real_cost,
+            "development_target_h20_hours_lower_bound": min(
+                0.8 * five_cost, 0.7 * sim_real_cost
+            ),
+        },
+        "two_percent_regret": {
+            "twenty_percent_below_current_guided": 0.8 * two_cost,
+            "thirty_percent_below_posthoc_sim_slo_real_final": 0.7 * sim_real_cost,
+            "development_target_h20_hours_lower_bound": min(
+                0.8 * two_cost, 0.7 * sim_real_cost
+            ),
+        },
+    }
+
+    all_trial_costs = [
+        float(trial["engine_h20_hours_lower_bound"])
+        for campaign in sequential.values()
+        for trial in campaign["trials"]
+    ] + [
+        float(trial["engine_h20_hours_lower_bound"])
+        for campaign in (primary, companion)
+        for trial in campaign["trials"]
+    ]
+    all_regrets = [
+        float(point["regret"])
+        for campaign in sequential.values()
+        for point in campaign["curve"]["points"]
+        if point["regret"] is not None
+    ]
+    monotonic_logs = all(
+        trial["engine_timestamps_monotonic"]
+        for campaign in [*sequential.values(), primary, companion]
+        for trial in campaign["trials"]
+    )
+    invariants = {
+        "dash0_task_contexts_equal_except_method_and_port": len(
+            {json.dumps(value, sort_keys=True) for value in task_contexts.values()}
+        )
+        == 1,
+        "surface_has_expected_cell_count": len(cells)
+        == int(surface_manifest["expected_cells"]),
+        "surface_matches_simulator_real_scores": surface_matches_simulator,
+        "all_costs_non_negative": all(value >= 0 for value in all_trial_costs),
+        "all_regrets_in_0_1": all(0 <= value <= 1 for value in all_regrets),
+        "surface_scores_not_all_identical": len(set(real_scores.values())) > 1,
+        "sequential_scores_not_all_identical": len(set(all_sequential_scores)) > 1,
+        "engine_log_timestamps_monotonic": monotonic_logs,
+        "sequential_trial_counts_match_manifest": all(
+            len(sequential[name]["trials"]) == int(run["expected_trials"])
+            for name, run in manifest["sequential_runs"].items()
+        ),
+        "simulator_suite_has_no_failed_runs": int(simulator["execution"]["failed_runs"])
+        == 0,
+        "simulator_scores_not_all_identical": len(set(throughput_scores.values())) > 1,
+    }
+    failed_invariants = [name for name, passed in invariants.items() if not passed]
+    if failed_invariants:
+        raise RuntimeError(f"data sanity invariant failed: {failed_invariants}")
+
+    failed_primary_attempts = [
+        trial for trial in primary["trials"] if trial["status"] != "completed"
+    ]
+    return {
+        "schema": SCHEMA,
+        "cost_definition": {
+            "reported_metric": "engine H20-hours lower bound",
+            "formula": "parallel_size * (last_engine_log_timestamp - first_engine_log_timestamp) / 3600",
+            "included": ["engine startup after first timestamp", "warm-up/probes until last timestamp"],
+            "not_reconstructable": [
+                "GPU allocation before first engine timestamp",
+                "controller/LLM latency",
+                "cleanup after last engine timestamp",
+                "one-time simulator operator profiling GPU-hours",
+            ],
+            "future_all_in_metric": "allocation_start_to_GPU_idle * allocated_GPU_count, including failures",
+        },
+        "comparison_scope": {
+            "pure_llm_vs_guided_harness": "direct: same dash0 fixed task context",
+            "simulator_vs_surface": "direct: simulator predictions and exact dash1 12-cell real surface",
+            "dash0_methods_vs_dash1_simulator": "indicative only: matched model/engine/workload/GPU type, different host and campaign",
+        },
+        "empirical_reference": {
+            "score_req_s_per_gpu": empirical_reference,
+            "meaning": "best observed across the two dash0 sequential runs; not a global oracle",
+        },
+        "sequential_runs": sequential,
+        "direct_dash0_comparison": direct_comparison,
+        "real_surface": {
+            "cells": {
+                cell: {
+                    "score_req_s_per_gpu": real_scores[cell],
+                    "engine_h20_hours_lower_bound": cell_costs[cell],
+                }
+                for cell in sorted(cells)
+            },
+            "oracle_score_req_s_per_gpu": surface_oracle_score,
+            "oracle_cells": surface_oracle_cells,
+            "completed_annotation_engine_h20_hours_lower_bound": sum(cell_costs.values()),
+            "failed_primary_attempt_n": len(failed_primary_attempts),
+            "failed_primary_attempt_engine_h20_hours_lower_bound": sum(
+                float(trial["engine_h20_hours_lower_bound"])
+                for trial in failed_primary_attempts
+            ),
+            "primary_state_sha256": primary["state_sha256"],
+            "tp4_companion_state_sha256": companion["state_sha256"],
+        },
+        "simulator": {
+            "marginal_gpu_hours_without_real_verification": 0.0,
+            "observed_fidelity_suite_cpu_hours": float(
+                simulator["execution"]["suite_elapsed_seconds"]
+            )
+            / 3600.0,
+            "observed_fidelity_suite_runs": int(simulator["execution"]["attempted_runs"]),
+            "one_time_profile_gpu_hours": None,
+            "one_time_profile_cost_status": "not recorded; total cold-start cost is unknown",
+            "decision_bearing_throughput_proxy_sim_only_top1": {
+                "selected_cell": simulator_only_cell,
+                "selected_real_score_req_s_per_gpu": real_scores[simulator_only_cell],
+                "real_regret": regret(real_scores[simulator_only_cell], surface_oracle_score),
+                "gpu_hours": 0.0,
+            },
+            "decision_bearing_throughput_proxy_plus_real_final": throughput_real_final,
+            "posthoc_slo_gated_plus_real_final": {
+                **slo_diagnostic,
+                "status": "diagnostic/post-hoc, not a preregistered prospective policy",
+                "false_feasible": int(slo["false_feasibility"]["overall"]["false_feasible"]),
+                "false_infeasible": int(slo["false_feasibility"]["overall"]["false_infeasible"]),
+            },
+            "metrics_sha256": sha256_text(simulator_text),
+        },
+        "provisional_development_targets": {
+            "status": "lower-bound, single-task targets; require same-host prospective validation",
+            **target_bars,
+        },
+        "data_sanity": {
+            "invariants": invariants,
+            "sequential_score_summary": numeric_summary(all_sequential_scores),
+            "surface_score_summary": numeric_summary(list(real_scores.values())),
+            "simulator_throughput_score_summary": numeric_summary(
+                list(throughput_scores.values())
+            ),
+            "trial_cost_summary": numeric_summary(all_trial_costs),
+            "regret_summary": numeric_summary(all_regrets),
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--manifest", type=Path, default=Path(__file__).with_name("manifest.json"))
+    parser.add_argument("--output", type=Path, default=Path(__file__).with_name("metrics.json"))
+    args = parser.parse_args()
+    analysis = build_analysis(args.manifest)
+    args.output.write_text(json.dumps(analysis, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    print(json.dumps({
+        "status": "ok",
+        "output": str(args.output),
+        "empirical_reference": analysis["empirical_reference"],
+        "surface_oracle": analysis["real_surface"]["oracle_score_req_s_per_gpu"],
+        "sanity": analysis["data_sanity"],
+    }, indent=2, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()
diff --git a/runs/tuning-cost/manifest.json b/runs/tuning-cost/manifest.json
new file mode 100644
index 0000000..fabfb53
--- /dev/null
+++ b/runs/tuning-cost/manifest.json
@@ -0,0 +1,42 @@
+{
+  "schema": "aituner-tuning-cost-v1",
+  "year": 2026,
+  "task": {
+    "model": "Qwen/Qwen3-30B-A3B",
+    "engine": "community-vLLM 0.20.0",
+    "gpu": "NVIDIA H20",
+    "trace_window": "chat_w20260311_1000",
+    "input_tokens": [0, 8192],
+    "output_tokens": 128,
+    "replay_time_scale": 0.1,
+    "target_pass_rate": 0.95,
+    "ttft_ms": [2000, 4000, 6000],
+    "tpot_ms": 50
+  },
+  "sequential_runs": {
+    "pure_llm": {
+      "root": "ssh://dash0/home/admin/cpfs/wjh/aituner/aituner/.aituner-community-vllm020/dash0-qwen30b-a3b-community-vllm020-chat-0-8k-out128-scale01-high1-noharness",
+      "expected_trials": 12,
+      "missing_log_duration_s": {
+        "trial-0003": 13.0
+      },
+      "missing_log_reason": {
+        "trial-0003": "conservative trial_spec-to-result mtime envelope for a pre-ready CLI failure"
+      }
+    },
+    "guided_harness": {
+      "root": "ssh://dash0/home/admin/cpfs/wjh/aituner/aituner/.aituner-community-vllm020/dash0-qwen30b-a3b-community-vllm020-chat-0-8k-out128-scale01-high1-harness-guided-v2",
+      "expected_trials": 4,
+      "missing_log_duration_s": {},
+      "missing_log_reason": {}
+    }
+  },
+  "real_surface": {
+    "primary_root": "recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-mixed-qwen30b-tp-mns-surface-high1-dash1-d8899c5-20260701T095858Z/store/interaction-mixed-qwen30b-tp-mns-surface-high1-dash1-d8899c5-20260701T095858Z",
+    "tp4_companion_root": "recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-mixed-qwen30b-tp4-mns-nocap-qps20-dash1-d8899c5-20260701T161900Z/store/interaction-mixed-qwen30b-tp4-mns-nocap-qps20-dash1-d8899c5-20260701T161900Z",
+    "expected_cells": 12,
+    "fixed_max_num_batched_tokens": 8192
+  },
+  "simulator_metrics": "/home/gahow/phd/replayserve/runs/simfid_s2rb/results/metrics.json",
+  "threshold_regrets": [0.05, 0.02, 0.01, 0.0]
+}
diff --git a/runs/tuning-cost/metrics.json b/runs/tuning-cost/metrics.json
new file mode 100644
index 0000000..e246bea
--- /dev/null
+++ b/runs/tuning-cost/metrics.json
@@ -0,0 +1,736 @@
+{
+  "comparison_scope": {
+    "dash0_methods_vs_dash1_simulator": "indicative only: matched model/engine/workload/GPU type, different host and campaign",
+    "pure_llm_vs_guided_harness": "direct: same dash0 fixed task context",
+    "simulator_vs_surface": "direct: simulator predictions and exact dash1 12-cell real surface"
+  },
+  "cost_definition": {
+    "formula": "parallel_size * (last_engine_log_timestamp - first_engine_log_timestamp) / 3600",
+    "future_all_in_metric": "allocation_start_to_GPU_idle * allocated_GPU_count, including failures",
+    "included": [
+      "engine startup after first timestamp",
+      "warm-up/probes until last timestamp"
+    ],
+    "not_reconstructable": [
+      "GPU allocation before first engine timestamp",
+      "controller/LLM latency",
+      "cleanup after last engine timestamp",
+      "one-time simulator operator profiling GPU-hours"
+    ],
+    "reported_metric": "engine H20-hours lower bound"
+  },
+  "data_sanity": {
+    "invariants": {
+      "all_costs_non_negative": true,
+      "all_regrets_in_0_1": true,
+      "dash0_task_contexts_equal_except_method_and_port": true,
+      "engine_log_timestamps_monotonic": true,
+      "sequential_scores_not_all_identical": true,
+      "sequential_trial_counts_match_manifest": true,
+      "simulator_scores_not_all_identical": true,
+      "simulator_suite_has_no_failed_runs": true,
+      "surface_has_expected_cell_count": true,
+      "surface_matches_simulator_real_scores": true,
+      "surface_scores_not_all_identical": true
+    },
+    "regret_summary": {
+      "distinct_n": 6,
+      "max": 0.34328358208955223,
+      "min": 0.0,
+      "n": 16
+    },
+    "sequential_score_summary": {
+      "distinct_n": 7,
+      "max": 3.35,
+      "min": 1.1041666666666667,
+      "n": 9
+    },
+    "simulator_throughput_score_summary": {
+      "distinct_n": 10,
+      "max": 4.356763578770651,
+      "min": 1.5449814460277083,
+      "n": 12
+    },
+    "surface_score_summary": {
+      "distinct_n": 8,
+      "max": 3.283333333333333,
+      "min": 1.2833333333333334,
+      "n": 12
+    },
+    "trial_cost_summary": {
+      "distinct_n": 26,
+      "max": 0.49777777777777776,
+      "min": 0.0,
+      "n": 32
+    }
+  },
+  "direct_dash0_comparison": {
+    "regret_le_0.02": {
+      "guided_harness_h20_hours_lower_bound": 0.4458333333333333,
+      "guided_saving_vs_pure_llm": 0.6109090909090908,
+      "pure_llm_h20_hours_lower_bound": 1.1458333333333333
+    },
+    "regret_le_0.05": {
+      "guided_harness_h20_hours_lower_bound": 0.26805555555555555,
+      "guided_saving_vs_pure_llm": 0.0585365853658536,
+      "pure_llm_h20_hours_lower_bound": 0.2847222222222222
+    }
+  },
+  "empirical_reference": {
+    "meaning": "best observed across the two dash0 sequential runs; not a global oracle",
+    "score_req_s_per_gpu": 3.35
+  },
+  "provisional_development_targets": {
+    "five_percent_regret": {
+      "development_target_h20_hours_lower_bound": 0.21444444444444444,
+      "thirty_percent_below_posthoc_sim_slo_real_final": 0.36088888888888887,
+      "twenty_percent_below_current_guided": 0.21444444444444444
+    },
+    "status": "lower-bound, single-task targets; require same-host prospective validation",
+    "two_percent_regret": {
+      "development_target_h20_hours_lower_bound": 0.3566666666666667,
+      "thirty_percent_below_posthoc_sim_slo_real_final": 0.36088888888888887,
+      "twenty_percent_below_current_guided": 0.3566666666666667
+    }
+  },
+  "real_surface": {
+    "cells": {
+      "tp1_mns16": {
+        "engine_h20_hours_lower_bound": 0.14083333333333334,
+        "score_req_s_per_gpu": 2.35
+      },
+      "tp1_mns32": {
+        "engine_h20_hours_lower_bound": 0.13194444444444445,
+        "score_req_s_per_gpu": 2.283333333333333
+      },
+      "tp1_mns64": {
+        "engine_h20_hours_lower_bound": 0.13527777777777777,
+        "score_req_s_per_gpu": 2.283333333333333
+      },
+      "tp1_mns8": {
+        "engine_h20_hours_lower_bound": 0.13333333333333333,
+        "score_req_s_per_gpu": 2.1
+      },
+      "tp2_mns16": {
+        "engine_h20_hours_lower_bound": 0.29944444444444446,
+        "score_req_s_per_gpu": 2.275
+      },
+      "tp2_mns32": {
+        "engine_h20_hours_lower_bound": 0.2816666666666667,
+        "score_req_s_per_gpu": 3.283333333333333
+      },
+      "tp2_mns64": {
+        "engine_h20_hours_lower_bound": 0.2338888888888889,
+        "score_req_s_per_gpu": 3.2583333333333333
+      },
+      "tp2_mns8": {
+        "engine_h20_hours_lower_bound": 0.2777777777777778,
+        "score_req_s_per_gpu": 2.275
+      },
+      "tp4_mns16": {
+        "engine_h20_hours_lower_bound": 0.4866666666666667,
+        "score_req_s_per_gpu": 2.441666666666667
+      },
+      "tp4_mns32": {
+        "engine_h20_hours_lower_bound": 0.49777777777777776,
+        "score_req_s_per_gpu": 2.441666666666667
+      },
+      "tp4_mns64": {
+        "engine_h20_hours_lower_bound": 0.49666666666666665,
+        "score_req_s_per_gpu": 2.441666666666667
+      },
+      "tp4_mns8": {
+        "engine_h20_hours_lower_bound": 0.48,
+        "score_req_s_per_gpu": 1.2833333333333334
+      }
+    },
+    "completed_annotation_engine_h20_hours_lower_bound": 3.5952777777777776,
+    "failed_primary_attempt_engine_h20_hours_lower_bound": 0.0,
+    "failed_primary_attempt_n": 4,
+    "oracle_cells": [
+      "tp2_mns32"
+    ],
+    "oracle_score_req_s_per_gpu": 3.283333333333333,
+    "primary_state_sha256": "790bf6df9045e22c46a8abdc022390bf842d884ef13ade4b627936722b9c3366",
+    "tp4_companion_state_sha256": "1fc4f584a69a90fb0ff1d64f6a07a8383c368f4e306a0b7a3c19d2304042fd73"
+  },
+  "schema": "aituner-tuning-cost-analysis-v1",
+  "sequential_runs": {
+    "guided_harness": {
+      "curve": {
+        "cost_to_threshold": {
+          "regret_le_0": null,
+          "regret_le_0.01": null,
+          "regret_le_0.02": {
+            "best_score_req_s_per_gpu": 3.283333333333333,
+            "cumulative_engine_h20_hours_lower_bound": 0.4458333333333333,
+            "regret": 0.01990049751243783,
+            "trial_id": "trial-0003"
+          },
+          "regret_le_0.05": {
+            "best_score_req_s_per_gpu": 3.2583333333333333,
+            "cumulative_engine_h20_hours_lower_bound": 0.26805555555555555,
+            "regret": 0.027363184079602032,
+            "trial_id": "trial-0002"
+          }
+        },
+        "points": [
+          {
+            "best_score_req_s_per_gpu": 2.3833333333333333,
+            "cumulative_engine_h20_hours_lower_bound": 0.09305555555555556,
+            "regret": 0.2885572139303483,
+            "trial_id": "trial-0001"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.2583333333333333,
+            "cumulative_engine_h20_hours_lower_bound": 0.26805555555555555,
+            "regret": 0.027363184079602032,
+            "trial_id": "trial-0002"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.283333333333333,
+            "cumulative_engine_h20_hours_lower_bound": 0.4458333333333333,
+            "regret": 0.01990049751243783,
+            "trial_id": "trial-0003"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.3,
+            "cumulative_engine_h20_hours_lower_bound": 0.6230555555555555,
+            "regret": 0.014925373134328401,
+            "trial_id": "trial-0004"
+          }
+        ],
+        "reference_score_req_s_per_gpu": 3.35,
+        "total_engine_h20_hours_lower_bound": 0.6230555555555555
+      },
+      "root": "ssh://dash0/home/admin/cpfs/wjh/aituner/aituner/.aituner-community-vllm020/dash0-qwen30b-a3b-community-vllm020-chat-0-8k-out128-scale01-high1-harness-guided-v2",
+      "state_sha256": "76e439d1f20e0f9af54785005258cbe39f205ae107fc2398955e360e263c7718",
+      "study_spec_sha256": "03241089052f1a07dbbb2ba6100c6737635877b683d275813b3c59f2769fc2d7",
+      "trials": [
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {}
+          },
+          "duration_note": "",
+          "duration_s": 335.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.09305555555555556,
+          "engine_log_sha256": "6d8b908085c03d4796e78679eed13737581fc7d2e18947fb72d7031a9f557c9d",
+          "engine_timestamp_n": 108,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 1,
+          "score_req_s_per_gpu": 2.3833333333333333,
+          "status": "completed",
+          "trial_id": "trial-0001"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 315.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.175,
+          "engine_log_sha256": "b09caadd930495c27effea57e4164e52f58c6461cbbc0f4e873a309d45ec3443",
+          "engine_timestamp_n": 134,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": 3.2583333333333333,
+          "status": "completed",
+          "trial_id": "trial-0002"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "enable-chunked-prefill": true,
+              "gpu-memory-utilization": 0.95,
+              "max-num-batched-tokens": 16384,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 320.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.17777777777777778,
+          "engine_log_sha256": "8f6f2637cff8566d39b8e6c17de71ed37a01f2aa6a4460ff36ecc9728d033c3a",
+          "engine_timestamp_n": 134,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": 3.283333333333333,
+          "status": "completed",
+          "trial_id": "trial-0003"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "enable-chunked-prefill": true,
+              "gpu-memory-utilization": 0.95,
+              "max-num-batched-tokens": 24576,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 319.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.17722222222222223,
+          "engine_log_sha256": "29b8fc588b5422ceae60456188d6373070c179610243258fa213aeead9f07a00",
+          "engine_timestamp_n": 137,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": 3.3,
+          "status": "completed",
+          "trial_id": "trial-0004"
+        }
+      ]
+    },
+    "pure_llm": {
+      "curve": {
+        "cost_to_threshold": {
+          "regret_le_0": {
+            "best_score_req_s_per_gpu": 3.35,
+            "cumulative_engine_h20_hours_lower_bound": 1.3719444444444444,
+            "regret": 0.0,
+            "trial_id": "trial-0007"
+          },
+          "regret_le_0.01": {
+            "best_score_req_s_per_gpu": 3.35,
+            "cumulative_engine_h20_hours_lower_bound": 1.3719444444444444,
+            "regret": 0.0,
+            "trial_id": "trial-0007"
+          },
+          "regret_le_0.02": {
+            "best_score_req_s_per_gpu": 3.3,
+            "cumulative_engine_h20_hours_lower_bound": 1.1458333333333333,
+            "regret": 0.014925373134328401,
+            "trial_id": "trial-0006"
+          },
+          "regret_le_0.05": {
+            "best_score_req_s_per_gpu": 3.2583333333333333,
+            "cumulative_engine_h20_hours_lower_bound": 0.2847222222222222,
+            "regret": 0.027363184079602032,
+            "trial_id": "trial-0002"
+          }
+        },
+        "points": [
+          {
+            "best_score_req_s_per_gpu": 2.2,
+            "cumulative_engine_h20_hours_lower_bound": 0.10861111111111112,
+            "regret": 0.34328358208955223,
+            "trial_id": "trial-0001"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.2583333333333333,
+            "cumulative_engine_h20_hours_lower_bound": 0.2847222222222222,
+            "regret": 0.027363184079602032,
+            "trial_id": "trial-0002"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.2583333333333333,
+            "cumulative_engine_h20_hours_lower_bound": 0.29194444444444445,
+            "regret": 0.027363184079602032,
+            "trial_id": "trial-0003"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.2583333333333333,
+            "cumulative_engine_h20_hours_lower_bound": 0.4280555555555555,
+            "regret": 0.027363184079602032,
+            "trial_id": "trial-0004"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.2583333333333333,
+            "cumulative_engine_h20_hours_lower_bound": 0.9247222222222222,
+            "regret": 0.027363184079602032,
+            "trial_id": "trial-0005"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.3,
+            "cumulative_engine_h20_hours_lower_bound": 1.1458333333333333,
+            "regret": 0.014925373134328401,
+            "trial_id": "trial-0006"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.35,
+            "cumulative_engine_h20_hours_lower_bound": 1.3719444444444444,
+            "regret": 0.0,
+            "trial_id": "trial-0007"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.35,
+            "cumulative_engine_h20_hours_lower_bound": 1.561388888888889,
+            "regret": 0.0,
+            "trial_id": "trial-0008"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.35,
+            "cumulative_engine_h20_hours_lower_bound": 1.7497222222222222,
+            "regret": 0.0,
+            "trial_id": "trial-0009"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.35,
+            "cumulative_engine_h20_hours_lower_bound": 1.9047222222222222,
+            "regret": 0.0,
+            "trial_id": "trial-0010"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.35,
+            "cumulative_engine_h20_hours_lower_bound": 2.0930555555555554,
+            "regret": 0.0,
+            "trial_id": "trial-0011"
+          },
+          {
+            "best_score_req_s_per_gpu": 3.35,
+            "cumulative_engine_h20_hours_lower_bound": 2.2824999999999998,
+            "regret": 0.0,
+            "trial_id": "trial-0012"
+          }
+        ],
+        "reference_score_req_s_per_gpu": 3.35,
+        "total_engine_h20_hours_lower_bound": 2.2824999999999998
+      },
+      "root": "ssh://dash0/home/admin/cpfs/wjh/aituner/aituner/.aituner-community-vllm020/dash0-qwen30b-a3b-community-vllm020-chat-0-8k-out128-scale01-high1-noharness",
+      "state_sha256": "342cde3546299cd312ebc078b1c9589773b841d23c4ed4632370883be598fd4a",
+      "study_spec_sha256": "3238dc89477108405c09b25eee434c3c10c0f5be245d2bfa3b90509a372c088e",
+      "trials": [
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {}
+          },
+          "duration_note": "",
+          "duration_s": 391.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.10861111111111112,
+          "engine_log_sha256": "c5caf9e9f6faa73a4a39adb71a7a6c04bcf623d5c0c91b231116bb596c8a11be",
+          "engine_timestamp_n": 113,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 1,
+          "score_req_s_per_gpu": 2.2,
+          "status": "completed",
+          "trial_id": "trial-0001"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 317.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.1761111111111111,
+          "engine_log_sha256": "239ff7349d7b9a1b24334fbe6e2a45a967bfa2d16380e98da4f81520d422cd0d",
+          "engine_timestamp_n": 134,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": 3.2583333333333333,
+          "status": "completed",
+          "trial_id": "trial-0002"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "enable-expert-parallel": true,
+              "expert-parallel-size": 2,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "conservative trial_spec-to-result mtime envelope for a pre-ready CLI failure",
+          "duration_s": 13.0,
+          "duration_source": "conservative_fallback",
+          "engine_h20_hours_lower_bound": 0.007222222222222222,
+          "engine_log_sha256": "4707735f33c8bf9030054eccd36b66985a3879050d80a853f93f166282414d8f",
+          "engine_timestamp_n": 0,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "engine_launch",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": null,
+          "status": "failed",
+          "trial_id": "trial-0003"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "data-parallel-size": 2,
+              "tensor-parallel-size": 1
+            }
+          },
+          "duration_note": "",
+          "duration_s": 245.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.1361111111111111,
+          "engine_log_sha256": "215ed31c6d227e4a6771ecb637c2b4e3e38e766d73a8882a923d1fa8b97a23da",
+          "engine_timestamp_n": 170,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": null,
+          "status": "completed",
+          "trial_id": "trial-0004"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "data-parallel-size": 2,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 447.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.49666666666666665,
+          "engine_log_sha256": "6057b3477ccca14e0ebd467ac96b777662df67b57827c851d58caf3a77b95ca4",
+          "engine_timestamp_n": 201,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 4,
+          "score_req_s_per_gpu": 1.1041666666666667,
+          "status": "completed",
+          "trial_id": "trial-0005"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "enable-chunked-prefill": true,
+              "gpu-memory-utilization": 0.95,
+              "max-num-batched-tokens": 16384,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 398.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.22111111111111112,
+          "engine_log_sha256": "c5c336f74d419645639b469eea9f92575780164637fdda078fa5c84c96915f1f",
+          "engine_timestamp_n": 138,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": 3.3,
+          "status": "completed",
+          "trial_id": "trial-0006"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "enable-chunked-prefill": true,
+              "gpu-memory-utilization": 0.95,
+              "max-num-batched-tokens": 24576,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 407.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.22611111111111112,
+          "engine_log_sha256": "5e2829479ecd4771e7c781d40209f74e4c2843bbca15522974e435e06daa12a2",
+          "engine_timestamp_n": 142,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": 3.35,
+          "status": "completed",
+          "trial_id": "trial-0007"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "enable-chunked-prefill": true,
+              "gpu-memory-utilization": 0.95,
+              "max-num-batched-tokens": 32768,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 341.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.18944444444444444,
+          "engine_log_sha256": "fde628968fba10fccfe5b09675aa57cf725f507be83c10dd3c0b6c109b5d72eb",
+          "engine_timestamp_n": 136,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": null,
+          "status": "completed",
+          "trial_id": "trial-0008"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "enable-chunked-prefill": true,
+              "gpu-memory-utilization": 0.95,
+              "max-num-batched-tokens": 28672,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 339.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.18833333333333332,
+          "engine_log_sha256": "fc60765578d57af08e0310983ff42e2a200739a90f11c45f9d4b1f6b7b789cc8",
+          "engine_timestamp_n": 136,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": null,
+          "status": "completed",
+          "trial_id": "trial-0009"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "enable-chunked-prefill": true,
+              "enable-prefix-caching": true,
+              "gpu-memory-utilization": 0.95,
+              "max-num-batched-tokens": 24576,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 279.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.155,
+          "engine_log_sha256": "22c473b39662df6c8c4c64a616346652ab889d63256efab18257b1f4445ce13b",
+          "engine_timestamp_n": 133,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": null,
+          "status": "completed",
+          "trial_id": "trial-0010"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "enable-chunked-prefill": true,
+              "gpu-memory-utilization": 0.95,
+              "max-num-batched-tokens": 25600,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 339.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.18833333333333332,
+          "engine_log_sha256": "19b19de4a0a11a5a112b3e8ef04c3ca4d90d6c6c9097b5180c024d8570c9a608",
+          "engine_timestamp_n": 136,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": null,
+          "status": "completed",
+          "trial_id": "trial-0011"
+        },
+        {
+          "config_patch": {
+            "env_patch": {},
+            "flag_patch": {
+              "enable-chunked-prefill": true,
+              "gpu-memory-utilization": 0.95,
+              "max-num-batched-tokens": 25088,
+              "tensor-parallel-size": 2
+            }
+          },
+          "duration_note": "",
+          "duration_s": 341.0,
+          "duration_source": "engine_log_span",
+          "engine_h20_hours_lower_bound": 0.18944444444444444,
+          "engine_log_sha256": "40f7307d6677f2bb55c0132d42fc82fce78ffa6a558d5c69b01ec60d1e40b3ae",
+          "engine_timestamp_n": 136,
+          "engine_timestamps_monotonic": true,
+          "failure_stage": "",
+          "parallel_size": 2,
+          "score_req_s_per_gpu": null,
+          "status": "completed",
+          "trial_id": "trial-0012"
+        }
+      ]
+    }
+  },
+  "simulator": {
+    "decision_bearing_throughput_proxy_plus_real_final": {
+      "nominal_k_1": {
+        "candidate_cells": [
+          "tp1_mns64"
+        ],
+        "engine_h20_hours_lower_bound": 0.13527777777777777,
+        "real_evaluations": 1,
+        "real_regret": 0.30456852791878175,
+        "selected_cell": "tp1_mns64",
+        "selected_real_score_req_s_per_gpu": 2.283333333333333
+      },
+      "nominal_k_2": {
+        "candidate_cells": [
+          "tp1_mns64",
+          "tp1_mns32"
+        ],
+        "engine_h20_hours_lower_bound": 0.26722222222222225,
+        "real_evaluations": 2,
+        "real_regret": 0.30456852791878175,
+        "selected_cell": "tp1_mns64",
+        "selected_real_score_req_s_per_gpu": 2.283333333333333
+      },
+      "nominal_k_3": {
+        "candidate_cells": [
+          "tp1_mns64",
+          "tp1_mns32",
+          "tp2_mns32",
+          "tp2_mns64"
+        ],
+        "engine_h20_hours_lower_bound": 0.7827777777777778,
+        "real_evaluations": 4,
+        "real_regret": 0.0,
+        "selected_cell": "tp2_mns32",
+        "selected_real_score_req_s_per_gpu": 3.283333333333333
+      }
+    },
+    "decision_bearing_throughput_proxy_sim_only_top1": {
+      "gpu_hours": 0.0,
+      "real_regret": 0.30456852791878175,
+      "selected_cell": "tp1_mns64",
+      "selected_real_score_req_s_per_gpu": 2.283333333333333
+    },
+    "marginal_gpu_hours_without_real_verification": 0.0,
+    "metrics_sha256": "55edb37d5692e979ab6f6dc6c65913a9db0aa0a836c350e4c05d9c38eee78206",
+    "observed_fidelity_suite_cpu_hours": 2.055026211017717,
+    "observed_fidelity_suite_runs": 184,
+    "one_time_profile_cost_status": "not recorded; total cold-start cost is unknown",
+    "one_time_profile_gpu_hours": null,
+    "posthoc_slo_gated_plus_real_final": {
+      "candidate_cells": [
+        "tp2_mns32",
+        "tp2_mns64"
+      ],
+      "engine_h20_hours_lower_bound": 0.5155555555555555,
+      "false_feasible": 21,
+      "false_infeasible": 7,
+      "real_evaluations": 2,
+      "real_regret": 0.0,
+      "selected_cell": "tp2_mns32",
+      "selected_real_score_req_s_per_gpu": 3.283333333333333,
+      "status": "diagnostic/post-hoc, not a preregistered prospective policy"
+    }
+  }
+}
diff --git a/runs/tuning-cost/test_analysis.py b/runs/tuning-cost/test_analysis.py
new file mode 100644
index 0000000..ca698c7
--- /dev/null
+++ b/runs/tuning-cost/test_analysis.py
@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import importlib.util
+import math
+import sys
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def load_analysis():
+    spec = importlib.util.spec_from_file_location("tuning_cost", HERE / "analyze.py")
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def main() -> None:
+    analysis = load_analysis()
+    duration, monotonic, count = analysis.timestamp_span(
+        "INFO 07-01 10:00:00 x\nINFO 07-01 10:05:30 y\n", 2026
+    )
+    assert duration == 330.0
+    assert monotonic
+    assert count == 2
+
+    trials = [
+        {"trial_id": "t1", "engine_h20_hours_lower_bound": 0.1, "score_req_s_per_gpu": 8.0},
+        {"trial_id": "t2", "engine_h20_hours_lower_bound": 0.2, "score_req_s_per_gpu": None},
+        {"trial_id": "t3", "engine_h20_hours_lower_bound": 0.3, "score_req_s_per_gpu": 9.6},
+    ]
+    curve = analysis.sequential_curve(trials, 10.0, [0.05, 0.04])
+    assert curve["cost_to_threshold"]["regret_le_0.05"]["trial_id"] == "t3"
+    assert curve["cost_to_threshold"]["regret_le_0.04"]["trial_id"] == "t3"
+    assert math.isclose(curve["total_engine_h20_hours_lower_bound"], 0.6)
+
+    candidates = analysis.tie_expanded_candidates(
+        {"a": 3.0, "b": 2.0, "c": 2.0, "d": 1.0}, 2
+    )
+    assert candidates == ["a", "b", "c"]
+    policy = analysis.real_final_policy(
+        candidates,
+        {"a": 1.0, "b": 4.0, "c": 2.0, "d": 3.0},
+        {"a": 0.1, "b": 0.2, "c": 0.3, "d": 0.4},
+    )
+    assert policy["selected_cell"] == "b"
+    assert policy["real_regret"] == 0.0
+    assert math.isclose(policy["engine_h20_hours_lower_bound"], 0.6)
+    print("tuning cost analysis: PASS")
+
+
+if __name__ == "__main__":
+    main()