gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Audit tuning cost and core challenges

Browse Source
This commit is contained in:
Gahow Wang
2026-07-15 01:41:25 +08:00
parent 8c930ba3a1
commit 0d16838097
5 changed files with 1733 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

391
docs/tuning-core-challenges-cost-audit-20260715.md Normal file
View File

@@ -0,0 +1,391 @@
# AITuner tuning：核心挑战、统一成本口径与研究路线
日期：2026-07-15（Asia/Singapore）
状态：**问题定义与历史成本审计完成；新的 tuner 贡献尚未建立。**
## 结论先行
我们不应该把 tuning 定义成“根据当前 telemetry 判断哪个 cap 满了，再调对应 knob”。这个定义同时遗漏了 knob interaction、反事实识别、实验成本和跨任务失配。更准确的问题是：
> 给定模型、engine version、hardware、workload、SLO 和一个声明好的合法配置空间，tuner 如何用最少的真实 GPU 成本，依次选择可能包含多个 knob 的 intervention，找到 SLO-goodput regret 不超过 `epsilon` 的配置？
AITuner 可以形成的系统贡献应当是：
> **一个 intervention-calibrated、action-conditioned、cost-aware 的 tuner：它从真实 engine trajectory 和已测 intervention 中学习联合 config action 的反事实收益分布，并以 cost-to-oracle 而非规则命中率作为目标。Harness 只负责实验语义、合法性、配对、记账和可复现性，不负责用人工 bottleneck rule 决定 action。**
现有结果支持这个问题值得做，但不支持宣称它已经解决：
- 在真实 `TP x MNS` surface 上，one-knob-at-a-time 会停在比 oracle 低 **25.6%** 的 coordinate-wise local optimum。
- 在 action-aware pilot 中，增加 MBBT 在“几乎从未独占打满 MBBT cap”的情况下仍把 source goodput 提高 **48.0%--77.1%**；因此 `cap -> knob` 不是完整模型。
- 同一 dash0 任务上，当前 guided harness 到 5% empirical regret 只比纯 LLM 少 **5.85%** H20-hours；到 2% regret 则少 **61.09%**。这说明必须比较完整 cost--regret curve，不能只比较最终最好值。
- Frontier 的 decision-bearing throughput top-1 在 12-cell surface 上有 **30.46%** real regret。Simulator 本身的边际 GPU cost 是 0，但通过 real-final 恢复 oracle 需要 tie-expanded 4 个真实 cell，即 **0.7828 reconstructed H20-hours**。
## 1. Tuning 问题和成功标准
固定 task context：
```text
T = {model, engine build, hardware, workload/trace, SLO, legal config space C}
```
每个完整配置 `c in C` 的目标为：
```text
f_T(c) = max request_rate_per_gpu
subject to request SLO pass rate >= target
```
有限空间 oracle 为：
```text
f*_T = max_{c in C} f_T(c)
regret(c) = 1 - f_T(c) / f*_T
```
顺序 tuner 在第 `t` 步基于历史 `D_t` 选择一个完整 config intervention：
```text
a_t = c_t -> c_{t+1}
```
成功不是“最后找到一个不错的值”，而是同时满足：
1. `regret(best_t) <= epsilon`；
2. 达到该点之前的 all-in H20-hours 最小；
3. launch、correctness、SLO 和失败率约束不退化；
4. 结论在 held-out task 上成立，而不是在用于设计规则的 task 上成立。
### 1.1 GPU cost 的统一定义
未来实验的 task-marginal cost 应定义为：
```text
C_task = sum_j allocated_GPU_count_j
* (GPU_idle_or_release_time_j - allocation_start_time_j)
```
它包括 method 实际触发的 startup、warm-up、prefix/full replay、confirmation、failure、cleanup；如果 LLM 思考期间 GPU 仍被占用，也计入。Simulator/模型的一次性 onboarding 成本单独报告：
```text
C_e2e(N tasks) = C_profile_or_training / N + C_task
```
另外报告 CPU-hours、LLM API latency/cost，但不把它们伪装成 GPU-hours。构建 benchmark oracle 的 exhaustive annotation cost 是公共评测成本，单独报告，不计入任何方法；同时可给一个将其等量加回所有方法的 conservative view。
历史记录没有 allocation start/release timestamp。本次只能从每个 `engine.log` 的首末时间戳重建：
```text
C_engine_lower_bound = parallel_size * engine_log_span / 3600
```
因此下面所有历史 H20-hour 数字都是 **engine-lifetime lower bound**，不是 all-in cost。尤其 simulator 的一次性 H20 operator profiling 成本没有记录，不能称为完全免费。
### 1.2 两种 oracle 必须分开
- **Exact finite-surface oracle**：声明好的 12-cell `TP x MNS` 空间全部真实测量，oracle 是 `TP2/MNS32 = 3.2833 req/s/GPU`。
- **Broader empirical reference**：dash0 两个 sequential run 中观察到的最好值 `3.35 req/s/GPU`。它包含 surface 外的 MBBT/chunk/GMU action，但只是 best observed，不是全局 oracle。
不能把 empirical best 写成 global oracle，也不能让每个方法使用不同的 oracle 定义。
## 2. 现有方案的 cost-to-oracle 审计
可复算输入和完整结果在：
- `runs/tuning-cost/manifest.json`
- `runs/tuning-cost/analyze.py`
- `runs/tuning-cost/metrics.json`
### 2.1 严格同任务对照：纯 LLM vs 当前 guided harness
两组均为 dash0、Qwen3-30B-A3B、community-vLLM 0.20.0、8xH20 可见、`chat_w20260311_1000`、input 0--8k、output 128、replay scale 0.1、TTFT 2/4/6s、TPOT 50ms、pass rate 0.95。除 tuner method 和服务端口外，固定 task spec 相同。
Reference 是两组中 best observed `3.35 req/s/GPU`：
| Method | 到 <=5% regret | 到 <=2% regret | 到 <=1% regret | 完整 run 成本 | 最终 best |
|---|---:|---:|---:|---:|---:|
| Pure LLM, no harness | 0.2847 H20h，trial 2，regret 2.736% | 1.1458，trial 6，regret 1.493% | 1.3719，trial 7，regret 0% | 2.2825 | 3.35 |
| Guided harness v2 | 0.2681 H20h，trial 2，regret 2.736% | 0.4458，trial 3，regret 1.990% | 未达到 | 0.6231 | 3.30，regret 1.493% |
直接结论：
- 5% endpoint：guided 比 pure LLM 少 **5.85%**，不是 material contribution。
- 2% endpoint：guided 比 pure LLM 少 **61.09%**，有明显 headroom signal，但只有一个 task，不能外推。
- Pure LLM 在 trial 7 已找到 best observed，之后又花了 `2.2825 - 1.3719 = 0.9106 H20h` 而没有改进，说明 trustworthy stopping 本身就是成本来源。
- Pure LLM 的 trial 3 使用当前 binary 不支持的 `--expert-parallel-size` 并在 launch 前失败。当前 harness 的 legality/version contract 有实际价值，但它仍不是性能 action-ranking 贡献。
### 2.2 Simulator：零边际 GPU cost 不等于零 tuning cost
Frontier fidelity suite 在 CPU 上执行 184 个 simulation，耗时 **2.055 CPU-hours**，simulation 本身为 0 marginal H20-hours。其对应的 exact dash1 12-cell real surface annotation lower bound 为 **3.5953 H20-hours**。
Decision-bearing `frozen-calibrated/throughput-proxy`：
| Policy | Real cells evaluated | Real-final H20h lower bound | Selected real regret |
|---|---:|---:|---:|
| Simulator-only top-1 | 0 | 0 | **30.46%**，选 TP1/MNS64 |
| Throughput top-1 + real final | 1 | 0.1353 | **30.46%** |
| Throughput top-2 + real final | 2 | 0.2672 | **30.46%** |
| Throughput nominal top-3 + real final | tie-expanded 4 | 0.7828 | 0%，找到 TP2/MNS32 |
Post-hoc `SLO-gated` reading 把 `{TP2/MNS32, TP2/MNS64}` 放在 top tie bucket；测两个 cell 需 **0.5156 H20h** 并能找到 oracle。但它不是 preregistered decision-bearing policy，而且 anchor verdict 中有 21 个 false-feasible、7 个 false-infeasible，只能作为诊断上界，不能反写成 prospective simulator 结果。
Pure LLM/harness 数据来自 dash0，simulator exact surface 来自 dash1。模型、engine、trace、GPU type 匹配，但 host 和 campaign 不同。因此两块内部可以直接比较，跨块只能做 development-level 指示；paper 结论必须在同 host、同 task execution protocol 下重跑。
### 2.3 我们要达到的成本目标
在当前 reconstructed lower-bound 口径下，一个有意义的单任务 development bar 是：
| Endpoint | 当前最强同任务 baseline | 20% reduction bar | 兼顾 post-hoc sim+real 的 30% bar | 暂定目标 |
|---|---:|---:|---:|---:|
| <=5% empirical regret | guided 0.2681 | 0.2144 | 0.3609 | **<=0.2144 H20h** |
| <=2% empirical regret | guided 0.4458 | 0.3567 | 0.3609 | **<=0.3567 H20h** |
这两个数字不是 paper result，只用于检查 proposed method 是否有足够 headroom：
- 5% endpoint 已经由 baseline + TP2 两个完整 trial 达到。任何必须先跑 source 再跑 target 的 telemetry tuner 都不能靠减少 trial count 获得 20% 优势；它必须能够 one-shot warm-start、跳过 baseline，或安全地缩短其中一次测量。
- 2% endpoint 有更合理的结构性空间：从一个 source 直接选择 joint `TP2 + MBBT/chunk` target，可能跳过当前中间 trial；如果仍按当前三次完整 trial 顺序执行，就不会达到 bar。
Paper-facing gate 不使用这些跨 campaign 绝对数，而使用 prospective same-host all-in cost：在每个 held-out task 上 regret <=5%，相对最强 safe outcome-only/current harness 至少省 20%，相对 frozen simulator+real 至少省 30%，并报告 task-level paired confidence interval。
## 3. 四个最核心的 tuning challenge
### Challenge 1：响应面是联合、条件化且 regime-dependent 的
#### 问题本质
一般情况下：
```text
f(c) != base + sum_k effect_k(c_k)
```
一个 knob 的 effect 是当前完整 context 的函数：
```text
Delta_x(c, workload, engine state)
```
它可能随 topology、另一个 runtime knob、load、SLO 或 engine version 改变大小甚至改变符号。因此不能先分别求每个 knob 的最优值再 merge，也不能固定一个低质量 context 去判断另一个 knob。
#### 已有真实证据
在 C1 12-cell real surface：
- `MNS 8 -> 32` 在 TP1/TP2/TP4 下分别提升约 **8.7% / 44.3% / 90.3%**。
- 从同一 `TP1/MNS8` 起点，先 tune MNS 再 TP 会停在 `TP4/MNS16 = 2.4417`；该点沿任一单维都没有 strictly improving move，但 joint/global surface oracle `TP2/MNS32 = 3.2833` 高 **34.5% relative to the local point**，即 local point 对 oracle 有 **25.6% regret**。
- C3 中 `MBT 256 -> 384` 的 effect 根据 topology/MNS 从 0 到约 -9.2%；`MNS 64 -> 128` 从 0 到约 +10.1%。
- Action-aware Regime A 中 MBBT 几乎从不作为 exclusive cap，但 MBBT action 仍把 source goodput 提高 48.0%--77.1%。它通过 chunk size、prefill packing 和 scarce MNS slot residency 的联合变化获得收益。
这直接否定两类通用策略：OAT/coordinate greedy，以及 `which cap is full -> tune that knob`。
#### Tuner 必须具备的能力
- Action 的基本单位是完整 `config delta`，允许 sparse joint action，而不是孤立 knob/value。
- 对 topology/runtime family 使用 crossed anchors 或信息增益设计，主动测 interaction；不是默认所有 interaction 都强。
- 能从数据判断 task 是 topology-dominant、runtime-interaction-dominant 还是 flat/noisy，并据此分配实验，而不是把固定 search order 写进规则。
### Challenge 2：当前状态是 observational signal，tuning 需要 counterfactual identification
#### 问题本质
一次 telemetry trace 只能告诉我们：
```text
P(engine trajectory | current config, workload)
```
Tuning 真正需要的是：
```text
P(Delta SLO-goodput, failure, cost
| source trajectory, proposed full-config action)
```
Queue、KV、padding、split prefill 等状态既可能是原因，也可能是 workload/config 的结果。看见某种状态，不等于知道哪个 action 能修复它。一个 action 也可能同时改变多条机制；例如 MBBT 同时改变总 token budget、per-request chunk 和 multi-request packing，现有 telemetry 的解释是 mechanism-consistent，不是已完成的 causal decomposition。
#### 已有真实证据
- 5/10 秒 telemetry 确实太短；300 秒 phase-aware experiment 中，MNS action 的 queue/padding 机制直到 replay 75%--100% 才稳定出现。
- 但 external TTFT outcome 在 25% 已完美区分该 action 是否修复 SLO。Telemetry 解释了 why，却没有比 outcome 更早或更可靠地指导 tuning。
- 3.125 req/s/GPU 的 source 无法在 timeout 内 drain；另一组 source 已达 offered ceiling 的 99.1%--100%，数学上不可能通过 10% improvement gate。没有 exposure/headroom 和 censoring control，模型学到的不是 action response。
- Same-config repeats 与 matched intervention 的波动不可忽略；只比较两个未经配对的 run 会混入 arrival/order/warm-state noise。
#### Tuner 必须具备的能力
- 训练样本必须是 exact-workload paired intervention：`(source trajectory, action) -> target delta`，保留失败和 censoring。
- 使用 phase-binned continuous trajectory，而不是人工 bottleneck label 或 threshold rule。
- 输出 response distribution 和 uncertainty；证据不足时 abstain，而不是强行给 diagnosis。
- Telemetry 的价值必须通过同 cutoff、同 model capacity 的 outcome-only ablation 证明。若不能降低 end-to-end H20-hours，instrumentation 只保留为 debugging/解释工具。
### Challenge 3：这是异构成本下的 sequential experimental design，不是静态 ranking
#### 问题本质
每个 trial 的成本不同：TP4 是 TP1 的四倍 GPU multiplier，startup/warm-up 可能主导短 probe，失败也有成本；同时 tuner 不知道 oracle，只能在 exploitation、information gain 和 cost 之间权衡。选对 top-1 的 accuracy 不能代表 tuning 效果。
必须回答三个连续问题：
1. 下一次测哪个联合 action？
2. 测多久，何时 continuation/confirmation？
3. 什么证据允许停止，并声称 best 已在 `epsilon` 内？
#### 已有真实证据
- Pure LLM 达到 best observed 后仍浪费 0.9106 reconstructed H20h。
- Simulator top-1 虽然 0 marginal GPUh，却因 rank error 损失 30.46%；real-final 的 k 增大又迅速增加 H20h。
- 5% endpoint 上两个方法都只需两个 trial，selection-count headroom 很小；2% endpoint 才暴露 action quality 和 stopping 的巨大差异。
- Prefix 不是天然便宜：如果 startup、warm-up 和稳定状态形成占主要成本，缩短 replay window 未必带来等比例 H20h reduction。
#### Tuner 必须具备的能力
- Acquisition 直接优化 expected regret reduction / predicted H20 cost，并把 failure probability 纳入约束。
- 在 run 前做与 tuning policy 分离的 workload admissibility check：避免 outcome ceiling、无法 drain、无请求或 measurement cap。
- 使用 uncertainty-aware continuation 和 stop；stop criterion 针对声明的 candidate set 中“仍存在 >epsilon improvement 的概率”，而不是连续几次没提升。
- 主结果报告 H20-hours-to-5%/2%/1%、fixed-budget regret 和 cost-normalized regret AUC，不 metric shopping。
### Challenge 4：任何 mechanism model 都有 fidelity 和 transfer boundary
#### 问题本质
Simulator、learned surrogate、LLM prior 都是近似。Workload、SLO、model、hardware、engine version 改变后，operator cost、scheduler state transition、合法 flag 和 response surface 都可能变化。模型在 calibration task 上解释得好，不表示能在 held-out task 上排序正确。
#### 已有真实证据
- Frontier throughput reading 在完全匹配的 12-cell task 上仍把 real oracle 排错，top-1 regret 30.46%。这说明预测绝对 throughput 还不够，局部 rank fidelity 才是 tuning 关键。
- Post-hoc SLO reading 的 top bucket 正确，但有大量 anchor feasibility error，也没有 prospective policy status。
- Pure LLM 提出了当前 community-vLLM binary 不支持的 flag；engine/API version knowledge 本身会漂移。
- 已有 cross-version experiment 中 vLLM 0.20 的强配置在 0.24 上出现大幅退化，说明 response prior 不能无条件迁移。
#### Tuner 必须具备的能力
- Simulator 只能作为 prior mean 或 candidate prior；真实 outcome 是 authoritative update。
- 学习 simulator residual：把 `sim prediction + source state + action` 映射到 real response，而不是用 telemetry 重新实现另一个无校准 simulator。
- 对 task-level OOD 显式提高 uncertainty/abstain；train/test 按完整 task 分割，不能按 request、anchor 或同一 surface cell 随机分割。
- 分开报告 cold-start profile/training cost 与 per-task marginal cost，并在 N=1/10/100 等 amortization horizon 下展示。
## 4. 对应的系统设计
### 4.1 Harness：从 rule-based tuner 收缩成 experimental control plane
Harness 保留以下确定性职责：
- engine-version-aware config schema、合法性和资源约束；
- 完整 config/action canonicalization，禁止隐式 merge 和重复试验；
- exact trace/request/arrival/length hash，配对、随机化和 counter-rotation；
- engine trajectory、external outcome、failure/censoring 的统一时间轴；
- all-in GPU cost ledger、oracle annotation 分账、budget enforcement；
- data sanity、coverage、SLO/correctness 和 stop-proof audit。
Harness **不**包含 `queue > N -> increase MNS`、`cap full -> tune knob` 或人工 diagnosis-to-action mapping。这里的规则是实验语义和安全 invariant，不是性能决策 heuristic。
### 4.2 Action-conditioned response model
每条学习记录为：
```text
x = {source full config,
workload/SLO context,
source external outcome,
phase-binned engine trajectory}
a = normalized full-config delta
y = {Delta SLO-goodput, target feasibility/failure, measured H20 cost}
```
学习：
```text
p_theta(y | x, a, optional simulator prediction)
```
第一版应使用适合小数据且有 uncertainty 的 action-conditioned Gaussian-process/bootstrapped surrogate；kernel/feature ablation包括：
1. config + external outcome；
2. 同样输入 + telemetry trajectory；
3. simulator + config + outcome；
4. 同样输入 + telemetry residual features。
Telemetry 保留 continuous phase distributions：queue/running residency、MNS/token slack、prefill/decode composition、partial/split prefill、step duration、KV、graph/padding。模型学习它们与 action 的 interaction；不先压成 bottleneck label。
### 4.3 Cost-aware policy
在合法的 single/joint candidate set 上选择：
```text
a* = argmax_a
expected constrained improvement(a)
/ expected all-in H20 cost(a)
```
探索项来自 posterior uncertainty/information gain；launch/SLO failure 有显式 penalty。Simulator 可提供 prior mean，但 simulator 与 real discrepancy 会被 posterior residual 更新。一次 target measurement 后更新 response model，并重新计算下一步 action 或停止概率。
LLM 在这个 tuning core 中不是 telemetry classifier。它最多作为可移除的 candidate/prior source，提出 schema 内的 sparse joint actions 或检索 engine mechanism；每个 proposal 都由同一个 response model、cost acquisition 和 real validator 评分。只有 `with LLM` 相对 `same tuner without LLM` 在 held-out tasks 上继续降低 cost-to-oracle，才能讨论 LLM 必要性。
### 4.4 Stop 条件
对一个预先声明的有限 candidate set，满足以下条件才 stop：
```text
P(exists c: f(c) > best_observed / (1 - epsilon) | D_t) < alpha
```
并且 best config 通过独立 confirmation、SLO/correctness gate，remaining candidate 的 cost-aware value of information 低于阈值。停止原因、posterior coverage 和未测区域必须写入 audit。
## 5. 下一阶段如何证明，而不是再次构造 heuristic
### R0：已有数据 retrospective premise check
- 用 C1/C3 response surfaces 检查 joint model 是否能避免 OAT trap。
- 用 action-aware paired records 比较 outcome-only 与 +telemetry 的 action-delta calibration。
- 用 SimFid surface 比较 direct model 与 simulator-residual model 的 rank/regret。
- 所有 feature、kernel、candidate encoding 在 held-out task 结果之前冻结。
R0 只能筛选 model family，不能作为 paper result，因为现有 tasks 已参与路线设计。
### R1：prospective same-host cost-to-oracle pilot
- dash0 8xH20，固定 engine build/model；serialized placement，禁止共置干扰。
- 至少一个未参与 feature/threshold 选择的新 trace window；选择非 ceiling、可 drain 的 offered load。
- 声明一个可穷举的小 surface，至少包含 topology/runtime crossed actions，而不是只有一个 MNS ladder。
- Oracle annotation 与 tuner online actions 分开记账；method 只能看到当时可用的数据。
- 运行 random/search、OAT、纯 LLM、当前 guided harness、frozen simulator+real、outcome-only response、+telemetry response、sim-residual +telemetry。
- 比较完整 H20 cost-to-regret curve，而不是 action classification accuracy。
Pilot opening gate：
1. telemetry model 相对相同 response model 去掉 telemetry，确实改变至少一个正确的 prospective action ranking；
2. 最终 regret <=5%，无 false-safe accept；
3. all-in H20-hours 相对 strongest safe outcome-only 至少下降 20%；
4. 如果使用 simulator，需相对 frozen simulator+real 至少下降 30%；
5. instrumentation overhead <=1%，所有成本和失败均计入。
若 1--5 任一失败，就不能把 telemetry/harness 写成 tuning contribution；保留其 debugging/measurement 价值即可。
### R2：task-held-out replication
至少 3 个 workload window x 2 个 SLO regime，按完整 task 做 leave-one-task-out 或固定 train/test split。报告每个 task 的 regret、安全和成本，以及 task-level paired bootstrap CI。只有 R2 通过，才能把单 task 的 61.09% lower-bound saving 升级为项目贡献。
## 6. 当前能说与不能说的贡献
当前能说：
- 我们有真实反例证明 OAT 和 cap-to-knob mapping 不是通用 tuning strategy。
- Harness 的 legality、exact replay、failure/cost accounting 有必要的实验基础设施价值。
- 当前 guided sequence 在一个严格同任务比较中显著减少了达到 2% empirical regret 的 reconstructed engine cost。
- Simulator 的边际计算便宜，但 rank error 会转化成显著 real regret 或更多 real-final 成本。
当前不能说：
- telemetry 已经对 end-to-end tuning 提供独立增益；现有 direct pilot 对此为 negative。
- 当前 harness 的 heuristic action ranking 是系统贡献；5% endpoint 只省 5.85%。
- LLM 是必要组件；尚无同 policy 的 with/without LLM held-out ablation。
- simulator 总 tuning cost 是 0；profile GPU cost 未审计，real verification 不能忽略。
- 3.35 是 global oracle，或 dash0 与 dash1 数字是完全 controlled comparison。
## Data sanity
- Dash0 sequential numeric scores：n=9，min/max `1.1042/3.35`，distinct=7；两组 config outcome 不全相同。
- Exact surface scores：n=12，min/max `1.2833/3.2833`，distinct=8；12 cells 完整且与 simulator metrics 中的 real scores 一致。
- Reconstructed trial/cell attempts 包括 4 个无 engine timestamp 的失败：n=32，min/max `0/0.49778 H20h`，distinct=26；所有可重建成本均非负。
- Sequential regret observations：n=16，min/max `0/0.34328`，distinct=6，全部在 `[0,1]`。
- Checked invariants：dash0 fixed task contexts 相同（除 method/port）；trial counts 与 manifest 相符；engine log timestamps monotonic；surface cell 唯一且 MBT=8192；simulator 无失败且 predictions 不全相同；scores/results 不全相同；cost 非负；regret bounded。
- Measurement limitation：primary 12-cell campaign 的 4 个 TP4 pre-ready failure 没有 engine timestamp，随后由 companion campaign 完整重跑；其失败成本在 engine-lifetime reconstruction 中为 0。因此 `3.5953 H20h` 是 completed annotation lower bound，不能作为 all-in annotation cost。这个缺口已显式保留，没有在其上建立 total-cost claim。

507
runs/tuning-cost/analyze.py Normal file
View File

@@ -0,0 +1,507 @@
#!/usr/bin/env python3
"""Reconstruct tuning cost and cost-to-oracle curves from existing runs.
The historical engine logs do not contain controller setup/cleanup timestamps.
Consequently the reported GPU cost is an engine-lifetime lower bound:
parallel_size * (last engine timestamp - first engine timestamp). It must not
be presented as all-in method cost. New experiments should record allocation
start/end directly.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import math
import re
import subprocess
from datetime import datetime
from pathlib import Path
from typing import Any
from urllib.parse import urlsplit
SCHEMA = "aituner-tuning-cost-analysis-v1"
TIMESTAMP = re.compile(r"\b(\d{2}-\d{2} \d{2}:\d{2}:\d{2})(?:\.\d+)?\b")
def numeric_summary(values: list[float]) -> dict[str, Any]:
values = [float(value) for value in values]
return {
"n": len(values),
"min": min(values) if values else None,
"max": max(values) if values else None,
"distinct_n": len(set(values)),
}
def sha256_text(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()
class Reader:
def __init__(self, repo_root: Path):
self.repo_root = repo_root
def _split(self, locator: str) -> tuple[str | None, str]:
if locator.startswith("ssh://"):
parsed = urlsplit(locator)
if not parsed.hostname or not parsed.path.startswith("/"):
raise ValueError(f"invalid SSH locator: {locator}")
return parsed.hostname, parsed.path
path = Path(locator)
if not path.is_absolute():
path = self.repo_root / path
return None, str(path)
def read_text(self, locator: str) -> str:
host, path = self._split(locator)
if host is None:
return Path(path).read_text(encoding="utf-8", errors="replace")
completed = subprocess.run(
["ssh", host, "cat", "--", path],
check=True,
capture_output=True,
text=True,
)
return completed.stdout
def join_locator(root: str, *parts: str) -> str:
return "/".join([root.rstrip("/"), *(part.strip("/") for part in parts)])
def timestamp_span(log_text: str, year: int) -> tuple[float | None, bool, int]:
parsed = [
datetime.strptime(f"{year}-{match}", "%Y-%m-%d %H:%M:%S")
for match in TIMESTAMP.findall(log_text)
]
if not parsed:
return None, True, 0
monotonic = all(right >= left for left, right in zip(parsed, parsed[1:]))
return (max(parsed) - min(parsed)).total_seconds(), monotonic, len(parsed)
def load_campaign(
reader: Reader,
root: str,
year: int,
missing_duration_s: dict[str, float] | None = None,
missing_reason: dict[str, str] | None = None,
) -> dict[str, Any]:
missing_duration_s = missing_duration_s or {}
missing_reason = missing_reason or {}
state_text = reader.read_text(join_locator(root, "state.json"))
state = json.loads(state_text)
trials = []
for trial in state["trials"]:
trial_id = str(trial["trial_id"])
log_text = reader.read_text(join_locator(root, "trials", trial_id, "engine.log"))
duration_s, monotonic, timestamp_n = timestamp_span(log_text, year)
duration_source = "engine_log_span"
duration_note = ""
if duration_s is None:
duration_s = float(missing_duration_s.get(trial_id, 0.0))
duration_source = (
"conservative_fallback" if trial_id in missing_duration_s else "no_engine_timestamp"
)
duration_note = missing_reason.get(trial_id, "")
parallel_size = int(trial["parallel_size"])
score = trial.get("best_request_rate_per_gpu")
trials.append(
{
"trial_id": trial_id,
"status": trial["status"],
"failure_stage": trial.get("failure_stage", ""),
"parallel_size": parallel_size,
"score_req_s_per_gpu": None if score is None else float(score),
"config_patch": trial["config_patch"],
"duration_s": duration_s,
"duration_source": duration_source,
"duration_note": duration_note,
"engine_timestamp_n": timestamp_n,
"engine_timestamps_monotonic": monotonic,
"engine_h20_hours_lower_bound": duration_s * parallel_size / 3600.0,
"engine_log_sha256": sha256_text(log_text),
}
)
return {
"root": root,
"state_sha256": sha256_text(state_text),
"trials": trials,
}
def fixed_task_context(spec: dict[str, Any]) -> dict[str, Any]:
flags = dict(spec["engine"]["base_flags"])
flags.pop("port", None)
return {
"model": spec["model"],
"hardware": spec["hardware"],
"engine_version": spec["engine"]["engine_version"],
"launch_args": spec["engine"]["launch_args"],
"base_flags_without_port": flags,
"search": spec["search"],
"slo": spec["slo"],
"trace": spec["trace"],
}
def regret(score: float, reference: float) -> float:
if reference <= 0:
raise ValueError("reference score must be positive")
return max(0.0, 1.0 - float(score) / float(reference))
def sequential_curve(
trials: list[dict[str, Any]], reference: float, thresholds: list[float]
) -> dict[str, Any]:
cumulative_cost = 0.0
best_score: float | None = None
points = []
for trial in trials:
cumulative_cost += float(trial["engine_h20_hours_lower_bound"])
score = trial["score_req_s_per_gpu"]
if score is not None:
best_score = score if best_score is None else max(best_score, score)
points.append(
{
"trial_id": trial["trial_id"],
"cumulative_engine_h20_hours_lower_bound": cumulative_cost,
"best_score_req_s_per_gpu": best_score,
"regret": None if best_score is None else regret(best_score, reference),
}
)
hits = {}
for threshold in thresholds:
hit = next(
(
point
for point in points
if point["regret"] is not None
and float(point["regret"]) <= float(threshold) + 1e-12
),
None,
)
hits[f"regret_le_{threshold:g}"] = hit
return {
"reference_score_req_s_per_gpu": reference,
"points": points,
"cost_to_threshold": hits,
"total_engine_h20_hours_lower_bound": cumulative_cost,
}
def surface_cell(trial: dict[str, Any]) -> str:
flags = trial["config_patch"]["flag_patch"]
return f"tp{int(flags['tensor-parallel-size'])}_mns{int(flags['max-num-seqs'])}"
def tie_expanded_candidates(scores: dict[str, float], nominal_k: int) -> list[str]:
ordered = sorted(scores, key=lambda cell: (-float(scores[cell]), cell))
if nominal_k <= 0 or nominal_k > len(ordered):
raise ValueError("nominal k outside score surface")
cutoff = float(scores[ordered[nominal_k - 1]])
tolerance = max(1e-12, abs(cutoff) * 1e-12)
return [cell for cell in ordered if float(scores[cell]) >= cutoff - tolerance]
def real_final_policy(
candidates: list[str], real_scores: dict[str, float], cell_costs: dict[str, float]
) -> dict[str, Any]:
oracle = max(real_scores.values())
selected = max(candidates, key=lambda cell: (real_scores[cell], cell))
return {
"candidate_cells": candidates,
"real_evaluations": len(candidates),
"selected_cell": selected,
"selected_real_score_req_s_per_gpu": real_scores[selected],
"real_regret": regret(real_scores[selected], oracle),
"engine_h20_hours_lower_bound": sum(cell_costs[cell] for cell in candidates),
}
def percentage_saving(new: float, old: float) -> float:
if old <= 0:
raise ValueError("baseline cost must be positive")
return 1.0 - new / old
def build_analysis(manifest_path: Path) -> dict[str, Any]:
repo_root = manifest_path.resolve().parents[2]
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
reader = Reader(repo_root)
year = int(manifest["year"])
sequential = {}
task_contexts = {}
for name, run in manifest["sequential_runs"].items():
sequential[name] = load_campaign(
reader,
run["root"],
year,
run.get("missing_log_duration_s"),
run.get("missing_log_reason"),
)
spec_text = reader.read_text(join_locator(run["root"], "study_spec.snapshot.json"))
sequential[name]["study_spec_sha256"] = sha256_text(spec_text)
task_contexts[name] = fixed_task_context(json.loads(spec_text))
all_sequential_scores = [
trial["score_req_s_per_gpu"]
for campaign in sequential.values()
for trial in campaign["trials"]
if trial["score_req_s_per_gpu"] is not None
]
empirical_reference = max(all_sequential_scores)
thresholds = [float(value) for value in manifest["threshold_regrets"]]
for campaign in sequential.values():
campaign["curve"] = sequential_curve(
campaign["trials"], empirical_reference, thresholds
)
surface_manifest = manifest["real_surface"]
primary = load_campaign(reader, surface_manifest["primary_root"], year)
companion = load_campaign(reader, surface_manifest["tp4_companion_root"], year)
completed_surface_trials = [
trial
for trial in primary["trials"] + companion["trials"]
if trial["status"] == "completed" and trial["score_req_s_per_gpu"] is not None
]
cells: dict[str, dict[str, Any]] = {}
for trial in completed_surface_trials:
flags = trial["config_patch"]["flag_patch"]
if int(flags["max-num-batched-tokens"]) != int(
surface_manifest["fixed_max_num_batched_tokens"]
):
raise ValueError("surface MBT invariant failed")
cell = surface_cell(trial)
if cell in cells:
raise ValueError(f"duplicate completed surface cell: {cell}")
cells[cell] = trial
real_scores = {cell: float(trial["score_req_s_per_gpu"]) for cell, trial in cells.items()}
cell_costs = {
cell: float(trial["engine_h20_hours_lower_bound"]) for cell, trial in cells.items()
}
surface_oracle_score = max(real_scores.values())
surface_oracle_cells = [
cell for cell, score in real_scores.items() if math.isclose(score, surface_oracle_score)
]
simulator_text = reader.read_text(manifest["simulator_metrics"])
simulator = json.loads(simulator_text)
simulator_real_scores = {
cell: float(score) for cell, score in simulator["real_scores"].items()
}
surface_matches_simulator = set(real_scores) == set(simulator_real_scores) and all(
math.isclose(real_scores[cell], simulator_real_scores[cell], abs_tol=1e-12)
for cell in real_scores
)
throughput = simulator["analyses"]["frozen-calibrated/throughput-proxy"]
throughput_scores = {
cell: float(score) for cell, score in throughput["simulated_scores"].items()
}
throughput_real_final = {
f"nominal_k_{k}": real_final_policy(
tie_expanded_candidates(throughput_scores, k), real_scores, cell_costs
)
for k in (1, 2, 3)
}
throughput_top1 = tie_expanded_candidates(throughput_scores, 1)
simulator_only_cell = throughput_top1[0]
slo = simulator["analyses"]["frozen-calibrated/SLO-gated"]
slo_top_bucket = list(slo["metrics"]["top1"]["candidate_cells"])
slo_diagnostic = real_final_policy(slo_top_bucket, real_scores, cell_costs)
pure_hits = sequential["pure_llm"]["curve"]["cost_to_threshold"]
guided_hits = sequential["guided_harness"]["curve"]["cost_to_threshold"]
direct_comparison = {}
for threshold in (0.05, 0.02):
key = f"regret_le_{threshold:g}"
pure_hit = pure_hits[key]
guided_hit = guided_hits[key]
if pure_hit is None or guided_hit is None:
continue
pure_cost = float(pure_hit["cumulative_engine_h20_hours_lower_bound"])
guided_cost = float(guided_hit["cumulative_engine_h20_hours_lower_bound"])
direct_comparison[key] = {
"pure_llm_h20_hours_lower_bound": pure_cost,
"guided_harness_h20_hours_lower_bound": guided_cost,
"guided_saving_vs_pure_llm": percentage_saving(guided_cost, pure_cost),
}
five_cost = direct_comparison["regret_le_0.05"][
"guided_harness_h20_hours_lower_bound"
]
two_cost = direct_comparison["regret_le_0.02"][
"guided_harness_h20_hours_lower_bound"
]
sim_real_cost = float(slo_diagnostic["engine_h20_hours_lower_bound"])
target_bars = {
"five_percent_regret": {
"twenty_percent_below_current_guided": 0.8 * five_cost,
"thirty_percent_below_posthoc_sim_slo_real_final": 0.7 * sim_real_cost,
"development_target_h20_hours_lower_bound": min(
0.8 * five_cost, 0.7 * sim_real_cost
),
},
"two_percent_regret": {
"twenty_percent_below_current_guided": 0.8 * two_cost,
"thirty_percent_below_posthoc_sim_slo_real_final": 0.7 * sim_real_cost,
"development_target_h20_hours_lower_bound": min(
0.8 * two_cost, 0.7 * sim_real_cost
),
},
}
all_trial_costs = [
float(trial["engine_h20_hours_lower_bound"])
for campaign in sequential.values()
for trial in campaign["trials"]
] + [
float(trial["engine_h20_hours_lower_bound"])
for campaign in (primary, companion)
for trial in campaign["trials"]
]
all_regrets = [
float(point["regret"])
for campaign in sequential.values()
for point in campaign["curve"]["points"]
if point["regret"] is not None
]
monotonic_logs = all(
trial["engine_timestamps_monotonic"]
for campaign in [*sequential.values(), primary, companion]
for trial in campaign["trials"]
)
invariants = {
"dash0_task_contexts_equal_except_method_and_port": len(
{json.dumps(value, sort_keys=True) for value in task_contexts.values()}
)
== 1,
"surface_has_expected_cell_count": len(cells)
== int(surface_manifest["expected_cells"]),
"surface_matches_simulator_real_scores": surface_matches_simulator,
"all_costs_non_negative": all(value >= 0 for value in all_trial_costs),
"all_regrets_in_0_1": all(0 <= value <= 1 for value in all_regrets),
"surface_scores_not_all_identical": len(set(real_scores.values())) > 1,
"sequential_scores_not_all_identical": len(set(all_sequential_scores)) > 1,
"engine_log_timestamps_monotonic": monotonic_logs,
"sequential_trial_counts_match_manifest": all(
len(sequential[name]["trials"]) == int(run["expected_trials"])
for name, run in manifest["sequential_runs"].items()
),
"simulator_suite_has_no_failed_runs": int(simulator["execution"]["failed_runs"])
== 0,
"simulator_scores_not_all_identical": len(set(throughput_scores.values())) > 1,
}
failed_invariants = [name for name, passed in invariants.items() if not passed]
if failed_invariants:
raise RuntimeError(f"data sanity invariant failed: {failed_invariants}")
failed_primary_attempts = [
trial for trial in primary["trials"] if trial["status"] != "completed"
]
return {
"schema": SCHEMA,
"cost_definition": {
"reported_metric": "engine H20-hours lower bound",
"formula": "parallel_size * (last_engine_log_timestamp - first_engine_log_timestamp) / 3600",
"included": ["engine startup after first timestamp", "warm-up/probes until last timestamp"],
"not_reconstructable": [
"GPU allocation before first engine timestamp",
"controller/LLM latency",
"cleanup after last engine timestamp",
"one-time simulator operator profiling GPU-hours",
],
"future_all_in_metric": "allocation_start_to_GPU_idle * allocated_GPU_count, including failures",
},
"comparison_scope": {
"pure_llm_vs_guided_harness": "direct: same dash0 fixed task context",
"simulator_vs_surface": "direct: simulator predictions and exact dash1 12-cell real surface",
"dash0_methods_vs_dash1_simulator": "indicative only: matched model/engine/workload/GPU type, different host and campaign",
},
"empirical_reference": {
"score_req_s_per_gpu": empirical_reference,
"meaning": "best observed across the two dash0 sequential runs; not a global oracle",
},
"sequential_runs": sequential,
"direct_dash0_comparison": direct_comparison,
"real_surface": {
"cells": {
cell: {
"score_req_s_per_gpu": real_scores[cell],
"engine_h20_hours_lower_bound": cell_costs[cell],
}
for cell in sorted(cells)
},
"oracle_score_req_s_per_gpu": surface_oracle_score,
"oracle_cells": surface_oracle_cells,
"completed_annotation_engine_h20_hours_lower_bound": sum(cell_costs.values()),
"failed_primary_attempt_n": len(failed_primary_attempts),
"failed_primary_attempt_engine_h20_hours_lower_bound": sum(
float(trial["engine_h20_hours_lower_bound"])
for trial in failed_primary_attempts
),
"primary_state_sha256": primary["state_sha256"],
"tp4_companion_state_sha256": companion["state_sha256"],
},
"simulator": {
"marginal_gpu_hours_without_real_verification": 0.0,
"observed_fidelity_suite_cpu_hours": float(
simulator["execution"]["suite_elapsed_seconds"]
)
/ 3600.0,
"observed_fidelity_suite_runs": int(simulator["execution"]["attempted_runs"]),
"one_time_profile_gpu_hours": None,
"one_time_profile_cost_status": "not recorded; total cold-start cost is unknown",
"decision_bearing_throughput_proxy_sim_only_top1": {
"selected_cell": simulator_only_cell,
"selected_real_score_req_s_per_gpu": real_scores[simulator_only_cell],
"real_regret": regret(real_scores[simulator_only_cell], surface_oracle_score),
"gpu_hours": 0.0,
},
"decision_bearing_throughput_proxy_plus_real_final": throughput_real_final,
"posthoc_slo_gated_plus_real_final": {
**slo_diagnostic,
"status": "diagnostic/post-hoc, not a preregistered prospective policy",
"false_feasible": int(slo["false_feasibility"]["overall"]["false_feasible"]),
"false_infeasible": int(slo["false_feasibility"]["overall"]["false_infeasible"]),
},
"metrics_sha256": sha256_text(simulator_text),
},
"provisional_development_targets": {
"status": "lower-bound, single-task targets; require same-host prospective validation",
**target_bars,
},
"data_sanity": {
"invariants": invariants,
"sequential_score_summary": numeric_summary(all_sequential_scores),
"surface_score_summary": numeric_summary(list(real_scores.values())),
"simulator_throughput_score_summary": numeric_summary(
list(throughput_scores.values())
),
"trial_cost_summary": numeric_summary(all_trial_costs),
"regret_summary": numeric_summary(all_regrets),
},
}
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--manifest", type=Path, default=Path(__file__).with_name("manifest.json"))
parser.add_argument("--output", type=Path, default=Path(__file__).with_name("metrics.json"))
args = parser.parse_args()
analysis = build_analysis(args.manifest)
args.output.write_text(json.dumps(analysis, indent=2, sort_keys=True) + "\n", encoding="utf-8")
print(json.dumps({
"status": "ok",
"output": str(args.output),
"empirical_reference": analysis["empirical_reference"],
"surface_oracle": analysis["real_surface"]["oracle_score_req_s_per_gpu"],
"sanity": analysis["data_sanity"],
}, indent=2, sort_keys=True))
if __name__ == "__main__":
main()

42
runs/tuning-cost/manifest.json Normal file
View File

@@ -0,0 +1,42 @@
{
"schema": "aituner-tuning-cost-v1",
"year": 2026,
"task": {
"model": "Qwen/Qwen3-30B-A3B",
"engine": "community-vLLM 0.20.0",
"gpu": "NVIDIA H20",
"trace_window": "chat_w20260311_1000",
"input_tokens": [0, 8192],
"output_tokens": 128,
"replay_time_scale": 0.1,
"target_pass_rate": 0.95,
"ttft_ms": [2000, 4000, 6000],
"tpot_ms": 50
},
"sequential_runs": {
"pure_llm": {
"root": "ssh://dash0/home/admin/cpfs/wjh/aituner/aituner/.aituner-community-vllm020/dash0-qwen30b-a3b-community-vllm020-chat-0-8k-out128-scale01-high1-noharness",
"expected_trials": 12,
"missing_log_duration_s": {
"trial-0003": 13.0
},
"missing_log_reason": {
"trial-0003": "conservative trial_spec-to-result mtime envelope for a pre-ready CLI failure"
}
},
"guided_harness": {
"root": "ssh://dash0/home/admin/cpfs/wjh/aituner/aituner/.aituner-community-vllm020/dash0-qwen30b-a3b-community-vllm020-chat-0-8k-out128-scale01-high1-harness-guided-v2",
"expected_trials": 4,
"missing_log_duration_s": {},
"missing_log_reason": {}
}
},
"real_surface": {
"primary_root": "recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-mixed-qwen30b-tp-mns-surface-high1-dash1-d8899c5-20260701T095858Z/store/interaction-mixed-qwen30b-tp-mns-surface-high1-dash1-d8899c5-20260701T095858Z",
"tp4_companion_root": "recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-mixed-qwen30b-tp4-mns-nocap-qps20-dash1-d8899c5-20260701T161900Z/store/interaction-mixed-qwen30b-tp4-mns-nocap-qps20-dash1-d8899c5-20260701T161900Z",
"expected_cells": 12,
"fixed_max_num_batched_tokens": 8192
},
"simulator_metrics": "/home/gahow/phd/replayserve/runs/simfid_s2rb/results/metrics.json",
"threshold_regrets": [0.05, 0.02, 0.01, 0.0]
}

736
runs/tuning-cost/metrics.json Normal file
View File

@@ -0,0 +1,736 @@
{
"comparison_scope": {
"dash0_methods_vs_dash1_simulator": "indicative only: matched model/engine/workload/GPU type, different host and campaign",
"pure_llm_vs_guided_harness": "direct: same dash0 fixed task context",
"simulator_vs_surface": "direct: simulator predictions and exact dash1 12-cell real surface"
},
"cost_definition": {
"formula": "parallel_size * (last_engine_log_timestamp - first_engine_log_timestamp) / 3600",
"future_all_in_metric": "allocation_start_to_GPU_idle * allocated_GPU_count, including failures",
"included": [
"engine startup after first timestamp",
"warm-up/probes until last timestamp"
],
"not_reconstructable": [
"GPU allocation before first engine timestamp",
"controller/LLM latency",
"cleanup after last engine timestamp",
"one-time simulator operator profiling GPU-hours"
],
"reported_metric": "engine H20-hours lower bound"
},
"data_sanity": {
"invariants": {
"all_costs_non_negative": true,
"all_regrets_in_0_1": true,
"dash0_task_contexts_equal_except_method_and_port": true,
"engine_log_timestamps_monotonic": true,
"sequential_scores_not_all_identical": true,
"sequential_trial_counts_match_manifest": true,
"simulator_scores_not_all_identical": true,
"simulator_suite_has_no_failed_runs": true,
"surface_has_expected_cell_count": true,
"surface_matches_simulator_real_scores": true,
"surface_scores_not_all_identical": true
},
"regret_summary": {
"distinct_n": 6,
"max": 0.34328358208955223,
"min": 0.0,
"n": 16
},
"sequential_score_summary": {
"distinct_n": 7,
"max": 3.35,
"min": 1.1041666666666667,
"n": 9
},
"simulator_throughput_score_summary": {
"distinct_n": 10,
"max": 4.356763578770651,
"min": 1.5449814460277083,
"n": 12
},
"surface_score_summary": {
"distinct_n": 8,
"max": 3.283333333333333,
"min": 1.2833333333333334,
"n": 12
},
"trial_cost_summary": {
"distinct_n": 26,
"max": 0.49777777777777776,
"min": 0.0,
"n": 32
}
},
"direct_dash0_comparison": {
"regret_le_0.02": {
"guided_harness_h20_hours_lower_bound": 0.4458333333333333,
"guided_saving_vs_pure_llm": 0.6109090909090908,
"pure_llm_h20_hours_lower_bound": 1.1458333333333333
},
"regret_le_0.05": {
"guided_harness_h20_hours_lower_bound": 0.26805555555555555,
"guided_saving_vs_pure_llm": 0.0585365853658536,
"pure_llm_h20_hours_lower_bound": 0.2847222222222222
}
},
"empirical_reference": {
"meaning": "best observed across the two dash0 sequential runs; not a global oracle",
"score_req_s_per_gpu": 3.35
},
"provisional_development_targets": {
"five_percent_regret": {
"development_target_h20_hours_lower_bound": 0.21444444444444444,
"thirty_percent_below_posthoc_sim_slo_real_final": 0.36088888888888887,
"twenty_percent_below_current_guided": 0.21444444444444444
},
"status": "lower-bound, single-task targets; require same-host prospective validation",
"two_percent_regret": {
"development_target_h20_hours_lower_bound": 0.3566666666666667,
"thirty_percent_below_posthoc_sim_slo_real_final": 0.36088888888888887,
"twenty_percent_below_current_guided": 0.3566666666666667
}
},
"real_surface": {
"cells": {
"tp1_mns16": {
"engine_h20_hours_lower_bound": 0.14083333333333334,
"score_req_s_per_gpu": 2.35
},
"tp1_mns32": {
"engine_h20_hours_lower_bound": 0.13194444444444445,
"score_req_s_per_gpu": 2.283333333333333
},
"tp1_mns64": {
"engine_h20_hours_lower_bound": 0.13527777777777777,
"score_req_s_per_gpu": 2.283333333333333
},
"tp1_mns8": {
"engine_h20_hours_lower_bound": 0.13333333333333333,
"score_req_s_per_gpu": 2.1
},
"tp2_mns16": {
"engine_h20_hours_lower_bound": 0.29944444444444446,
"score_req_s_per_gpu": 2.275
},
"tp2_mns32": {
"engine_h20_hours_lower_bound": 0.2816666666666667,
"score_req_s_per_gpu": 3.283333333333333
},
"tp2_mns64": {
"engine_h20_hours_lower_bound": 0.2338888888888889,
"score_req_s_per_gpu": 3.2583333333333333
},
"tp2_mns8": {
"engine_h20_hours_lower_bound": 0.2777777777777778,
"score_req_s_per_gpu": 2.275
},
"tp4_mns16": {
"engine_h20_hours_lower_bound": 0.4866666666666667,
"score_req_s_per_gpu": 2.441666666666667
},
"tp4_mns32": {
"engine_h20_hours_lower_bound": 0.49777777777777776,
"score_req_s_per_gpu": 2.441666666666667
},
"tp4_mns64": {
"engine_h20_hours_lower_bound": 0.49666666666666665,
"score_req_s_per_gpu": 2.441666666666667
},
"tp4_mns8": {
"engine_h20_hours_lower_bound": 0.48,
"score_req_s_per_gpu": 1.2833333333333334
}
},
"completed_annotation_engine_h20_hours_lower_bound": 3.5952777777777776,
"failed_primary_attempt_engine_h20_hours_lower_bound": 0.0,
"failed_primary_attempt_n": 4,
"oracle_cells": [
"tp2_mns32"
],
"oracle_score_req_s_per_gpu": 3.283333333333333,
"primary_state_sha256": "790bf6df9045e22c46a8abdc022390bf842d884ef13ade4b627936722b9c3366",
"tp4_companion_state_sha256": "1fc4f584a69a90fb0ff1d64f6a07a8383c368f4e306a0b7a3c19d2304042fd73"
},
"schema": "aituner-tuning-cost-analysis-v1",
"sequential_runs": {
"guided_harness": {
"curve": {
"cost_to_threshold": {
"regret_le_0": null,
"regret_le_0.01": null,
"regret_le_0.02": {
"best_score_req_s_per_gpu": 3.283333333333333,
"cumulative_engine_h20_hours_lower_bound": 0.4458333333333333,
"regret": 0.01990049751243783,
"trial_id": "trial-0003"
},
"regret_le_0.05": {
"best_score_req_s_per_gpu": 3.2583333333333333,
"cumulative_engine_h20_hours_lower_bound": 0.26805555555555555,
"regret": 0.027363184079602032,
"trial_id": "trial-0002"
}
},
"points": [
{
"best_score_req_s_per_gpu": 2.3833333333333333,
"cumulative_engine_h20_hours_lower_bound": 0.09305555555555556,
"regret": 0.2885572139303483,
"trial_id": "trial-0001"
},
{
"best_score_req_s_per_gpu": 3.2583333333333333,
"cumulative_engine_h20_hours_lower_bound": 0.26805555555555555,
"regret": 0.027363184079602032,
"trial_id": "trial-0002"
},
{
"best_score_req_s_per_gpu": 3.283333333333333,
"cumulative_engine_h20_hours_lower_bound": 0.4458333333333333,
"regret": 0.01990049751243783,
"trial_id": "trial-0003"
},
{
"best_score_req_s_per_gpu": 3.3,
"cumulative_engine_h20_hours_lower_bound": 0.6230555555555555,
"regret": 0.014925373134328401,
"trial_id": "trial-0004"
}
],
"reference_score_req_s_per_gpu": 3.35,
"total_engine_h20_hours_lower_bound": 0.6230555555555555
},
"root": "ssh://dash0/home/admin/cpfs/wjh/aituner/aituner/.aituner-community-vllm020/dash0-qwen30b-a3b-community-vllm020-chat-0-8k-out128-scale01-high1-harness-guided-v2",
"state_sha256": "76e439d1f20e0f9af54785005258cbe39f205ae107fc2398955e360e263c7718",
"study_spec_sha256": "03241089052f1a07dbbb2ba6100c6737635877b683d275813b3c59f2769fc2d7",
"trials": [
{
"config_patch": {
"env_patch": {},
"flag_patch": {}
},
"duration_note": "",
"duration_s": 335.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.09305555555555556,
"engine_log_sha256": "6d8b908085c03d4796e78679eed13737581fc7d2e18947fb72d7031a9f557c9d",
"engine_timestamp_n": 108,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 1,
"score_req_s_per_gpu": 2.3833333333333333,
"status": "completed",
"trial_id": "trial-0001"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 315.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.175,
"engine_log_sha256": "b09caadd930495c27effea57e4164e52f58c6461cbbc0f4e873a309d45ec3443",
"engine_timestamp_n": 134,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": 3.2583333333333333,
"status": "completed",
"trial_id": "trial-0002"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"enable-chunked-prefill": true,
"gpu-memory-utilization": 0.95,
"max-num-batched-tokens": 16384,
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 320.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.17777777777777778,
"engine_log_sha256": "8f6f2637cff8566d39b8e6c17de71ed37a01f2aa6a4460ff36ecc9728d033c3a",
"engine_timestamp_n": 134,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": 3.283333333333333,
"status": "completed",
"trial_id": "trial-0003"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"enable-chunked-prefill": true,
"gpu-memory-utilization": 0.95,
"max-num-batched-tokens": 24576,
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 319.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.17722222222222223,
"engine_log_sha256": "29b8fc588b5422ceae60456188d6373070c179610243258fa213aeead9f07a00",
"engine_timestamp_n": 137,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": 3.3,
"status": "completed",
"trial_id": "trial-0004"
}
]
},
"pure_llm": {
"curve": {
"cost_to_threshold": {
"regret_le_0": {
"best_score_req_s_per_gpu": 3.35,
"cumulative_engine_h20_hours_lower_bound": 1.3719444444444444,
"regret": 0.0,
"trial_id": "trial-0007"
},
"regret_le_0.01": {
"best_score_req_s_per_gpu": 3.35,
"cumulative_engine_h20_hours_lower_bound": 1.3719444444444444,
"regret": 0.0,
"trial_id": "trial-0007"
},
"regret_le_0.02": {
"best_score_req_s_per_gpu": 3.3,
"cumulative_engine_h20_hours_lower_bound": 1.1458333333333333,
"regret": 0.014925373134328401,
"trial_id": "trial-0006"
},
"regret_le_0.05": {
"best_score_req_s_per_gpu": 3.2583333333333333,
"cumulative_engine_h20_hours_lower_bound": 0.2847222222222222,
"regret": 0.027363184079602032,
"trial_id": "trial-0002"
}
},
"points": [
{
"best_score_req_s_per_gpu": 2.2,
"cumulative_engine_h20_hours_lower_bound": 0.10861111111111112,
"regret": 0.34328358208955223,
"trial_id": "trial-0001"
},
{
"best_score_req_s_per_gpu": 3.2583333333333333,
"cumulative_engine_h20_hours_lower_bound": 0.2847222222222222,
"regret": 0.027363184079602032,
"trial_id": "trial-0002"
},
{
"best_score_req_s_per_gpu": 3.2583333333333333,
"cumulative_engine_h20_hours_lower_bound": 0.29194444444444445,
"regret": 0.027363184079602032,
"trial_id": "trial-0003"
},
{
"best_score_req_s_per_gpu": 3.2583333333333333,
"cumulative_engine_h20_hours_lower_bound": 0.4280555555555555,
"regret": 0.027363184079602032,
"trial_id": "trial-0004"
},
{
"best_score_req_s_per_gpu": 3.2583333333333333,
"cumulative_engine_h20_hours_lower_bound": 0.9247222222222222,
"regret": 0.027363184079602032,
"trial_id": "trial-0005"
},
{
"best_score_req_s_per_gpu": 3.3,
"cumulative_engine_h20_hours_lower_bound": 1.1458333333333333,
"regret": 0.014925373134328401,
"trial_id": "trial-0006"
},
{
"best_score_req_s_per_gpu": 3.35,
"cumulative_engine_h20_hours_lower_bound": 1.3719444444444444,
"regret": 0.0,
"trial_id": "trial-0007"
},
{
"best_score_req_s_per_gpu": 3.35,
"cumulative_engine_h20_hours_lower_bound": 1.561388888888889,
"regret": 0.0,
"trial_id": "trial-0008"
},
{
"best_score_req_s_per_gpu": 3.35,
"cumulative_engine_h20_hours_lower_bound": 1.7497222222222222,
"regret": 0.0,
"trial_id": "trial-0009"
},
{
"best_score_req_s_per_gpu": 3.35,
"cumulative_engine_h20_hours_lower_bound": 1.9047222222222222,
"regret": 0.0,
"trial_id": "trial-0010"
},
{
"best_score_req_s_per_gpu": 3.35,
"cumulative_engine_h20_hours_lower_bound": 2.0930555555555554,
"regret": 0.0,
"trial_id": "trial-0011"
},
{
"best_score_req_s_per_gpu": 3.35,
"cumulative_engine_h20_hours_lower_bound": 2.2824999999999998,
"regret": 0.0,
"trial_id": "trial-0012"
}
],
"reference_score_req_s_per_gpu": 3.35,
"total_engine_h20_hours_lower_bound": 2.2824999999999998
},
"root": "ssh://dash0/home/admin/cpfs/wjh/aituner/aituner/.aituner-community-vllm020/dash0-qwen30b-a3b-community-vllm020-chat-0-8k-out128-scale01-high1-noharness",
"state_sha256": "342cde3546299cd312ebc078b1c9589773b841d23c4ed4632370883be598fd4a",
"study_spec_sha256": "3238dc89477108405c09b25eee434c3c10c0f5be245d2bfa3b90509a372c088e",
"trials": [
{
"config_patch": {
"env_patch": {},
"flag_patch": {}
},
"duration_note": "",
"duration_s": 391.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.10861111111111112,
"engine_log_sha256": "c5caf9e9f6faa73a4a39adb71a7a6c04bcf623d5c0c91b231116bb596c8a11be",
"engine_timestamp_n": 113,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 1,
"score_req_s_per_gpu": 2.2,
"status": "completed",
"trial_id": "trial-0001"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 317.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.1761111111111111,
"engine_log_sha256": "239ff7349d7b9a1b24334fbe6e2a45a967bfa2d16380e98da4f81520d422cd0d",
"engine_timestamp_n": 134,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": 3.2583333333333333,
"status": "completed",
"trial_id": "trial-0002"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"enable-expert-parallel": true,
"expert-parallel-size": 2,
"tensor-parallel-size": 2
}
},
"duration_note": "conservative trial_spec-to-result mtime envelope for a pre-ready CLI failure",
"duration_s": 13.0,
"duration_source": "conservative_fallback",
"engine_h20_hours_lower_bound": 0.007222222222222222,
"engine_log_sha256": "4707735f33c8bf9030054eccd36b66985a3879050d80a853f93f166282414d8f",
"engine_timestamp_n": 0,
"engine_timestamps_monotonic": true,
"failure_stage": "engine_launch",
"parallel_size": 2,
"score_req_s_per_gpu": null,
"status": "failed",
"trial_id": "trial-0003"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"data-parallel-size": 2,
"tensor-parallel-size": 1
}
},
"duration_note": "",
"duration_s": 245.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.1361111111111111,
"engine_log_sha256": "215ed31c6d227e4a6771ecb637c2b4e3e38e766d73a8882a923d1fa8b97a23da",
"engine_timestamp_n": 170,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": null,
"status": "completed",
"trial_id": "trial-0004"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"data-parallel-size": 2,
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 447.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.49666666666666665,
"engine_log_sha256": "6057b3477ccca14e0ebd467ac96b777662df67b57827c851d58caf3a77b95ca4",
"engine_timestamp_n": 201,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 4,
"score_req_s_per_gpu": 1.1041666666666667,
"status": "completed",
"trial_id": "trial-0005"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"enable-chunked-prefill": true,
"gpu-memory-utilization": 0.95,
"max-num-batched-tokens": 16384,
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 398.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.22111111111111112,
"engine_log_sha256": "c5c336f74d419645639b469eea9f92575780164637fdda078fa5c84c96915f1f",
"engine_timestamp_n": 138,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": 3.3,
"status": "completed",
"trial_id": "trial-0006"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"enable-chunked-prefill": true,
"gpu-memory-utilization": 0.95,
"max-num-batched-tokens": 24576,
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 407.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.22611111111111112,
"engine_log_sha256": "5e2829479ecd4771e7c781d40209f74e4c2843bbca15522974e435e06daa12a2",
"engine_timestamp_n": 142,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": 3.35,
"status": "completed",
"trial_id": "trial-0007"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"enable-chunked-prefill": true,
"gpu-memory-utilization": 0.95,
"max-num-batched-tokens": 32768,
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 341.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.18944444444444444,
"engine_log_sha256": "fde628968fba10fccfe5b09675aa57cf725f507be83c10dd3c0b6c109b5d72eb",
"engine_timestamp_n": 136,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": null,
"status": "completed",
"trial_id": "trial-0008"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"enable-chunked-prefill": true,
"gpu-memory-utilization": 0.95,
"max-num-batched-tokens": 28672,
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 339.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.18833333333333332,
"engine_log_sha256": "fc60765578d57af08e0310983ff42e2a200739a90f11c45f9d4b1f6b7b789cc8",
"engine_timestamp_n": 136,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": null,
"status": "completed",
"trial_id": "trial-0009"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"enable-chunked-prefill": true,
"enable-prefix-caching": true,
"gpu-memory-utilization": 0.95,
"max-num-batched-tokens": 24576,
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 279.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.155,
"engine_log_sha256": "22c473b39662df6c8c4c64a616346652ab889d63256efab18257b1f4445ce13b",
"engine_timestamp_n": 133,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": null,
"status": "completed",
"trial_id": "trial-0010"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"enable-chunked-prefill": true,
"gpu-memory-utilization": 0.95,
"max-num-batched-tokens": 25600,
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 339.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.18833333333333332,
"engine_log_sha256": "19b19de4a0a11a5a112b3e8ef04c3ca4d90d6c6c9097b5180c024d8570c9a608",
"engine_timestamp_n": 136,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": null,
"status": "completed",
"trial_id": "trial-0011"
},
{
"config_patch": {
"env_patch": {},
"flag_patch": {
"enable-chunked-prefill": true,
"gpu-memory-utilization": 0.95,
"max-num-batched-tokens": 25088,
"tensor-parallel-size": 2
}
},
"duration_note": "",
"duration_s": 341.0,
"duration_source": "engine_log_span",
"engine_h20_hours_lower_bound": 0.18944444444444444,
"engine_log_sha256": "40f7307d6677f2bb55c0132d42fc82fce78ffa6a558d5c69b01ec60d1e40b3ae",
"engine_timestamp_n": 136,
"engine_timestamps_monotonic": true,
"failure_stage": "",
"parallel_size": 2,
"score_req_s_per_gpu": null,
"status": "completed",
"trial_id": "trial-0012"
}
]
}
},
"simulator": {
"decision_bearing_throughput_proxy_plus_real_final": {
"nominal_k_1": {
"candidate_cells": [
"tp1_mns64"
],
"engine_h20_hours_lower_bound": 0.13527777777777777,
"real_evaluations": 1,
"real_regret": 0.30456852791878175,
"selected_cell": "tp1_mns64",
"selected_real_score_req_s_per_gpu": 2.283333333333333
},
"nominal_k_2": {
"candidate_cells": [
"tp1_mns64",
"tp1_mns32"
],
"engine_h20_hours_lower_bound": 0.26722222222222225,
"real_evaluations": 2,
"real_regret": 0.30456852791878175,
"selected_cell": "tp1_mns64",
"selected_real_score_req_s_per_gpu": 2.283333333333333
},
"nominal_k_3": {
"candidate_cells": [
"tp1_mns64",
"tp1_mns32",
"tp2_mns32",
"tp2_mns64"
],
"engine_h20_hours_lower_bound": 0.7827777777777778,
"real_evaluations": 4,
"real_regret": 0.0,
"selected_cell": "tp2_mns32",
"selected_real_score_req_s_per_gpu": 3.283333333333333
}
},
"decision_bearing_throughput_proxy_sim_only_top1": {
"gpu_hours": 0.0,
"real_regret": 0.30456852791878175,
"selected_cell": "tp1_mns64",
"selected_real_score_req_s_per_gpu": 2.283333333333333
},
"marginal_gpu_hours_without_real_verification": 0.0,
"metrics_sha256": "55edb37d5692e979ab6f6dc6c65913a9db0aa0a836c350e4c05d9c38eee78206",
"observed_fidelity_suite_cpu_hours": 2.055026211017717,
"observed_fidelity_suite_runs": 184,
"one_time_profile_cost_status": "not recorded; total cold-start cost is unknown",
"one_time_profile_gpu_hours": null,
"posthoc_slo_gated_plus_real_final": {
"candidate_cells": [
"tp2_mns32",
"tp2_mns64"
],
"engine_h20_hours_lower_bound": 0.5155555555555555,
"false_feasible": 21,
"false_infeasible": 7,
"real_evaluations": 2,
"real_regret": 0.0,
"selected_cell": "tp2_mns32",
"selected_real_score_req_s_per_gpu": 3.283333333333333,
"status": "diagnostic/post-hoc, not a preregistered prospective policy"
}
}
}

57
runs/tuning-cost/test_analysis.py Normal file
View File

@@ -0,0 +1,57 @@
#!/usr/bin/env python3
from __future__ import annotations
import importlib.util
import math
import sys
from pathlib import Path
HERE = Path(__file__).resolve().parent
def load_analysis():
spec = importlib.util.spec_from_file_location("tuning_cost", HERE / "analyze.py")
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
def main() -> None:
analysis = load_analysis()
duration, monotonic, count = analysis.timestamp_span(
"INFO 07-01 10:00:00 x\nINFO 07-01 10:05:30 y\n", 2026
)
assert duration == 330.0
assert monotonic
assert count == 2
trials = [
{"trial_id": "t1", "engine_h20_hours_lower_bound": 0.1, "score_req_s_per_gpu": 8.0},
{"trial_id": "t2", "engine_h20_hours_lower_bound": 0.2, "score_req_s_per_gpu": None},
{"trial_id": "t3", "engine_h20_hours_lower_bound": 0.3, "score_req_s_per_gpu": 9.6},
]
curve = analysis.sequential_curve(trials, 10.0, [0.05, 0.04])
assert curve["cost_to_threshold"]["regret_le_0.05"]["trial_id"] == "t3"
assert curve["cost_to_threshold"]["regret_le_0.04"]["trial_id"] == "t3"
assert math.isclose(curve["total_engine_h20_hours_lower_bound"], 0.6)
candidates = analysis.tie_expanded_candidates(
{"a": 3.0, "b": 2.0, "c": 2.0, "d": 1.0}, 2
)
assert candidates == ["a", "b", "c"]
policy = analysis.real_final_policy(
candidates,
{"a": 1.0, "b": 4.0, "c": 2.0, "d": 3.0},
{"a": 0.1, "b": 0.2, "c": 0.3, "d": 0.4},
)
assert policy["selected_cell"] == "b"
assert policy["real_regret"] == 0.0
assert math.isclose(policy["engine_h20_hours_lower_bound"], 0.6)
print("tuning cost analysis: PASS")
if __name__ == "__main__":
main()