Prepare Frontier code trace fidelity campaign

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.research/experiments/frontier-split-rootcause-s0-20260720.md

@@ -0,0 +1,60 @@
+# 实验 S0：good/bad case 分裂的统一分解（margin vs differential residual）
+
+> **状态：** 已完成（2026-07-20；含 S0b 方向化修正与一轮 strict review 修复）
+>
+> 用户指令：核心要务是分析为什么部分 case 下 Frontier work、部分不 work 的 system 根因；本 card 是该诊断 campaign 的第一个 slice，仅使用 frozen artifacts，零 GPU 成本。人工 review 由用户的直接指令（"只有做好这个分析我们才能推进下一步"）满足。
+
+## Claim 与决策
+
+- **Parent claim：** ongoing.md H2——误差机制是 action-conditioned residual；本实验把它细化为"分裂从哪来"。
+- **现象（已冻结）：** 同一 best-effort Frontier 栈上，Q30/Q235 的 Trace-PD 与多数 PO 面 selection 近优，而 Fixed-PD 面 14–58% regret；失败 objective 随负载档切换（Q30 低压 TPOT/E2E 全反、高压 TTFT 56–58%）；A1 collective profile 修复了 Q235 Trace-PO p90 21.2%→0.3% 但对 Fixed-PD 33% 完全无效（本 card frozen-inputs/q235-ablation-a1）。
+- **Competing hypotheses：**
+  - **H-SCALE：** 分裂完全由「config-differential residual vs 真机 decision margin」的关系解释：good case 的 sim/real 比值跨 config 近似均匀（乘性偏移，argmin 不变），bad case 的比值跨 config 分散且超过 margin。workload shape 本身不需要出现在解释里。
+  - **H-THRESH：** 绝对 service-time 高估近似均匀，但与离散机制（MNS admission cap、MoE token-bucket、graph bucket）交互后被转换为 config-differential 误差；fixed uniform workload 把所有请求同步到同一 state 轨迹，使阈值交叉对整个 cell 相干生效；trace 的长度/到达异质性把阈值效应摊平。
+  - **H-STATE：** 失败由 simulator 闭环 batch state 分布漂移主导（Q235：sim decode batch 13.5 vs real 3.9 + B4→B5 profile cliff 正反馈）；即使打破 workload 同步性，闭环漂移仍可翻转排序。
+  - 三者关系：H-SCALE 是现象层（必要条件），H-THRESH/H-STATE 是 differential residual 的两种产生机制，可共存但可判别（见事前预测）。
+- **事前预测：**
+  - H-SCALE 成立 ⟺ 对每个 case×objective，failure 恰好发生在「top 邻域 log-ratio spread > log1p(真机相对 margin)」处（两侧同为 log-space 尺度），无反例。
+  - H-THRESH 独有：bad case 的 differential 误差集中于阈值语义分量（first-scheduling wait、bucket 跳变段），且 sim-only 反事实（去阈值/加 jitter）恢复排序——Q30 高压 TTFT 的 admission 反事实已支持一例。
+  - H-STATE 独有：差异化误差在去掉阈值分量后仍在 execution 项内（Q235 Fixed-PD 的 own-composition −20.07 vs exact-state +10.90 已支持一例）。
+- **判定规则：** S0 只裁决 H-SCALE 与「分量定位」（queue vs execution）；H-THRESH/H-STATE 的干预判别属 S1+（sim-only 反事实）与 GPU 实验（需另行 review）。若 H-SCALE 出现反例（good case 有 spread>margin 仍选对，或 bad case spread<margin），必须原样报告，不得平滑。
+
+## Setup
+
+- **输入（全部 frozen，runs/frontier-split-rootcause-v0/frozen-inputs/）：** q30-trace-pd（graph-piecewise comparison）、q30-fixed-hi（fixed-pd/fixed-po 高压面）、q30-expansion-lo（低压 fixed-pd/fixed-po/trace-po）、q235-fourcase-a0、q235-ablation-a1（+provenance）、q235-state-diag、q30-admission-diag。来源 cpfs 路径与 SHA 见各目录内 manifest/launch 记录。
+- **计算（每 case×objective）：** per-config 比值 r_c=sim_c/real_c；config-uniform scale=geomean(r_c)；differential residual=log-ratio spread（全 surface 与真机 top-3 邻域各一）；真机 relative margin（best 与 2nd-best、best 与 sim-winner 的真机值差）；failure flag=regret>5%；H-SCALE 检验=failure ⟺ 邻域 spread>log1p(margin)（review 修正：初版直接以 ln 差比较普通 relative margin，尺度不一致；修正后 70 行 verdict 不变）。
+- **分量定位：** q30-admission-diag 提供 TTFT=first-scheduling wait+prefill execution 分解；q235-state-diag 提供 own-composition vs exact-state contrast；把这些已知分量证据合并进统一表。
+- **交叉核对：** 重算的 regret 必须与各 frozen comparison.md 表一致（抽查 58.0%、33.0%、0.0%）；A0 vs A1 的 Q235 对比必须复现 trace-po p90 21.2%→0.3%、fixed-pd 四项不变。
+
+## 预期产物与 review
+
+- runs/frontier-split-rootcause-v0/analyze_split_decomposition.py（只读 frozen-inputs，确定性输出）
+- runs/frontier-split-rootcause-v0/results/decomposition.{json,md}：统一表，每行 case×objective，列出 winner、regret、scale、spread（全/邻域）、margin、H-SCALE verdict、已知分量归因
+- runs/frontier-split-rootcause-v0/results/margin-vs-residual.png：x=真机 margin，y=邻域 differential residual，点色=selection 对错；H-SCALE 成立则对错点被对角线分离
+- 人工验收：编排者亲自重跑脚本、抽查交叉核对数字、亲自查看渲染图
+
+## 复现信息
+
+- **Code：** AITuner branch feature/sim，自 HEAD 18c0b25 起；worker/reviewer job-id 见下方「Review 与 provenance 补记」；脚本与产物随本 card 同一 commit 入库（含 frozen-inputs 本地拷贝）。
+- **Environment：** 本地 workstation，CPU-only，python3+matplotlib；不访问远端。
+- **已知 deviation：** frozen-inputs 是 cpfs 原件的本地拷贝（scp，2026-07-20）；q30-expansion-lo 的低压 Fixed-PD 面已被高压面取代为 primary，本分析将两档并列为独立观测，不混合。
+
+## 结果
+
+- **观察事实：**
+  - 70 行（14 个 case surface）全部算出，无数据缺口；四组硬性交叉核对通过；连续运行产物 SHA 一致。
+  - **H-SCALE 判为必要非充分**：23 个 material failure（regret>5%）全部满足「top-3 邻域 log spread > log1p(margin)」，无一例失败发生在 residual 小于 margin 处；但另有 40/70 行同样满足该条件却均非 material failure（其中仅 16 行 exact winner match，其余 24 行是小 regret 的 winner 错位——14 个 MNS 精确 tie 与 10 个 strict reversal）——无方向 spread 不携带决策信息。
+  - **S0b 方向化后的机制普查**：23 个 material failure 的 winner-deciding pair 分布为 tp-axis 11、mixed 10、mns-axis 2（Q235 A0/A1 Fixed-PD 的 8 个 TPOT/E2E failure 全为 tp-axis；Q30 Fixed-PD 高低压为 tp/mixed）；仅有的 2 个 mns-axis failure 是 Q235 A0/A1 Trace-PD E2E p90（regret 6.2%，勉强越过 5% 门槛）。trace 面严格反序中 tp-axis 为 0（q30 trace-pd 三轴全 0）。
+  - **A1 对照的轴分解**：serving-matched collective profile 把 Q235 两个 PO 面的 tp-axis 反序从 4/4 清零（trace-po p90 regret 21.2%→0.3%），但 Fixed-PD 仅 5→4、四项 regret 一位小数不动——prefill 路径的 TP-differential 误差源=collective profile（可修），decode 耦合的 TP-differential 误差另有来源。
+  - **MNS 不敏感缺陷**：14 个 winner-label mismatch 是 simulator 逐位相等的 tie，全部 mns-axis（如 q30 fixed-po 的 MNS16↔32、q235 fixed-pd 的 MNS64↔128）；tie 计入后 MNS 边界误差 31 与 TP 严格反序 32 相当，但 MNS 侧 regret 小。
+  - **成功的鲁棒性**：23 个 exact-winner success 中 17 个 margin-robust（margin≥1%），6 个 fragile（含 q30 trace-pd E2E p90 的 0.1% margin 与 q235 A1 fixed-po 四项）。
+  - **面级 scale 对照**：prefill-only 面 geomean scale 0.96–1.37×（绝对预测基本准确），含 decode 的面 4.3–130×——绝对误差灾难集中于 decode。
+- **异常：** 无数据异常。strict review（FAIL：3 Major/1 Minor）指出 H-SCALE 尺度混用（log spread vs relative margin）、tie 轴普查缺失、card 状态过期、Q235 一致性表述过强；全部修复，修复后 70 行 verdict 逐行不变。
+- **含义：** 分裂的现象层解释是「margin 保护 + config-differential 误差」。机制层上，21/23 个 material failure 由 TP/mixed pair 决定，且所有大 regret（≥13%）failure 都发生在含 decode 的面上：其中 Q235 Fixed-PD 有 state-drift 直接证据、Q30 高压 TTFT 有 admission 反事实证据，而 **Q30 低压 TPOT/E2E 反转的机制尚未诊断**（S1 目标）。「decode 耦合的 TP-differential 误差是主要载体」是当前最强归纳，不是对全部 failure 的已证机制归因；2 个 mns-axis 边缘 failure（6.2%）在该归纳之外。trace 面成功伴随「TP 反序为零 + TP margin 宽」，但「误差小」与「margin 宽」谁是主因仍未判——这正是 H-THRESH vs H-STATE 的判别缺口。轴标签与机制不一一对应（Q30 admission 是 MNS 阈值机制但 deciding pair 为 tp/mixed，因 TP 改变到达压力）。
+- **Claim update：** H2（action-conditioned residual）supported 且被细化：residual 的决策相关分量集中在 TP 轴、由 decode 状态耦合产生；H-SCALE 降级为必要条件；H-THRESH/H-STATE 保持 competing，待 S1 判别。
+- **下一步：** S1（sim-only 反事实：Q30 低压 Fixed-PD TPOT 反转的分量定位——这是唯一无机制解释的 material failure；fixed workload jitter 判别 H-THRESH vs H-STATE）；GPU 判别实验（加压 Trace-PD、jittered Fixed-PD 真机面，dash1–4）另行出 card 供 review。
+
+## Review 与 provenance 补记
+
+- S0 worker：codex `task-mrsn1s9c-z6ha0w`；S0b：`task-mrsnjzn6-k18x4n`（resume）；strict reviewer（fresh 只读）：`task-mrsocmlb-386btc`（verdict FAIL）；修复轮：`task-mrsop06n-pwxo0b`（fresh writable）。编排者独立验收：脚本重跑、SHA 比对、两图目视检查。
+- 产物 SHA（修复后）：decomposition.json `81ea56b2…`、decomposition.md `4d19af54…`、margin-vs-residual.png `df2110c4…`、decision-pair-axis.png `2e787301…`。

.research/experiments/frontier-workload-regime-taxonomy-20260720.md

@@ -0,0 +1,183 @@
+# Frontier workload-regime taxonomy
+
+- Date: 2026-07-20
+- Status: proposed; awaiting review before workload generation or GPU runs
+- Scope: explain when Frontier preserves the real-system config ranking, rather than merely comparing Fixed with Trace
+
+## Claim under test
+
+Frontier reliability is controlled by three quantities:
+
+1. the latency-model residual between simulator and real execution;
+2. the closed-loop gain from timing to scheduler state (batch, MoE routing, CUDA-graph bucket, MNS occupancy, admission/KV pressure);
+3. the real decision margin between configurations.
+
+For a config pair `a,b`, define
+
+```text
+D_real(a,b) = log L_real(a) - log L_real(b)
+delta(a,b)  = [log L_sim(a)-log L_real(a)]
+              - [log L_sim(b)-log L_real(b)]
+slack(a,b)  = sign(D_real) * [D_real + delta]
+```
+
+`slack < 0` means the simulator reverses the real pairwise ordering. The primary hypothesis is that reversals occur when simulator and real execution land on different sides of a scheduler-state knee, or when the real decision margin is too small to absorb the differential residual. `Fixed` and `Trace` are not themselves the causal classes.
+
+## Existing evidence motivating the experiment
+
+- Q30 Trace-PD preserves all six objective winners, but many pairwise residuals oppose the real winner. Its success is therefore often margin protection, not zero residual.
+- Q235 Trace-PD preserves TTFT/TPOT winners but misses E2E p90 by 6.2%; Trace is not universally safe.
+- Q30/Q235 Fixed-PD decode objectives show negative minimum signed slack and 13--37% regret.
+- In Q30 low-load Fixed-PD, Frontier's batch-1 TP ordering is correct, while the closed-loop simulator increases TP4's effective batch and changes the MoE cost enough to reverse the ordering. This identifies a concrete state knee, but does not yet establish a general rule.
+
+## Workload families
+
+All comparisons use the same request multiset where applicable, the same total observation window, and the same normalized offered decode load
+
+```text
+rho = request_rate * E[output_tokens] / measured_reference_decode_capacity.
+```
+
+This avoids equating equal request rates with equal load.
+
+| ID | Shape / request lengths | Arrival process | Prefix/session state | Isolated effect |
+|---|---|---|---|---|
+| W0 | short fixed `2048 -> 128` | uniform | off | known low-residence failure anchor |
+| W1 | trace-mean fixed ISL/OSL | uniform | off | homogeneous baseline |
+| W2 | trace-mean fixed ISL/OSL | trace timestamps | off | arrival burst only |
+| W3 | exact trace ISL/OSL multiset | uniform | off | length heterogeneity only |
+| W4 | exact trace ISL/OSL multiset | trace timestamps | off | length + burst |
+| W5 | exact trace prompts/ISL/OSL | uniform | exact prefix/session identity | prefix state without burst |
+| W6 | exact trace prompts/ISL/OSL | trace timestamps | exact prefix/session identity | full production trace |
+
+Prefix is intentionally a nested factor: enabling a synthetic prefix graph on fixed identical requests would introduce a different workload rather than isolate production prefix reuse. Therefore this is not presented as a full `2^3` factorial.
+
+## Load sweep and expected patterns
+
+Simulator discovery sweep: `rho in {0.05, 0.25, 0.50, 0.90, 1.20}`. The points mean deep low load, light batching, moderate batching, capacity knee, and overload; their request rates are derived independently for every workload family.
+
+| Pattern | Observable state | Prediction for Frontier |
+|---|---|---|
+| P1 singleton-linear | real and sim stay below the first batch/graph knee | works if the batch-1 operator ordering is correct |
+| P2 knee-straddling | real and sim occupy opposite sides of a batch/MoE/graph/MNS knee | fails systematically; Fixed-PD is the current example |
+| P3 same-side batched | both systems cross the same knee and remain below admission pressure | works if batch-conditioned operator ordering is correct |
+| P4 capacity/admission aligned | both systems are governed by the same capacity bottleneck | TTFT/config winner may work despite large absolute error; E2E/MNS can remain fragile |
+| P5 heterogeneity-smoothed | broad lengths reduce coherent threshold occupancy at matched `rho` | may work; this is a hypothesis, not an established explanation |
+| P6 burst-sensitive | same request multiset, but transient queue/MNS occupancy differs | mean ranking may work while TTFT/E2E tail ranking fails |
+| P7 prefix-state-sensitive | hit/eviction and reused-token distributions differ | TTFT ranking fails unless prefix-state transitions are modeled; decode TPOT may remain stable |
+| P8 decision-boundary | real config margin is comparable to run variance/residual | fragile; an exact winner match is not reliable evidence |
+
+## Hypotheses and distinguishing tests
+
+### H1: state-regime hypothesis (primary)
+
+I believe config-ranking failures occur when the latency residual moves a workload across a scheduler-state knee, because the residual is then amplified into a different batch/resource trajectory. I will verify this by checking whether signed-slack zero crossings co-locate with measured real/simulator state-knee crossings.
+
+### H2: heterogeneity-smoothing hypothesis
+
+I believe length heterogeneity can reduce coherent threshold amplification, because requests reach scheduler boundaries at dispersed times. I will verify it with W1 vs W3 and W2 vs W4 at matched `rho`, requiring a smaller real/sim state-distribution gap rather than merely a correct winner.
+
+### H3: bottleneck/margin-protection alternative
+
+Trace success may instead be explained entirely by a large real decision margin or a shared capacity bottleneck. This hypothesis wins over H2 if W3/W4 do not reduce state-distribution error after matching load and margin, while ranking correctness remains predicted by margin alone.
+
+### H4: burst and prefix are independent failure channels
+
+I believe arrival bursts primarily affect waiting/admission and tail TTFT/E2E, whereas prefix mismatch primarily affects prefill/TTFT state. I will verify this with W1/W2, W3/W4, and W3/W5 paired comparisons.
+
+## Configuration and model scope
+
+Discovery uses Qwen30B because its 12-cell `TP x MNS` surface already has simulator and real anchors:
+
+- TP: `{1, 2, 4}`
+- MNS: `{8, 16, 32, 64}`
+- objectives: mean/p90 TTFT, TPOT, E2E
+
+Qwen235B is a held-out confirmation, not pooled into discovery:
+
+- existing four feasible TP/MNS configurations;
+- only the workload/load patterns that discriminate H1--H4 after Q30 converges.
+
+## Measurements
+
+End-to-end:
+
+- completed/failed requests and achieved request/token rate;
+- TTFT, TPOT, E2E mean/p50/p90/p95;
+- config regret, pairwise agreement, signed decision slack;
+- run-to-run winner stability.
+
+Closed-loop state:
+
+- prefill/decode batch-size histograms and time-weighted batch;
+- Running/Waiting distributions and admission delay;
+- MNS active-token occupancy and KV/context pressure;
+- CUDA-graph bucket residency and fallback frequency;
+- prefix hit/reused-token/eviction distributions for W5/W6.
+
+## Decision rules
+
+A workload/load region is:
+
+- **reliable** if regret is at most 5%, pairwise agreement is at least 0.8 at two adjacent load points, and the winner is stable across confirmation trials;
+- **fragile** if regret is at most 5% but the real margin overlaps run uncertainty, or a small rate/timing perturbation changes the winner;
+- **failed** if regret exceeds 5% or a decision-critical pair has negative signed slack;
+- **mechanistically explained by H1** only if the ranking transition co-locates with an observed state-regime transition. Correlation with the Fixed/Trace label is insufficient.
+
+H2 is supported only if the heterogeneous member of a matched pair reduces state-distribution error and shifts the failure boundary in repeated trials. A correct winner alone does not support smoothing.
+
+## Execution plan after review
+
+1. Materialize W0--W6 with one manifest recording request multiset, arrival timestamps, prefix identity, rate contract, and hashes.
+2. Run the simulator sweep across `rho` and the Q30 surface; emit a per-stage state ledger.
+3. Select real-machine pilot points only around the predicted knees plus one safe-side control. Use guard configs `TP1/MNS64`, `TP4/MNS8`, and `TP4/MNS64`; add `TP2/MNS32` only if the transition is not bracketed.
+4. Use only `dash1`, `dash2`, `dash3`, and `dash4`, each verified as an 8×H20 host. `dash0` is excluded from probing, synchronization, and execution. Pin one independent experiment group to each host so at most four groups run in parallel; do not split one trial across hosts.
+5. Run one pilot trial per selected point. Confirm only hypothesis-discriminating points with three fresh-server trials and rotated order.
+6. Apply the resulting classifier unchanged to the Q235 held-out cases.
+
+Provisional four-way allocation after the simulator identifies the discriminating points:
+
+| Host | Experiment group | Primary contrast |
+|---|---|---|
+| dash1 | homogeneous controls | W0/W1 across safe side and first knee |
+| dash2 | arrival effect | W1 vs W2 and W3 vs W4 |
+| dash3 | length heterogeneity | W1 vs W3 and W2 vs W4 |
+| dash4 | prefix/full trace | W3 vs W5 and W4 vs W6 |
+
+The groups are logical queues, not permanent ownership: if a host probe fails, that host is excluded and its group waits or moves to another permitted idle host. Cross-host latency values are not pooled until a common canary config verifies that host effects are within run uncertainty.
+
+No GPU run is authorized by this card yet. The review decision is whether the workload decomposition and decision rules are sufficient to implement the materializer and launch Phase 1.
+
+## Expected figure
+
+The accompanying mock figure is schematic, not data. Panel A shows the state knee that real and simulator trajectories may cross at different loads. Panel B shows the corresponding minimum signed decision slack; a negative value denotes a ranking reversal. The claim is supported only if measured zero crossings and state knees align across workload families.
+
+## Risks and controls
+
+- Equal `rho` does not guarantee equal prefill pressure; report both prefill and decode offered work and stratify if necessary.
+- Full-trace overload can collapse all configs to similarly poor latency. Such points identify a capacity-limited region but cannot validate fine-grained ranking.
+- MNS ties and censored/failed requests can create false winners; exclude invalid cells before calculating regret and report the exclusion.
+- One trace cannot establish generality. The initial result is a mechanism boundary for this trace/model/hardware, followed by held-out Q235 validation.
+
+## Execution log
+
+### 2026-07-20: materialization and simulator launch
+
+- Code baseline: `feature/sim@157bf36` for the valid v4 sweep.
+- Hosts probed: `dash1`, `dash2`, `dash3`, `dash4`; each exposed 8 NVIDIA H20 GPUs with 0 MiB used at probe time. `dash0` was not probed or used.
+- Source cohort: 129 Q30 Trace-PD requests. The private artifact supplies exact prompts, lengths, outputs, timestamps, sessions, and runtime block identities; the simulator projection retains only the first `floor(ISL/16)` complete block identities.
+- Materialized: 35 cases = W0--W6 × `rho {0.05,0.25,0.50,0.90,1.20}`. Audit passed request count, exact decode offered load, empirical arrival rate, prefix block count, and prefix-off empty identity vectors.
+- Simulator smoke: W0 / `rho=0.05` / TP4-MNS64 completed 129/129. Simulator TTFT mean/p90 was 109.81/124.16 ms and TPOT mean/p90 was 36.26/36.79 ms. This is a harness check, not real-system fidelity evidence.
+- Invalid attempts retained for audit: v1 had a Bash argument-expansion error; v2 mixed multiple workload families into a runner that requires strictly increasing anchors from one family; v3 exposed a scikit-learn cache-version mismatch. None is used as scientific evidence.
+- Valid v4 controls: isolated output/predictor cache per TP/prefix group; scikit-learn 1.9.0 matching the predictor cache format; per-family five-point runner invocations; stage batch ledger enabled; TP1 exempted from the collective fallback gate because a single rank has no all-reduce.
+- Active v4 allocation: dash1=TP1 prefix off/on, dash2=TP2 prefix off/on, dash3=TP4 prefix off, dash4=TP4 prefix on. The four fleet jobs are running from fresh `sim-v4` output roots. First-process audit found the explicit isolated `--metrics_config_cache_dir` on all hosts and zero cross-version warnings.
+- First valid v4 tranche: 16/16 observed cells completed, each with 129 requests, request metrics, and a stage-batch ledger; no traceback, fallback, or version warning was found. The tranche covers all five W0 load points at TP1/TP2/TP4-MNS8 plus the first W5 prefix points at TP4-MNS8.
+- Early load-boundary observation: W0 at `rho=0.05` is low-latency for TP4-MNS8 (simulator TTFT mean 109.25 ms) but already queues for TP1-MNS8 (25.70 s); at `rho=0.25`, even TP4-MNS8 reaches 27.13 s mean TTFT. Because `rho` normalizes decode tokens only, high-rate short-output W0 also raises prefill and active-sequence pressure. These points map the overload boundary and are not eligible as reasonable-latency real pilots.
+- Real-runtime gate: a stock vLLM 0.20.0 environment passed import/H20 checks but used CUDA 13.0, so it is excluded from comparison with the historical CUDA 12.9 baseline. The replacement environment `vllm-0.20.0-cu129-workload-regime-v2` passes `vllm CLI=0.20.0+cu129`, torch `2.11.0+cu129`, CUDA runtime 12.9, H20 visibility, and all 179 package dependency checks. The first CPFS install used file copies and was stopped after download because it was still copying roughly 7 GB after 12 minutes; its incomplete directory is retained with an `invalid-copy-incomplete` suffix, while v2 uses same-filesystem hardlinks from the validated cache.
+- Load-contract correction: the original Fixed-PD surface held request rate per GPU constant, so global arrival rate scaled with TP. The v4 sweep holds global arrival rate constant and is retained as the control that isolates service-topology changes. A matched per-GPU sweep is now required to reproduce the original closed-loop intervention: TP1/TP2/TP4 receive `1x/2x/4x` global arrival rate at the same per-GPU `rho`.
+- Per-GPU low-load materialization: 105 cases = W0--W6 × `rho {0.0025,0.005,0.01,0.02,0.05}` × TP `{1,2,4}` were generated under `traces-per-gpu-low`. Audit passed 105 unique paths, 129 public/private rows per case, digests, arrival alignment, and exact `global_rate / TP = per_gpu_rate`. W0 `rho=0.01` is 0.239375 req/s/GPU, bracketing the original 0.215 req/s/GPU Fixed-PD point with `rho=0.005`.
+- The per-GPU sweep writes to a separate `sim-per-gpu-v1` result root but reuses the completed v4 predictor cache for the same TP/prefix/config. Predictor cache provenance is explicit in every surface manifest; workload results and state ledgers are never shared.
+- `wait_and_dispatch_per_gpu.sh` is active locally as a serial gate. It requires all four exact v4 run directories to contain `finished_at` and exit code zero before probing dash1--dash4 and dispatching the four per-GPU jobs; it does not launch a second sweep while v4 is still consuming CPU.
+- A first materialization attempt rounded both `rho=0.005` and `rho=0.01` to the same `rho0p01` directory. Digest validation stopped before simulator launch; the invalid directories were retained with an `invalid-rho-label-collision` suffix. The label function now preserves up to 12 significant digits and has a regression test.
+
+Current decision: finish the v4 fixed-global-rate control, then reuse its trained predictors for the low-load per-GPU sweep before selecting discriminating real-machine pilot points. No real latency result from vLLM 0.20.2 will be compared with the historical vLLM 0.20.0 baseline until the runtime-version gate is resolved.

.research/ongoing.md

@@ -3,14 +3,20 @@
 > 2026-07-17｜写给未参与项目的读者，可直接作为 presentation 讲稿。历史过程与复现信息见 `../runs/*/` 各 experiment card、`../docs/` 各 campaign 文档。
 >
 > **2026-07-19 update：** Qwen235 Fixed-PD 的错误排序在 exact real state composition 下已经翻正，主因是 simulator closed-loop batch state，而不是 collective。Qwen30 Fixed-PD 的 56--58% TTFT regret 也已定位：Frontier 将 decode service time 高估 4--8×，使 TP4 的 modeled concurrency 越过 MNS admission cap并产生虚假排队；去掉该等待后 Frontier 与真机都判定 TP4 topology 更快。详见 [`experiments/qwen30-fixed-pd-ttft-admission-diagnosis-20260719.md`](experiments/qwen30-fixed-pd-ttft-admission-diagnosis-20260719.md)。
+>
+> **2026-07-20 update：** 对全部 14 个 frozen case surface（70 个 case×objective）做了统一的 margin-vs-residual 分解与方向化机制普查（[`experiments/frontier-split-rootcause-s0-20260720.md`](experiments/frontier-split-rootcause-s0-20260720.md)）。三个要点：(1) 「residual 超过 margin」是失败的必要条件但远非充分——good/bad 分裂不能用无方向误差量解释；(2) 23 个 material failure 的 winner-deciding pair 中 21 个落在 TP 轴或 mixed（其余 2 个是 6.2% regret 的边缘 mns-axis case），trace 面的 TP 反序为零；A1 measured collective 把 Qwen235 两个 prefill-only 面的 TP 反序清零（trace-PO p90 regret 21.2%→0.3%）却对 Fixed-PD 完全无效——prefill 路径的 TP 差异化误差源是 collective profile（可修），decode 耦合的 TP 差异化误差是当前所有 material failure 的载体；(3) 「Fixed-PD 失败因为高压」被否证：失败 Fixed-PD 的真机 in-flight（14.05）低于全对的 Trace-PD（38.69），且低压 Fixed-PD 同样失败、失败 objective 随负载切换。另有次要缺陷：14 个 winner 错位来自 simulator 对 MNS 逐位不敏感的精确 tie。
+>
+> **2026-07-20 root-cause update：** Q30 低压 Fixed-PD 的 exact stage ledger 关闭了最后一个未解释的 material failure。相同 batch=1 state 下 Frontier full predictor 给 TP4 `18.3515 ms/step`、TP1 `19.3506 ms/step`，方向正确；但 per-GPU 固定到达率使 cluster arrival 随 TP 增长，叠加 decode residence 高估后，TP4 在 simulator 内自激到 time-weighted batch `3.0437`（96.13% decode 时间 batch≥3），own-state step 变为 `28.1712 ms`。其中相对 batch=1 的 `+9.8197 ms` 有 `+8.9297 ms` 来自 batch-conditioned MoE，collective 仅 `+0.0121 ms`。因此 Fixed-PD 的根因不是“固定 workload”或“高压力”本身，而是 **execution-time residual 进入离散事件时钟后改变 future scheduler state；该 state 再通过 MoE/profile/graph 或 MNS admission 非线性放大，形成 action-dependent signed residual 并穿过 decision margin**。Q30 低压是平滑 state-feedback，Q30 高压是跨 MNS cap 的 threshold amplification，Q235 是 composition drift；三者为同一闭环机制族。
+>
+> **2026-07-20 load-audit update：** Trace-PD overload 不是 Fixed/Trace good-bad 分裂的统一解释。旧 Q30 Trace-PD decode offered/observed-peak throughput≈`1.00×`、peak Running/Waiting=`47/0`；降到 `0.10 req/s/GPU` 后 TTFT `245.95/685.51 → 228.14/835.38 ms`（mean/p90，不出现 tail collapse），TPOT `13.18/15.39 → 7.91/8.90 ms`。旧 Q235 则是 `3.44×` 明确过载、peak=`116/3`；降到 `0.035 req/s/GPU` 后 TTFT `1141.54/2616.69 → 478.14/1347.75 ms`，TPOT `61.89/78.62 → 24.00/28.49 ms`。旧 surface 仍有 `417×/32.6×` mean-TTFT spread，否定“所有配置一样差”。八 case baseline 与 claim boundary 见 [`experiments/frontier-eightcase-load-audit-20260720.md`](experiments/frontier-eightcase-load-audit-20260720.md)。
 
 ## 一眼看懂
 
 - **Topic / problem：** LLM serving 的自动、低成本配置调优（AITuner）。当前主线问题：用 simulator 给部署配置（并行度、批量上限等）排序，什么时候可信？需要补多少真机证据？算上这些成本还划算吗？
 - **Central claim：** simulator 要能帮助配置调优，必须先满足 scheduler transition 的 liveness/coverage，再满足「配置相关残差小于真机 decision margin」；前者决定 capacity 是否有定义，后者决定排序是否正确。(ID: C0)
 - **当前结论：** 早先 35 个 trace stall 不是 Frontier scheduler liveness failure：adapter 为不满 16-token 的 prefix block 错误生成了 cache identity，Frontier 又没有 fail-fast。修正为完整 block、使用真实 graph buckets/KV blocks 和 `piecewise`/`KERNEL_ONLY` profile 后，Qwen30 Trace-PD 的全部 12 个 cell 完成 129/129 request，Frontier 对 TTFT/TPOT/E2E 的 6 个 argmin 均与三次 fresh-server 真机一致；但绝对 latency 仍高估 4--511×。这只证明一个 MoE Trace-PD surface 的 selection fidelity，不能外推到 prefill-only、fixed workload 或 235B。
-- **最大 uncertainty / risk：** 这个正确选择是否依赖 Trace-PD 的 queue/margin 偶然性；Fixed-PD、Trace-P、Fixed-P 与大型 FP8 MoE 是否仍保留正确 config ranking。Q235 还缺少同栈 profile/runtime contract，不能复用旧 vLLM 0.10.2 数据。
-- **下一项 critical action：** 完成 Qwen30 其余三项 fixed/trace × PD/P surface。Fixed case 已先启动无请求 vLLM runtime-state preflight；Trace-P 的 graph-aligned Frontier 12-cell CPU surface 正在运行。Q235 先做 vLLM 0.20 TP4/TP8 compatibility gate，随后才允许新 profile。
+- **最大 uncertainty / risk：** 根因已收敛，且 overload 已被排除为统一解释，但可信域边界仍未画清：trace 面的 heterogeneity 是否让 closed-loop state residual 变小，还是当前 success 主要由 capacity/MNS margin 保护？两个降载点只建立 reference-config latency baseline，不能证明新负载下全 surface 仍选对。
+- **下一项 critical action：** 不再做无锚点的 jitter 猜测；保持 request shape 不变，在预测的 MoE/MNS knee 两侧做小规模 rate sweep，并用少量真机 state/batch anchor 验证 `λR(B)` fixed point。成功标准是同时预测 state-regime、排名与 knee，而不只是某个点的 regret。
 - **停止条件：** T1 出 verdict 且成本账本建立后：pass 且摊销论证成立 → 转向「sim 剪枝 + 真机终选」的 hybrid 机制设计；fail → 转入失败机制归因；两条路都无 insight 增量 → 收敛写作。
 
 ## 核心概念
@@ -34,9 +40,10 @@
       - **Supporting：** 235B prefill-only regret=0；235B fixed-shape mixed 的 top set 全中；30B 加 per-TP 校准后 regret 0.76%（但这是外部端到端 scale 给出的上界，不是原生 profile 保真度）。
       - **Counterevidence：** 修正 prefix trace contract 后的 TP2/MNS16 `none`-graph run 完成但 p50 TPOT 约 96 ms，真机为约 14 ms；然而该比较尚未对齐 real vLLM 的 `FULL_AND_PIECEWISE` graph path。
       - **下一项 discriminative experiment：** 补齐 `KERNEL_ONLY` graph family并以 `piecewise` 重跑相同 trace；若 full surface 仍错，graph omission 不再是可用解释。
-    - **Hypothesis（机制，active）：** 误差机制是 action-conditioned residual——执行状态的转移（并行拓扑、kernel family、graph mode、batch 组成）使按算子 profile 的组合预测跨配置不可复合；残差大于 margin 时排序失败。(ID: H2；supported)
+    - **Hypothesis（机制，active）：** 误差机制是 action-conditioned residual——执行状态的转移（并行拓扑、kernel family、graph mode、batch 组成）使按算子 profile 的组合预测跨配置不可复合；残差大于 margin 时排序失败。(ID: H2；supported，已细化)
       - **Supporting：** 三个 TP 档的端到端校准系数为 0.72/0.47/0.35，残差确实随配置剧烈变化；235B 的批量上限交互预测错误但被 2× margin 容忍；30B prefill-only 在低负载近似对齐、饱和后按 TP 反向放大，最终 τ-b=−1。
-      - **下一步：** 按 collective 通信 → batch 组成 → 调度器逐步轨迹的单变量顺序定位组合性失效点（对应 experiment card 中 A1/A2/A3 消融阶梯）。
+      - **细化（2026-07-20 统一普查）：** 决策相关的残差分量集中在 TP 轴且由 decode 状态耦合产生——prefill-only 面的绝对 scale 仅 0.96–1.37× 且 measured collective 即可清除其 TP 反序，而含 decode 的面 scale 4.3–130×、全部 material failure 都由 TP/mixed pair 决定。「residual>margin」只是必要条件；失败还需要残差对准 winner-deciding pair。
+      - **机制 verdict（2026-07-20）：** closed-loop state drift 是根因，离散阈值是其放大器而非 competing explanation。Q30 低压 exact ledger 显示同 state 的 TP 方向正确，但 TP4 被模拟 residence 反馈推到 batch 3--4，MoE step 增长后反序；Q30 高压进一步跨过 MNS admission cap；Q235 换成 exact real composition 后排序翻正。下一步从“找根因”转为测量 state-regime/knee 的可信边界。
   - **Subclaim：** 成本论证只有在摊销前提下成立。(ID: C3)
     - **Hypothesis（active）：** 每个 model×硬件×runtime 的一次性对齐成本，摊销到大配置面、频繁重调（引擎版本 churn 的频率证据见 claim map）或禁止在线实验的场景后，低于重复真机调优。(ID: H3；untested——分母已实测，分子未入账)
       - **下一步：** 建 cost ledger（见「下一步」）。
@@ -45,27 +52,27 @@
 
 - **Question：** 生产 trace 忠实回放（prefix 打开、原始到达时间与会话结构）下，best-effort Frontier 能否满足 low-regret gate？
 - **为什么现在做：** 这是 H1 的判决实验；所有已完成的机制分解都在人工 workload 上，不能替代这个 verdict。
-- **当前状态：** Trace-PD 的 graph-aligned surface 已通过 selection gate，但绝对 latency 不通过 calibration。现在以不共享结果的 Fixed-PD、Trace-P、Fixed-P surface 检验它的泛化边界。
+- **当前状态：** Trace-PD 的 graph-aligned surface 已通过原负载 selection gate，但绝对 latency 不通过 calibration；Fixed-PD 的 failure 已定位为 closed-loop state drift。两个降载 Trace-PD anchor 已通过完成率/admission/backlog gate，下一步需要 full surface rate sweep 才能检验 ranking 是否跨 load regime 保持。
 - **Result → decision：** 若其它 surface 排序失败，保留 Trace-PD success 为条件化 envelope，并按 fixed/trace/prefill/decode 的差异定位 state composition；若都通过，才扩大到 Q235 或寻找 simulator 已解决范围之外的新问题。
 - **Experiment card：** [`../runs/frontier-fidelity-envelope-v1/experiment-card.md`](../runs/frontier-fidelity-envelope-v1/experiment-card.md)
 
 ## Key evidence（最多 3 条）
 
 - **E1（否证「prefill-only 是充分 easy condition」，支持 H2）：** 30B BF16、去掉 decode/prefix/混合 batch 后，真机最优是 TP4（8 vs 7 req/s/GPU），simulator 却把 TP4 排最差（6 vs 8）：top set 无交集，regret 12.5%，τ-b=−1。产物：`../runs/frontier-phase-factorial-v0/results/final/`（dash0，12.07 H20-GPUh）。
-- **E2（margin 可掩盖残差，支持 C0/H2）：** 235B FP8 的 prefill-only 与 fixed-shape mixed 都选对 top set，但 simulator 漏掉并发上限（max-num-seqs）×批 token 上限（max-num-batched-tokens）的交互棋盘格，并把 10/34 个真机 anchor 误判为不可行；真机头部 margin 有 2×，残差被容忍。产物：`../runs/frontier-multicase-sufficiency-v0/best_effort/fixed_cohort_evidence/`、`../runs/frontier-multicase-sufficiency-v1/results/t0-final/`。
-- **E3（selection 与 calibration 分开）：** 用完整 block projection、`piecewise` 和 graph-compatible KERNEL_ONLY profile 后，Qwen30 Trace-PD 的 12/12 sim cells 完成，6 个 mean/p90 latency objective 的 argmin 都与真机一致；但 sim/real latency ratio 仍为 4--511×。产物：[`../runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md`](../runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md)。
+- **E2（统一机制普查：material failure 全部由 decode 耦合的 TP 差异化误差决定，支持 H2 细化）：** 对 14 个 frozen surface、70 个 case×objective 的方向化分解显示：23 个 material failure 中 21 个由 TP/mixed pair 决定（仅 2 个 6.2% 边缘 mns-axis case）、trace 面 TP 反序为零；measured collective（A1）把 Qwen235 两个 prefill-only 面的 TP 反序清零（trace-PO p90 regret 21.2%→0.3%）但对 Fixed-PD 的 33% 无效；「residual>margin」仅为失败的必要条件。产物：[`../runs/frontier-split-rootcause-v0/results/`](../runs/frontier-split-rootcause-v0/results/decomposition.md)（实验 card：[`experiments/frontier-split-rootcause-s0-20260720.md`](experiments/frontier-split-rootcause-s0-20260720.md)）。
+- **E3（closed-loop state 是 Fixed-PD 根因，而非同 state predictor 反序）：** Q30 低压相同 batch=1 state 下 TP4 比 TP1 快约 1.00 ms/step，但 TP4 own state 的 time-weighted batch=3.0437，使 step 增加 9.8197 ms（其中 MoE +8.9297 ms）并反序；Q235 用 exact real composition 重放也把 TP8−TP4 从错向 −20.07 ms 翻为正确 +10.90 ms。Q30 高压再由 MNS cap 将同族 state/residence 误差放大成约 27 s 排队。产物：[`experiments/frontier-split-rootcause-s1-20260720.md`](experiments/frontier-split-rootcause-s1-20260720.md)。
 
 ## 下一步（最多 3 项）
 
-- [ ] **完成 Qwen30 remaining matrix（direct progress）：** Fixed-PD、Trace-P、Fixed-P 分别冻结 Frontier/真机 12-cell surface；P-only 的 TPOT 一律为 N/A。
+- [ ] **画可信域边界：** 固定 request shape，在预测的 MoE/MNS knee 两侧做最小 rate sweep；只在判别点补真机 batch/state anchor，验证 `B≈min(MNS, λR(B))` 是否同时解释 state 与 ranking。
 - [ ] **Q235 portability gate：** 先验证 vLLM0.20 TP4/TP8 FP8 runtime 和 deadc4a profile provenance，再决定是否允许其 Fixed-P sweep。
 - [ ] **建 cost ledger：** parent H3；完成标准 = 每 case 一行（profiling GPU-h、补丁工时、校准探测、sim CPU-h），与已实测的真机调优成本同表，随每个 case 更新。
 
 ## Blocker 或 anomaly
 
-- **下一启动已准备：** dash0 8×H20 当前空闲；graph-compatible attention/linear/MoE/router kernel-only profile 会先在一 GPU smoke，成功后以 3 张 GPU 并行完成 attention shards。完整 replay 仍为 CPU-only。
+- **当前运行状态：** 八 case load audit 的新增真机 run 已完成；未启动 full-surface rate sweep，避免把两个 single-config anchor 外推成 ranking claim。自 2026-07-20 起，本任务只允许使用 `dash1`--`dash4`（每台 8×H20、最多四组并行）；`dash0` 保留给其他同事，不做 probe、同步或运行。
 - **Anomaly（保留）：** 235B pilot 中 simulator 把 10/34 个 anchor 误判为不可行——false-infeasible 是 H1 的主要威胁模式，T1 分析时须单独报告。
-- **勘误（已固定）：** 统一平台为 dash0-only，早期 dash1 出处是文档错误；fixed-shape pilot 的主 SLO（TPOT 40ms）无判别力，150ms 是事后明示的敏感性分析，不得写成盲选的 primary。
+- **平台边界（已更新）：** 历史结果仍来自其各自 card 记录的平台，不改写 provenance；后续实验平台切换为 `dash1`--`dash4`。跨主机比较前必须跑相同 canary 并量化 host effect。fixed-shape pilot 的主 SLO（TPOT 40ms）无判别力，150ms 是事后明示的敏感性分析，不得写成盲选的 primary。
 
 ## Related work
 

AGENTS.md

@@ -1,10 +1,11 @@
 # Project Operating Notes
 
-## Remote experiment host
+## Remote experiment hosts
 
-- Default experiment machine: `dash0`.
-- Hardware expectation: 8 NVIDIA H20 GPUs.
-- SSH check: use `ssh dash0` before scheduling or debugging remote runs.
+- Experiment machines: `dash1`, `dash2`, `dash3`, and `dash4`.
+- Do not use or probe `dash0`; it is reserved for other users.
+- Hardware expectation: 8 NVIDIA H20 GPUs per host.
+- Before scheduling, probe only `dash1`--`dash4` and confirm all eight GPUs are idle and healthy.
 - Remote project path: `/home/admin/cpfs/wjh/aituner/aituner`.
 - If remote downloads are slow or fail, start the proxy from the remote `wjh`
   home directory with `./auto_proxy.sh`, then run downloads in a shell where
@@ -13,7 +14,8 @@
 
 ## Local/remote sync workflow
 
-- Treat this local repository and the `dash0` repository as the same project checkout.
+- Treat this local repository and the `dash1`--`dash4` repositories as the same project checkout.
 - Synchronize code through Git using `commit`, `push`, and `pull`.
-- For remote experiments, commit local changes, push to `origin`, then pull on `dash0` in `/home/admin/cpfs/wjh/aituner/aituner` before running.
+- For remote experiments, commit local changes, push to `origin`, then pull on each assigned host in `/home/admin/cpfs/wjh/aituner/aituner` before running.
+- Up to four independent 8-GPU experiment groups may run in parallel, one group per host; pin every job explicitly to one of `dash1`--`dash4`.
 - Do not ask for the remote host or project path again unless the user explicitly changes them.

runs/frontier-attn-structured-v0/.gitignore

@@ -0,0 +1,4 @@
+__pycache__/
+cache/
+replay/
+figure-prototype.svg

runs/frontier-attn-structured-v0/0001-Experiment-with-structured-attention-prefill-predict.patch

@@ -0,0 +1,249 @@
+From 1f8900a4ac64e45754b03d0aa7c1dddab65785cf Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Thu, 23 Jul 2026 15:27:40 +0800
+Subject: [PATCH] Experiment with structured attention prefill predictor
+
+---
+ .../shared_prediction_model_manager.py        | 16 +++-
+ .../sklearn_execution_time_predictor.py       | 16 +++-
+ .../structured_attention_prefill.py           | 79 +++++++++++++++++++
+ .../unit/test_structured_attention_prefill.py | 59 ++++++++++++++
+ 4 files changed, 165 insertions(+), 5 deletions(-)
+ create mode 100644 frontier/execution_time_predictor/structured_attention_prefill.py
+ create mode 100644 tests/unit/test_structured_attention_prefill.py
+
+diff --git a/frontier/execution_time_predictor/shared_prediction_model_manager.py b/frontier/execution_time_predictor/shared_prediction_model_manager.py
+index 8a65a49..4a21165 100644
+--- a/frontier/execution_time_predictor/shared_prediction_model_manager.py
++++ b/frontier/execution_time_predictor/shared_prediction_model_manager.py
+@@ -19,6 +19,9 @@ from frontier.execution_time_predictor.attention_tp_policy import (
+ from frontier.execution_time_predictor.attention_dataset_contract import (
+     enforce_mixed_attention_input_contract,
+ )
++from frontier.execution_time_predictor.structured_attention_prefill import (
++    StructuredAttentionPrefillRegressor,
++)
+ from frontier.logger import init_logger
+ from frontier.moe_gating_runtime import (
+     DEFAULT_MOE_GATING_RUNTIME_CONTEXT,
+@@ -1254,7 +1257,10 @@ class ExecutionTimePredictionModelManager:
+                 raise ValueError(
+                     "Missing required column 'prefill_chunk_size' in attention profiling data."
+                 )
+-            standard_prefill_df = prefill_df[prefill_df["prefill_chunk_size"] > 0].copy()
++            standard_prefill_df = prefill_df[
++                (prefill_df["prefill_chunk_size"] > 0)
++                & (prefill_df["batch_size"] == 1)
++            ].copy()
+ 
+             prefill_model_signature = f"attn_prefill_{attention_signature}"
+             if prefill_model_signature not in trained_model_signatures:
+@@ -1742,7 +1748,13 @@ class ExecutionTimePredictionModelManager:
+         # initialization to generate missing cache files.
+         # ============================================================
+ 
+-        estimator, grid_search_params = self._create_estimator_and_params(execution_time_predictor_config)
++        if model_name == "attn_prefill":
++            estimator = StructuredAttentionPrefillRegressor()
++            grid_search_params = {}
++        else:
++            estimator, grid_search_params = self._create_estimator_and_params(
++                execution_time_predictor_config
++            )
+ 
+         cv = min(execution_time_predictor_config.k_fold_cv_splits, len(df)) if len(df) >= 2 else 2
+ 
+diff --git a/frontier/execution_time_predictor/sklearn_execution_time_predictor.py b/frontier/execution_time_predictor/sklearn_execution_time_predictor.py
+index 27b62bf..b7f5350 100644
+--- a/frontier/execution_time_predictor/sklearn_execution_time_predictor.py
++++ b/frontier/execution_time_predictor/sklearn_execution_time_predictor.py
+@@ -45,6 +45,9 @@ from frontier.execution_time_predictor.attention_tp_policy import (
+ from frontier.execution_time_predictor.attention_dataset_contract import (
+     enforce_mixed_attention_input_contract,
+ )
++from frontier.execution_time_predictor.structured_attention_prefill import (
++    StructuredAttentionPrefillRegressor,
++)
+ from frontier.logger import init_logger
+ from frontier.moe_gating_runtime import get_moe_gating_base_model_name
+ from frontier.profiling.cpu_overhead.schema import (
+@@ -2573,8 +2576,12 @@ class SklearnExecutionTimePredictor(BaseExecutionTimePredictor):
+         if cached_model:
+             return cached_model
+ 
+-        model = self._get_estimator()
+-        grid_search_params = self._get_grid_search_params()
++        if model_name == "attn_prefill":
++            model = StructuredAttentionPrefillRegressor()
++            grid_search_params = {}
++        else:
++            model = self._get_estimator()
++            grid_search_params = self._get_grid_search_params()
+ 
+         if len(df) < self._config.k_fold_cv_splits:
+             cv = 2
+@@ -2869,7 +2876,10 @@ class SklearnExecutionTimePredictor(BaseExecutionTimePredictor):
+                 raise ValueError(
+                     "Missing required column 'prefill_chunk_size' in attention profiling data."
+                 )
+-            standard_prefill_df = prefill_df[prefill_df["prefill_chunk_size"] > 0].copy()
++            standard_prefill_df = prefill_df[
++                (prefill_df["prefill_chunk_size"] > 0)
++                & (prefill_df["batch_size"] == 1)
++            ].copy()
+             if len(standard_prefill_df) == 0:
+                 raise ValueError(
+                     "No standard prefill rows (prefill_chunk_size > 0) found in eager attention profiling data."
+diff --git a/frontier/execution_time_predictor/structured_attention_prefill.py b/frontier/execution_time_predictor/structured_attention_prefill.py
+new file mode 100644
+index 0000000..1829047
+--- /dev/null
++++ b/frontier/execution_time_predictor/structured_attention_prefill.py
+@@ -0,0 +1,79 @@
++"""Structured latency model for single-request chunked prefill attention."""
++
++from typing import Any
++
++import numpy as np
++from sklearn.base import BaseEstimator, RegressorMixin
++from sklearn.isotonic import IsotonicRegression
++from sklearn.linear_model import LinearRegression
++
++
++class StructuredAttentionPrefillRegressor(RegressorMixin, BaseEstimator):
++    """Model attention as a monotone base curve plus continuous KV growth.
++
++    Input columns retain the existing Frontier contract:
++    ``[kv_cache_size, prefill_chunk_size_squared]``.
++    """
++
++    def fit(self, X: Any, y: Any) -> "StructuredAttentionPrefillRegressor":
++        values = self._as_feature_array(X)
++        target = np.asarray(y, dtype=float)
++        kv_cache_size = values[:, 0]
++        prefill_chunk_size = np.sqrt(np.maximum(values[:, 1], 0.0))
++
++        base_mask = np.isclose(kv_cache_size, 0.0)
++        growth_mask = kv_cache_size > 0.0
++        if not np.any(base_mask) or not np.any(growth_mask):
++            raise ValueError(
++                "structured attn_prefill training requires both KV=0 base rows "
++                "and KV>0 growth rows"
++            )
++
++        base_q = prefill_chunk_size[base_mask]
++        base_y = target[base_mask]
++        unique_q = np.unique(base_q)
++        grouped_y = np.asarray(
++            [np.mean(base_y[np.isclose(base_q, q)]) for q in unique_q],
++            dtype=float,
++        )
++        self._base_model = IsotonicRegression(
++            increasing=True,
++            out_of_bounds="clip",
++        ).fit(unique_q, grouped_y)
++
++        growth_q = prefill_chunk_size[growth_mask]
++        growth_kv = kv_cache_size[growth_mask]
++        growth_base = self._base_model.predict(growth_q)
++        growth_features = np.column_stack(
++            (growth_kv, growth_q * growth_kv)
++        )
++        self._growth_model = LinearRegression(
++            fit_intercept=False,
++            positive=True,
++        ).fit(growth_features, target[growth_mask] - growth_base)
++
++        self.n_features_in_ = 2
++        self._frontier_base_q_min = float(unique_q.min())
++        self._frontier_base_q_max = float(unique_q.max())
++        self._frontier_growth_kv_max = float(growth_kv.max())
++        return self
++
++    def predict(self, X: Any) -> np.ndarray:
++        values = self._as_feature_array(X)
++        kv_cache_size = values[:, 0]
++        prefill_chunk_size = np.sqrt(np.maximum(values[:, 1], 0.0))
++        base = self._base_model.predict(prefill_chunk_size)
++        growth_features = np.column_stack(
++            (kv_cache_size, prefill_chunk_size * kv_cache_size)
++        )
++        return np.maximum(base + self._growth_model.predict(growth_features), 0.0)
++
++    @staticmethod
++    def _as_feature_array(X: Any) -> np.ndarray:
++        values = np.asarray(X, dtype=float)
++        if values.ndim != 2 or values.shape[1] != 2:
++            raise ValueError(
++                "structured attn_prefill expects exactly two features: "
++                "kv_cache_size and prefill_chunk_size_squared"
++            )
++        return values
+diff --git a/tests/unit/test_structured_attention_prefill.py b/tests/unit/test_structured_attention_prefill.py
+new file mode 100644
+index 0000000..12c4247
+--- /dev/null
++++ b/tests/unit/test_structured_attention_prefill.py
+@@ -0,0 +1,59 @@
++import pickle
++import unittest
++
++import numpy as np
++
++from frontier.execution_time_predictor.structured_attention_prefill import (
++    StructuredAttentionPrefillRegressor,
++)
++
++
++class StructuredAttentionPrefillRegressorTest(unittest.TestCase):
++    def setUp(self) -> None:
++        q = np.asarray([64, 128, 256, 512, 1024, 2048, 4096, 8192], dtype=float)
++        base = 0.05 + 1e-4 * q + 4e-8 * q**2
++        context_q = np.asarray([2048, 4096, 8192] * 3, dtype=float)
++        context_kv = np.repeat([8192, 16384, 24576], 3).astype(float)
++        context_y = (
++            np.interp(context_q, q, base)
++            + 1.5e-5 * context_kv
++            + 3e-8 * context_q * context_kv
++        )
++        self.X = np.column_stack(
++            (
++                np.concatenate((np.zeros_like(q), context_kv)),
++                np.concatenate((q**2, context_q**2)),
++            )
++        )
++        self.y = np.concatenate((base, context_y))
++
++    def test_recovers_structured_curve(self) -> None:
++        model = StructuredAttentionPrefillRegressor().fit(self.X, self.y)
++        np.testing.assert_allclose(model.predict(self.X), self.y, rtol=1e-6)
++
++    def test_prediction_is_nonnegative_and_monotone(self) -> None:
++        model = StructuredAttentionPrefillRegressor().fit(self.X, self.y)
++        q = np.arange(1, 8193, dtype=float)
++        for kv in (0, 8192, 32768, 40912):
++            X = np.column_stack((np.full_like(q, kv), q**2))
++            prediction = model.predict(X)
++            self.assertTrue(np.all(prediction >= 0))
++            self.assertTrue(np.all(np.diff(prediction) >= -1e-12))
++
++        kv = np.arange(0, 40913, 64, dtype=float)
++        for q_value in (64, 2048, 8192):
++            X = np.column_stack((kv, np.full_like(kv, q_value**2)))
++            self.assertTrue(np.all(np.diff(model.predict(X)) >= -1e-12))
++
++    def test_pickle_round_trip(self) -> None:
++        model = StructuredAttentionPrefillRegressor().fit(self.X, self.y)
++        restored = pickle.loads(pickle.dumps(model))
++        np.testing.assert_allclose(restored.predict(self.X), self.y, rtol=1e-6)
++
++    def test_requires_base_and_growth_rows(self) -> None:
++        with self.assertRaisesRegex(ValueError, "KV=0 base rows"):
++            StructuredAttentionPrefillRegressor().fit(self.X[:8], self.y[:8])
++
++
++if __name__ == "__main__":
++    unittest.main()
+-- 
+2.43.0
+

runs/frontier-attn-structured-v0/analyze_predictor_ablation.py

@@ -0,0 +1,256 @@
+#!/usr/bin/env python3
+"""Offline predictor ablation for EXP-ATTN-STRUCTURED.
+
+This is deliberately profile-only: it decides whether the structured model is
+good enough to justify the expensive 7-cell trace replay.
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+import pandas as pd
+from sklearn.ensemble import RandomForestRegressor
+
+ROOT = Path(__file__).resolve().parent
+REPO = ROOT.parents[1]
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--profile",
+        type=Path,
+        default=REPO
+        / "runs/frontier-prefill-kvgrowth-fix-v0/profiles/"
+        "profile-v5-kvgrowth/attention.csv",
+    )
+    parser.add_argument(
+        "--frontier-checkout",
+        type=Path,
+        default=Path("/tmp/frontier-attn-structured-v0"),
+    )
+    parser.add_argument("--output-root", type=Path, default=ROOT / "results")
+    return parser.parse_args()
+
+
+def normalize_bool(series: pd.Series) -> pd.Series:
+    return series.astype(str).str.strip().str.lower().isin(
+        {"1", "true", "t", "yes", "y"}
+    )
+
+
+def load_profile(path: Path) -> pd.DataFrame:
+    df = pd.read_csv(path).drop_duplicates()
+    for column in ("is_prefill", "is_true_mixed_batch"):
+        df[column] = normalize_bool(df[column])
+    df = df[
+        (df["n_embd"] == 2048)
+        & (df["n_q_head"] == 32)
+        & (df["n_kv_head"] == 4)
+        & (df["block_size"] == 16)
+        & df["is_prefill"]
+        & ~df["is_true_mixed_batch"]
+        & (df["prefill_chunk_size"] > 0)
+    ].copy()
+    df["prefill_chunk_size_squared"] = df["prefill_chunk_size"] ** 2
+    return df
+
+
+def mape(actual: np.ndarray, predicted: np.ndarray) -> float:
+    return float(np.mean(np.abs((predicted - actual) / actual)) * 100)
+
+
+def make_rf() -> RandomForestRegressor:
+    # Exact best parameters selected by the current profile-v5 GridSearchCV.
+    return RandomForestRegressor(
+        random_state=0,
+        n_estimators=250,
+        max_depth=8,
+        min_samples_split=2,
+    )
+
+
+def features(df: pd.DataFrame) -> pd.DataFrame:
+    return df[["kv_cache_size", "prefill_chunk_size_squared"]]
+
+
+def score_model(
+    name: str,
+    estimator: Any,
+    train: pd.DataFrame,
+    single: pd.DataFrame,
+    grid: pd.DataFrame,
+) -> dict[str, Any]:
+    target = "time_stats.attn_prefill.median"
+    estimator.fit(features(train), train[target])
+    grid_prediction = estimator.predict(features(grid))
+    single_prediction = estimator.predict(features(single))
+
+    heldout_actual: list[float] = []
+    heldout_prediction: list[float] = []
+    for context in sorted(grid["kv_cache_size"].unique()):
+        test = grid[grid["kv_cache_size"] == context]
+        fold_train = train.drop(index=test.index, errors="ignore")
+        fold_model = (
+            make_rf()
+            if name.startswith("rf")
+            else estimator.__class__()
+        )
+        fold_model.fit(features(fold_train), fold_train[target])
+        heldout_actual.extend(test[target].astype(float))
+        heldout_prediction.extend(fold_model.predict(features(test)))
+
+    q = np.arange(1, 8193, dtype=float)
+    q_deltas: list[float] = []
+    prediction_min: list[float] = []
+    for context in (0, 8192, 16384, 24576, 32768, 40912):
+        X = pd.DataFrame(
+            {
+                "kv_cache_size": np.full_like(q, context),
+                "prefill_chunk_size_squared": q**2,
+            }
+        )
+        prediction = estimator.predict(X)
+        prediction_min.append(float(prediction.min()))
+        q_deltas.append(float(np.diff(prediction).min()))
+
+    kv = np.arange(0, 40913, 64, dtype=float)
+    kv_deltas: list[float] = []
+    for query in (64, 512, 2048, 4096, 8192):
+        X = pd.DataFrame(
+            {
+                "kv_cache_size": kv,
+                "prefill_chunk_size_squared": np.full_like(kv, query**2),
+            }
+        )
+        kv_deltas.append(float(np.diff(estimator.predict(X)).min()))
+
+    heldout_actual_array = np.asarray(heldout_actual)
+    heldout_prediction_array = np.asarray(heldout_prediction)
+    return {
+        "candidate": name,
+        "training_rows": len(train),
+        "grid_fit_mape_pct": mape(
+            grid[target].to_numpy(), np.asarray(grid_prediction)
+        ),
+        "single_fit_mape_pct": mape(
+            single[target].to_numpy(), np.asarray(single_prediction)
+        ),
+        "heldout_context_mape_pct": mape(
+            heldout_actual_array, heldout_prediction_array
+        ),
+        "heldout_context_max_abs_error_pct": float(
+            np.max(
+                np.abs(
+                    (heldout_prediction_array - heldout_actual_array)
+                    / heldout_actual_array
+                )
+            )
+            * 100
+        ),
+        "prediction_min_ms": min(prediction_min),
+        "q_min_delta_ms": min(q_deltas),
+        "kv_min_delta_ms": min(kv_deltas),
+        "monotone_and_nonnegative": (
+            min(prediction_min) >= 0
+            and min(q_deltas) >= -1e-12
+            and min(kv_deltas) >= -1e-12
+        ),
+    }
+
+
+def main() -> None:
+    args = parse_args()
+    sys.path.insert(0, str(args.frontier_checkout))
+    from frontier.execution_time_predictor.structured_attention_prefill import (
+        StructuredAttentionPrefillRegressor,
+    )
+
+    df = load_profile(args.profile)
+    records: list[dict[str, Any]] = []
+    data_audit: dict[str, Any] = {}
+    for tp in (1, 2, 4):
+        tp_df = df[df["num_tensor_parallel_workers"] == tp].copy()
+        single = tp_df[tp_df["batch_size"] == 1].copy()
+        grid = single[
+            single["prefill_chunk_size"].isin((2048, 4096, 8192))
+            & (single["kv_cache_size"] > 0)
+        ].copy()
+        duplicate_groups = (
+            tp_df.groupby(
+                ["kv_cache_size", "prefill_chunk_size_squared"]
+            )
+            .size()
+            .gt(1)
+            .sum()
+        )
+        data_audit[f"tp{tp}"] = {
+            "standard_rows": len(tp_df),
+            "single_request_rows": len(single),
+            "target_grid_rows": len(grid),
+            "duplicate_feature_groups": int(duplicate_groups),
+        }
+
+        candidates = (
+            ("rf_all", make_rf(), tp_df),
+            ("rf_single", make_rf(), single),
+            (
+                "structured_single",
+                StructuredAttentionPrefillRegressor(),
+                single,
+            ),
+        )
+        for name, model, train in candidates:
+            result = score_model(name, model, train, single, grid)
+            result["tp"] = tp
+            records.append(result)
+
+    structured = [r for r in records if r["candidate"] == "structured_single"]
+    checks = {
+        "heldout_context_mape_le_5pct": all(
+            r["heldout_context_mape_pct"] <= 5 for r in structured
+        ),
+        "monotone_and_nonnegative": all(
+            r["monotone_and_nonnegative"] for r in structured
+        ),
+    }
+    checks["profile_gate"] = all(checks.values())
+    payload = {
+        "schema": "frontier-attn-structured-ablation-v1",
+        "profile": str(args.profile.resolve()),
+        "frontier_checkout": str(args.frontier_checkout.resolve()),
+        "data_audit": data_audit,
+        "results": records,
+        "checks": checks,
+    }
+
+    args.output_root.mkdir(parents=True, exist_ok=True)
+    (args.output_root / "predictor-ablation.json").write_text(
+        json.dumps(payload, indent=2)
+    )
+    with (args.output_root / "predictor-ablation.csv").open(
+        "w", newline=""
+    ) as stream:
+        writer = csv.DictWriter(stream, fieldnames=list(records[0]))
+        writer.writeheader()
+        writer.writerows(records)
+    print(json.dumps(checks, indent=2))
+    for row in records:
+        print(
+            f"TP{row['tp']} {row['candidate']:18s} "
+            f"grid={row['grid_fit_mape_pct']:.2f}% "
+            f"heldout={row['heldout_context_mape_pct']:.2f}% "
+            f"max={row['heldout_context_max_abs_error_pct']:.2f}% "
+            f"monotone={row['monotone_and_nonnegative']}"
+        )
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-attn-structured-v0/analyze_trace_verdict.py

@@ -0,0 +1,304 @@
+#!/usr/bin/env python3
+"""Trial-aware verdict for the seven structured-attention trace replays."""
+
+from __future__ import annotations
+
+import csv
+import json
+import math
+from pathlib import Path
+from typing import Any
+
+ROOT = Path(__file__).resolve().parent
+REPO = ROOT.parents[1]
+S3_REAL = REPO / "runs/frontier-s3-real-v0"
+V5 = REPO / "runs/frontier-prefill-kvgrowth-fix-v0"
+
+CELLS = {
+    "tp1_rho0p00125": {
+        "real": "frontier-tp1-real-r0p00125-t*",
+        "old": V5 / "sim-replay-tp1/v5/tp1_rho0p00125",
+    },
+    "tp1_rho0p0025": {
+        "real": "frontier-tp1-real-r0p0025-t*",
+        "old": V5 / "sim-replay-tp1/v5/tp1_rho0p0025",
+    },
+    "tp2_rho0p0025": {
+        "real": "frontier-s3-real-full-r0p0025-tp2-t*",
+        "old": V5 / "sim-replay/tp2_rho0p0025",
+    },
+    "tp2_rho0p005": {
+        "real": "frontier-s3-real-full-r0p005-tp2-t*",
+        "old": V5 / "sim-replay/tp2_rho0p005",
+    },
+    "tp4_rho0p0025": {
+        "real": "frontier-s3-real-full-r0p0025-tp4-t*",
+        "old": V5 / "sim-replay/tp4_rho0p0025",
+    },
+    "tp4_rho0p005": {
+        "real": "frontier-s3-real-full-r0p005-tp4-t*",
+        "old": V5 / "sim-replay/tp4_rho0p005",
+    },
+    "tp4_rho0p01": {
+        "real": "frontier-s3-real-full-r0p01-tp4-t*",
+        "old": V5 / "sim-replay/tp4_rho0p01",
+    },
+}
+METRICS = {
+    "ttft": ("ttft_ms", "ttft"),
+    "tpot": ("tpot_ms", "tpot"),
+    "e2e": ("e2e_ms", "request_e2e_time"),
+}
+QUANTILES = {"mean": None, "p50": 0.5, "p90": 0.9, "p99": 0.99}
+
+
+def percentile(values: list[float], quantile: float) -> float:
+    ordered = sorted(values)
+    position = (len(ordered) - 1) * quantile
+    lower, upper = math.floor(position), math.ceil(position)
+    if lower == upper:
+        return ordered[lower]
+    return (
+        ordered[lower] * (upper - position)
+        + ordered[upper] * (position - lower)
+    )
+
+
+def summarize(values: list[float]) -> dict[str, float]:
+    return {
+        name: (
+            sum(values) / len(values)
+            if quantile is None
+            else percentile(values, quantile)
+        )
+        for name, quantile in QUANTILES.items()
+    }
+
+
+def load_real_trials(pattern: str) -> list[list[dict[str, Any]]]:
+    trials = []
+    for run_root in sorted((S3_REAL / "fleet-artifacts").glob(pattern)):
+        results = list(
+            run_root.glob(
+                "artifacts/outputs/full-real/*/*/trial-*/results/result.json"
+            )
+        )
+        if len(results) != 1:
+            raise ValueError(f"expected one result in {run_root}, got {results}")
+        trials.append(json.loads(results[0].read_text())["requests"])
+    if len(trials) != 2:
+        raise ValueError(f"expected two real trials for {pattern}, got {len(trials)}")
+    return trials
+
+
+def load_sim(root: Path) -> list[dict[str, str]]:
+    matches = list((root / "metrics").rglob("request_metrics.csv"))
+    if len(matches) != 1:
+        raise ValueError(f"expected one request_metrics.csv below {root}: {matches}")
+    rows = list(csv.DictReader(matches[0].open()))
+    rows.sort(key=lambda row: int(float(row["Request Id"])))
+    return rows
+
+
+def distribution_bias(
+    real_rows: list[dict[str, Any]],
+    sim_rows: list[dict[str, str]],
+) -> dict[str, dict[str, float]]:
+    output: dict[str, dict[str, float]] = {}
+    for metric, (real_key, sim_key) in METRICS.items():
+        pairs = [
+            (float(real[real_key]), float(sim[sim_key]))
+            for real, sim in zip(real_rows, sim_rows)
+            if real.get("success")
+        ]
+        real_summary = summarize([pair[0] for pair in pairs])
+        sim_summary = summarize([pair[1] for pair in pairs])
+        output[metric] = {
+            name: (sim_summary[name] - real_summary[name]) / real_summary[name]
+            for name in QUANTILES
+        }
+    return output
+
+
+def paired_relative_error(
+    real_rows: list[dict[str, Any]],
+    sim_rows: list[dict[str, str]],
+) -> dict[str, dict[str, float]]:
+    output: dict[str, dict[str, float]] = {}
+    for metric, (real_key, sim_key) in METRICS.items():
+        errors = [
+            (float(sim[sim_key]) - float(real[real_key])) / float(real[real_key])
+            for real, sim in zip(real_rows, sim_rows)
+            if real.get("success") and float(real[real_key]) != 0
+        ]
+        output[metric] = summarize(errors)
+    return output
+
+
+def aggregate_trial_bias(
+    trial_biases: list[dict[str, dict[str, float]]],
+) -> dict[str, dict[str, dict[str, float]]]:
+    return {
+        metric: {
+            quantile: {
+                "mean": sum(values) / len(values),
+                "min": min(values),
+                "max": max(values),
+            }
+            for quantile in QUANTILES
+            for values in [
+                [trial[metric][quantile] for trial in trial_biases]
+            ]
+        }
+        for metric in METRICS
+    }
+
+
+def legacy_pooled_bias(
+    real_trials: list[list[dict[str, Any]]],
+    sim_rows: list[dict[str, str]],
+) -> dict[str, dict[str, float]]:
+    output: dict[str, dict[str, float]] = {}
+    for metric, (real_key, sim_key) in METRICS.items():
+        real_values = [
+            float(row[real_key])
+            for trial in real_trials
+            for row in trial[: len(sim_rows)]
+            if row.get("success")
+        ]
+        sim_values = [float(row[sim_key]) for row in sim_rows]
+        real_summary = summarize(real_values)
+        sim_summary = summarize(sim_values)
+        output[metric] = {
+            name: (sim_summary[name] - real_summary[name]) / real_summary[name]
+            for name in QUANTILES
+        }
+    return output
+
+
+def waiting_p99(sim_rows: list[dict[str, str]]) -> float:
+    return percentile(
+        [float(row["request_waiting_time_total"]) for row in sim_rows], 0.99
+    )
+
+
+def main() -> None:
+    results: dict[str, Any] = {}
+    flat_rows: list[dict[str, Any]] = []
+    for label, paths in CELLS.items():
+        real_trials = load_real_trials(paths["real"])
+        old_sim = load_sim(paths["old"])
+        new_sim = load_sim(ROOT / "replay" / label)
+        old_trial_bias = [
+            distribution_bias(trial, old_sim) for trial in real_trials
+        ]
+        new_trial_bias = [
+            distribution_bias(trial, new_sim) for trial in real_trials
+        ]
+        old_legacy = legacy_pooled_bias(real_trials, old_sim)
+        new_legacy = legacy_pooled_bias(real_trials, new_sim)
+        wait_p99 = waiting_p99(new_sim)
+        results[label] = {
+            "old": {
+                "trialwise_distribution_bias": old_trial_bias,
+                "trialwise_distribution_bias_summary": aggregate_trial_bias(
+                    old_trial_bias
+                ),
+                "legacy_pooled_distribution_bias": old_legacy,
+            },
+            "new": {
+                "trialwise_distribution_bias": new_trial_bias,
+                "trialwise_distribution_bias_summary": aggregate_trial_bias(
+                    new_trial_bias
+                ),
+                "paired_relative_error": [
+                    paired_relative_error(trial, new_sim)
+                    for trial in real_trials
+                ],
+                "legacy_pooled_distribution_bias": new_legacy,
+                "waiting_p99_ms": wait_p99,
+                "validity": (
+                    "PASS_SUBCRITICAL"
+                    if wait_p99 < 1000
+                    else "GATE_FAIL_DIAGNOSTIC"
+                ),
+            },
+        }
+        for metric in METRICS:
+            for quantile in QUANTILES:
+                flat_rows.append(
+                    {
+                        "cell": label,
+                        "metric": metric,
+                        "quantile": quantile,
+                        "old_bias": old_legacy[metric][quantile],
+                        "new_bias": new_legacy[metric][quantile],
+                        "abs_bias_delta_pp": 100
+                        * (
+                            abs(new_legacy[metric][quantile])
+                            - abs(old_legacy[metric][quantile])
+                        ),
+                        "validity": results[label]["new"]["validity"],
+                    }
+                )
+
+    tp1_checks = []
+    for cell in ("tp1_rho0p00125", "tp1_rho0p0025"):
+        for quantile in ("mean", "p99"):
+            old = results[cell]["old"]["legacy_pooled_distribution_bias"]["ttft"][
+                quantile
+            ]
+            new = results[cell]["new"]["legacy_pooled_distribution_bias"]["ttft"][
+                quantile
+            ]
+            tp1_checks.append(abs(old) - abs(new) >= 0.05)
+
+    regressions = [
+        row
+        for row in flat_rows
+        if row["cell"].startswith(("tp2", "tp4"))
+        and row["metric"] in ("ttft", "e2e")
+        and row["abs_bias_delta_pp"] > 5
+    ]
+    checks = {
+        "tp1_ttft_mean_p99_improve_ge_5pp": all(tp1_checks),
+        "tp2_tp4_ttft_e2e_no_abs_regression_gt_5pp": not regressions,
+        "regressions": regressions,
+    }
+    checks["trace_gate"] = (
+        checks["tp1_ttft_mean_p99_improve_ge_5pp"]
+        and checks["tp2_tp4_ttft_e2e_no_abs_regression_gt_5pp"]
+    )
+
+    payload = {
+        "schema": "frontier-attn-structured-trial-aware-verdict-v1",
+        "metric_note": (
+            "Primary values are per-real-trial distribution biases with request "
+            "alignment by index. legacy_pooled reproduces the old milestone "
+            "quantile convention only for direct comparison."
+        ),
+        "cells": results,
+        "checks": checks,
+    }
+    output = ROOT / "results"
+    output.mkdir(parents=True, exist_ok=True)
+    (output / "trace-verdict.json").write_text(json.dumps(payload, indent=2))
+    with (output / "trace-verdict.csv").open("w", newline="") as stream:
+        writer = csv.DictWriter(stream, fieldnames=list(flat_rows[0]))
+        writer.writeheader()
+        writer.writerows(flat_rows)
+
+    print(json.dumps(checks, indent=2))
+    for label, result in results.items():
+        old = result["old"]["legacy_pooled_distribution_bias"]["ttft"]
+        new = result["new"]["legacy_pooled_distribution_bias"]["ttft"]
+        print(
+            f"{label}: TTFT mean {old['mean']:+.1%}->{new['mean']:+.1%}, "
+            f"p99 {old['p99']:+.1%}->{new['p99']:+.1%}, "
+            f"waiting_p99={result['new']['waiting_p99_ms']:.0f}ms "
+            f"{result['new']['validity']}"
+        )
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-attn-structured-v0/experiment-card.md

@@ -0,0 +1,82 @@
+# 实验 EXP-ATTN-STRUCTURED：结构化 predictor 能否关闭大 KV 端的 RF 欠拟合
+
+> **状态：** 已完成（profile gate PASS；global merge gate FAIL）
+>
+> Parent campaign：[`../frontier-simulator-gap-campaign-v0/README.md`](../frontier-simulator-gap-campaign-v0/README.md)
+
+## Claim 与决策
+
+- **Parent claim：** profile-v5 已补齐 chunked-prefill KV-context 测量，但当前 RF 仍在 TP1/2/4 的新网格上产生约 12%--14% self-fit MAPE，并在 TP1 真实 trace 中留下 −13% 到 −22% TTFT 偏差。
+- **目的：** 检查该 residual 是否来自可工程修复的 predictor representation，而不是 profile 数据或 serving path。
+- **Competing hypotheses：**
+  - H1：standard prefill 模型错误混入 pure multi-request rows，且 RF 对连续 attention scaling 作阶梯平滑；使用单请求数据和结构化 `base(q)+KV×(a+bq)` 可关闭残余。
+  - H2：残余主要来自未建模的 serving-path 组件；替换 predictor 不会改善 7-cell trace fidelity。
+- **事前预测：**
+  - H1：held-out context MAPE ≤5%，TP1 TTFT mean/p99 绝对偏差至少改善 5 pp。
+  - H2：profile gate 失败，或 profile gate 通过但 trace TTFT 几乎不动。
+- **判定规则：**
+  - profile gate：TP1/2/4 held-out context MAPE 均 ≤5%；q/KV 单调且预测非负。
+  - trace gate：两个 TP1 cell 的 TTFT mean/p99 |bias| 各改善 ≥5 pp；TP2/TP4 任一 TTFT/E2E quantile 不恶化 >5 pp。
+  - profile gate 失败即停止；trace gate 失败则回退 patch，不进入 EXP-2。
+
+## Setup
+
+- **自变量：**
+  - A：现有 RF，standard prefill 全部非 true-mixed rows。
+  - B：现有 RF，但仅 `batch_size=1`。
+  - C：仅 `batch_size=1` 的 structured predictor：
+    - `base(q)`：KV=0 profile 的单调分段线性插值；
+    - growth：非负 least-squares `KV×(a+bq)`。
+- **控制变量：** attention/linear/MoE/collective profile、trace、prefix cache、scheduler、graph mode、KV blocks、MNS、全部 argv。
+- **System context：** Qwen3-30B-A3B BF16；H20；Frontier `deadc4a3`；TP1/2/4；MNS16；chunk 8192；prefix caching。
+- **Workload 或 trace：** 现有 7-cell 60-min production chat trace matrix；real 侧每 cell 两个 trial。
+- **Baselines：** `docs/assets/frontier-fidelity/full-matrix.csv` 的 sim-v5。
+- **Metrics：**
+  - profile：grid fit MAPE、leave-one-context MAPE/max error、q/KV monotonicity；
+  - trace：request-ID paired bias；每个 real trial 单独计算后报告 mean 与 trial interval；
+  - queue validity：waiting p99，TP1 超过 1 s 的 cell 标为 diagnostic。
+
+## 预期产物与 review
+
+- **预期数据：** `results/predictor-ablation.{json,csv}`、`replay/<cell>/`、`results/paired-verdict.json`。
+- **Figure prototype：** `figure-prototype.png`；左图为 q8k 随 KV 增长的 actual/RF/structured，右图为 7-cell TTFT bias 的事前期望。
+- **人工 review：** 已按 campaign 顺序批准执行。
+- **Review 意见：** 只改 standard single-request predictor；不得改 mixed predictor 或任何 profile row。
+
+## 复现信息
+
+- **Code：** Frontier base `deadc4a321f0baaa534c6ebd17f974123733cdc2`；实验 patch 将保存为 `frontier-structured-attn.patch` 并记录 SHA256。
+- **Environment：** 本地 CPU replay；Python dependency roots 复用 `runs/frontier-collective-joint-v0/counterfactual/joint-r2/manifest.json`。
+- **产物路径：** 本目录。
+- **已知 deviation：** milestone 文档将 7-cell 口径称为“逐 request paired”，但旧脚本实际 pool 两个 real trial 后比较 quantile；本实验会修正分析口径，不改旧结果文件。
+
+## 预分析事实
+
+- 现有训练代码使用 `["kv_cache_size", "prefill_chunk_size_squared"]` 与 RF grid search。
+- runtime cache 注释明确 standard model 是 per-request；多请求 prefill 在模型存在时走 `attn_prefill_mixed`。
+- profile-v5 的 standard 训练集每 TP 有 29 行，其中单请求 23 行；有 4 组相同 `(KV,q²)` feature 对应多个 pure-batch 标签。
+- 初步 structured candidate 的 leave-one-context MAPE：TP1 0.84%、TP2 1.61%、TP4 3.01%；max error 分别 2.04%、3.47%、5.49%。这些是实现前的临时计算，须由版本化脚本复现后才进入结果。
+
+## 结果
+
+- **观察事实：**
+  - structured held-out-context MAPE 为 TP1/2/4=`0.84%/1.60%/3.01%`；
+    当前 RF 为 `44.40%/44.20%/43.54%`。单调/非负 gate 通过。
+  - TP1 两点 TTFT mean bias `−13.5/−17.7% → −6.3/−9.5%`，p99
+    `−16.8/−22.1% → −8.6/−14.4%`。
+  - TP2 两点 TTFT mean bias `−11.2/−14.1% → −4.5/−7.1%`，p99
+    `−17.3/−19.4% → −7.7/−9.0%`。
+  - TP4 三点 TTFT mean bias `+2.5/+2.9/−0.1% → +7.8/+8.4/+6.0%`；
+    三点均使绝对误差恶化 `5.3--5.8 pp`，触发预设回归 gate。
+  - validity 重新审计：TP1 两点 waiting p99=`1.34/1.89 s`；TP2
+    ρ=.005=`1.17 s`，均标为 `GATE_FAIL_DIAGNOSTIC`。其余四点通过。
+- **异常：** TP4 ρ=.005 的 TPOT p99 从 `+31.2%` 变为 `+36.6%`，
+  表明该 tail 对 prefill/mixed-decode 相位敏感，不是本 patch 能关闭的稳定
+  decode predictor 偏差。
+- **含义：** H1 的 representation 机制得到支持，但“全局替换 RF 可直接提升
+  7-cell fidelity”被反驳。TP4 原先接近零的 mean TTFT 含有 predictor
+  欠拟合与其它正向 residual 的误差抵消；单独修正 attention 会揭开后者。
+- **Claim update：** structured predictor 是明确的工程候选，但必须与 TP4
+  residual 联合收敛后才可 merge；当前 patch 只保留为 ablation。
+- **下一步：** EXP-2 先重算 structured 分支的 TP2 chunk-level residual；
+  仅 residual ≥10% 才运行 GPU serving-path 三臂 profile。

runs/frontier-attn-structured-v0/frontier-reference.json

@@ -0,0 +1,525 @@
+{
+  "cc_cache": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/cc-cache",
+  "cells": {
+    "tp1_mns16": {
+      "argv": [
+        "/usr/bin/python3",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/run_frontier_with_curves.py",
+        "--simulation_mode",
+        "online",
+        "--sys_arch",
+        "co-location",
+        "--cc_backend_config_type",
+        "vidur",
+        "--cluster_config_num_replicas",
+        "1",
+        "--cluster_scheduler_config_type",
+        "sticky_round_robin",
+        "--replica_config_model_name",
+        "qwen3-a3b-30b-moe",
+        "--replica_config_device",
+        "h20",
+        "--replica_config_network_device",
+        "h20_dgx",
+        "--replica_config_attn_tensor_parallel_size",
+        "1",
+        "--replica_config_attn_data_parallel_size",
+        "1",
+        "--replica_config_moe_tensor_parallel_size",
+        "1",
+        "--replica_config_moe_expert_parallel_size",
+        "1",
+        "--replica_config_num_pipeline_stages",
+        "1",
+        "--replica_scheduler_config_type",
+        "vllm_v1",
+        "--decode_cuda_graph_mode",
+        "piecewise",
+        "--vllm_v1_scheduler_config_batch_size_cap",
+        "16",
+        "--vllm_v1_scheduler_config_max_tokens_in_batch",
+        "8192",
+        "--vllm_v1_scheduler_config_long_prefill_token_threshold",
+        "0",
+        "--vllm_v1_scheduler_config_block_size",
+        "16",
+        "--vllm_v1_scheduler_config_num_blocks_mode",
+        "explicit",
+        "--vllm_v1_scheduler_config_gpu_memory_utilization",
+        "0.92",
+        "--vllm_v1_scheduler_config_non_kv_cache_overhead_bytes",
+        "0",
+        "--request_generator_config_type",
+        "trace_replay",
+        "--trace_request_generator_config_trace_file",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/inputs/tp1-frontier.csv",
+        "--trace_request_generator_config_max_tokens",
+        "40960",
+        "--metrics_config_output_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/sim/tp1_mns16/metrics",
+        "--metrics_config_run_id",
+        "joint_tp1_mns16",
+        "--metrics_config_write_metrics",
+        "--metrics_config_store_request_metrics",
+        "--metrics_config_store_batch_metrics",
+        "--metrics_config_store_token_completion_metrics",
+        "--metrics_config_store_utilization_metrics",
+        "--no-metrics_config_store_plots",
+        "--no-metrics_config_enable_chrome_trace",
+        "--no-metrics_config_write_json_trace",
+        "--metrics_config_store_frontier_stage_batch_ledger",
+        "--no-random_forrest_execution_time_predictor_config_enable_dummy_mode",
+        "--random_forrest_execution_time_predictor_config_linear_op_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/linear_op.csv",
+        "--random_forrest_execution_time_predictor_config_atten_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/attention.csv",
+        "--random_forrest_execution_time_predictor_config_moe_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/moe.csv",
+        "--random_forrest_execution_time_predictor_config_linear_op_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/linear_op.csv",
+        "--random_forrest_execution_time_predictor_config_atten_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/attention.csv",
+        "--random_forrest_execution_time_predictor_config_moe_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/moe.csv",
+        "--random_forrest_execution_time_predictor_config_prediction_max_prefill_chunk_size",
+        "8192",
+        "--random_forrest_execution_time_predictor_config_prediction_max_batch_size",
+        "32",
+        "--random_forrest_execution_time_predictor_config_prediction_max_tokens_per_request",
+        "40960",
+        "--random_forrest_execution_time_predictor_config_no_cache",
+        "--random_forrest_execution_time_predictor_config_skip_cpu_overhead_modeling",
+        "--vllm_v1_scheduler_config_num_blocks",
+        "20128",
+        "--vllm_v1_scheduler_config_enable_chunked_prefill",
+        "--random_forrest_execution_time_predictor_config_num_training_job_threads",
+        "4",
+        "--cudagraph_capture_sizes",
+        "1",
+        "2",
+        "4",
+        "8",
+        "16",
+        "24",
+        "32",
+        "--vidur_cc_backend_config_all_reduce_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/measured-allreduce.csv",
+        "--vidur_cc_backend_config_cache_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/cc-cache",
+        "--vidur_cc_backend_config_k_fold_cv_splits",
+        "6",
+        "--vidur_cc_backend_config_num_training_job_threads",
+        "1",
+        "--metrics_config_cache_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/model-cache"
+      ],
+      "log": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/logs/tp1_mns16.log",
+      "source_command": "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-lo-fixed-pd-cells/sim/fixed-pd/runs/tp1_mns16/tp1/command.json",
+      "source_command_sha256": "a9815797b1601bf6f6cdf0269e84acb376a84945609e338868dc8347aab650e6",
+      "usage": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/usage/tp1_mns16.json"
+    },
+    "tp2_mns16": {
+      "argv": [
+        "/usr/bin/python3",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/run_frontier_with_curves.py",
+        "--simulation_mode",
+        "online",
+        "--sys_arch",
+        "co-location",
+        "--cc_backend_config_type",
+        "vidur",
+        "--cluster_config_num_replicas",
+        "1",
+        "--cluster_scheduler_config_type",
+        "sticky_round_robin",
+        "--replica_config_model_name",
+        "qwen3-a3b-30b-moe",
+        "--replica_config_device",
+        "h20",
+        "--replica_config_network_device",
+        "h20_dgx",
+        "--replica_config_attn_tensor_parallel_size",
+        "2",
+        "--replica_config_attn_data_parallel_size",
+        "1",
+        "--replica_config_moe_tensor_parallel_size",
+        "2",
+        "--replica_config_moe_expert_parallel_size",
+        "1",
+        "--replica_config_num_pipeline_stages",
+        "1",
+        "--replica_scheduler_config_type",
+        "vllm_v1",
+        "--decode_cuda_graph_mode",
+        "piecewise",
+        "--vllm_v1_scheduler_config_batch_size_cap",
+        "16",
+        "--vllm_v1_scheduler_config_max_tokens_in_batch",
+        "8192",
+        "--vllm_v1_scheduler_config_long_prefill_token_threshold",
+        "0",
+        "--vllm_v1_scheduler_config_block_size",
+        "16",
+        "--vllm_v1_scheduler_config_num_blocks_mode",
+        "explicit",
+        "--vllm_v1_scheduler_config_gpu_memory_utilization",
+        "0.92",
+        "--vllm_v1_scheduler_config_non_kv_cache_overhead_bytes",
+        "0",
+        "--request_generator_config_type",
+        "trace_replay",
+        "--trace_request_generator_config_trace_file",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/inputs/tp2-frontier.csv",
+        "--trace_request_generator_config_max_tokens",
+        "40960",
+        "--metrics_config_output_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/sim/tp2_mns16/metrics",
+        "--metrics_config_run_id",
+        "joint_tp2_mns16",
+        "--metrics_config_write_metrics",
+        "--metrics_config_store_request_metrics",
+        "--metrics_config_store_batch_metrics",
+        "--metrics_config_store_token_completion_metrics",
+        "--metrics_config_store_utilization_metrics",
+        "--no-metrics_config_store_plots",
+        "--no-metrics_config_enable_chrome_trace",
+        "--no-metrics_config_write_json_trace",
+        "--metrics_config_store_frontier_stage_batch_ledger",
+        "--no-random_forrest_execution_time_predictor_config_enable_dummy_mode",
+        "--random_forrest_execution_time_predictor_config_linear_op_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/linear_op.csv",
+        "--random_forrest_execution_time_predictor_config_atten_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/attention.csv",
+        "--random_forrest_execution_time_predictor_config_moe_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/moe.csv",
+        "--random_forrest_execution_time_predictor_config_linear_op_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/linear_op.csv",
+        "--random_forrest_execution_time_predictor_config_atten_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/attention.csv",
+        "--random_forrest_execution_time_predictor_config_moe_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/moe.csv",
+        "--random_forrest_execution_time_predictor_config_prediction_max_prefill_chunk_size",
+        "8192",
+        "--random_forrest_execution_time_predictor_config_prediction_max_batch_size",
+        "32",
+        "--random_forrest_execution_time_predictor_config_prediction_max_tokens_per_request",
+        "40960",
+        "--random_forrest_execution_time_predictor_config_no_cache",
+        "--random_forrest_execution_time_predictor_config_skip_cpu_overhead_modeling",
+        "--vllm_v1_scheduler_config_num_blocks",
+        "76620",
+        "--vllm_v1_scheduler_config_enable_chunked_prefill",
+        "--random_forrest_execution_time_predictor_config_num_training_job_threads",
+        "4",
+        "--cudagraph_capture_sizes",
+        "1",
+        "2",
+        "4",
+        "8",
+        "16",
+        "24",
+        "32",
+        "--vidur_cc_backend_config_all_reduce_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/measured-allreduce.csv",
+        "--vidur_cc_backend_config_cache_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/cc-cache",
+        "--vidur_cc_backend_config_k_fold_cv_splits",
+        "6",
+        "--vidur_cc_backend_config_num_training_job_threads",
+        "1",
+        "--metrics_config_cache_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/model-cache"
+      ],
+      "log": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/logs/tp2_mns16.log",
+      "source_command": "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-lo-fixed-pd-cells/sim/fixed-pd/runs/tp2_mns16/tp2/command.json",
+      "source_command_sha256": "61788a8810be301c9dbc006624aa19b6a932bc44d341b836861087833cffc3df",
+      "usage": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/usage/tp2_mns16.json"
+    },
+    "tp4_mns16": {
+      "argv": [
+        "/usr/bin/python3",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/run_frontier_with_curves.py",
+        "--simulation_mode",
+        "online",
+        "--sys_arch",
+        "co-location",
+        "--cc_backend_config_type",
+        "vidur",
+        "--cluster_config_num_replicas",
+        "1",
+        "--cluster_scheduler_config_type",
+        "sticky_round_robin",
+        "--replica_config_model_name",
+        "qwen3-a3b-30b-moe",
+        "--replica_config_device",
+        "h20",
+        "--replica_config_network_device",
+        "h20_dgx",
+        "--replica_config_attn_tensor_parallel_size",
+        "4",
+        "--replica_config_attn_data_parallel_size",
+        "1",
+        "--replica_config_moe_tensor_parallel_size",
+        "4",
+        "--replica_config_moe_expert_parallel_size",
+        "1",
+        "--replica_config_num_pipeline_stages",
+        "1",
+        "--replica_scheduler_config_type",
+        "vllm_v1",
+        "--decode_cuda_graph_mode",
+        "piecewise",
+        "--vllm_v1_scheduler_config_batch_size_cap",
+        "16",
+        "--vllm_v1_scheduler_config_max_tokens_in_batch",
+        "8192",
+        "--vllm_v1_scheduler_config_long_prefill_token_threshold",
+        "0",
+        "--vllm_v1_scheduler_config_block_size",
+        "16",
+        "--vllm_v1_scheduler_config_num_blocks_mode",
+        "explicit",
+        "--vllm_v1_scheduler_config_gpu_memory_utilization",
+        "0.92",
+        "--vllm_v1_scheduler_config_non_kv_cache_overhead_bytes",
+        "0",
+        "--request_generator_config_type",
+        "trace_replay",
+        "--trace_request_generator_config_trace_file",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/inputs/tp4-frontier.csv",
+        "--trace_request_generator_config_max_tokens",
+        "40960",
+        "--metrics_config_output_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/sim/tp4_mns16/metrics",
+        "--metrics_config_run_id",
+        "joint_tp4_mns16",
+        "--metrics_config_write_metrics",
+        "--metrics_config_store_request_metrics",
+        "--metrics_config_store_batch_metrics",
+        "--metrics_config_store_token_completion_metrics",
+        "--metrics_config_store_utilization_metrics",
+        "--no-metrics_config_store_plots",
+        "--no-metrics_config_enable_chrome_trace",
+        "--no-metrics_config_write_json_trace",
+        "--metrics_config_store_frontier_stage_batch_ledger",
+        "--no-random_forrest_execution_time_predictor_config_enable_dummy_mode",
+        "--random_forrest_execution_time_predictor_config_linear_op_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/linear_op.csv",
+        "--random_forrest_execution_time_predictor_config_atten_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/attention.csv",
+        "--random_forrest_execution_time_predictor_config_moe_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/moe.csv",
+        "--random_forrest_execution_time_predictor_config_linear_op_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/linear_op.csv",
+        "--random_forrest_execution_time_predictor_config_atten_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/attention.csv",
+        "--random_forrest_execution_time_predictor_config_moe_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/moe.csv",
+        "--random_forrest_execution_time_predictor_config_prediction_max_prefill_chunk_size",
+        "8192",
+        "--random_forrest_execution_time_predictor_config_prediction_max_batch_size",
+        "32",
+        "--random_forrest_execution_time_predictor_config_prediction_max_tokens_per_request",
+        "40960",
+        "--random_forrest_execution_time_predictor_config_no_cache",
+        "--random_forrest_execution_time_predictor_config_skip_cpu_overhead_modeling",
+        "--vllm_v1_scheduler_config_num_blocks",
+        "191882",
+        "--vllm_v1_scheduler_config_enable_chunked_prefill",
+        "--random_forrest_execution_time_predictor_config_num_training_job_threads",
+        "4",
+        "--cudagraph_capture_sizes",
+        "1",
+        "2",
+        "4",
+        "8",
+        "16",
+        "24",
+        "32",
+        "--vidur_cc_backend_config_all_reduce_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/measured-allreduce.csv",
+        "--vidur_cc_backend_config_cache_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/cc-cache",
+        "--vidur_cc_backend_config_k_fold_cv_splits",
+        "6",
+        "--vidur_cc_backend_config_num_training_job_threads",
+        "1",
+        "--metrics_config_cache_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/model-cache"
+      ],
+      "log": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/logs/tp4_mns16.log",
+      "source_command": "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-lo-fixed-pd-cells/sim/fixed-pd/runs/tp4_mns16/tp4/command.json",
+      "source_command_sha256": "9bbcf10446336ba5885193f391dd628cd18ff64d91a51ebcd463ffd24be95532",
+      "usage": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/usage/tp4_mns16.json"
+    },
+    "tp4_mns32": {
+      "argv": [
+        "/usr/bin/python3",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/run_frontier_with_curves.py",
+        "--simulation_mode",
+        "online",
+        "--sys_arch",
+        "co-location",
+        "--cc_backend_config_type",
+        "vidur",
+        "--cluster_config_num_replicas",
+        "1",
+        "--cluster_scheduler_config_type",
+        "sticky_round_robin",
+        "--replica_config_model_name",
+        "qwen3-a3b-30b-moe",
+        "--replica_config_device",
+        "h20",
+        "--replica_config_network_device",
+        "h20_dgx",
+        "--replica_config_attn_tensor_parallel_size",
+        "4",
+        "--replica_config_attn_data_parallel_size",
+        "1",
+        "--replica_config_moe_tensor_parallel_size",
+        "4",
+        "--replica_config_moe_expert_parallel_size",
+        "1",
+        "--replica_config_num_pipeline_stages",
+        "1",
+        "--replica_scheduler_config_type",
+        "vllm_v1",
+        "--decode_cuda_graph_mode",
+        "piecewise",
+        "--vllm_v1_scheduler_config_batch_size_cap",
+        "32",
+        "--vllm_v1_scheduler_config_max_tokens_in_batch",
+        "8192",
+        "--vllm_v1_scheduler_config_long_prefill_token_threshold",
+        "0",
+        "--vllm_v1_scheduler_config_block_size",
+        "16",
+        "--vllm_v1_scheduler_config_num_blocks_mode",
+        "explicit",
+        "--vllm_v1_scheduler_config_gpu_memory_utilization",
+        "0.92",
+        "--vllm_v1_scheduler_config_non_kv_cache_overhead_bytes",
+        "0",
+        "--request_generator_config_type",
+        "trace_replay",
+        "--trace_request_generator_config_trace_file",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/inputs/tp4-frontier.csv",
+        "--trace_request_generator_config_max_tokens",
+        "40960",
+        "--metrics_config_output_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/sim/tp4_mns32/metrics",
+        "--metrics_config_run_id",
+        "joint_tp4_mns32",
+        "--metrics_config_write_metrics",
+        "--metrics_config_store_request_metrics",
+        "--metrics_config_store_batch_metrics",
+        "--metrics_config_store_token_completion_metrics",
+        "--metrics_config_store_utilization_metrics",
+        "--no-metrics_config_store_plots",
+        "--no-metrics_config_enable_chrome_trace",
+        "--no-metrics_config_write_json_trace",
+        "--metrics_config_store_frontier_stage_batch_ledger",
+        "--no-random_forrest_execution_time_predictor_config_enable_dummy_mode",
+        "--random_forrest_execution_time_predictor_config_linear_op_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/linear_op.csv",
+        "--random_forrest_execution_time_predictor_config_atten_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/attention.csv",
+        "--random_forrest_execution_time_predictor_config_moe_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/profile-v4-trace-final/moe.csv",
+        "--random_forrest_execution_time_predictor_config_linear_op_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/linear_op.csv",
+        "--random_forrest_execution_time_predictor_config_atten_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/attention.csv",
+        "--random_forrest_execution_time_predictor_config_moe_kernel_only_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/frozen-kernel-only/moe.csv",
+        "--random_forrest_execution_time_predictor_config_prediction_max_prefill_chunk_size",
+        "8192",
+        "--random_forrest_execution_time_predictor_config_prediction_max_batch_size",
+        "64",
+        "--random_forrest_execution_time_predictor_config_prediction_max_tokens_per_request",
+        "40960",
+        "--random_forrest_execution_time_predictor_config_no_cache",
+        "--random_forrest_execution_time_predictor_config_skip_cpu_overhead_modeling",
+        "--vllm_v1_scheduler_config_num_blocks",
+        "191786",
+        "--vllm_v1_scheduler_config_enable_chunked_prefill",
+        "--random_forrest_execution_time_predictor_config_num_training_job_threads",
+        "4",
+        "--cudagraph_capture_sizes",
+        "1",
+        "2",
+        "4",
+        "8",
+        "16",
+        "24",
+        "32",
+        "40",
+        "48",
+        "56",
+        "64",
+        "--vidur_cc_backend_config_all_reduce_input_file",
+        "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-profiles/measured-allreduce.csv",
+        "--vidur_cc_backend_config_cache_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/cc-cache",
+        "--vidur_cc_backend_config_k_fold_cv_splits",
+        "6",
+        "--vidur_cc_backend_config_num_training_job_threads",
+        "1",
+        "--metrics_config_cache_dir",
+        "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/model-cache"
+      ],
+      "log": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/logs/tp4_mns32.log",
+      "source_command": "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-lo-fixed-pd-cells/sim/fixed-pd/runs/tp4_mns32/tp4/command.json",
+      "source_command_sha256": "fbc7dee55590b415ed1cde8072de835ed155a0c20ba0eb305c3cb22aa8065a51",
+      "usage": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/usage/tp4_mns32.json"
+    }
+  },
+  "collective_curve": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/results/collective-curve.json",
+  "collective_curve_sha256": "f9543649d4ea78f08240bf1284ab74083aa5cf5671ed47e386047f1453300b36",
+  "collective_curve_variant": "drop_mean",
+  "frontier_checkout": "/tmp/frontier-attn-structured-v0",
+  "frontier_commit": "1f8900a4ac64e45754b03d0aa7c1dddab65785cf",
+  "mode": "joint",
+  "model_cache": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/model-cache",
+  "moe_curve": "/home/gahow/phd/aituner/runs/frontier-fused-moe-profile-v0/results/fused-moe-curve.json",
+  "moe_curve_sha256": "b94d65d9d581adefcc6c14ed4920cce6a1136f74f1014737dc3e4249bc8250d2",
+  "python": "/usr/bin/python3",
+  "python_dependency_roots": [
+    "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/python-deps",
+    "/home/gahow/.cache/uv/archive-v0/-_kzErLcPO5nASZFX8b9k",
+    "/home/gahow/.cache/uv/archive-v0/FbaBs_QJ9QKEbQ9V_4aIR",
+    "/home/gahow/.cache/uv/archive-v0/fuHsGXD0Lv_UjFC8yI4-7",
+    "/home/gahow/.cache/uv/archive-v0/jFGdqQLpB1eopfm9VxT3j",
+    "/home/gahow/.cache/uv/archive-v0/YWW6ExSJuPVvv4-qYQTin",
+    "/home/gahow/.cache/uv/archive-v0/3_qxZ5Ll-EpVAGZfbksfe"
+  ],
+  "traces": {
+    "1": {
+      "first_arrival_s": 0.0,
+      "last_arrival_s": 595.348837209302,
+      "requests": 129,
+      "source_request_metrics": "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-lo-fixed-pd-cells/sim/fixed-pd/runs/tp1_mns16/tp1/metrics/qwen3_a3b_30b_moe/online_serving/qwen30_trace_tp1_mns16_tp1/request_metrics.csv",
+      "source_sha256": "0b82e09644a5884fcd10d894b68495daefdabb32b770146c2f9ece37b8469f4f",
+      "trace": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/inputs/tp1-frontier.csv",
+      "trace_sha256": "59dd8996ff879ef94330004104dfdf515b791bce4036576eccc93290e9206dad"
+    },
+    "2": {
+      "first_arrival_s": 0.0,
+      "last_arrival_s": 297.674418604651,
+      "requests": 129,
+      "source_request_metrics": "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-lo-fixed-pd-cells/sim/fixed-pd/runs/tp2_mns16/tp2/metrics/qwen3_a3b_30b_moe/online_serving/qwen30_trace_tp2_mns16_tp2/request_metrics.csv",
+      "source_sha256": "33983081bb20dd5e2053e9e3d13def8732e958150c9b47a8609ba345123f2316",
+      "trace": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/inputs/tp2-frontier.csv",
+      "trace_sha256": "64fc077b38274a76a8279884ac4115836cd1157c95119c64fabac50d81124f69"
+    },
+    "4": {
+      "first_arrival_s": 0.0,
+      "last_arrival_s": 148.837209302326,
+      "requests": 129,
+      "source_request_metrics": "/home/gahow/phd/aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-lo-fixed-pd-cells/sim/fixed-pd/runs/tp4_mns16/tp4/metrics/qwen3_a3b_30b_moe/online_serving/qwen30_trace_tp4_mns16_tp4/request_metrics.csv",
+      "source_sha256": "b36cd383c07b546d2c1f2fac754d5dbb92efd6880316b4233a7aef9fa1115a36",
+      "trace": "/home/gahow/phd/aituner/runs/frontier-collective-joint-v0/counterfactual/joint-r2/inputs/tp4-frontier.csv",
+      "trace_sha256": "adb3d6f3932a44c86c3d9e7cf1e57739594e8c48d19a26b5aa54b37dce0e0c19"
+    }
+  }
+}

runs/frontier-attn-structured-v0/plot_figure_prototype.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+"""Schematic figure frozen before EXP-ATTN-STRUCTURED execution."""
+
+from pathlib import Path
+
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+
+ROOT = Path(__file__).resolve().parent
+SURFACE = "#fcfcfb"
+INK = "#111111"
+MUTED = "#77736c"
+GRID = "#dedbd2"
+RF = "#d95f02"
+STRUCTURED = "#1b75bc"
+
+fig, axes = plt.subplots(1, 2, figsize=(10.8, 4.2), dpi=160)
+fig.patch.set_facecolor(SURFACE)
+
+for ax in axes:
+    ax.set_facecolor(SURFACE)
+    ax.grid(axis="y", color=GRID, linewidth=0.8)
+    ax.set_axisbelow(True)
+    ax.spines[["top", "right"]].set_visible(False)
+    ax.tick_params(colors=MUTED, labelsize=8)
+
+kv = np.array([8, 16, 24], dtype=float)
+actual = np.array([11.97, 19.92, 27.84])
+rf = np.array([9.46, 16.40, 24.52])
+structured_expected = np.array([12.0, 19.9, 27.9])
+axes[0].plot(kv, actual, "o-", color=INK, label="profile actual")
+axes[0].plot(kv, rf, "s--", color=RF, label="current RF")
+axes[0].plot(
+    kv,
+    structured_expected,
+    "^:",
+    color=STRUCTURED,
+    label="structured (expected)",
+)
+axes[0].set_xlabel("KV context (ktok)")
+axes[0].set_ylabel("TP1 q8k attention time (ms)")
+axes[0].set_title("(a) Continuous KV growth", loc="left", fontsize=10)
+axes[0].legend(frameon=False, fontsize=8)
+
+labels = ["TP1\n.00125", "TP1\n.0025", "TP2\n.0025", "TP2\n.005",
+          "TP4\n.0025", "TP4\n.005", "TP4\n.01"]
+x = np.arange(len(labels))
+v5_mean = np.array([-13.5, -17.7, -11.2, -14.1, 2.5, 2.9, -0.2])
+expected = np.array([-5, -8, -9, -11, 3, 3, 0])
+axes[1].axhspan(-15, 15, color=GRID, alpha=0.5)
+axes[1].axhline(0, color=MUTED, linewidth=0.8)
+axes[1].plot(x, v5_mean, "o-", color=RF, label="sim-v5 measured")
+axes[1].plot(x, expected, "s--", color=STRUCTURED, label="H1 expected")
+axes[1].set_xticks(x, labels)
+axes[1].set_ylabel("TTFT mean bias (%)")
+axes[1].set_title("(b) 7-cell trace gate", loc="left", fontsize=10)
+axes[1].legend(frameon=False, fontsize=8)
+
+fig.suptitle(
+    "MOCK / schematic — EXP-ATTN-STRUCTURED (not measured results)",
+    x=0.01,
+    ha="left",
+    color=RF,
+    fontsize=9,
+)
+fig.tight_layout(rect=(0, 0, 1, 0.95))
+fig.savefig(ROOT / "figure-prototype.png", facecolor=SURFACE)
+fig.savefig(ROOT / "figure-prototype.svg", facecolor=SURFACE)
+print(ROOT / "figure-prototype.png")

runs/frontier-attn-structured-v0/results/predictor-ablation.csv

@@ -0,0 +1,10 @@
+candidate,training_rows,grid_fit_mape_pct,single_fit_mape_pct,heldout_context_mape_pct,heldout_context_max_abs_error_pct,prediction_min_ms,q_min_delta_ms,kv_min_delta_ms,monotone_and_nonnegative,tp
+rf_all,29,15.074275515235467,19.842994757611535,44.40139318281943,82.78279487156401,0.06029164119272453,0.0,-4.2841601371801374e-05,False,1
+rf_single,23,11.834992526698676,22.893412972496023,34.45356428541224,62.49334437588834,0.059327708247725125,-0.00022153525203457564,-1.4336001873005433e-05,False,1
+structured_single,23,0.8432511364168856,2.227110646811972,0.841926169535806,2.0408978739639134,0.05679146709541477,0.0,0.0016745062683911627,True,1
+rf_all,29,13.812689689573157,17.879870012606048,44.20143320278334,82.0656368501208,0.06000113548192927,-0.004361070463210395,-0.009949388915300408,False,2
+rf_single,23,10.738962684891058,20.818820413545826,34.84360402100271,66.67253880294443,0.059971319361210015,-0.003811210796127021,-0.009949388915300408,False,2
+structured_single,23,1.518412258922364,6.226292654776418,1.6047958673086566,3.464671475193195,0.05767893331746252,0.0,0.0013929374121726124,True,2
+rf_all,29,14.117015331381916,15.49366794487052,43.53650868837558,81.33508178007524,0.05866772018640992,-0.0009967416035880083,-5.5955198407176e-05,False,4
+rf_single,23,12.07517666989496,17.672692652721008,34.14562332866605,62.31478818862995,0.058213693721655094,-0.0012244979345549661,-8.259841203689389e-05,False,4
+structured_single,23,3.088740104976349,5.969109476280061,3.0103379904473164,5.490598706238697,0.05747733327249683,0.0,0.0012048051417407057,True,4

runs/frontier-attn-structured-v0/results/predictor-ablation.json

@@ -0,0 +1,149 @@
+{
+  "schema": "frontier-attn-structured-ablation-v1",
+  "profile": "/home/gahow/phd/aituner/runs/frontier-prefill-kvgrowth-fix-v0/profiles/profile-v5-kvgrowth/attention.csv",
+  "frontier_checkout": "/tmp/frontier-attn-structured-v0",
+  "data_audit": {
+    "tp1": {
+      "standard_rows": 29,
+      "single_request_rows": 23,
+      "target_grid_rows": 10,
+      "duplicate_feature_groups": 4
+    },
+    "tp2": {
+      "standard_rows": 29,
+      "single_request_rows": 23,
+      "target_grid_rows": 10,
+      "duplicate_feature_groups": 4
+    },
+    "tp4": {
+      "standard_rows": 29,
+      "single_request_rows": 23,
+      "target_grid_rows": 10,
+      "duplicate_feature_groups": 4
+    }
+  },
+  "results": [
+    {
+      "candidate": "rf_all",
+      "training_rows": 29,
+      "grid_fit_mape_pct": 15.074275515235467,
+      "single_fit_mape_pct": 19.842994757611535,
+      "heldout_context_mape_pct": 44.40139318281943,
+      "heldout_context_max_abs_error_pct": 82.78279487156401,
+      "prediction_min_ms": 0.06029164119272453,
+      "q_min_delta_ms": 0.0,
+      "kv_min_delta_ms": -4.2841601371801374e-05,
+      "monotone_and_nonnegative": false,
+      "tp": 1
+    },
+    {
+      "candidate": "rf_single",
+      "training_rows": 23,
+      "grid_fit_mape_pct": 11.834992526698676,
+      "single_fit_mape_pct": 22.893412972496023,
+      "heldout_context_mape_pct": 34.45356428541224,
+      "heldout_context_max_abs_error_pct": 62.49334437588834,
+      "prediction_min_ms": 0.059327708247725125,
+      "q_min_delta_ms": -0.00022153525203457564,
+      "kv_min_delta_ms": -1.4336001873005433e-05,
+      "monotone_and_nonnegative": false,
+      "tp": 1
+    },
+    {
+      "candidate": "structured_single",
+      "training_rows": 23,
+      "grid_fit_mape_pct": 0.8432511364168856,
+      "single_fit_mape_pct": 2.227110646811972,
+      "heldout_context_mape_pct": 0.841926169535806,
+      "heldout_context_max_abs_error_pct": 2.0408978739639134,
+      "prediction_min_ms": 0.05679146709541477,
+      "q_min_delta_ms": 0.0,
+      "kv_min_delta_ms": 0.0016745062683911627,
+      "monotone_and_nonnegative": true,
+      "tp": 1
+    },
+    {
+      "candidate": "rf_all",
+      "training_rows": 29,
+      "grid_fit_mape_pct": 13.812689689573157,
+      "single_fit_mape_pct": 17.879870012606048,
+      "heldout_context_mape_pct": 44.20143320278334,
+      "heldout_context_max_abs_error_pct": 82.0656368501208,
+      "prediction_min_ms": 0.06000113548192927,
+      "q_min_delta_ms": -0.004361070463210395,
+      "kv_min_delta_ms": -0.009949388915300408,
+      "monotone_and_nonnegative": false,
+      "tp": 2
+    },
+    {
+      "candidate": "rf_single",
+      "training_rows": 23,
+      "grid_fit_mape_pct": 10.738962684891058,
+      "single_fit_mape_pct": 20.818820413545826,
+      "heldout_context_mape_pct": 34.84360402100271,
+      "heldout_context_max_abs_error_pct": 66.67253880294443,
+      "prediction_min_ms": 0.059971319361210015,
+      "q_min_delta_ms": -0.003811210796127021,
+      "kv_min_delta_ms": -0.009949388915300408,
+      "monotone_and_nonnegative": false,
+      "tp": 2
+    },
+    {
+      "candidate": "structured_single",
+      "training_rows": 23,
+      "grid_fit_mape_pct": 1.518412258922364,
+      "single_fit_mape_pct": 6.226292654776418,
+      "heldout_context_mape_pct": 1.6047958673086566,
+      "heldout_context_max_abs_error_pct": 3.464671475193195,
+      "prediction_min_ms": 0.05767893331746252,
+      "q_min_delta_ms": 0.0,
+      "kv_min_delta_ms": 0.0013929374121726124,
+      "monotone_and_nonnegative": true,
+      "tp": 2
+    },
+    {
+      "candidate": "rf_all",
+      "training_rows": 29,
+      "grid_fit_mape_pct": 14.117015331381916,
+      "single_fit_mape_pct": 15.49366794487052,
+      "heldout_context_mape_pct": 43.53650868837558,
+      "heldout_context_max_abs_error_pct": 81.33508178007524,
+      "prediction_min_ms": 0.05866772018640992,
+      "q_min_delta_ms": -0.0009967416035880083,
+      "kv_min_delta_ms": -5.5955198407176e-05,
+      "monotone_and_nonnegative": false,
+      "tp": 4
+    },
+    {
+      "candidate": "rf_single",
+      "training_rows": 23,
+      "grid_fit_mape_pct": 12.07517666989496,
+      "single_fit_mape_pct": 17.672692652721008,
+      "heldout_context_mape_pct": 34.14562332866605,
+      "heldout_context_max_abs_error_pct": 62.31478818862995,
+      "prediction_min_ms": 0.058213693721655094,
+      "q_min_delta_ms": -0.0012244979345549661,
+      "kv_min_delta_ms": -8.259841203689389e-05,
+      "monotone_and_nonnegative": false,
+      "tp": 4
+    },
+    {
+      "candidate": "structured_single",
+      "training_rows": 23,
+      "grid_fit_mape_pct": 3.088740104976349,
+      "single_fit_mape_pct": 5.969109476280061,
+      "heldout_context_mape_pct": 3.0103379904473164,
+      "heldout_context_max_abs_error_pct": 5.490598706238697,
+      "prediction_min_ms": 0.05747733327249683,
+      "q_min_delta_ms": 0.0,
+      "kv_min_delta_ms": 0.0012048051417407057,
+      "monotone_and_nonnegative": true,
+      "tp": 4
+    }
+  ],
+  "checks": {
+    "heldout_context_mape_le_5pct": true,
+    "monotone_and_nonnegative": true,
+    "profile_gate": true
+  }
+}

runs/frontier-attn-structured-v0/results/trace-verdict.csv

@@ -0,0 +1,85 @@
+cell,metric,quantile,old_bias,new_bias,abs_bias_delta_pp,validity
+tp1_rho0p00125,ttft,mean,-0.13461915993830945,-0.06328385143753473,-7.133530850077471,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,ttft,p50,-0.1888032883165517,-0.1117698463137391,-7.7033442002812595,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,ttft,p90,-0.23223319000679374,-0.0884727580604037,-14.376043194639005,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,ttft,p99,-0.16824856840168442,-0.08585570369326061,-8.239286470842382,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,tpot,mean,0.1305653876178269,0.14303898259628217,1.2473594978455265,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,tpot,p50,0.16837991778063455,0.1688780144991309,0.04980967184963492,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,tpot,p90,0.012855564422932954,0.02293841255516459,1.0082848132231637,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,tpot,p99,-0.08151080694091946,-0.06480973978913974,-1.6701067151779714,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,e2e,mean,0.06539157436639341,0.0838094906426525,1.8417916276259092,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,e2e,p50,0.10867934171755954,0.11142188099086506,0.2742539273305519,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,e2e,p90,0.10946179160934073,0.11872607178176105,0.9264280172420314,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p00125,e2e,p99,-0.0657146472433028,-0.04585643943465822,-1.9858207808644577,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,ttft,mean,-0.17677087458166194,-0.09487906467830536,-8.189180990335657,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,ttft,p50,0.008311436147272566,0.015508635458377175,0.7197199311104608,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,ttft,p90,-0.2608553178054708,-0.129865645852306,-13.098967195316478,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,ttft,p99,-0.22116442296103195,-0.14410151018893788,-7.706291277209407,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,tpot,mean,0.014380733682430142,0.052302860177306544,3.7922126494876403,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,tpot,p50,0.13351145963877706,0.14244154194999165,0.8930082311214588,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,tpot,p90,-0.05456950130438105,0.018069551387063856,-3.649994991731719,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,tpot,p99,-0.23635406197836167,-0.1790206784227079,-5.733338355565376,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,e2e,mean,-0.020164189983441452,0.01563919326555167,-0.4524996717889782,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,e2e,p50,0.07632815851795742,0.10505951594320918,2.8731357425251765,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,e2e,p90,0.049809009042946335,0.08165607475145953,3.1847065708513194,GATE_FAIL_DIAGNOSTIC
+tp1_rho0p0025,e2e,p99,-0.18303183751478602,-0.15333409130015813,-2.96977462146279,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p0025,ttft,mean,-0.11161154024124531,-0.045433491316312524,-6.617804892493279,PASS_SUBCRITICAL
+tp2_rho0p0025,ttft,p50,-0.18420934047220774,-0.0875990583320057,-9.661028214020204,PASS_SUBCRITICAL
+tp2_rho0p0025,ttft,p90,-0.21836264490202395,-0.1142200855839675,-10.414255931805645,PASS_SUBCRITICAL
+tp2_rho0p0025,ttft,p99,-0.17283197594971011,-0.07674087497700505,-9.609110097270507,PASS_SUBCRITICAL
+tp2_rho0p0025,tpot,mean,0.13711499081487563,0.15073709978689778,1.362210897202215,PASS_SUBCRITICAL
+tp2_rho0p0025,tpot,p50,0.17555321305308488,0.18527285925405948,0.9719646200974597,PASS_SUBCRITICAL
+tp2_rho0p0025,tpot,p90,0.0591579975137338,0.07863996768588354,1.9481970172149734,PASS_SUBCRITICAL
+tp2_rho0p0025,tpot,p99,0.1300483675091633,0.1661995397125918,3.6151172203428503,PASS_SUBCRITICAL
+tp2_rho0p0025,e2e,mean,0.10925353865257875,0.12696146154030977,1.7707922887731016,PASS_SUBCRITICAL
+tp2_rho0p0025,e2e,p50,0.1339165600755408,0.14796251184179712,1.404595176625631,PASS_SUBCRITICAL
+tp2_rho0p0025,e2e,p90,0.10666828724664539,0.12767235986604622,2.1004072619400835,PASS_SUBCRITICAL
+tp2_rho0p0025,e2e,p99,-0.07437942975605877,-0.03965040123988098,-3.472902851617779,PASS_SUBCRITICAL
+tp2_rho0p005,ttft,mean,-0.14129871969878843,-0.07082500585057615,-7.047371384821228,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,ttft,p50,-0.2042045530944649,-0.1284088888361358,-7.579566425832909,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,ttft,p90,-0.19003454588767263,-0.111136573344055,-7.889797254361763,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,ttft,p99,-0.19373351009539902,-0.09012556161973535,-10.360794847566366,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,tpot,mean,0.03117337438562004,0.05457336599027876,2.3399991604658723,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,tpot,p50,0.07668249597302182,0.0872512146277895,1.0568718654767675,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,tpot,p90,-0.03724827658429621,0.01132352325010614,-2.5924753334190074,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,tpot,p99,-0.10050436157089844,-0.05948329733667248,-4.102106423422596,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,e2e,mean,0.023867807624916495,0.050164236759908075,2.629642913499158,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,e2e,p50,0.07536668131278851,0.0912493994395978,1.588271812680929,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,e2e,p90,-0.03100470462321266,-0.0004979191794830456,-3.0506785443729614,GATE_FAIL_DIAGNOSTIC
+tp2_rho0p005,e2e,p99,-0.04909581632382212,-0.0014369075488634014,-4.765890877495872,GATE_FAIL_DIAGNOSTIC
+tp4_rho0p0025,ttft,mean,0.025022574738277282,0.07766646565061346,5.2643890912336175,PASS_SUBCRITICAL
+tp4_rho0p0025,ttft,p50,-0.040042171846277425,0.029153379043297147,-1.0888792802980278,PASS_SUBCRITICAL
+tp4_rho0p0025,ttft,p90,-0.056230097634382616,0.025222989748299444,-3.100710788608317,PASS_SUBCRITICAL
+tp4_rho0p0025,ttft,p99,-0.07535478089127973,0.0205549324689376,-5.479984842234213,PASS_SUBCRITICAL
+tp4_rho0p0025,tpot,mean,0.2170232618103144,0.2295170016795841,1.2493739869269715,PASS_SUBCRITICAL
+tp4_rho0p0025,tpot,p50,0.22406751004936917,0.22406940610958842,0.00018960602192474862,PASS_SUBCRITICAL
+tp4_rho0p0025,tpot,p90,0.1725668492041552,0.1795965483385546,0.7029699134399409,PASS_SUBCRITICAL
+tp4_rho0p0025,tpot,p99,0.1603764334794579,0.254199450927048,9.382301744759008,PASS_SUBCRITICAL
+tp4_rho0p0025,e2e,mean,0.18328079455378776,0.19190660476104554,0.8625810207257778,PASS_SUBCRITICAL
+tp4_rho0p0025,e2e,p50,0.2057109615696404,0.21253242300694286,0.6821461437302473,PASS_SUBCRITICAL
+tp4_rho0p0025,e2e,p90,0.1869703879211648,0.19279855916666536,0.5828171245500557,PASS_SUBCRITICAL
+tp4_rho0p0025,e2e,p99,0.14838304065885655,0.15160588346499027,0.3222842806133719,PASS_SUBCRITICAL
+tp4_rho0p005,ttft,mean,0.028648879997638963,0.08356973519379125,5.492085519615229,PASS_SUBCRITICAL
+tp4_rho0p005,ttft,p50,0.028237979190582876,0.09021324737193111,6.197526818134823,PASS_SUBCRITICAL
+tp4_rho0p005,ttft,p90,-0.055961238578361966,-0.0006012940489499138,-5.535994452941205,PASS_SUBCRITICAL
+tp4_rho0p005,ttft,p99,-0.045491187615110146,0.05253708684194205,0.7045899226831902,PASS_SUBCRITICAL
+tp4_rho0p005,tpot,mean,0.1707193936623511,0.1813506819061229,1.0631288243771824,PASS_SUBCRITICAL
+tp4_rho0p005,tpot,p50,0.16851374859025317,0.1743324539248605,0.5818705334607321,PASS_SUBCRITICAL
+tp4_rho0p005,tpot,p90,0.10492621353626864,0.11816822907155744,1.3242015535288796,PASS_SUBCRITICAL
+tp4_rho0p005,tpot,p99,0.31231888769471766,0.3662267201704473,5.390783247572961,PASS_SUBCRITICAL
+tp4_rho0p005,e2e,mean,0.151102823467785,0.1630793421767109,1.197651870892591,PASS_SUBCRITICAL
+tp4_rho0p005,e2e,p50,0.1594213531394918,0.17360527090575292,1.418391776626113,PASS_SUBCRITICAL
+tp4_rho0p005,e2e,p90,0.1266352410718406,0.13652104764058856,0.9885806568747962,PASS_SUBCRITICAL
+tp4_rho0p005,e2e,p99,0.15576537699445703,0.17677450343779522,2.100912644333819,PASS_SUBCRITICAL
+tp4_rho0p01,ttft,mean,-0.0014652143973501086,0.059650620031540064,5.8185405634189955,PASS_SUBCRITICAL
+tp4_rho0p01,ttft,p50,0.22936601881498542,0.24159106387124998,1.2225045056264565,PASS_SUBCRITICAL
+tp4_rho0p01,ttft,p90,-0.0646093465409875,-0.011548419893895705,-5.30609266470918,PASS_SUBCRITICAL
+tp4_rho0p01,ttft,p99,-0.09621678853313553,-0.015475807392170575,-8.074098114096495,PASS_SUBCRITICAL
+tp4_rho0p01,tpot,mean,0.06675609059150077,0.10596101263201793,3.9204922040517163,PASS_SUBCRITICAL
+tp4_rho0p01,tpot,p50,0.08477302587551214,0.09703754188844527,1.2264516012933129,PASS_SUBCRITICAL
+tp4_rho0p01,tpot,p90,0.0379183273767328,0.08369800201750718,4.577967464077439,PASS_SUBCRITICAL
+tp4_rho0p01,tpot,p99,-0.010663954751357074,0.06667813160571406,5.601417685435699,PASS_SUBCRITICAL
+tp4_rho0p01,e2e,mean,0.07212904317306096,0.09777288053702092,2.564383736395996,PASS_SUBCRITICAL
+tp4_rho0p01,e2e,p50,0.12069215463307655,0.1388377217196832,1.8145567086606653,PASS_SUBCRITICAL
+tp4_rho0p01,e2e,p90,0.03317293927357903,0.058060760567474216,2.4887821293895183,PASS_SUBCRITICAL
+tp4_rho0p01,e2e,p99,0.0338378271163308,0.06380410696893425,2.996627985260345,PASS_SUBCRITICAL

runs/frontier-attn-structured-v0/run_replay.py

@@ -0,0 +1,126 @@
+#!/usr/bin/env python3
+"""Replay one real-trace cell with the structured-attention experiment commit."""
+
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent
+REPO = ROOT.parents[1]
+S3_REAL = REPO / "runs/frontier-s3-real-v0"
+BASE_REFERENCE = (
+    REPO
+    / "runs/frontier-collective-joint-v0/counterfactual/joint-r2/manifest.json"
+)
+BASE_COMMIT = "deadc4a321f0baaa534c6ebd17f974123733cdc2"
+EXPERIMENT_COMMIT = "1f8900a4ac64e45754b03d0aa7c1dddab65785cf"
+PATCH = ROOT / "0001-Experiment-with-structured-attention-prefill-predict.patch"
+
+
+def load_s3_module():
+    spec = importlib.util.spec_from_file_location(
+        "s3_prefix_replay", S3_REAL / "run_frontier_prefix_replay.py"
+    )
+    module = importlib.util.module_from_spec(spec)
+    sys.path.insert(0, str(S3_REAL))
+    spec.loader.exec_module(module)
+    return module
+
+
+def git(checkout: Path, *args: str) -> str:
+    return subprocess.check_output(
+        ["git", "-C", str(checkout), *args], text=True
+    ).strip()
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--trace", type=Path, required=True)
+    parser.add_argument("--output-root", type=Path, required=True)
+    parser.add_argument(
+        "--config",
+        choices=("tp4_mns16", "tp2_mns16", "tp1_mns16"),
+        required=True,
+    )
+    parser.add_argument("--label", required=True)
+    parser.add_argument("--max-tokens", type=int, required=True)
+    parser.add_argument("--duration-s", type=float)
+    parser.add_argument("--cache-root", type=Path, required=True)
+    parser.add_argument(
+        "--frontier-checkout",
+        type=Path,
+        default=Path("/tmp/frontier-attn-structured-v0"),
+    )
+    parser.add_argument(
+        "--attention-profile",
+        type=Path,
+        default=REPO
+        / "runs/frontier-prefill-kvgrowth-fix-v0/profiles/"
+        "profile-v5-kvgrowth/attention.csv",
+    )
+    args = parser.parse_args()
+
+    frontier = args.frontier_checkout.resolve()
+    profile = args.attention_profile.resolve()
+    if git(frontier, "rev-parse", "HEAD") != EXPERIMENT_COMMIT:
+        raise SystemExit(f"unexpected experiment checkout HEAD: {frontier}")
+    if git(frontier, "rev-parse", "HEAD^") != BASE_COMMIT:
+        raise SystemExit("experiment commit is not directly based on frozen Frontier")
+    if git(frontier, "status", "--porcelain"):
+        raise SystemExit("experiment Frontier checkout must be clean")
+    if not profile.is_file():
+        raise SystemExit(f"attention profile missing: {profile}")
+
+    reference = json.loads(BASE_REFERENCE.read_text())
+    reference["frontier_checkout"] = str(frontier)
+    reference["frontier_commit"] = EXPERIMENT_COMMIT
+    generated_reference = ROOT / "frontier-reference.json"
+    generated_reference.write_text(json.dumps(reference, indent=2))
+
+    module = load_s3_module()
+    module.REFERENCE = generated_reference
+    module.EXPECTED_FRONTIER_COMMIT = EXPERIMENT_COMMIT
+    original_replace = module.replace_flag
+
+    def replace_and_override(argv: list[str], flag: str, value: str) -> None:
+        original_replace(argv, flag, value)
+        if flag.endswith("trace_file"):
+            atten_flag = (
+                "--random_forrest_execution_time_predictor_config_atten_input_file"
+            )
+            original_replace(argv, atten_flag, str(profile))
+            no_cache = (
+                "--random_forrest_execution_time_predictor_config_no_cache"
+            )
+            if no_cache in argv:
+                argv.remove(no_cache)
+
+    module.replace_flag = replace_and_override
+    module.parse_args = lambda: args
+    module.main()
+
+    manifest_path = args.output_root / "manifest.json"
+    manifest = json.loads(manifest_path.read_text())
+    manifest.update(
+        {
+            "schema": "frontier-attn-structured-replay-v1",
+            "frontier_base_commit": BASE_COMMIT,
+            "frontier_experiment_commit": EXPERIMENT_COMMIT,
+            "frontier_patch": str(PATCH.resolve()),
+            "frontier_patch_sha256": module.sha256(PATCH),
+            "attention_profile_override": str(profile),
+            "attention_profile_sha256": module.sha256(profile),
+            "model_cache_enabled": True,
+        }
+    )
+    manifest_path.write_text(json.dumps(manifest, indent=2))
+    print(f"structured replay done: {args.output_root}")
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-code-trace-v0/README.md

@@ -0,0 +1,118 @@
+# Frontier code-trace campaign handoff
+
+本目录已经准备好无 GPU 的 data preflight、512→16 参数化映射、prefill-only 转换和 `max_model_len` 显式适配。当前没有启动或探测 `dash1`–`dash4`。
+
+完整设计与 gate 见 [`experiment-card.md`](experiment-card.md)。
+
+## 当前已知阻塞
+
+本机 `/home/gahow/ali-trace/trace-glm5.1-formatted/` 不存在。仓库历史记录的远端路径是：
+
+```text
+/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/
+```
+
+有机器后先确认用户给出的 `~/ali-trace/...` 是否解析到同一目录，再运行以下命令。
+
+## 1. 审计所有 1h+ code source
+
+在持有 trace 的机器、repo 根目录执行：
+
+```bash
+python3 runs/frontier-code-trace-v0/audit_code_trace.py \
+  --trace-root ~/ali-trace/trace-glm5.1-formatted \
+  --output runs/frontier-code-trace-v0/inputs/code-audit.json
+```
+
+如果目录里混有非 request JSONL，先只读列举文件，再用多个 `--source` 显式指定。审计输出必须满足：
+
+```text
+data_gate = PASS
+selected.hash_contract.exact_source_block_size != null
+max_model_len_recommendation != null
+selected.selected_window_stats.max_model_len_coverage[推荐值].coverage = 1.0
+```
+
+旧记录预计 source block size 为 512、推荐 max model len 为 131072，但禁止把这两个值写死为实验事实。
+
+## 2. 物化稳定窗口
+
+```bash
+python3 runs/frontier-code-trace-v0/prepare_code_window.py \
+  --audit runs/frontier-code-trace-v0/inputs/code-audit.json \
+  --output-root runs/frontier-code-trace-v0/inputs/code-window
+```
+
+输出是 60–75min `code-raw-window.jsonl` 和 manifest。source 文件不修改。
+
+## 3. 生成 P+D paired trace
+
+若没有 prompt sidecar，先生成 shape/prefix-faithful synthetic prompts：
+
+```bash
+python3 runs/frontier-s3-real-v0/remap_hash_blocks.py \
+  --input runs/frontier-code-trace-v0/inputs/code-window/code-raw-window.jsonl \
+  --output-root runs/frontier-code-trace-v0/inputs/code-pd-rho-max \
+  --source-block-size 512 \
+  --workload-mode prefill_decode \
+  --rho 1.0 \
+  --max-total-tokens 131072 \
+  --validate-parents
+```
+
+命令中的 `512` 和 `131072` 必须替换为 audit manifest 值。若存在对齐 prompt sidecar，加 `--prompt` 与 `--tokenizer`，并要求 synthetic fallback 为 0。
+
+正式 rho 不能直接用 1.0；先从最大 remap cache 按 session-coherent `sampling_u` 过滤，分别标定 low/mid/near-knee。
+
+## 4. 生成 prefill-only paired trace
+
+对 chat/code 使用同一个转换接口：
+
+```bash
+python3 runs/frontier-s3-real-v0/remap_hash_blocks.py \
+  --input INPUT_WINDOW.jsonl \
+  --output-root OUTPUT_ROOT \
+  --source-block-size SOURCE_BLOCK_SIZE \
+  --workload-mode prefill_only \
+  --rho RHO \
+  --max-total-tokens MAX_MODEL_LEN \
+  --validate-parents
+```
+
+该模式会同时把 Frontier `num_decode_tokens`、real request `min/max_tokens` 和 remapped row 的 `output_length` 固定为 1。
+
+## 5. max-model-len 真机 gate
+
+现有 real runner 新增了三个显式环境变量，chat 默认行为不变：
+
+```bash
+MAX_MODEL_LEN=131072 \
+TRACE_INPUT_ROOT=/absolute/path/to/materialized/code-cell \
+ALLOW_SYNTHETIC_PROMPTS=true \
+OUTPUT_ROOT=/absolute/path/to/new/output \
+bash runs/frontier-s3-real-v0/run_full_real.sh RHO_LABEL tp4_mns16 1 PORT
+```
+
+- `MAX_MODEL_LEN` 必须等于 manifest 推荐值；
+- `TRACE_INPUT_ROOT` 内必须有 `real_requests.jsonl` 和 `manifest.json`；
+- synthetic prompt 默认拒绝，只有在 experiment card 明确降级 claim 后才设为 `true`；
+- runner 会在启动前扫描 paired requests，若任何 `ISL+OSL` 超 cap 立即失败。
+
+正式 full job 前，先按 experiment card 的 G4 补 32k–128k attention profile，再做 TP4→TP2 的 p50/p99/max 单请求与 5min canary。
+
+## 6. decode-only
+
+当前 materializer 故意不提供 `decode_only` 选项。严格 decode-only 需要 initial-KV state，而不是把 prompt 改短。只有 real `DecodeBenchConnector`（或等价能力）与 Frontier initial-KV contract 都通过 G7 后，才创建 decode-only jobs。
+
+## 本地验证
+
+```bash
+python3 -m unittest -v \
+  runs/frontier-code-trace-v0/test_code_trace_preflight.py \
+  runs/frontier-s3-real-v0/test_remap_hash_blocks.py \
+  runs/frontier-s3-real-v0/test_select_chat_window.py
+python3 -m py_compile \
+  runs/frontier-code-trace-v0/*.py \
+  runs/frontier-s3-real-v0/*.py
+bash -n runs/frontier-s3-real-v0/run_full_real.sh
+```

runs/frontier-code-trace-v0/audit_code_trace.py

@@ -0,0 +1,361 @@
+#!/usr/bin/env python3
+"""Audit long code traces before choosing a replay window and max model length."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import statistics
+from collections import Counter
+from pathlib import Path
+from typing import Any, Iterable, Sequence
+
+
+BLOCK_SIZE_CANDIDATES = (16, 32, 64, 128, 256, 512, 1024)
+MAX_MODEL_LEN_CANDIDATES = (40960, 65536, 98304, 131072, 262144)
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--trace-root", type=Path)
+    parser.add_argument("--source", type=Path, action="append")
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--min-minutes", type=int, default=60)
+    parser.add_argument("--max-minutes", type=int, default=75)
+    parser.add_argument("--bin-seconds", type=int, default=60)
+    parser.add_argument("--max-acceptable-gap-s", type=float, default=5.0)
+    parser.add_argument("--model-position-limit", type=int, default=262144)
+    return parser.parse_args()
+
+
+def percentile(values: Sequence[int | float], fraction: float) -> float | None:
+    if not values:
+        return None
+    ordered = sorted(float(value) for value in values)
+    position = (len(ordered) - 1) * fraction
+    lower = math.floor(position)
+    upper = math.ceil(position)
+    if lower == upper:
+        return ordered[lower]
+    return ordered[lower] * (upper - position) + ordered[upper] * (position - lower)
+
+
+def distribution(values: Sequence[int | float]) -> dict[str, int | float | None]:
+    return {
+        "count": len(values),
+        "min": min(values) if values else None,
+        "p50": percentile(values, 0.50),
+        "p90": percentile(values, 0.90),
+        "p95": percentile(values, 0.95),
+        "p99": percentile(values, 0.99),
+        "max": max(values) if values else None,
+        "mean": statistics.fmean(values) if values else None,
+    }
+
+
+def parse_hash_ids(value: Any) -> list[Any]:
+    if isinstance(value, list):
+        return value
+    if isinstance(value, str):
+        stripped = value.strip()
+        if not stripped:
+            return []
+        if stripped.startswith("["):
+            decoded = json.loads(stripped)
+            if not isinstance(decoded, list):
+                raise ValueError("hash_ids JSON must decode to a list")
+            return decoded
+        delimiter = "|" if "|" in stripped else ","
+        return [part for part in stripped.split(delimiter) if part.strip()]
+    if value is None:
+        return []
+    return [value]
+
+
+def iter_jsonl(path: Path) -> Iterable[tuple[int, dict[str, Any]]]:
+    with path.open() as stream:
+        for line_number, line in enumerate(stream, 1):
+            if not line.strip():
+                continue
+            row = json.loads(line)
+            if not isinstance(row, dict):
+                raise ValueError(f"{path}:{line_number}: row must be an object")
+            yield line_number, row
+
+
+def choose_window(
+    *,
+    counts: Sequence[int],
+    max_gaps: Sequence[float],
+    first_timestamp: float,
+    min_minutes: int,
+    max_minutes: int,
+    bin_seconds: int,
+    max_acceptable_gap_s: float,
+) -> dict[str, Any] | None:
+    candidates = []
+    for minutes in range(max_minutes, min_minutes - 1, -1):
+        bins = math.ceil(minutes * 60 / bin_seconds)
+        for start_bin in range(0, len(counts) - bins + 1):
+            selected = counts[start_bin : start_bin + bins]
+            mean = statistics.fmean(selected)
+            cv = statistics.pstdev(selected) / mean if mean else math.inf
+            max_gap = max(max_gaps[start_bin : start_bin + bins], default=0.0)
+            candidates.append(
+                {
+                    "_score": (
+                        max_gap > max_acceptable_gap_s,
+                        cv,
+                        max_gap,
+                        -minutes,
+                        start_bin,
+                    ),
+                    "start_bin": start_bin,
+                    "minutes": minutes,
+                    "count_mean_per_bin": mean,
+                    "count_cv": cv,
+                    "count_min_per_bin": min(selected),
+                    "count_max_per_bin": max(selected),
+                    "max_gap_s": max_gap,
+                }
+            )
+    if not candidates:
+        return None
+    chosen = min(candidates, key=lambda item: item["_score"])
+    chosen.pop("_score")
+    chosen["start_timestamp"] = first_timestamp + chosen["start_bin"] * bin_seconds
+    chosen["end_timestamp"] = chosen["start_timestamp"] + chosen["minutes"] * 60
+    return chosen
+
+
+def scan_source(path: Path, args: argparse.Namespace) -> dict[str, Any]:
+    rows = 0
+    first_timestamp = None
+    last_timestamp = None
+    previous_timestamp = None
+    counts: Counter[int] = Counter()
+    max_gaps: dict[int, float] = {}
+    input_lengths: list[int] = []
+    output_lengths: list[int] = []
+    total_lengths: list[int] = []
+    hash_rows = 0
+    hash_matches = Counter()
+    prompt_rows = 0
+    sampling_rows = 0
+    schema_keys: Counter[str] = Counter()
+    for line_number, row in iter_jsonl(path):
+        missing = [
+            key
+            for key in ("timestamp", "input_length", "output_length")
+            if key not in row
+        ]
+        if missing:
+            raise ValueError(f"{path}:{line_number}: missing required fields {missing}")
+        timestamp = float(row["timestamp"])
+        if first_timestamp is None:
+            first_timestamp = timestamp
+        if previous_timestamp is not None and timestamp < previous_timestamp:
+            raise ValueError(
+                f"{path}:{line_number}: timestamp {timestamp} < {previous_timestamp}"
+            )
+        bin_index = math.floor((timestamp - first_timestamp) / args.bin_seconds)
+        counts[bin_index] += 1
+        if previous_timestamp is not None:
+            previous_bin = math.floor(
+                (previous_timestamp - first_timestamp) / args.bin_seconds
+            )
+            max_gaps[previous_bin] = max(
+                max_gaps.get(previous_bin, 0.0),
+                timestamp - previous_timestamp,
+            )
+        input_tokens = int(row["input_length"])
+        output_tokens = max(1, int(row["output_length"]))
+        if input_tokens <= 0:
+            raise ValueError(f"{path}:{line_number}: input_length must be positive")
+        input_lengths.append(input_tokens)
+        output_lengths.append(output_tokens)
+        total_lengths.append(input_tokens + output_tokens)
+        hashes = parse_hash_ids(row.get("hash_ids"))
+        if hashes:
+            hash_rows += 1
+            for block_size in BLOCK_SIZE_CANDIDATES:
+                if len(hashes) == math.ceil(input_tokens / block_size):
+                    hash_matches[block_size] += 1
+        prompt_rows += int(
+            isinstance(row.get("prompt"), (str, list)) and bool(row.get("prompt"))
+        )
+        sampling_rows += int("sampling_u" in row)
+        schema_keys.update(row.keys())
+        rows += 1
+        previous_timestamp = timestamp
+        last_timestamp = timestamp
+    if not rows or first_timestamp is None or last_timestamp is None:
+        raise ValueError(f"{path}: empty trace")
+    total_bins = math.floor((last_timestamp - first_timestamp) / args.bin_seconds) + 1
+    chosen = choose_window(
+        counts=[counts[index] for index in range(total_bins)],
+        max_gaps=[max_gaps.get(index, 0.0) for index in range(total_bins)],
+        first_timestamp=first_timestamp,
+        min_minutes=args.min_minutes,
+        max_minutes=args.max_minutes,
+        bin_seconds=args.bin_seconds,
+        max_acceptable_gap_s=args.max_acceptable_gap_s,
+    )
+    return {
+        "source": str(path.resolve()),
+        "rows": rows,
+        "first_timestamp": first_timestamp,
+        "last_timestamp": last_timestamp,
+        "span_s": last_timestamp - first_timestamp,
+        "request_rate_per_s": rows / max(last_timestamp - first_timestamp, 1.0),
+        "input_length": distribution(input_lengths),
+        "output_length": distribution(output_lengths),
+        "total_length": distribution(total_lengths),
+        "over_max_model_len": {
+            str(limit): {
+                "requests": sum(value > limit for value in total_lengths),
+                "fraction": sum(value > limit for value in total_lengths) / rows,
+            }
+            for limit in MAX_MODEL_LEN_CANDIDATES
+        },
+        "hash_contract": {
+            "rows_with_hash_ids": hash_rows,
+            "candidate_exact_match_rows": {
+                str(size): hash_matches[size] for size in BLOCK_SIZE_CANDIDATES
+            },
+            "exact_source_block_size": next(
+                (
+                    size
+                    for size in BLOCK_SIZE_CANDIDATES
+                    if hash_rows and hash_matches[size] == hash_rows
+                ),
+                None,
+            ),
+        },
+        "prompt_rows": prompt_rows,
+        "sampling_u_rows": sampling_rows,
+        "schema_field_counts": dict(sorted(schema_keys.items())),
+        "stable_window": chosen,
+    }
+
+
+def scan_window(source: Path, window: dict[str, Any]) -> dict[str, Any]:
+    start = float(window["start_timestamp"])
+    end = float(window["end_timestamp"])
+    inputs: list[int] = []
+    outputs: list[int] = []
+    totals: list[int] = []
+    for _, row in iter_jsonl(source):
+        timestamp = float(row["timestamp"])
+        if timestamp < start:
+            continue
+        if timestamp >= end:
+            break
+        input_tokens = int(row["input_length"])
+        output_tokens = max(1, int(row["output_length"]))
+        inputs.append(input_tokens)
+        outputs.append(output_tokens)
+        totals.append(input_tokens + output_tokens)
+    return {
+        "requests": len(totals),
+        "input_length": distribution(inputs),
+        "output_length": distribution(outputs),
+        "total_length": distribution(totals),
+        "max_model_len_coverage": {
+            str(limit): {
+                "covered_requests": sum(value <= limit for value in totals),
+                "excluded_requests": sum(value > limit for value in totals),
+                "coverage": sum(value <= limit for value in totals) / len(totals),
+            }
+            for limit in MAX_MODEL_LEN_CANDIDATES
+        },
+    }
+
+
+def resolve_sources(args: argparse.Namespace) -> list[Path]:
+    if args.source:
+        return [path.resolve() for path in args.source]
+    if args.trace_root is None:
+        raise ValueError("provide --trace-root or one or more --source")
+    sources = sorted(
+        path.resolve()
+        for path in args.trace_root.glob("*.jsonl")
+        if "prompt" not in path.stem.lower()
+    )
+    if not sources:
+        raise FileNotFoundError(f"no non-prompt JSONL files under {args.trace_root}")
+    return sources
+
+
+def main() -> None:
+    args = parse_args()
+    if not 0 < args.min_minutes <= args.max_minutes:
+        raise ValueError("require 0 < min_minutes <= max_minutes")
+    sources = resolve_sources(args)
+    files = [scan_source(path, args) for path in sources]
+    eligible = [item for item in files if item["stable_window"] is not None]
+    if not eligible:
+        chosen = None
+        data_gate = "BLOCKED_NO_1H_WINDOW"
+    else:
+        chosen = min(
+            eligible,
+            key=lambda item: (
+                item["stable_window"]["max_gap_s"] > args.max_acceptable_gap_s,
+                item["stable_window"]["count_cv"],
+                -item["stable_window"]["minutes"],
+                item["source"],
+            ),
+        )
+        chosen["selected_window_stats"] = scan_window(
+            Path(chosen["source"]), chosen["stable_window"]
+        )
+        exact_block_size = chosen["hash_contract"]["exact_source_block_size"]
+        max_total = chosen["selected_window_stats"]["total_length"]["max"]
+        data_gate = (
+            "PASS"
+            if exact_block_size is not None
+            and max_total is not None
+            and max_total <= args.model_position_limit
+            else "BLOCKED_HASH_OR_POSITION_CONTRACT"
+        )
+    recommendation = None
+    if chosen is not None:
+        maximum = chosen["selected_window_stats"]["total_length"]["max"]
+        recommendation = next(
+            (
+                limit
+                for limit in MAX_MODEL_LEN_CANDIDATES
+                if maximum <= limit <= args.model_position_limit
+            ),
+            None,
+        )
+    payload = {
+        "schema": "frontier-code-trace-audit-v1",
+        "trace_root": str(args.trace_root.resolve()) if args.trace_root else None,
+        "sources": [str(path) for path in sources],
+        "window_policy": {
+            "min_minutes": args.min_minutes,
+            "max_minutes": args.max_minutes,
+            "bin_seconds": args.bin_seconds,
+            "max_acceptable_gap_s": args.max_acceptable_gap_s,
+            "selection": "lowest density CV after rejecting anomalous-gap windows",
+        },
+        "model_position_limit": args.model_position_limit,
+        "files": files,
+        "selected": chosen,
+        "max_model_len_recommendation": recommendation,
+        "data_gate": data_gate,
+        "runtime_gate": (
+            "PENDING: vLLM startup must prove enough KV blocks and nonzero "
+            "max concurrency at the recommended max_model_len for each TP"
+        ),
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    print(json.dumps({"data_gate": data_gate, "output": str(args.output)}))
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-code-trace-v0/experiment-card.md

@@ -0,0 +1,204 @@
+# EXP-CODE-TRACE：从 chat 1h trace 扩展到 code 与 phase-separated replay
+
+> **状态：READY_FOR_DATA PREFLIGHT，未启动 GPU。** 当前只完成本地适配与实验冻结；`dash1`--`dash4` 有整机空闲后按本文 gate 顺序推进。禁止使用 `dash0`。
+
+## 目标与成功定义
+
+当前 1h+ 证据只覆盖 Qwen3-30B-A3B 的生产 chat trace、prefill+decode（P+D）和亚临界负载。本 campaign 分两步扩展：
+
+1. **主任务：** 使用 `~/ali-trace/trace-glm5.1-formatted/` 中的 1h+ code trace，先完成 P+D real-vs-Frontier 回放；
+2. **后续 phase matrix：** 对 chat/code 都补 prefill-only 和严格 decode-only。
+
+本轮不是只看“能否跑完”。每个正式 cell 必须满足：同一 request vector、同一 arrival、同一 token shape、同一 prefix/initial-KV 合约、real 零失败、无持续 backlog，并同时报告 TTFT/TPOT/E2E、queue/batch、KV/prefix state 与 5min 分窗漂移。
+
+## 三种 workload mode 的冻结定义
+
+| Mode | 保留 | 改写 | 主指标 | 明确不声称 |
+|---|---|---|---|---|
+| P+D | 原 ISL/OSL、arrival、session/prefix | 仅做 source block→16-token runtime block 映射 | TTFT、TPOT、E2E、hit ratio、batch/queue | 不代表 PD 分离 |
+| prefill-only | 原 ISL、arrival、session/prefix | OSL 固定为 1，real `min_tokens=max_tokens=1`，sim decode tokens=1 | TTFT、prefill service/tokens/s、prefix hit、queue | TPOT 不定义；1-token decode 只用于完成请求 |
+| strict decode-only | 原 OSL、context length、arrival burst | arrival 定义为 **KV-ready time**；请求进入 decode 时已有 ISL 长度的 initial KV | TPOT、decode tokens/s、batch/queue、preemption | 不包含 prefill 与 KV transfer latency，不把短 prompt proxy 称为 decode-only |
+
+strict decode-only 必须同时具备：
+
+- real：vLLM `DecodeBenchConnector`（或等价、经验证的 initial-KV 注入）；
+- sim：Frontier request 在 admission 时已拥有相同长度/块布局的 computed KV；
+- 两侧都不在 decode critical path 重做 prefill；
+- arrival 以 KV-ready time 对齐。若只保留原 trace 的相对到达形状，结论限定为 decode engine compute/scheduling fidelity。
+
+在该合约完成前，只允许跑并标注为 **decode-dominant proxy**，不能进入 strict decode-only 结果表。
+
+## 为什么 code P+D 不能直接复用 chat 配置
+
+已知历史探查显示 code trace ISL p90 约 81.9k，约 32.6% 请求超过旧 `40960` 上限；真实数值必须由本 campaign 重新审计。至少有四个独立适配面：
+
+1. **Serving cap：** `max_model_len` 必须覆盖 `ISL+OSL`，不能只看 ISL，也不能静默丢掉超长请求；
+2. **KV capacity：** Qwen3-30B 模型 position limit 为 262144，但 TP1/2/4 在 H20 上是否有足够 KV blocks 是 runtime gate，不由 config.json 自动保证；
+3. **Prefix block：** code source hash 预计为 512-token block，chat harness 原先固定 64→16；
+4. **Profile support：** 当前修复后的 attention profile 只覆盖到约 32k KV context。即使 vLLM 能跑 128k，Frontier 对 32k–128k 仍会出 profile 支撑域；在补 long-context 网格前只能做诊断 replay，不能做 fidelity claim。
+
+## Hypotheses
+
+- **H-code-generalizes：** 在补齐 long-context profile 支撑域后，code P+D 的 TTFT/TPOT/E2E 分布统计偏差仍处于当前 chat 量级，且 1h 残差不发散。
+- **H-longctx-gap：** code 的主要新增 gap 来自 32k 以上 KV-context 外推；补到 trace p99/max 对应的网格后，TTFT bias 随 ISL 的二次项显著收敛。
+- **H-phase-specific：** prefill-only 主要暴露 long-context/profile gap；strict decode-only 主要暴露 batch-conditioned whole-layer service 与 scheduler fixed-point gap。二者不能用 P+D 的误差抵消来互相证明准确。
+
+## Preflight gates（按顺序，任一失败即停止后续真机矩阵）
+
+### G0：数据位置与 provenance
+
+- 只读列举 `trace-glm5.1-formatted/*.jsonl`，记录文件大小与 SHA256；
+- 确认至少两个独立日期段：一个作为 development，一个 held-out；
+- 本机当前没有该目录；仓库历史记录的远端位置为
+  `/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/`。恢复机器后先确认 `~/ali-trace/...` 是否为同一路径/软链，不能假设。
+
+### G1：1h window、schema 与 block contract
+
+运行 `audit_code_trace.py`，要求：
+
+- timestamp 单调，存在 60–75min 连续稳定窗口；
+- `timestamp/input_length/output_length` 全行存在；
+- `hash_ids` 数量与某个 source block size 在全行严格满足
+  `ceil(ISL/source_block_size)`；预计值 512，但以审计结果为准；
+- 记录 ISL/OSL/ISL+OSL 的 p50/p90/p95/p99/max、gap、request rate、prompt/sampling 字段覆盖。
+
+选择窗口后用 `prepare_code_window.py` 物化只读派生文件，并按 session root 生成确定性的 `sampling_u`。另一日期段不参与 rho 与 profile 选择。
+
+### G2：`max_model_len` data gate
+
+候选 cap 固定为 `40960/65536/98304/131072/262144`，选能 **100% 覆盖选中窗口 `ISL+OSL`** 的最小值。规则：
+
+- 若 max≤131072，主路径使用 131072 或更小的审计推荐值；
+- 若存在 >131072 请求，不允许悄悄过滤。优先验证 262144；若 runtime 不可行，必须预注册过滤比例，并把 claim 改为“≤131072 子群”；
+- Frontier 的 trace max tokens、predictor max tokens/request、vLLM `--max-model-len` 三处使用同一个 manifest 值。
+
+### G3：prompt 与 prefix fidelity
+
+优先级：
+
+1. 有对齐 prompt text sidecar：用 Qwen tokenizer 重分词，要求 token length 与 trace ISL 全行一致；
+2. trace 内已有 prompt text/token IDs：同样做长度与 hash relation 检查；
+3. 两者都没有：允许用 source hash 确定性展开为 synthetic Qwen token IDs，但结果降级为 **length/arrival/prefix-shape faithful**，不声称 prompt-content 或 MoE routing faithful。
+
+不论走哪条路径，source→16 映射冲突、runtime identity collision、parent prefix violation 都必须为 0。P+D/prefill-only 两侧 prefix caching 同开；先用 5–10min TP4/MNS16 做 hit-ratio audit。
+
+### G4：long-context profile support
+
+现有 profile-v5 的 KV context 上界约 32k，对 code 不足。根据 development window 的 uncached-ISL 分布生成 profile-v6-code-longctx：
+
+- full chunk：`q8k`，context 至少覆盖 32k/48k/64k/80k/96k/112k/120k；
+- tail chunk：从真实 `ISL mod 8192` 的 p50/p90 选择 2–4k/4–6k 代表点；
+- TP1/2/4 分开采集，复测 `q1ks8k/q8ks32k` anchor；
+- 每点至少两次 fresh-process repeat；CV≤5%，anchor drift≤10%；
+- profile max context 必须 ≥ development window p99；正式 max claim 要求 ≥ max。若只覆盖 p99，max 以上请求单独列为 out-of-support，不进入总体准确度数字。
+
+这是 code P+D 正式 fidelity 的硬 gate。可以先用旧 profile 跑 diagnostic sim 来估 load，但不得与真机组成最终 gap。
+
+### G5：vLLM max-length/KV runtime gate
+
+对每个候选 topology（先 TP4，再 TP2，TP1 后置）：
+
+1. fresh server，以 manifest cap 启动；
+2. 记录 vLLM 版本、model config、GPU KV blocks、maximum concurrency、启动日志；
+3. 发 3 个单请求：ISL p50、p99、max（OSL=1），usage 必须逐 token 对齐；
+4. 发 5min sampled P+D canary，零 OOM/timeout/preemption storm；
+5. 只有 maximum concurrency>1 且 canary drain tail≤窗口时长 10% 才进入 rho calibration。
+
+`max_model_len` 变大不等于每个请求都预占最大 KV，但会改变启动合法性与可表达的单请求上界；实际 KV 压力仍由并发 token state 决定。
+
+### G6：每种 mode 独立标定 rho
+
+不能复用 P+D rho：
+
+- P+D 同时按 raw/prefix-adjusted prefill tokens/s 与 decode tokens/s 看 knee；
+- prefill-only 因 OSL=1，重新按 prefill work 标定；
+- strict decode-only 因无 prefill，按 decode tokens/s 和 batch fixed point 标定。
+
+每种 workload×mode 选择 `low/mid/near-knee` 三点；正式点必须亚临界：全请求完成、无持续 backlog、drain tail≤10%、waiting p99 不单调随时间增长。跨 knee 点若运行，只作为 overload boundary，不支持“不发散”结论。
+
+### G7：strict decode-only capability gate
+
+先在 10min synthetic trace 上验证：
+
+- real connector 确认没有执行 prefill kernel；
+- Frontier ledger 第一个阶段就是 decode，computed tokens=ISL；
+- 相同 context length 下两侧 KV block count 一致；
+- connector preload/transfer 时间独立记账，不混入 TPOT；
+- decode batch telemetry 能覆盖 b1 到目标 batch。
+
+若 vLLM 0.20 community stack 没有等价 connector，严格 case 保持 BLOCKED；可另跑 decode-dominant proxy，但单独命名和汇报。
+
+## 正式实验矩阵与推进顺序
+
+### Phase A：code P+D（第一优先级）
+
+1. **A0 CPU/data：** G0–G4；
+2. **A1 max-len smoke：** TP4→TP2；TP1 只在 KV gate 通过后加入；
+3. **A2 paired 10min canary：** TP4/MNS16，low rho，real+sim；
+4. **A3 calibration：** 各 rho 只先跑 sim，冻结 low/mid/near-knee；
+5. **A4 full：** TP4/MNS16、TP2/MNS16 × 3 rho × 2 trial × 60–75min；
+6. **A5 held-out：** 只在 development window 判据冻结后，对第二日期段跑 TP4 的 mid/near-knee。
+
+若某 topology 的 near-knee 过载，像现有 chat TP2/ρ0.01 一样排除，不为凑齐矩阵强跑。
+
+### Phase B：chat/code prefill-only
+
+- 复用各自已物化 window，只把 OSL 改为 1；
+- primary：TP4/MNS16、TP2/MNS16 × 3 独立 rho × 2 trial；
+- 报 TTFT/CDF/quantiles、prefill tokens/s、prefix hit、waiting、chunk/context 分带 residual；
+- TPOT 记为 N/A，E2E 仅作为“一 token completion”辅助值；
+- code 必须继续使用 profile-v6 long-context；chat 使用已验证 profile-v5。
+
+### Phase C：chat/code strict decode-only
+
+先做 batch-sensitive screening，再决定是否铺满：
+
+- **C0 capability canary：** 两 workload × TP4 × MNS{16,128}，10min；
+- **C1 core full：** TP{2,4} × MNS{16,128} × rho{low,near-knee} × 2 trial；
+- **C2 conditional expansion：** 只有当 C1 的 batch 分布从 b≤8 跨到 b>8，或 accuracy gap 随 MNS 改变>5pp，才补 MNS{32,64} 与 mid rho。
+
+decode profile/serving anchors 至少覆盖实际 batch p99。当前 whole-layer grid 只对少数 b≤8 有证据，且 b6 有长尾；在 MNS128 case 前必须补 b{1,2,4,8,16,32,64,128} 或实际访问 bucket，不能把 b8 常数外推到 b128。
+
+## 指标与判据
+
+共同口径：
+
+- 分布统计偏差：`(sim statistic-real statistic)/real statistic`，不是 per-request MAPE；
+- mean/p50/p90/p99 与 empirical CDF；
+- 5min 分窗，前 15min warmup 不进漂移 slope；
+- batch histogram、time-weighted running/waiting、drain tail、preemption；
+- 两 trial pooled 结果和 trial-to-trial noise floor 分开报告。
+
+判据分两层：
+
+1. **准确度：** primary latency mean/p90/p99 的 |bias|≤15% 为强通过，15–30% 为有界但需标注 correction，>30% 立 bad case；任何 topology 排序或 SLO feasibility 翻转都单独判 failure，不能被平均值掩盖。
+2. **长时稳定：** `|residual Theil–Sen slope|×12 / real noise floor < 1` 为 H-BOUNDED；只适用于亚临界 cell。
+
+mode-specific：
+
+- P+D：TTFT/TPOT/E2E 全部 primary；
+- prefill-only：TTFT primary，TPOT N/A；
+- strict decode-only：TPOT primary，TTFT 仅表示 admission/connector overhead，不进入 compute-fidelity gate。
+
+## 成本与调度
+
+- Phase A core：12 个 60–75min jobs（2 topology×3 load×2 trial），约 15 host-hours；按 TP 加权约 45 H20-GPU-hours，加 2–4 个 smoke/canary；
+- Phase B 两 workload：24 个 full jobs，按相同 75min 上界约 90 H20-GPU-hours；
+- Phase C 不一次铺满。C0 4 个 10min canary；C1 32 个 full jobs；C2 按触发条件追加。
+
+每个 job fresh server；只在 `dash1`–`dash4` 全 8 卡 idle/healthy 时启动。即使 TP2/TP4 job 只用部分 GPU，也不在同一 host 并跑，避免 fresh-server 空窗竞态。每一批使用新的 jobs TOML，现有 dispatcher 非幂等。
+
+## 预期产物
+
+- `inputs/code-audit.json`、`inputs/code-window/window-manifest.json`；
+- P+D/prefill-only 的 paired `frontier.csv`、`real_requests.jsonl` 与 manifest；
+- profile-v6-code-longctx raw/merged profile 与 variance report；
+- 每 cell real/sim request metrics、server telemetry、stage ledger；
+- `results/code-pd-fidelity.md`；
+- 最终 `chat/code × P+D/prefill-only/decode-only` compatibility table。
+
+## 已知边界
+
+- code trace 来自 GLM5.1 业务，serving model 是 Qwen3-30B；若无原 prompt text，测试只能保持 shape/prefix 结构，不能证明内容相关 routing fidelity；
+- `max_model_len=128k/256k` 解决的是接入上界，不自动解决 32k 以上 profile 外推；
+- strict decode-only 只测 decode engine；完整 PD 分离还需要单独建模 prefill、KV transfer、backpressure 与 KV-ready arrival。

runs/frontier-code-trace-v0/prepare_code_window.py

@@ -0,0 +1,114 @@
+#!/usr/bin/env python3
+"""Materialize the stable code window selected by audit_code_trace.py."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+from pathlib import Path
+from typing import Any
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--audit", type=Path, required=True)
+    parser.add_argument("--output-root", type=Path, required=True)
+    parser.add_argument("--sample-seed", type=int, default=20260723)
+    return parser.parse_args()
+
+
+def session_uniform(seed: int, window_id: str, session_root: Any) -> float:
+    payload = json.dumps(
+        {"seed": seed, "window_id": window_id, "session_root": session_root},
+        sort_keys=True,
+        separators=(",", ":"),
+    ).encode()
+    return int.from_bytes(hashlib.blake2b(payload, digest_size=8).digest(), "big") / (
+        1 << 64
+    )
+
+
+def sha256(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as stream:
+        for chunk in iter(lambda: stream.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def main() -> None:
+    args = parse_args()
+    audit = json.loads(args.audit.read_text())
+    if audit["data_gate"] != "PASS":
+        raise ValueError(f"trace data gate is not PASS: {audit['data_gate']}")
+    selected = audit["selected"]
+    source = Path(selected["source"])
+    window = selected["stable_window"]
+    start = float(window["start_timestamp"])
+    end = float(window["end_timestamp"])
+    if args.output_root.exists():
+        raise ValueError(f"refusing to overwrite {args.output_root}")
+    args.output_root.mkdir(parents=True)
+    destination = args.output_root / "code-raw-window.jsonl"
+    root_of: dict[Any, Any] = {}
+    request_count = 0
+    with source.open() as input_stream, destination.open("w") as output_stream:
+        for source_index, line in enumerate(input_stream):
+            if not line.strip():
+                continue
+            row = json.loads(line)
+            timestamp = float(row["timestamp"])
+            if timestamp < start:
+                continue
+            if timestamp >= end:
+                break
+            chat = row.get("chat_id", source_index)
+            parent = row.get("parent_chat_id")
+            has_parent = parent not in (None, "", -1, "-1")
+            session_root = root_of.get(parent, parent) if has_parent else chat
+            root_of[chat] = session_root
+            materialized = {
+                **row,
+                "source_index": source_index,
+                "session_root": session_root,
+                "sampling_u": session_uniform(
+                    args.sample_seed,
+                    f"code-{start:.6f}-{end:.6f}",
+                    session_root,
+                ),
+            }
+            output_stream.write(
+                json.dumps(materialized, ensure_ascii=False, separators=(",", ":"))
+                + "\n"
+            )
+            request_count += 1
+    expected = int(selected["selected_window_stats"]["requests"])
+    if request_count != expected:
+        raise ValueError(f"window request mismatch: materialized={request_count}, audit={expected}")
+    manifest = {
+        "schema": "frontier-code-window-v1",
+        "audit": str(args.audit.resolve()),
+        "audit_sha256": sha256(args.audit),
+        "source": str(source.resolve()),
+        "source_block_size": selected["hash_contract"]["exact_source_block_size"],
+        "target_block_size": 16,
+        "start_timestamp": start,
+        "end_timestamp": end,
+        "duration_s": end - start,
+        "requests": request_count,
+        "sample_seed": args.sample_seed,
+        "sampling_rule": "session-coherent deterministic sampling_u",
+        "max_model_len": audit["max_model_len_recommendation"],
+        "window_stats": selected["selected_window_stats"],
+        "raw_window": str(destination.resolve()),
+        "raw_window_sha256": sha256(destination),
+    }
+    (args.output_root / "window-manifest.json").write_text(
+        json.dumps(manifest, indent=2, sort_keys=True) + "\n"
+    )
+    print(json.dumps({"requests": request_count, "output_root": str(args.output_root)}))
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-code-trace-v0/test_code_trace_preflight.py

@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parent
+
+
+class CodeTracePreflightTest(unittest.TestCase):
+    def test_audit_and_materialize_512_block_window(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            root = Path(tmp)
+            source = root / "051315-051317.jsonl"
+            with source.open("w") as stream:
+                for index in range(4501):
+                    input_tokens = 513 if index % 2 else 512
+                    stream.write(
+                        json.dumps(
+                            {
+                                "chat_id": index,
+                                "parent_chat_id": index - 1 if index % 2 else -1,
+                                "timestamp": float(index),
+                                "input_length": input_tokens,
+                                "output_length": 32,
+                                "hash_ids": [index // 2]
+                                if input_tokens == 512
+                                else [index // 2, 100000 + index],
+                            }
+                        )
+                        + "\n"
+                    )
+            audit = root / "audit.json"
+            subprocess.run(
+                [
+                    sys.executable,
+                    str(ROOT / "audit_code_trace.py"),
+                    "--source",
+                    str(source),
+                    "--output",
+                    str(audit),
+                ],
+                check=True,
+            )
+            payload = json.loads(audit.read_text())
+            self.assertEqual(payload["data_gate"], "PASS")
+            self.assertEqual(
+                payload["selected"]["hash_contract"]["exact_source_block_size"], 512
+            )
+            self.assertEqual(payload["max_model_len_recommendation"], 40960)
+            output = root / "window"
+            subprocess.run(
+                [
+                    sys.executable,
+                    str(ROOT / "prepare_code_window.py"),
+                    "--audit",
+                    str(audit),
+                    "--output-root",
+                    str(output),
+                ],
+                check=True,
+            )
+            manifest = json.loads((output / "window-manifest.json").read_text())
+            rows = [
+                json.loads(line)
+                for line in (output / "code-raw-window.jsonl").read_text().splitlines()
+            ]
+        self.assertEqual(manifest["requests"], len(rows))
+        self.assertGreaterEqual(manifest["duration_s"], 3600)
+        self.assertEqual(rows[0]["sampling_u"], rows[1]["sampling_u"])
+
+
+if __name__ == "__main__":
+    unittest.main()

runs/frontier-decode-batch-grid-v0/.gitignore

@@ -0,0 +1,4 @@
+fleet-artifacts/
+fleet-state/
+remote-outputs/
+replay/

runs/frontier-decode-batch-grid-v0/analyze_bc8_replay.py

@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+"""Compare original and whole-layer-curve Frontier against the BC-8 real pilot."""
+
+from __future__ import annotations
+
+import json
+import statistics
+from collections import Counter
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parent
+REPO = ROOT.parents[1]
+CONFIGS = ("tp1_mns16", "tp2_mns16", "tp4_mns16", "tp4_mns32")
+REPLAY = ROOT / "replay/bc8"
+ORIGINAL = REPO / "runs/frontier-knee-sweep-v0/raw/fixed/rho0p02"
+REAL = REPO / "runs/frontier-pilot-v0/results/pilot-surface.json"
+
+
+def find_one(root: Path, name: str) -> Path:
+    matches = list(root.glob(f"**/{name}"))
+    if len(matches) != 1:
+        raise ValueError(f"expected one {name} under {root}, got {matches}")
+    return matches[0]
+
+
+def summarize_sim(root: Path) -> dict:
+    metrics = json.loads(find_one(root, "system_metrics.json").read_text())
+    ledger = find_one(root, "frontier_stage_batch_ledger.jsonl")
+    histogram: Counter[int] = Counter()
+    service_ms: dict[int, set[float]] = {}
+    for line in ledger.read_text().splitlines():
+        row = json.loads(line)
+        tokens = row["request_num_tokens"]
+        if tokens and all(int(value) == 1 for value in tokens):
+            batch = len(tokens)
+            histogram[batch] += 1
+            service_ms.setdefault(batch, set()).add(
+                float(row["execution_time"]["model_time_ms"])
+            )
+    return {
+        "tpot_mean_ms": metrics["tpot_statistics"]["mean"],
+        "ttft_mean_ms": metrics["ttft_statistics"]["mean"],
+        "e2e_mean_ms": metrics["request_e2e_time_statistics"]["mean"],
+        "decode_batch_histogram": dict(sorted(histogram.items())),
+        "decode_service_ms": {
+            str(batch): sorted(values) for batch, values in sorted(service_ms.items())
+        },
+        "decode_b_gt_1_fraction": sum(
+            count for batch, count in histogram.items() if batch > 1
+        )
+        / sum(histogram.values()),
+    }
+
+
+def main() -> None:
+    real_rows = [
+        row
+        for row in json.loads(REAL.read_text())
+        if row["load"] == "p4" and row["canonical"]
+    ]
+    points = []
+    for config in CONFIGS:
+        real = [row for row in real_rows if row["config"] == config]
+        original = summarize_sim(ORIGINAL / config)
+        corrected = summarize_sim(REPLAY / "raw" / config)
+        usage = json.loads((REPLAY / "raw" / config / "usage.json").read_text())
+        points.append(
+            {
+                "config": config,
+                "tp": int(config[2]),
+                "mns": int(config.split("mns")[1]),
+                "real_tpot_mean_ms": statistics.fmean(
+                    row["tpot_mean_ms"] for row in real
+                ),
+                "original": original,
+                "whole_curve": corrected,
+                "original_tpot_residual_ms": (
+                    original["tpot_mean_ms"]
+                    - statistics.fmean(row["tpot_mean_ms"] for row in real)
+                ),
+                "whole_curve_tpot_residual_ms": (
+                    corrected["tpot_mean_ms"]
+                    - statistics.fmean(row["tpot_mean_ms"] for row in real)
+                ),
+                "usage": usage,
+            }
+        )
+
+    def ranking(key):
+        return [
+            point["config"]
+            for point in sorted(points, key=lambda point: (key(point), point["config"]))
+            if point["mns"] == 16
+        ]
+
+    real_ranking = ranking(lambda point: point["real_tpot_mean_ms"])
+    original_ranking = ranking(lambda point: point["original"]["tpot_mean_ms"])
+    corrected_ranking = ranking(lambda point: point["whole_curve"]["tpot_mean_ms"])
+    payload = {
+        "schema": "frontier-decode-grid-bc8-verdict.v1",
+        "points": points,
+        "mns16_rankings_fast_to_slow": {
+            "real": real_ranking,
+            "original": original_ranking,
+            "whole_curve": corrected_ranking,
+        },
+        "whole_curve_restores_real_mns16_ranking": corrected_ranking == real_ranking,
+        "all_corrected_batches_within_measured_support": all(
+            max(map(int, point["whole_curve"]["decode_batch_histogram"])) <= 8
+            for point in points
+        ),
+        "decision": (
+            "STATIC_WHOLE_CURVE_FIXES_BC8"
+            if corrected_ranking == real_ranking
+            else "ESCALATE_EVENT_LEVEL_STATE"
+        ),
+    }
+    output = ROOT / "results/bc8-replay-verdict.json"
+    output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+
+    lines = [
+        "# BC-8 whole-layer decode-curve replay",
+        "",
+        "| Config | Real TPOT | Original sim | Whole-curve sim | Residual before -> after | b>1 |",
+        "|---|---:|---:|---:|---:|---:|",
+    ]
+    for point in points:
+        lines.append(
+            f"| {point['config']} | {point['real_tpot_mean_ms']:.3f} | "
+            f"{point['original']['tpot_mean_ms']:.3f} | "
+            f"{point['whole_curve']['tpot_mean_ms']:.3f} | "
+            f"{point['original_tpot_residual_ms']:+.3f} -> "
+            f"{point['whole_curve_tpot_residual_ms']:+.3f} | "
+            f"{point['whole_curve']['decode_b_gt_1_fraction']:.2%} |"
+        )
+    lines.extend(
+        [
+            "",
+            f"- Real MNS16: `{' < '.join(real_ranking)}`",
+            f"- Original: `{' < '.join(original_ranking)}`",
+            f"- Whole curve: `{' < '.join(corrected_ranking)}`",
+            f"- Decision: **{payload['decision']}**.",
+            "",
+        ]
+    )
+    (ROOT / "results/bc8-replay-verdict.md").write_text("\n".join(lines))
+    print(json.dumps({"output": str(output), "decision": payload["decision"]}))
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-decode-batch-grid-v0/analyze_decode_trace.py

@@ -0,0 +1,195 @@
+#!/usr/bin/env python3
+"""Summarize graph-on vLLM Torch traces by decode component."""
+
+from __future__ import annotations
+
+import argparse
+import gzip
+import json
+import math
+import statistics
+from collections import Counter, defaultdict
+from pathlib import Path
+
+
+COMPONENT_NAMES = (
+    "attention",
+    "linear_norm_rope",
+    "router",
+    "moe",
+    "collective",
+    "output_head",
+    "other",
+)
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--trace-root", type=Path, required=True)
+    parser.add_argument("--label", required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    return parser.parse_args()
+
+
+def classify(name: str, occurrences: int, steps: int) -> str:
+    lower = name.lower()
+    if any(token in lower for token in ("nccl", "allreduce", "all_reduce")):
+        return "collective"
+    if "topkgating" in lower or "fused_topk" in lower:
+        return "router"
+    if any(
+        token in lower
+        for token in (
+            "fused_moe",
+            "moefcgemm",
+            "tensorrt_llm::kernels::cutlass_kernels",
+            "groupproblemshape",
+            "memcpy32_post",
+        )
+    ):
+        return "moe"
+    if any(
+        token in lower
+        for token in (
+            "flashattn",
+            "flashattnfwd",
+            "reshape_and_cache",
+            "prepare_varlen_num_blocks",
+        )
+    ):
+        return "attention"
+    if "nvjet" in lower and occurrences <= steps * 2:
+        return "output_head"
+    if any(
+        token in lower
+        for token in (
+            "nvjet",
+            "cublaslt",
+            "rms_norm",
+            "rsqrt",
+            "triton_red_fused_2",
+            "triton_poi_fused_3",
+            "triton_red_fused_0",
+            "triton_poi_fused_1",
+        )
+    ):
+        return "linear_norm_rope"
+    return "other"
+
+
+def stats(values: list[float]) -> dict[str, float | int]:
+    ordered = sorted(values)
+    return {
+        "n": len(values),
+        "mean_ms": statistics.fmean(values),
+        "population_std_ms": statistics.pstdev(values),
+        "p50_ms": statistics.median(ordered),
+        "p95_ms": ordered[math.ceil(0.95 * len(ordered)) - 1],
+    }
+
+
+def load_events(path: Path) -> list[dict]:
+    opener = gzip.open if path.suffix == ".gz" else open
+    with opener(path, "rt") as source:
+        return json.load(source)["traceEvents"]
+
+
+def analyze_rank(path: Path) -> dict:
+    events = load_events(path)
+    kernels = [event for event in events if event.get("cat") == "kernel"]
+    all_windows = sorted(
+        (
+            event
+            for event in events
+            if event.get("cat") == "gpu_user_annotation"
+            and str(event.get("name", "")).startswith("execute_")
+        ),
+        key=lambda event: float(event["ts"]),
+    )
+    if not all_windows:
+        raise ValueError(f"{path}: no GPU execute annotations")
+    window_names = Counter(str(window["name"]) for window in all_windows)
+    selected_name = window_names.most_common(1)[0][0]
+    windows = [
+        window for window in all_windows if str(window["name"]) == selected_name
+    ]
+
+    selected: list[dict] = []
+    step_kernels: list[list[dict]] = []
+    for window in windows:
+        start = float(window["ts"])
+        end = start + float(window["dur"])
+        current = [
+            kernel for kernel in kernels if start <= float(kernel["ts"]) < end
+        ]
+        selected.extend(current)
+        step_kernels.append(current)
+
+    occurrences = Counter(str(kernel["name"]) for kernel in selected)
+    component_steps: dict[str, list[float]] = defaultdict(list)
+    busy_steps: list[float] = []
+    wall_steps = [float(window["dur"]) / 1000.0 for window in windows]
+    for current in step_kernels:
+        per_component: dict[str, float] = defaultdict(float)
+        for kernel in current:
+            name = str(kernel["name"])
+            per_component[classify(name, occurrences[name], len(windows))] += (
+                float(kernel["dur"]) / 1000.0
+            )
+        for name in COMPONENT_NAMES:
+            component_steps[name].append(per_component[name])
+        busy_steps.append(sum(per_component.values()))
+
+    return {
+        "trace": str(path),
+        "selected_execute_annotation": selected_name,
+        "execute_annotation_histogram": dict(sorted(window_names.items())),
+        "steps": len(windows),
+        "execute_wall": stats(wall_steps),
+        "gpu_kernel_busy": stats(busy_steps),
+        "non_kernel_gap": stats(
+            [wall - busy for wall, busy in zip(wall_steps, busy_steps)]
+        ),
+        "components": {
+            name: stats(values) for name, values in component_steps.items()
+        },
+    }
+
+
+def main() -> None:
+    args = parse_args()
+    traces = sorted(args.trace_root.rglob("*.pt.trace.json*"))
+    if not traces:
+        raise SystemExit(f"no traces under {args.trace_root}")
+    ranks = [analyze_rank(path) for path in traces]
+    payload = {
+        "schema": "frontier-decode-batch-trace.v1",
+        "label": args.label,
+        "contract": {
+            "timing": "CUDA graph-on GPU execute annotations",
+            "component_time": "sum of CUDA kernel durations inside execute range",
+            "tp_aggregation": "per-rank; slowest rank mean approximates critical path",
+        },
+        "ranks": ranks,
+        "rank_summary": {
+            "ranks": len(ranks),
+            "slowest_rank_execute_mean_ms": max(
+                rank["execute_wall"]["mean_ms"] for rank in ranks
+            ),
+            "slowest_rank_kernel_busy_mean_ms": max(
+                rank["gpu_kernel_busy"]["mean_ms"] for rank in ranks
+            ),
+            "component_rank_mean_ms": {
+                name: statistics.fmean(
+                    rank["components"][name]["mean_ms"] for rank in ranks
+                )
+                for name in COMPONENT_NAMES
+            },
+        },
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-decode-batch-grid-v0/analyze_grid.py

@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+"""Aggregate fresh-process decode profiles and enforce the repeat gate."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import statistics
+from pathlib import Path
+
+
+COMPONENT_NAMES = (
+    "attention",
+    "linear_norm_rope",
+    "router",
+    "moe",
+    "collective",
+    "output_head",
+    "other",
+)
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--results-dir", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    cells = []
+    all_resolved = True
+    for tp in (2, 4):
+        for batch in (2, 4, 6, 8):
+            repeats = []
+            for repeat in (1, 2, 3):
+                path = args.results_dir / f"tp{tp}-b{batch}-r{repeat}.json"
+                if repeat == 3 and not path.exists():
+                    continue
+                payload = json.loads(path.read_text())
+                repeats.append(
+                    {
+                        "repeat": repeat,
+                        "source": str(path),
+                        "execute_mean_ms": payload["rank_summary"][
+                            "slowest_rank_execute_mean_ms"
+                        ],
+                        "component_rank_mean_ms": payload["rank_summary"][
+                            "component_rank_mean_ms"
+                        ],
+                    }
+                )
+            values = [row["execute_mean_ms"] for row in repeats]
+            initial_values = values[:2]
+            initial_mean = statistics.fmean(initial_values)
+            initial_cv_pct = (
+                statistics.pstdev(initial_values) / initial_mean * 100.0
+            )
+            needs_repeat_3 = initial_cv_pct > 10.0
+            repeat_gate_resolved = not needs_repeat_3 or len(repeats) == 3
+            all_resolved = all_resolved and repeat_gate_resolved
+            cells.append(
+                {
+                    "tp": tp,
+                    "batch": batch,
+                    "repeats": repeats,
+                    "median_execute_ms": statistics.median(values),
+                    "initial_repeat_cv_pct": initial_cv_pct,
+                    "all_repeat_cv_pct": (
+                        statistics.pstdev(values)
+                        / statistics.fmean(values)
+                        * 100.0
+                    ),
+                    "needs_repeat_3": needs_repeat_3,
+                    "repeat_gate_resolved": repeat_gate_resolved,
+                    "median_component_ms": {
+                        name: statistics.median(
+                            row["component_rank_mean_ms"][name] for row in repeats
+                        )
+                        for name in COMPONENT_NAMES
+                    },
+                }
+            )
+    payload = {
+        "schema": "frontier-decode-batch-grid.v1",
+        "contract": {
+            "workload": "Qwen3-30B-A3B BF16, 2048->128, graph-on, MNS=16",
+            "timing": "slowest-rank execute mean over 16 pure-decode steps",
+            "repeat_aggregation": "median of two fresh processes; three if unstable",
+            "stability_gate": "r1/r2 population CV <=10%; else require r3 and take median",
+        },
+        "all_repeat_gates_resolved": all_resolved,
+        "cells": cells,
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    if not all_resolved:
+        raise SystemExit("one or more cells require repeat 3")
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-decode-batch-grid-v0/experiment-card.md

@@ -0,0 +1,102 @@
+# 实验 EXP-DECODE-BATCH-GRID：BC-8 是否可由稳定 whole-layer b2--b8 curve 修复
+
+> **状态：** COMPLETED；BC-8 排序工程修复成立，global curve merge 暂停
+>
+> Parent campaign：[`../frontier-simulator-gap-campaign-v0/README.md`](../frontier-simulator-gap-campaign-v0/README.md)
+
+## Claim 与决策
+
+- **Parent claim：** 最新 1h trace 中 TP4 的 TPOT/E2E 仍稳定正偏约 10--23%；
+  历史 BC-8 knee-right 真机排序为 `TP2<TP4<TP1`，sim 为
+  `TP2<TP1<TP4`。这是当前最明显的 decode/state residual。
+- **目的：** 判断 BC-8 是 b2--b8 whole-layer service curve 可工程修复，
+  还是必须进入 event-level admission/batch-formation 建模。
+- **Competing hypotheses：**
+  - H1：现有 b2 service sample 不稳定或 b>4 常数外推，导致 TP4 residence
+    被高估；稳定 whole-layer curve 可恢复排序。
+  - H2：给定 batch 的 whole-layer curve 已准，错误来自 simulator 与 vLLM
+    batch formation/event semantics 不同。
+- **事前预测：** 若 H1 成立，TP4/b2 repeat CV≤10%，其 median 比当前进入
+  BC-8 的 curve 低至少 0.5 ms，注入后 `TP4<TP1`；若 H2 成立，
+  same-state residual <0.5 ms，但 BC-8 排序仍错。
+- **判定规则：**
+  - 先做 TP4/b2 两个 fresh-process repeats；CV>10% 时追加第三次，取
+    median，不允许从单次 favorable sample 选值。
+  - smoke 稳定后补 TP2/TP4 × b{2,4,6,8}；b>4 不再使用 b4 常数外推。
+  - whole-layer 注入只替换 pure-decode service curve，不改变 prefill。
+  - BC-8 排序仍错则停止 profile 修补，升级 event-level state telemetry。
+
+## Setup
+
+- **自变量：** TP `{2,4}`；pure-decode batch `{2,4,6,8}`；fresh-process
+  repeat `{1,2}`，不稳定 cell 追加 repeat 3。
+- **控制变量：** Qwen3-30B-A3B BF16；H20；vLLM 0.20；context=2048；
+  output=128；graph-on；MNS=16；no prefix caching；同模型/runtime/cache。
+- **Hardware：** dash1--dash4；每轮调度前要求目标主机 8×H20
+  idle/healthy。每主机同时只启动一个 engine，避免 CPU/JIT contention。
+- **Metrics：** critical-rank execute wall；MoE/attention/collective/other
+  kernels；repeat median/CV；sim same-state residual；BC-8 TPOT 完整排序。
+- **Baselines：**
+  - 旧 serving grid：TP4/b2 execute=`5.044 ms`，单 process 内
+    population std=`1.561 ms`，不足以进入判决链。
+  - BC-8：real `4.305<4.449<4.876 ms`；sim
+    `5.12<5.49<5.86 ms`（TP2/TP4/TP1 对应完整排序见结果审计）。
+
+## 预期产物与 review
+
+- **预期数据：** 每 cell trace、request result、runtime/GPU provenance；
+  `results/grid.json`；BC-8 counterfactual 与 verdict。
+- **Figure prototype：** `figure-prototype.png`。左图画 b2 repeat instability
+  与目标 CI，右图画 BC-8 real/sim/corrected 排序。
+- **人工 review：** campaign 已批准按第三优先级推进。
+- **Review 意见：** 先 TP4/b2 smoke；不在稳定性 gate 前铺完整网格。
+
+## 复现信息
+
+- **Frontier baseline：** `deadc4a321f0baaa534c6ebd17f974123733cdc2`；
+  CPU replay 使用既有 joint serving-path curves，未合入 EXP-1 rejected patch。
+- **Remote：** dash1--dash4 共享 clean detached experiment worktree
+  `ecc559938132851d61bf22daa55cea9658807dfa`；canonical checkout 的用户
+  dirty changes 未改动。
+- **Known limits：** 固定 2048→128 state-matched workload；本实验只支持
+  decode curve 与 BC-8，不外推到 chat prefill shape 或其它模型。
+
+## 结果
+
+- **Entry gate：** PASS。TP4 1h trace TPOT mean 正偏
+  `+10.6%--+23.0%`，E2E mean `+9.8%--+19.2%`；BC-8 完整排序仍错。
+- **TP4/b2 smoke：** 两个 fresh-process critical-rank execute mean 分别为
+  `4.6263 ms` 和 `4.6201 ms`，跨 repeat CV=`0.067%`，稳定性 gate
+  PASS，无需 repeat 3。median=`4.6232 ms`，比旧样本 `5.044 ms` 低
+  `0.421 ms`；方向支持 H1，但未达到事前 `>=0.5 ms` 的强证据阈值。
+- **Full grid：**
+
+  | Cell | fresh-process execute mean (ms) | median (ms) | initial CV |
+  |---|---:|---:|---:|
+  | TP2/b2 | 4.761 / 4.730 | 4.746 | 0.33% |
+  | TP2/b4 | 5.294 / 5.607 | 5.451 | 2.87% |
+  | TP2/b6 | 6.571 / 6.938 | 6.754 | 2.72% |
+  | TP2/b8 | 5.967 / 6.309 | 6.138 | 2.78% |
+  | TP4/b2 | 4.626 / 4.620 | 4.623 | 0.07% |
+  | TP4/b4 | 4.516 / 4.727 | 4.622 | 2.27% |
+  | TP4/b6 | 5.433 / 10.157 / 8.634 | 8.634 | 30.30% |
+  | TP4/b8 | 4.898 / 5.047 | 4.972 | 1.50% |
+
+- **TP4/b6 instability：** r2/r3 的 MoE 与 attention 基本不变，均值波动由
+  同步 collective 长尾造成；三个 run 的 step p50 仍约 `5.18--5.49 ms`。
+  依预注册规则追加 r3 并取三次中位数，但该 cell 不能视为稳定 deterministic
+  service constant。
+- **BC-8 exact replay：** replay 仅访问 b1/b2，因此没有使用 b6 或任何
+  插值/外推。TP4/b2 whole-layer override 将 TP4 sim TPOT
+  `5.864→5.367 ms`，残差 `+1.414→+0.918 ms`（MNS16）和
+  `+1.498→+1.001 ms`（MNS32）；MNS16 完整排序从
+  `TP2<TP1<TP4` 恢复为真实的 `TP2<TP4<TP1`。sim b>1 stage fraction
+  `48.15%→37.47%`。
+- **7-cell support audit：** 最新 1h cells 的 decode batch 最大到
+  `b6--b15`，而本实验 measured support 只到 b8；TP4/b6 又未通过
+  deterministic-mean 稳定性。为避免重新引入 b8 constant extrapolation，
+  不执行全局 7-cell injection，也不把整条 b2--b8 curve 合入默认 predictor。
+- **Decision：** H1 对 BC-8 成立，排序 gap 属于可工程修复的 TP4/b2
+  whole-layer service residual；但绝对残差、MNS tie 和高 batch collective
+  tail 仍需 event/distribution-aware 模型。建议先合入受作用域保护的 TP4/b2
+  correction，再补 b9--b16 与 collective-tail telemetry 后开启 global merge。

runs/frontier-decode-batch-grid-v0/fleet.toml

@@ -0,0 +1,46 @@
+version = 1
+
+[paths]
+state_dir = "runs/frontier-decode-batch-grid-v0/fleet-state"
+artifacts_dir = "runs/frontier-decode-batch-grid-v0/fleet-artifacts"
+
+[ssh]
+connect_timeout_sec = 10
+
+[scheduler]
+gpu_free_memory_mb = 1024
+gpu_free_utilization_pct = 10
+prefer_pack = true
+
+# Code is synchronized by Git. The mandatory scp pass uses an empty directory.
+[sync]
+mode = "scp"
+local_path = "/tmp/frontier-decode-batch-grid-v0-empty-sync"
+
+[[hosts]]
+name = "dash1"
+ssh_alias = "dash1"
+enabled = true
+sync_remote_path = "/home/admin/cpfs/wjh/aituner/aituner-frontier-decode-batch-grid-v0"
+fleet_root = "/home/admin/cpfs/wjh/aituner/gpu-fleet-decode-batch-grid-v0/dash1"
+
+[[hosts]]
+name = "dash2"
+ssh_alias = "dash2"
+enabled = true
+sync_remote_path = "/home/admin/cpfs/wjh/aituner/aituner-frontier-decode-batch-grid-v0"
+fleet_root = "/home/admin/cpfs/wjh/aituner/gpu-fleet-decode-batch-grid-v0/dash2"
+
+[[hosts]]
+name = "dash3"
+ssh_alias = "dash3"
+enabled = true
+sync_remote_path = "/home/admin/cpfs/wjh/aituner/aituner-frontier-decode-batch-grid-v0"
+fleet_root = "/home/admin/cpfs/wjh/aituner/gpu-fleet-decode-batch-grid-v0/dash3"
+
+[[hosts]]
+name = "dash4"
+ssh_alias = "dash4"
+enabled = true
+sync_remote_path = "/home/admin/cpfs/wjh/aituner/aituner-frontier-decode-batch-grid-v0"
+fleet_root = "/home/admin/cpfs/wjh/aituner/gpu-fleet-decode-batch-grid-v0/dash4"

runs/frontier-decode-batch-grid-v0/jobs-grid-r1a.toml

@@ -0,0 +1,61 @@
+version = 1
+
+[[jobs]]
+name = "decode-grid-tp2-b2-r1-20260723"
+gpus = 2
+gpu_model = "H20"
+hosts = ["dash1"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b2-r1"]
+
+[jobs.env]
+TP = "2"
+PROFILE_BATCH = "2"
+SERVER_PORT = "9841"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b2-r1"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"
+
+[[jobs]]
+name = "decode-grid-tp2-b4-r1-20260723"
+gpus = 2
+gpu_model = "H20"
+hosts = ["dash2"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b4-r1"]
+
+[jobs.env]
+TP = "2"
+PROFILE_BATCH = "4"
+SERVER_PORT = "9842"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b4-r1"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"
+
+[[jobs]]
+name = "decode-grid-tp4-b4-r1-20260723"
+gpus = 4
+gpu_model = "H20"
+hosts = ["dash3"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r1"]
+
+[jobs.env]
+TP = "4"
+PROFILE_BATCH = "4"
+SERVER_PORT = "9843"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r1"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"
+
+[[jobs]]
+name = "decode-grid-tp4-b6-r1-20260723"
+gpus = 4
+gpu_model = "H20"
+hosts = ["dash4"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r1"]
+
+[jobs.env]
+TP = "4"
+PROFILE_BATCH = "6"
+SERVER_PORT = "9844"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r1"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"

runs/frontier-decode-batch-grid-v0/jobs-grid-r1b.toml

@@ -0,0 +1,46 @@
+version = 1
+
+[[jobs]]
+name = "decode-grid-tp2-b6-r1-20260723"
+gpus = 2
+gpu_model = "H20"
+hosts = ["dash1"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b6-r1"]
+
+[jobs.env]
+TP = "2"
+PROFILE_BATCH = "6"
+SERVER_PORT = "9845"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b6-r1"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"
+
+[[jobs]]
+name = "decode-grid-tp2-b8-r1-20260723"
+gpus = 2
+gpu_model = "H20"
+hosts = ["dash2"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b8-r1"]
+
+[jobs.env]
+TP = "2"
+PROFILE_BATCH = "8"
+SERVER_PORT = "9846"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b8-r1"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"
+
+[[jobs]]
+name = "decode-grid-tp4-b8-r1-20260723"
+gpus = 4
+gpu_model = "H20"
+hosts = ["dash3"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r1"]
+
+[jobs.env]
+TP = "4"
+PROFILE_BATCH = "8"
+SERVER_PORT = "9847"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r1"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"

runs/frontier-decode-batch-grid-v0/jobs-grid-r2a.toml

@@ -0,0 +1,61 @@
+version = 1
+
+[[jobs]]
+name = "decode-grid-tp2-b2-r2-20260723"
+gpus = 2
+gpu_model = "H20"
+hosts = ["dash1"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b2-r2"]
+
+[jobs.env]
+TP = "2"
+PROFILE_BATCH = "2"
+SERVER_PORT = "9841"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b2-r2"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"
+
+[[jobs]]
+name = "decode-grid-tp2-b4-r2-20260723"
+gpus = 2
+gpu_model = "H20"
+hosts = ["dash2"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b4-r2"]
+
+[jobs.env]
+TP = "2"
+PROFILE_BATCH = "4"
+SERVER_PORT = "9842"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b4-r2"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"
+
+[[jobs]]
+name = "decode-grid-tp4-b4-r2-20260723"
+gpus = 4
+gpu_model = "H20"
+hosts = ["dash3"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r2"]
+
+[jobs.env]
+TP = "4"
+PROFILE_BATCH = "4"
+SERVER_PORT = "9843"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r2"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"
+
+[[jobs]]
+name = "decode-grid-tp4-b6-r2-20260723"
+gpus = 4
+gpu_model = "H20"
+hosts = ["dash4"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r2"]
+
+[jobs.env]
+TP = "4"
+PROFILE_BATCH = "6"
+SERVER_PORT = "9844"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r2"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"

runs/frontier-decode-batch-grid-v0/jobs-grid-r2b.toml

@@ -0,0 +1,46 @@
+version = 1
+
+[[jobs]]
+name = "decode-grid-tp2-b6-r2-20260723"
+gpus = 2
+gpu_model = "H20"
+hosts = ["dash1"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b6-r2"]
+
+[jobs.env]
+TP = "2"
+PROFILE_BATCH = "6"
+SERVER_PORT = "9845"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b6-r2"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"
+
+[[jobs]]
+name = "decode-grid-tp2-b8-r2-20260723"
+gpus = 2
+gpu_model = "H20"
+hosts = ["dash2"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b8-r2"]
+
+[jobs.env]
+TP = "2"
+PROFILE_BATCH = "8"
+SERVER_PORT = "9846"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b8-r2"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"
+
+[[jobs]]
+name = "decode-grid-tp4-b8-r2-20260723"
+gpus = 4
+gpu_model = "H20"
+hosts = ["dash3"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r2"]
+
+[jobs.env]
+TP = "4"
+PROFILE_BATCH = "8"
+SERVER_PORT = "9847"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r2"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"

runs/frontier-decode-batch-grid-v0/jobs-grid-r3.toml

@@ -0,0 +1,16 @@
+version = 1
+
+[[jobs]]
+name = "decode-grid-tp4-b6-r3-20260723"
+gpus = 4
+gpu_model = "H20"
+hosts = ["dash4"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r3"]
+
+[jobs.env]
+TP = "4"
+PROFILE_BATCH = "6"
+SERVER_PORT = "9844"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r3"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"

runs/frontier-decode-batch-grid-v0/jobs-smoke-r1.toml

@@ -0,0 +1,16 @@
+version = 1
+
+[[jobs]]
+name = "decode-batch-tp4-b2-r1-20260723"
+gpus = 4
+gpu_model = "H20"
+hosts = ["dash1"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r1"]
+
+[jobs.env]
+TP = "4"
+PROFILE_BATCH = "2"
+SERVER_PORT = "9831"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r1"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"

runs/frontier-decode-batch-grid-v0/jobs-smoke-r2.toml

@@ -0,0 +1,16 @@
+version = 1
+
+[[jobs]]
+name = "decode-batch-tp4-b2-r2-20260723"
+gpus = 4
+gpu_model = "H20"
+hosts = ["dash1"]
+command = "timeout --signal=TERM --kill-after=60s 1800 bash runs/frontier-decode-batch-grid-v0/run_decode_profile.sh"
+artifacts = ["runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r2"]
+
+[jobs.env]
+TP = "4"
+PROFILE_BATCH = "2"
+SERVER_PORT = "9832"
+OUTPUT_ROOT = "runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r2"
+FLASHINFER_WORKSPACE_BASE = "/tmp/frontier-component-flashinfer-v4"

runs/frontier-decode-batch-grid-v0/materialize_decode_batch.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Create simultaneous, prefix-disjoint requests for a fixed decode batch."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", type=Path, required=True)
+    parser.add_argument("--batch", type=int, required=True)
+    parser.add_argument("--input-tokens", type=int, default=2048)
+    parser.add_argument("--output-tokens", type=int, default=128)
+    parser.add_argument("--output", type=Path, required=True)
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    if min(args.batch, args.input_tokens, args.output_tokens) <= 0:
+        raise ValueError("batch and token counts must be positive")
+    if args.input_tokens + args.output_tokens > 40960:
+        raise ValueError("request exceeds the server max model length")
+
+    from transformers import AutoTokenizer
+
+    tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
+    special = set(tokenizer.all_special_ids)
+    candidates = [
+        token for token in range(tokenizer.vocab_size) if token not in special
+    ]
+    if len(candidates) < args.batch + 1:
+        raise ValueError("tokenizer has too few non-special token IDs")
+
+    base = candidates[0]
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    with args.output.open("w") as output:
+        for index in range(args.batch):
+            row = {
+                "source_index": index,
+                "arrived_at": 0.0,
+                "input_length": args.input_tokens,
+                "output_length": args.output_tokens,
+                "session_id": index,
+                "runtime_block_ids": [],
+                "body": {
+                    "prompt": [
+                        candidates[index + 1],
+                        *([base] * (args.input_tokens - 1)),
+                    ],
+                    "min_tokens": args.output_tokens,
+                    "max_tokens": args.output_tokens,
+                    "ignore_eos": True,
+                },
+            }
+            output.write(json.dumps(row, separators=(",", ":")) + "\n")
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-decode-batch-grid-v0/plot_figure_prototype.py

@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+"""Render the preregistered EXP-DECODE-BATCH-GRID schematic."""
+
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import numpy as np
+
+ROOT = Path(__file__).resolve().parent
+
+
+def main() -> None:
+    fig, axes = plt.subplots(1, 2, figsize=(9.6, 4.0))
+
+    ax = axes[0]
+    repeats = np.array([3.7, 5.0, 6.4])
+    ax.scatter([1, 2, 3], repeats, color="#d95f02", s=45, label="old/possible repeats")
+    ax.axhspan(4.1, 4.6, color="#1b9e77", alpha=0.18, label="stable target band")
+    ax.axhline(np.median(repeats), color="#7570b3", ls="--", label="median")
+    ax.set(xlabel="TP4/b2 fresh-process repeat", ylabel="execute wall (ms)")
+    ax.set_title("A. Stability gate before curve fit")
+    ax.set_xticks([1, 2, 3])
+    ax.legend(frameon=False, fontsize=8)
+
+    ax = axes[1]
+    labels = ["TP2", "TP4", "TP1"]
+    real = [4.305, 4.449, 4.876]
+    sim = [5.12, 5.86, 5.49]
+    corrected = [4.7, 4.9, 5.3]
+    x = np.arange(3)
+    width = 0.25
+    ax.bar(x - width, real, width, label="real", color="#1b9e77")
+    ax.bar(x, sim, width, label="current sim", color="#d95f02")
+    ax.bar(x + width, corrected, width, label="target corrected", color="#7570b3")
+    ax.set_xticks(x, labels)
+    ax.set_ylabel("BC-8 TPOT (ms)")
+    ax.set_title("B. Complete-order recovery gate")
+    ax.legend(frameon=False, fontsize=8)
+
+    fig.suptitle("EXP-DECODE-BATCH-GRID preregistered figure prototype")
+    fig.tight_layout()
+    fig.savefig(ROOT / "figure-prototype.png", dpi=180)
+    fig.savefig(ROOT / "figure-prototype.svg")
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-decode-batch-grid-v0/results/bc8-replay-verdict.json

@@ -0,0 +1,222 @@
+{
+  "all_corrected_batches_within_measured_support": true,
+  "decision": "STATIC_WHOLE_CURVE_FIXES_BC8",
+  "mns16_rankings_fast_to_slow": {
+    "original": [
+      "tp2_mns16",
+      "tp1_mns16",
+      "tp4_mns16"
+    ],
+    "real": [
+      "tp2_mns16",
+      "tp4_mns16",
+      "tp1_mns16"
+    ],
+    "whole_curve": [
+      "tp2_mns16",
+      "tp4_mns16",
+      "tp1_mns16"
+    ]
+  },
+  "points": [
+    {
+      "config": "tp1_mns16",
+      "mns": 16,
+      "original": {
+        "decode_b_gt_1_fraction": 0.0,
+        "decode_batch_histogram": {
+          "1": 16383
+        },
+        "decode_service_ms": {
+          "1": [
+            5.493362394
+          ]
+        },
+        "e2e_mean_ms": 864.3525078423435,
+        "tpot_mean_ms": 5.493362394020072,
+        "ttft_mean_ms": 166.6954838017941
+      },
+      "original_tpot_residual_ms": 0.616915450202109,
+      "real_tpot_mean_ms": 4.876446943817963,
+      "tp": 1,
+      "usage": {
+        "collective:moe:tp1-b1:structural-zero": 16383,
+        "moe:tp1-b1": 16383
+      },
+      "whole_curve": {
+        "decode_b_gt_1_fraction": 0.0,
+        "decode_batch_histogram": {
+          "1": 16383
+        },
+        "decode_service_ms": {
+          "1": [
+            5.493362394
+          ]
+        },
+        "e2e_mean_ms": 864.3525078423435,
+        "tpot_mean_ms": 5.493362394020072,
+        "ttft_mean_ms": 166.6954838017941
+      },
+      "whole_curve_tpot_residual_ms": 0.616915450202109
+    },
+    {
+      "config": "tp2_mns16",
+      "mns": 16,
+      "original": {
+        "decode_b_gt_1_fraction": 0.0,
+        "decode_batch_histogram": {
+          "1": 16383
+        },
+        "decode_service_ms": {
+          "1": [
+            4.928319445
+          ]
+        },
+        "e2e_mean_ms": 744.4521695355745,
+        "tpot_mean_ms": 4.9283194445308744,
+        "ttft_mean_ms": 118.55560008015348
+      },
+      "original_tpot_residual_ms": 0.6235698385840074,
+      "real_tpot_mean_ms": 4.304749605946867,
+      "tp": 2,
+      "usage": {
+        "collective:attention:tp2-b1": 16383,
+        "collective:moe:tp2-b1": 16383,
+        "fused_norm_deduction:attn:tp2-b1": 16383,
+        "fused_norm_deduction:mlp:tp2-b1": 16383,
+        "moe:tp2-b1": 16383
+      },
+      "whole_curve": {
+        "decode_b_gt_1_fraction": 0.0,
+        "decode_batch_histogram": {
+          "1": 16383
+        },
+        "decode_service_ms": {
+          "1": [
+            4.928319445
+          ]
+        },
+        "e2e_mean_ms": 744.4521695355745,
+        "tpot_mean_ms": 4.9283194445308744,
+        "ttft_mean_ms": 118.55560008015348
+      },
+      "whole_curve_tpot_residual_ms": 0.6235698385840074
+    },
+    {
+      "config": "tp4_mns16",
+      "mns": 16,
+      "original": {
+        "decode_b_gt_1_fraction": 0.4814983594604448,
+        "decode_batch_histogram": {
+          "1": 5689,
+          "2": 5283
+        },
+        "decode_service_ms": {
+          "1": [
+            4.738406304
+          ],
+          "2": [
+            5.411344412
+          ]
+        },
+        "e2e_mean_ms": 840.3052136178922,
+        "tpot_mean_ms": 5.863803518571768,
+        "ttft_mean_ms": 95.60216675927755
+      },
+      "original_tpot_residual_ms": 1.4143723087101598,
+      "real_tpot_mean_ms": 4.449431209861608,
+      "tp": 4,
+      "usage": {
+        "collective:attention:tp4-b1": 7393,
+        "collective:attention:tp4-b2": 4431,
+        "collective:moe:tp4-b1": 7393,
+        "collective:moe:tp4-b2": 4431,
+        "fused_norm_deduction:attn:tp4-b1": 7393,
+        "fused_norm_deduction:attn:tp4-b2": 4431,
+        "fused_norm_deduction:mlp:tp4-b1": 7393,
+        "fused_norm_deduction:mlp:tp4-b2": 4431,
+        "moe:tp4-b1": 7393,
+        "moe:tp4-b2": 4431,
+        "whole_decode:tp4-b2:target_ms=4.623176437": 4431
+      },
+      "whole_curve": {
+        "decode_b_gt_1_fraction": 0.37474627875507444,
+        "decode_batch_histogram": {
+          "1": 7393,
+          "2": 4431
+        },
+        "decode_service_ms": {
+          "1": [
+            4.738406304
+          ],
+          "2": [
+            4.623176437
+          ]
+        },
+        "e2e_mean_ms": 777.2050553866409,
+        "tpot_mean_ms": 5.367470055757525,
+        "ttft_mean_ms": 95.5363583054353
+      },
+      "whole_curve_tpot_residual_ms": 0.918038845895917
+    },
+    {
+      "config": "tp4_mns32",
+      "mns": 32,
+      "original": {
+        "decode_b_gt_1_fraction": 0.4814983594604448,
+        "decode_batch_histogram": {
+          "1": 5689,
+          "2": 5283
+        },
+        "decode_service_ms": {
+          "1": [
+            4.738406304
+          ],
+          "2": [
+            5.411344412
+          ]
+        },
+        "e2e_mean_ms": 840.3052136178922,
+        "tpot_mean_ms": 5.863803518571768,
+        "ttft_mean_ms": 95.60216675927755
+      },
+      "original_tpot_residual_ms": 1.4977765516959103,
+      "real_tpot_mean_ms": 4.366026966875857,
+      "tp": 4,
+      "usage": {
+        "collective:attention:tp4-b1": 7393,
+        "collective:attention:tp4-b2": 4431,
+        "collective:moe:tp4-b1": 7393,
+        "collective:moe:tp4-b2": 4431,
+        "fused_norm_deduction:attn:tp4-b1": 7393,
+        "fused_norm_deduction:attn:tp4-b2": 4431,
+        "fused_norm_deduction:mlp:tp4-b1": 7393,
+        "fused_norm_deduction:mlp:tp4-b2": 4431,
+        "moe:tp4-b1": 7393,
+        "moe:tp4-b2": 4431,
+        "whole_decode:tp4-b2:target_ms=4.623176437": 4431
+      },
+      "whole_curve": {
+        "decode_b_gt_1_fraction": 0.37474627875507444,
+        "decode_batch_histogram": {
+          "1": 7393,
+          "2": 4431
+        },
+        "decode_service_ms": {
+          "1": [
+            4.738406304
+          ],
+          "2": [
+            4.623176437
+          ]
+        },
+        "e2e_mean_ms": 777.2050553866409,
+        "tpot_mean_ms": 5.367470055757525,
+        "ttft_mean_ms": 95.5363583054353
+      },
+      "whole_curve_tpot_residual_ms": 1.0014430888816674
+    }
+  ],
+  "schema": "frontier-decode-grid-bc8-verdict.v1",
+  "whole_curve_restores_real_mns16_ranking": true
+}

runs/frontier-decode-batch-grid-v0/results/bc8-replay-verdict.md

@@ -0,0 +1,13 @@
+# BC-8 whole-layer decode-curve replay
+
+| Config | Real TPOT | Original sim | Whole-curve sim | Residual before -> after | b>1 |
+|---|---:|---:|---:|---:|---:|
+| tp1_mns16 | 4.876 | 5.493 | 5.493 | +0.617 -> +0.617 | 0.00% |
+| tp2_mns16 | 4.305 | 4.928 | 4.928 | +0.624 -> +0.624 | 0.00% |
+| tp4_mns16 | 4.449 | 5.864 | 5.367 | +1.414 -> +0.918 | 37.47% |
+| tp4_mns32 | 4.366 | 5.864 | 5.367 | +1.498 -> +1.001 | 37.47% |
+
+- Real MNS16: `tp2_mns16 < tp4_mns16 < tp1_mns16`
+- Original: `tp2_mns16 < tp1_mns16 < tp4_mns16`
+- Whole curve: `tp2_mns16 < tp4_mns16 < tp1_mns16`
+- Decision: **STATIC_WHOLE_CURVE_FIXES_BC8**.

runs/frontier-decode-batch-grid-v0/results/grid.json

@@ -0,0 +1,410 @@
+{
+  "all_repeat_gates_resolved": true,
+  "cells": [
+    {
+      "all_repeat_cv_pct": 0.33215115035261283,
+      "batch": 2,
+      "initial_repeat_cv_pct": 0.33215115035261283,
+      "median_component_ms": {
+        "attention": 0.7968654999999999,
+        "collective": 0.4589650625,
+        "linear_norm_rope": 0.9332466874999996,
+        "moe": 2.1423742343749987,
+        "other": 0.020707093750000002,
+        "output_head": 0.08698887500000001,
+        "router": 0.19049718749999994
+      },
+      "median_execute_ms": 4.74559925,
+      "needs_repeat_3": false,
+      "repeat_gate_resolved": true,
+      "repeats": [
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.7962288124999999,
+            "collective": 0.470928375,
+            "linear_norm_rope": 0.9330912187499996,
+            "moe": 2.136206281249999,
+            "other": 0.02065321875,
+            "output_head": 0.08696040625000001,
+            "router": 0.19041484374999995
+          },
+          "execute_mean_ms": 4.7613618125,
+          "repeat": 1,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp2-b2-r1.json"
+        },
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.7975021874999999,
+            "collective": 0.44700175,
+            "linear_norm_rope": 0.9334021562499997,
+            "moe": 2.1485421874999986,
+            "other": 0.02076096875,
+            "output_head": 0.08701734375,
+            "router": 0.19057953124999993
+          },
+          "execute_mean_ms": 4.7298366875,
+          "repeat": 2,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp2-b2-r2.json"
+        }
+      ],
+      "tp": 2
+    },
+    {
+      "all_repeat_cv_pct": 2.8664467148791393,
+      "batch": 4,
+      "initial_repeat_cv_pct": 2.8664467148791393,
+      "median_component_ms": {
+        "attention": 0.925814669047619,
+        "collective": 0.47613457857142855,
+        "linear_norm_rope": 0.926101651190476,
+        "moe": 2.5396719488095236,
+        "other": 0.021773490476190475,
+        "output_head": 0.08747840476190476,
+        "router": 0.18771020476190473
+      },
+      "median_execute_ms": 5.45066564047619,
+      "needs_repeat_3": false,
+      "repeat_gate_resolved": true,
+      "repeats": [
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.9284945714285713,
+            "collective": 0.4770398571428571,
+            "linear_norm_rope": 0.9251340357142854,
+            "moe": 2.5554174642857146,
+            "other": 0.021663214285714286,
+            "output_head": 0.08753164285714285,
+            "router": 0.1898966428571428
+          },
+          "execute_mean_ms": 5.294425214285715,
+          "repeat": 1,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp2-b4-r1.json"
+        },
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.9231347666666667,
+            "collective": 0.47522929999999997,
+            "linear_norm_rope": 0.9270692666666666,
+            "moe": 2.523926433333333,
+            "other": 0.021883766666666665,
+            "output_head": 0.08742516666666666,
+            "router": 0.18552376666666665
+          },
+          "execute_mean_ms": 5.606906066666666,
+          "repeat": 2,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp2-b4-r2.json"
+        }
+      ],
+      "tp": 2
+    },
+    {
+      "all_repeat_cv_pct": 2.718954157967049,
+      "batch": 6,
+      "initial_repeat_cv_pct": 2.718954157967049,
+      "median_component_ms": {
+        "attention": 1.0526737664835166,
+        "collective": 0.48269348763736264,
+        "linear_norm_rope": 0.9301222623626373,
+        "moe": 3.6929154436813176,
+        "other": 0.025437872252747254,
+        "output_head": 0.08800730082417582,
+        "router": 0.196665739010989
+      },
+      "median_execute_ms": 6.754197167582418,
+      "needs_repeat_3": false,
+      "repeat_gate_resolved": true,
+      "repeats": [
+        {
+          "component_rank_mean_ms": {
+            "attention": 1.0590365714285717,
+            "collective": 0.48200882142857143,
+            "linear_norm_rope": 0.9312401785714285,
+            "moe": 3.678020964285713,
+            "other": 0.02528782142857143,
+            "output_head": 0.08801067857142857,
+            "router": 0.1985437857142856
+          },
+          "execute_mean_ms": 6.570553642857143,
+          "repeat": 1,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp2-b6-r1.json"
+        },
+        {
+          "component_rank_mean_ms": {
+            "attention": 1.0463109615384618,
+            "collective": 0.48337815384615385,
+            "linear_norm_rope": 0.929004346153846,
+            "moe": 3.7078099230769217,
+            "other": 0.025587923076923078,
+            "output_head": 0.08800392307692309,
+            "router": 0.19478769230769236
+          },
+          "execute_mean_ms": 6.9378406923076925,
+          "repeat": 2,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp2-b6-r2.json"
+        }
+      ],
+      "tp": 2
+    },
+    {
+      "all_repeat_cv_pct": 2.7838528598972516,
+      "batch": 8,
+      "initial_repeat_cv_pct": 2.7838528598972516,
+      "median_component_ms": {
+        "attention": 1.408474403846154,
+        "collective": 0.5025935769230769,
+        "linear_norm_rope": 0.9324356730769228,
+        "moe": 2.6870017499999985,
+        "other": 0.02355823076923077,
+        "output_head": 0.08828742307692307,
+        "router": 0.19532132692307688
+      },
+      "median_execute_ms": 6.138216769230769,
+      "needs_repeat_3": false,
+      "repeat_gate_resolved": true,
+      "repeats": [
+        {
+          "component_rank_mean_ms": {
+            "attention": 1.4162915000000003,
+            "collective": 0.5046254615384616,
+            "linear_norm_rope": 0.933206192307692,
+            "moe": 2.681186538461537,
+            "other": 0.023443884615384616,
+            "output_head": 0.08826711538461537,
+            "router": 0.19981553846153838
+          },
+          "execute_mean_ms": 5.967337846153845,
+          "repeat": 1,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp2-b8-r1.json"
+        },
+        {
+          "component_rank_mean_ms": {
+            "attention": 1.400657307692308,
+            "collective": 0.5005616923076923,
+            "linear_norm_rope": 0.9316651538461536,
+            "moe": 2.69281696153846,
+            "other": 0.023672576923076925,
+            "output_head": 0.08830773076923076,
+            "router": 0.1908271153846154
+          },
+          "execute_mean_ms": 6.309095692307692,
+          "repeat": 2,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp2-b8-r2.json"
+        }
+      ],
+      "tp": 2
+    },
+    {
+      "all_repeat_cv_pct": 0.06699668597711499,
+      "batch": 2,
+      "initial_repeat_cv_pct": 0.06699668597711499,
+      "median_component_ms": {
+        "attention": 0.721186765625,
+        "collective": 0.5570387890625,
+        "linear_norm_rope": 0.8363623671875,
+        "moe": 1.7957157499999985,
+        "other": 0.020733953125000003,
+        "output_head": 0.045581921875,
+        "router": 0.18678388281249997
+      },
+      "median_execute_ms": 4.6231764375,
+      "needs_repeat_3": false,
+      "repeat_gate_resolved": true,
+      "repeats": [
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.72195021875,
+            "collective": 0.553751203125,
+            "linear_norm_rope": 0.8375226249999999,
+            "moe": 1.7999987343749986,
+            "other": 0.020717078125,
+            "output_head": 0.0456011875,
+            "router": 0.18689503124999995
+          },
+          "execute_mean_ms": 4.6262738125,
+          "repeat": 1,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp4-b2-r1.json"
+        },
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.7204233125,
+            "collective": 0.560326375,
+            "linear_norm_rope": 0.8352021093750001,
+            "moe": 1.7914327656249986,
+            "other": 0.020750828125000002,
+            "output_head": 0.04556265625,
+            "router": 0.18667273437499995
+          },
+          "execute_mean_ms": 4.6200790625,
+          "repeat": 2,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp4-b2-r2.json"
+        }
+      ],
+      "tp": 4
+    },
+    {
+      "all_repeat_cv_pct": 2.2737660875555155,
+      "batch": 4,
+      "initial_repeat_cv_pct": 2.2737660875555155,
+      "median_component_ms": {
+        "attention": 0.8017847142857144,
+        "collective": 0.5969553392857143,
+        "linear_norm_rope": 0.8391979553571427,
+        "moe": 1.9608389732142864,
+        "other": 0.021788607142857144,
+        "output_head": 0.04581030357142857,
+        "router": 0.18947951785714276
+      },
+      "median_execute_ms": 4.6215097499999995,
+      "needs_repeat_3": false,
+      "repeat_gate_resolved": true,
+      "repeats": [
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.8023426607142858,
+            "collective": 0.5361581428571428,
+            "linear_norm_rope": 0.8392915178571427,
+            "moe": 1.9595749285714295,
+            "other": 0.021800214285714284,
+            "output_head": 0.04583946428571428,
+            "router": 0.1894944999999999
+          },
+          "execute_mean_ms": 4.516427428571428,
+          "repeat": 1,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp4-b4-r1.json"
+        },
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.801226767857143,
+            "collective": 0.6577525357142857,
+            "linear_norm_rope": 0.8391043928571427,
+            "moe": 1.9621030178571435,
+            "other": 0.021777,
+            "output_head": 0.045781142857142854,
+            "router": 0.18946453571428562
+          },
+          "execute_mean_ms": 4.726592071428572,
+          "repeat": 2,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp4-b4-r2.json"
+        }
+      ],
+      "tp": 4
+    },
+    {
+      "all_repeat_cv_pct": 24.383571784490655,
+      "batch": 6,
+      "initial_repeat_cv_pct": 30.304513890111306,
+      "median_component_ms": {
+        "attention": 0.926535269230769,
+        "collective": 3.192824375,
+        "linear_norm_rope": 0.8382221538461538,
+        "moe": 2.383938624999999,
+        "other": 0.025532517857142854,
+        "output_head": 0.04836294642857143,
+        "router": 0.19243757142857137
+      },
+      "median_execute_ms": 8.633801928571428,
+      "needs_repeat_3": true,
+      "repeat_gate_resolved": true,
+      "repeats": [
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.926535269230769,
+            "collective": 0.5587644230769231,
+            "linear_norm_rope": 0.8382221538461538,
+            "moe": 2.378592788461537,
+            "other": 0.02582126923076923,
+            "output_head": 0.048266576923076926,
+            "router": 0.1886060961538461
+          },
+          "execute_mean_ms": 5.43262976923077,
+          "repeat": 1,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp4-b6-r1.json"
+        },
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.9215755535714284,
+            "collective": 4.099890464285714,
+            "linear_norm_rope": 0.8376546964285715,
+            "moe": 2.383938624999999,
+            "other": 0.02548019642857143,
+            "output_head": 0.04836294642857143,
+            "router": 0.19243757142857137
+          },
+          "execute_mean_ms": 10.1569875,
+          "repeat": 2,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp4-b6-r2.json"
+        },
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.9331604285714286,
+            "collective": 3.192824375,
+            "linear_norm_rope": 0.8404031785714285,
+            "moe": 2.399789857142857,
+            "other": 0.025532517857142854,
+            "output_head": 0.048368125,
+            "router": 0.19489794642857136
+          },
+          "execute_mean_ms": 8.633801928571428,
+          "repeat": 3,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp4-b6-r3.json"
+        }
+      ],
+      "tp": 4
+    },
+    {
+      "all_repeat_cv_pct": 1.4955512501139812,
+      "batch": 8,
+      "initial_repeat_cv_pct": 1.4955512501139812,
+      "median_component_ms": {
+        "attention": 0.9170377788461537,
+        "collective": 0.6205275096153846,
+        "linear_norm_rope": 0.8397761249999998,
+        "moe": 2.0145085576923076,
+        "other": 0.023612125,
+        "output_head": 0.04597551923076923,
+        "router": 0.1919322211538461
+      },
+      "median_execute_ms": 4.972498884615385,
+      "needs_repeat_3": false,
+      "repeat_gate_resolved": true,
+      "repeats": [
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.9206188846153844,
+            "collective": 0.6864052307692307,
+            "linear_norm_rope": 0.8407714807692306,
+            "moe": 2.0225017884615384,
+            "other": 0.023575519230769233,
+            "output_head": 0.04601186538461538,
+            "router": 0.19361455769230762
+          },
+          "execute_mean_ms": 4.898132615384616,
+          "repeat": 1,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp4-b8-r1.json"
+        },
+        {
+          "component_rank_mean_ms": {
+            "attention": 0.9134566730769228,
+            "collective": 0.5546497884615385,
+            "linear_norm_rope": 0.8387807692307689,
+            "moe": 2.0065153269230764,
+            "other": 0.02364873076923077,
+            "output_head": 0.04593917307692308,
+            "router": 0.19024988461538456
+          },
+          "execute_mean_ms": 5.046865153846154,
+          "repeat": 2,
+          "source": "runs/frontier-decode-batch-grid-v0/results/tp4-b8-r2.json"
+        }
+      ],
+      "tp": 4
+    }
+  ],
+  "contract": {
+    "repeat_aggregation": "median of two fresh processes; three if unstable",
+    "stability_gate": "r1/r2 population CV <=10%; else require r3 and take median",
+    "timing": "slowest-rank execute mean over 16 pure-decode steps",
+    "workload": "Qwen3-30B-A3B BF16, 2048->128, graph-on, MNS=16"
+  },
+  "schema": "frontier-decode-batch-grid.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp2-b2-r1.json

@@ -0,0 +1,185 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp2-b2-r1",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.7962288124999999,
+      "collective": 0.470928375,
+      "linear_norm_rope": 0.9330912187499996,
+      "moe": 2.136206281249999,
+      "other": 0.02065321875,
+      "output_head": 0.08696040625000001,
+      "router": 0.19041484374999995
+    },
+    "ranks": 2,
+    "slowest_rank_execute_mean_ms": 4.7613618125,
+    "slowest_rank_kernel_busy_mean_ms": 4.657866374999998
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7951932499999999,
+          "n": 16,
+          "p50_ms": 0.7978159999999996,
+          "p95_ms": 0.8009929999999998,
+          "population_std_ms": 0.004837302056673742
+        },
+        "collective": {
+          "mean_ms": 0.4532200625,
+          "n": 16,
+          "p50_ms": 0.44667250000000003,
+          "p95_ms": 0.579522,
+          "population_std_ms": 0.032947570582800075
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9318721874999996,
+          "n": 16,
+          "p50_ms": 0.9318319999999995,
+          "p95_ms": 0.9354359999999993,
+          "population_std_ms": 0.001684765236566671
+        },
+        "moe": {
+          "mean_ms": 2.133037374999999,
+          "n": 16,
+          "p50_ms": 2.260420499999999,
+          "p95_ms": 2.2688760000000006,
+          "population_std_ms": 0.1538413522138454
+        },
+        "other": {
+          "mean_ms": 0.0205200625,
+          "n": 16,
+          "p50_ms": 0.0205595,
+          "p95_ms": 0.020864,
+          "population_std_ms": 0.0002014638518289322
+        },
+        "output_head": {
+          "mean_ms": 0.086960375,
+          "n": 16,
+          "p50_ms": 0.087073,
+          "p95_ms": 0.087424,
+          "population_std_ms": 0.0003634009278675549
+        },
+        "router": {
+          "mean_ms": 0.19029662499999994,
+          "n": 16,
+          "p50_ms": 0.19019399999999992,
+          "p95_ms": 0.19120399999999993,
+          "population_std_ms": 0.00036799488362612264
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.7613618125,
+        "n": 16,
+        "p50_ms": 4.842258,
+        "p95_ms": 5.225267,
+        "population_std_ms": 0.20309347556335333
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.611099937499998,
+        "n": 16,
+        "p50_ms": 4.738485999999997,
+        "p95_ms": 4.880249,
+        "population_std_ms": 0.17253325791549196
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.15026187500000182,
+        "n": 16,
+        "p50_ms": 0.10683750000000192,
+        "p95_ms": 0.7480410000000006,
+        "population_std_ms": 0.15527935203322837
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b2-r1-20260723-20260723T091051245390Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b2-r1/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798066564146891.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7972643749999999,
+          "n": 16,
+          "p50_ms": 0.8003265,
+          "p95_ms": 0.8037429999999999,
+          "population_std_ms": 0.005557197617448423
+        },
+        "collective": {
+          "mean_ms": 0.4886366875,
+          "n": 16,
+          "p50_ms": 0.4386005,
+          "p95_ms": 1.0788600000000004,
+          "population_std_ms": 0.15686321799051514
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9343102499999997,
+          "n": 16,
+          "p50_ms": 0.9339574999999996,
+          "p95_ms": 0.9376900000000002,
+          "population_std_ms": 0.0018319578836589897
+        },
+        "moe": {
+          "mean_ms": 2.1393751874999984,
+          "n": 16,
+          "p50_ms": 2.2672624999999984,
+          "p95_ms": 2.276405999999998,
+          "population_std_ms": 0.15445752086424064
+        },
+        "other": {
+          "mean_ms": 0.020786375,
+          "n": 16,
+          "p50_ms": 0.020785,
+          "p95_ms": 0.021567999999999997,
+          "population_std_ms": 0.0003711985444139028
+        },
+        "output_head": {
+          "mean_ms": 0.0869604375,
+          "n": 16,
+          "p50_ms": 0.087008,
+          "p95_ms": 0.087937,
+          "population_std_ms": 0.0004370792217593405
+        },
+        "router": {
+          "mean_ms": 0.19053306249999993,
+          "n": 16,
+          "p50_ms": 0.19049349999999995,
+          "p95_ms": 0.19151799999999988,
+          "population_std_ms": 0.0003664594910678971
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.7613106875,
+        "n": 16,
+        "p50_ms": 4.8443950000000005,
+        "p95_ms": 5.2236769999999995,
+        "population_std_ms": 0.20362942305009554
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.657866374999998,
+        "n": 16,
+        "p50_ms": 4.739760999999998,
+        "p95_ms": 5.121143999999998,
+        "population_std_ms": 0.20551660892883905
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.10344431250000213,
+        "n": 16,
+        "p50_ms": 0.10326550000000179,
+        "p95_ms": 0.10955400000000282,
+        "population_std_ms": 0.002999292535722995
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b2-r1-20260723-20260723T091051245390Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b2-r1/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798066565640462.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp2-b2-r2.json

@@ -0,0 +1,185 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp2-b2-r2",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.7975021874999999,
+      "collective": 0.44700175,
+      "linear_norm_rope": 0.9334021562499997,
+      "moe": 2.1485421874999986,
+      "other": 0.02076096875,
+      "output_head": 0.08701734375,
+      "router": 0.19057953124999993
+    },
+    "ranks": 2,
+    "slowest_rank_execute_mean_ms": 4.7298366875,
+    "slowest_rank_kernel_busy_mean_ms": 4.625673687499998
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7957946874999998,
+          "n": 16,
+          "p50_ms": 0.7977499999999997,
+          "p95_ms": 0.8024729999999999,
+          "population_std_ms": 0.0050081016328389115
+        },
+        "collective": {
+          "mean_ms": 0.4559536875,
+          "n": 16,
+          "p50_ms": 0.44857899999999995,
+          "p95_ms": 0.5749780000000001,
+          "population_std_ms": 0.03100173565044134
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9314522499999998,
+          "n": 16,
+          "p50_ms": 0.9312219999999996,
+          "p95_ms": 0.9362239999999998,
+          "population_std_ms": 0.0018965892906741145
+        },
+        "moe": {
+          "mean_ms": 2.144734312499999,
+          "n": 16,
+          "p50_ms": 2.2638159999999994,
+          "p95_ms": 2.2729309999999985,
+          "population_std_ms": 0.16050482071137254
+        },
+        "other": {
+          "mean_ms": 0.0205803125,
+          "n": 16,
+          "p50_ms": 0.020513,
+          "p95_ms": 0.021342,
+          "population_std_ms": 0.00034404573074483917
+        },
+        "output_head": {
+          "mean_ms": 0.0869003125,
+          "n": 16,
+          "p50_ms": 0.086897,
+          "p95_ms": 0.087585,
+          "population_std_ms": 0.0003536547961554459
+        },
+        "router": {
+          "mean_ms": 0.19025812499999992,
+          "n": 16,
+          "p50_ms": 0.19022449999999994,
+          "p95_ms": 0.19071799999999992,
+          "population_std_ms": 0.00028991245984778943
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.72958625,
+        "n": 16,
+        "p50_ms": 4.841987,
+        "p95_ms": 4.974675,
+        "population_std_ms": 0.1744621378768544
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.625673687499998,
+        "n": 16,
+        "p50_ms": 4.7410514999999975,
+        "p95_ms": 4.871186999999999,
+        "population_std_ms": 0.1763796075070467
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.10391256250000158,
+        "n": 16,
+        "p50_ms": 0.10354800000000175,
+        "p95_ms": 0.10959400000000219,
+        "population_std_ms": 0.0028733005187579386
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b2-r2-20260723-20260723T092320882080Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b2-r2/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798812071346091.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7992096874999999,
+          "n": 16,
+          "p50_ms": 0.8017949999999999,
+          "p95_ms": 0.8056339999999999,
+          "population_std_ms": 0.00523807464292403
+        },
+        "collective": {
+          "mean_ms": 0.4380498125,
+          "n": 16,
+          "p50_ms": 0.4294910000000001,
+          "p95_ms": 0.56768,
+          "population_std_ms": 0.033749115542519664
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9353520624999997,
+          "n": 16,
+          "p50_ms": 0.9359214999999996,
+          "p95_ms": 0.9375480000000005,
+          "population_std_ms": 0.0017075802861340613
+        },
+        "moe": {
+          "mean_ms": 2.1523500624999987,
+          "n": 16,
+          "p50_ms": 2.2702614999999993,
+          "p95_ms": 2.284628,
+          "population_std_ms": 0.16187233535855888
+        },
+        "other": {
+          "mean_ms": 0.020941625000000002,
+          "n": 16,
+          "p50_ms": 0.0209265,
+          "p95_ms": 0.021598,
+          "population_std_ms": 0.0003365376789826063
+        },
+        "output_head": {
+          "mean_ms": 0.087134375,
+          "n": 16,
+          "p50_ms": 0.087281,
+          "p95_ms": 0.087585,
+          "population_std_ms": 0.0004719001317810776
+        },
+        "router": {
+          "mean_ms": 0.19090093749999995,
+          "n": 16,
+          "p50_ms": 0.19088099999999994,
+          "p95_ms": 0.19164899999999993,
+          "population_std_ms": 0.0003892426089134459
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.7298366875,
+        "n": 16,
+        "p50_ms": 4.8450835,
+        "p95_ms": 4.970972,
+        "population_std_ms": 0.1741141631820623
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.6239385624999985,
+        "n": 16,
+        "p50_ms": 4.7412614999999985,
+        "p95_ms": 4.869705999999997,
+        "population_std_ms": 0.17688744584409086
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.10589812500000201,
+        "n": 16,
+        "p50_ms": 0.10663300000000042,
+        "p95_ms": 0.11212500000000247,
+        "population_std_ms": 0.003380934446920671
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b2-r2-20260723-20260723T092320882080Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b2-r2/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798812076480600.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp2-b4-r1.json

@@ -0,0 +1,189 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp2-b4-r1",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.9284945714285713,
+      "collective": 0.4770398571428571,
+      "linear_norm_rope": 0.9251340357142854,
+      "moe": 2.5554174642857146,
+      "other": 0.021663214285714286,
+      "output_head": 0.08753164285714285,
+      "router": 0.1898966428571428
+    },
+    "ranks": 2,
+    "slowest_rank_execute_mean_ms": 5.294425214285715,
+    "slowest_rank_kernel_busy_mean_ms": 5.187257214285713
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9336339285714287,
+          "n": 14,
+          "p50_ms": 0.933751,
+          "p95_ms": 0.9427530000000001,
+          "population_std_ms": 0.0036295537007084476
+        },
+        "collective": {
+          "mean_ms": 0.47076142857142855,
+          "n": 14,
+          "p50_ms": 0.4640835000000001,
+          "p95_ms": 0.5701469999999997,
+          "population_std_ms": 0.0277061237241214
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9244245714285712,
+          "n": 14,
+          "p50_ms": 0.9243979999999998,
+          "p95_ms": 0.9272680000000004,
+          "population_std_ms": 0.0014581096232689887
+        },
+        "moe": {
+          "mean_ms": 2.554062214285715,
+          "n": 14,
+          "p50_ms": 2.594499500000001,
+          "p95_ms": 2.694474,
+          "population_std_ms": 0.11681852632925924
+        },
+        "other": {
+          "mean_ms": 0.02227242857142857,
+          "n": 14,
+          "p50_ms": 0.022274000000000002,
+          "p95_ms": 0.022625,
+          "population_std_ms": 0.00020562612198068582
+        },
+        "output_head": {
+          "mean_ms": 0.087598,
+          "n": 14,
+          "p50_ms": 0.08764849999999999,
+          "p95_ms": 0.088257,
+          "population_std_ms": 0.00042077038190171175
+        },
+        "router": {
+          "mean_ms": 0.1903450714285714,
+          "n": 14,
+          "p50_ms": 0.19036999999999993,
+          "p95_ms": 0.19091199999999997,
+          "population_std_ms": 0.00033161088735480956
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_4(4)": 14,
+        "execute_context_3(6144)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.292617,
+        "n": 14,
+        "p50_ms": 5.351565000000001,
+        "p95_ms": 5.420654000000001,
+        "population_std_ms": 0.12026193009071014
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 5.183097642857143,
+        "n": 14,
+        "p50_ms": 5.2450280000000005,
+        "p95_ms": 5.312519,
+        "population_std_ms": 0.12140493871020902
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.10951935714285696,
+        "n": 14,
+        "p50_ms": 0.1093335000000013,
+        "p95_ms": 0.11392299999999889,
+        "population_std_ms": 0.002305499592809895
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b4-r1-20260723-20260723T091055280404Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b4-r1/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798075195956333.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9233552142857139,
+          "n": 14,
+          "p50_ms": 0.9231514999999997,
+          "p95_ms": 0.92672,
+          "population_std_ms": 0.002063045328030955
+        },
+        "collective": {
+          "mean_ms": 0.4833182857142857,
+          "n": 14,
+          "p50_ms": 0.47641600000000006,
+          "p95_ms": 0.5747199999999999,
+          "population_std_ms": 0.025529410631741578
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9258434999999997,
+          "n": 14,
+          "p50_ms": 0.9259644999999999,
+          "p95_ms": 0.9298849999999995,
+          "population_std_ms": 0.0017809597633218826
+        },
+        "moe": {
+          "mean_ms": 2.556772714285714,
+          "n": 14,
+          "p50_ms": 2.5972645000000014,
+          "p95_ms": 2.697822,
+          "population_std_ms": 0.11678101165455461
+        },
+        "other": {
+          "mean_ms": 0.021054,
+          "n": 14,
+          "p50_ms": 0.021136000000000002,
+          "p95_ms": 0.021632000000000002,
+          "population_std_ms": 0.0003493892630617853
+        },
+        "output_head": {
+          "mean_ms": 0.08746528571428572,
+          "n": 14,
+          "p50_ms": 0.087424,
+          "p95_ms": 0.087936,
+          "population_std_ms": 0.00027621250322672924
+        },
+        "router": {
+          "mean_ms": 0.18944821428571418,
+          "n": 14,
+          "p50_ms": 0.1893899999999999,
+          "p95_ms": 0.18998899999999994,
+          "population_std_ms": 0.00036123541563669255
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_4(4)": 14,
+        "execute_context_3(6144)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.294425214285715,
+        "n": 14,
+        "p50_ms": 5.3559085,
+        "p95_ms": 5.420421,
+        "population_std_ms": 0.12112366521521874
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 5.187257214285713,
+        "n": 14,
+        "p50_ms": 5.247959,
+        "p95_ms": 5.314969999999999,
+        "population_std_ms": 0.12129808972773216
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.10716800000000092,
+        "n": 14,
+        "p50_ms": 0.10673850000000096,
+        "p95_ms": 0.11011199999999732,
+        "population_std_ms": 0.0018639973942341857
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b4-r1-20260723-20260723T091055280404Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b4-r1/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798075201036386.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp2-b4-r2.json

@@ -0,0 +1,187 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp2-b4-r2",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.9231347666666667,
+      "collective": 0.47522929999999997,
+      "linear_norm_rope": 0.9270692666666666,
+      "moe": 2.523926433333333,
+      "other": 0.021883766666666665,
+      "output_head": 0.08742516666666666,
+      "router": 0.18552376666666665
+    },
+    "ranks": 2,
+    "slowest_rank_execute_mean_ms": 5.606906066666666,
+    "slowest_rank_kernel_busy_mean_ms": 5.148062666666666
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9221591999999998,
+          "n": 15,
+          "p50_ms": 0.9204839999999996,
+          "p95_ms": 0.9305610000000001,
+          "population_std_ms": 0.0055392075630603605
+        },
+        "collective": {
+          "mean_ms": 0.47690353333333335,
+          "n": 15,
+          "p50_ms": 0.4750740000000001,
+          "p95_ms": 0.5079370000000001,
+          "population_std_ms": 0.008762698616801168
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9266067333333331,
+          "n": 15,
+          "p50_ms": 0.9269229999999996,
+          "p95_ms": 0.9281389999999996,
+          "population_std_ms": 0.001160841388342339
+        },
+        "moe": {
+          "mean_ms": 2.5183345333333333,
+          "n": 15,
+          "p50_ms": 2.535845999999999,
+          "p95_ms": 2.6756949999999993,
+          "population_std_ms": 0.11626729790436485
+        },
+        "other": {
+          "mean_ms": 0.0223382,
+          "n": 15,
+          "p50_ms": 0.022207,
+          "p95_ms": 0.022974,
+          "population_std_ms": 0.00033397688542772
+        },
+        "output_head": {
+          "mean_ms": 0.08742633333333334,
+          "n": 15,
+          "p50_ms": 0.08752,
+          "p95_ms": 0.088224,
+          "population_std_ms": 0.00039636644100573626
+        },
+        "router": {
+          "mean_ms": 0.18655373333333325,
+          "n": 15,
+          "p50_ms": 0.1865609999999999,
+          "p95_ms": 0.1867189999999999,
+          "population_std_ms": 0.00011684261018805013
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_4(4)": 15,
+        "execute_context_1(2048)_generation_3(3)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.606906066666666,
+        "n": 15,
+        "p50_ms": 5.649197,
+        "p95_ms": 5.755214,
+        "population_std_ms": 0.1162303895094949
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 5.140322266666667,
+        "n": 15,
+        "p50_ms": 5.182554999999998,
+        "p95_ms": 5.287060999999999,
+        "population_std_ms": 0.11673111694743465
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4665838000000001,
+        "n": 15,
+        "p50_ms": 0.466642000000002,
+        "p95_ms": 0.47348399999999913,
+        "population_std_ms": 0.003150324304998906
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 15,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b4-r2-20260723-20260723T092324802748Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b4-r2/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798821005851469.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9241103333333336,
+          "n": 15,
+          "p50_ms": 0.9240880000000004,
+          "p95_ms": 0.9288660000000003,
+          "population_std_ms": 0.0027653286403045273
+        },
+        "collective": {
+          "mean_ms": 0.47355506666666664,
+          "n": 15,
+          "p50_ms": 0.4710439999999998,
+          "p95_ms": 0.5040649999999999,
+          "population_std_ms": 0.008578622861249649
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9275318,
+          "n": 15,
+          "p50_ms": 0.927647,
+          "p95_ms": 0.9290880000000004,
+          "population_std_ms": 0.0009876329412624172
+        },
+        "moe": {
+          "mean_ms": 2.529518333333333,
+          "n": 15,
+          "p50_ms": 2.546805999999999,
+          "p95_ms": 2.682659999999999,
+          "population_std_ms": 0.11457118978560389
+        },
+        "other": {
+          "mean_ms": 0.021429333333333335,
+          "n": 15,
+          "p50_ms": 0.021312000000000005,
+          "p95_ms": 0.022112,
+          "population_std_ms": 0.0003245893131669962
+        },
+        "output_head": {
+          "mean_ms": 0.087424,
+          "n": 15,
+          "p50_ms": 0.087488,
+          "p95_ms": 0.087936,
+          "population_std_ms": 0.0003762779114785578
+        },
+        "router": {
+          "mean_ms": 0.1844938,
+          "n": 15,
+          "p50_ms": 0.18451400000000007,
+          "p95_ms": 0.18483299999999994,
+          "population_std_ms": 0.0001811099849998921
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_4(4)": 15,
+        "execute_context_1(2048)_generation_3(3)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.6060968,
+        "n": 15,
+        "p50_ms": 5.65066,
+        "p95_ms": 5.7521,
+        "population_std_ms": 0.1166928070969243
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 5.148062666666666,
+        "n": 15,
+        "p50_ms": 5.194570999999999,
+        "p95_ms": 5.294759999999998,
+        "population_std_ms": 0.11587941756076509
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.45803413333333365,
+        "n": 15,
+        "p50_ms": 0.4584260000000002,
+        "p95_ms": 0.4647389999999998,
+        "population_std_ms": 0.003347060996688631
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 15,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b4-r2-20260723-20260723T092324802748Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b4-r2/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798821004197352.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp2-b6-r1.json

@@ -0,0 +1,189 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp2-b6-r1",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 1.0590365714285717,
+      "collective": 0.48200882142857143,
+      "linear_norm_rope": 0.9312401785714285,
+      "moe": 3.678020964285713,
+      "other": 0.02528782142857143,
+      "output_head": 0.08801067857142857,
+      "router": 0.1985437857142856
+    },
+    "ranks": 2,
+    "slowest_rank_execute_mean_ms": 6.570553642857143,
+    "slowest_rank_kernel_busy_mean_ms": 6.462748499999998
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 1.056612428571429,
+          "n": 14,
+          "p50_ms": 1.0567985000000006,
+          "p95_ms": 1.0597780000000006,
+          "population_std_ms": 0.0016962762879018302
+        },
+        "collective": {
+          "mean_ms": 0.4931447857142857,
+          "n": 14,
+          "p50_ms": 0.4922715000000001,
+          "p95_ms": 0.5031650000000001,
+          "population_std_ms": 0.0056488543235214805
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9297670714285715,
+          "n": 14,
+          "p50_ms": 0.9300385,
+          "p95_ms": 0.9327710000000003,
+          "population_std_ms": 0.0016833948889195887
+        },
+        "moe": {
+          "mean_ms": 3.6738179999999985,
+          "n": 14,
+          "p50_ms": 3.667117499999998,
+          "p95_ms": 3.9196679999999993,
+          "population_std_ms": 0.1767424054840424
+        },
+        "other": {
+          "mean_ms": 0.025222642857142857,
+          "n": 14,
+          "p50_ms": 0.025185,
+          "p95_ms": 0.025983,
+          "population_std_ms": 0.00032333101286073835
+        },
+        "output_head": {
+          "mean_ms": 0.08808714285714285,
+          "n": 14,
+          "p50_ms": 0.08807999999999999,
+          "p95_ms": 0.08870399999999999,
+          "population_std_ms": 0.0003292721100381563
+        },
+        "router": {
+          "mean_ms": 0.19489707142857132,
+          "n": 14,
+          "p50_ms": 0.19094449999999988,
+          "p95_ms": 0.23763299999999998,
+          "population_std_ms": 0.012033189735443558
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_6(6)": 14,
+        "execute_context_2(2049)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 6.569652285714286,
+        "n": 14,
+        "p50_ms": 6.5619125,
+        "p95_ms": 6.809498,
+        "population_std_ms": 0.16639578247764245
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 6.461549142857142,
+        "n": 14,
+        "p50_ms": 6.452550999999998,
+        "p95_ms": 6.705629999999999,
+        "population_std_ms": 0.166627622430915
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.10810314285714392,
+        "n": 14,
+        "p50_ms": 0.10853050000000275,
+        "p95_ms": 0.1127690000000019,
+        "population_std_ms": 0.0025825165428091904
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b6-r1-20260723-20260723T091712228992Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b6-r1/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798447891492657.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 1.0614607142857144,
+          "n": 14,
+          "p50_ms": 1.0611505000000003,
+          "p95_ms": 1.0646370000000003,
+          "population_std_ms": 0.0017739766639056317
+        },
+        "collective": {
+          "mean_ms": 0.47087285714285715,
+          "n": 14,
+          "p50_ms": 0.46880499999999997,
+          "p95_ms": 0.48576,
+          "population_std_ms": 0.005842167135296126
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9327132857142855,
+          "n": 14,
+          "p50_ms": 0.9321509999999997,
+          "p95_ms": 0.9383449999999995,
+          "population_std_ms": 0.002275944997721476
+        },
+        "moe": {
+          "mean_ms": 3.6822239285714278,
+          "n": 14,
+          "p50_ms": 3.6781699999999997,
+          "p95_ms": 3.9217039999999996,
+          "population_std_ms": 0.17642357594982436
+        },
+        "other": {
+          "mean_ms": 0.025353000000000004,
+          "n": 14,
+          "p50_ms": 0.02536,
+          "p95_ms": 0.026433,
+          "population_std_ms": 0.0005135631829260575
+        },
+        "output_head": {
+          "mean_ms": 0.08793421428571428,
+          "n": 14,
+          "p50_ms": 0.0880325,
+          "p95_ms": 0.088833,
+          "population_std_ms": 0.0005008674159569053
+        },
+        "router": {
+          "mean_ms": 0.2021904999999999,
+          "n": 14,
+          "p50_ms": 0.1997179999999999,
+          "p95_ms": 0.237536,
+          "population_std_ms": 0.009894411602732434
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_6(6)": 14,
+        "execute_context_2(2049)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 6.570553642857143,
+        "n": 14,
+        "p50_ms": 6.5663730000000005,
+        "p95_ms": 6.809703000000001,
+        "population_std_ms": 0.16604228180350736
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 6.462748499999998,
+        "n": 14,
+        "p50_ms": 6.456941499999999,
+        "p95_ms": 6.703455999999998,
+        "population_std_ms": 0.16541565407168776
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.10780514285714433,
+        "n": 14,
+        "p50_ms": 0.10772100000000107,
+        "p95_ms": 0.1121450000000026,
+        "population_std_ms": 0.0019970330441916122
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b6-r1-20260723-20260723T091712228992Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b6-r1/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798447895471197.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp2-b6-r2.json

@@ -0,0 +1,191 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp2-b6-r2",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 1.0463109615384618,
+      "collective": 0.48337815384615385,
+      "linear_norm_rope": 0.929004346153846,
+      "moe": 3.7078099230769217,
+      "other": 0.025587923076923078,
+      "output_head": 0.08800392307692309,
+      "router": 0.19478769230769236
+    },
+    "ranks": 2,
+    "slowest_rank_execute_mean_ms": 6.9378406923076925,
+    "slowest_rank_kernel_busy_mean_ms": 6.477923153846153
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 1.0448520000000003,
+          "n": 13,
+          "p50_ms": 1.045254000000001,
+          "p95_ms": 1.0465970000000002,
+          "population_std_ms": 0.0012648827856913572
+        },
+        "collective": {
+          "mean_ms": 0.4906066923076923,
+          "n": 13,
+          "p50_ms": 0.48854500000000006,
+          "p95_ms": 0.5128349999999999,
+          "population_std_ms": 0.007319711768018179
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9276054615384616,
+          "n": 13,
+          "p50_ms": 0.927922,
+          "p95_ms": 0.9300229999999996,
+          "population_std_ms": 0.0012124997456231853
+        },
+        "moe": {
+          "mean_ms": 3.7032171538461522,
+          "n": 13,
+          "p50_ms": 3.6941509999999984,
+          "p95_ms": 3.9345319999999977,
+          "population_std_ms": 0.18506379688215108
+        },
+        "other": {
+          "mean_ms": 0.02531215384615385,
+          "n": 13,
+          "p50_ms": 0.025345,
+          "p95_ms": 0.025984,
+          "population_std_ms": 0.0003609239950148982
+        },
+        "output_head": {
+          "mean_ms": 0.08801007692307693,
+          "n": 13,
+          "p50_ms": 0.087968,
+          "p95_ms": 0.08851200000000001,
+          "population_std_ms": 0.0003446964018777317
+        },
+        "router": {
+          "mean_ms": 0.19831961538461543,
+          "n": 13,
+          "p50_ms": 0.1956510000000001,
+          "p95_ms": 0.23433300000000007,
+          "population_std_ms": 0.010532589860037432
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_6(6)": 13,
+        "execute_context_2(2049)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 6.937815615384616,
+        "n": 13,
+        "p50_ms": 6.920955,
+        "p95_ms": 7.177563,
+        "population_std_ms": 0.17465452547487034
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 6.477923153846153,
+        "n": 13,
+        "p50_ms": 6.464021999999998,
+        "p95_ms": 6.714968999999998,
+        "population_std_ms": 0.17390358211193135
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4598924615384625,
+        "n": 13,
+        "p50_ms": 0.46010100000000165,
+        "p95_ms": 0.4628480000000019,
+        "population_std_ms": 0.0022957863280165477
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b6-r2-20260723-20260723T093020293408Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b6-r2/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784799236964172371.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 1.0477699230769233,
+          "n": 13,
+          "p50_ms": 1.0473580000000002,
+          "p95_ms": 1.0502500000000003,
+          "population_std_ms": 0.0016845846994043283
+        },
+        "collective": {
+          "mean_ms": 0.47614961538461537,
+          "n": 13,
+          "p50_ms": 0.47488799999999987,
+          "p95_ms": 0.4852860000000001,
+          "population_std_ms": 0.006279926721013708
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9304032307692304,
+          "n": 13,
+          "p50_ms": 0.9300610000000001,
+          "p95_ms": 0.9343409999999996,
+          "population_std_ms": 0.0015794439213190966
+        },
+        "moe": {
+          "mean_ms": 3.712402692307691,
+          "n": 13,
+          "p50_ms": 3.7038869999999973,
+          "p95_ms": 3.951868000000003,
+          "population_std_ms": 0.18554981438267218
+        },
+        "other": {
+          "mean_ms": 0.02586369230769231,
+          "n": 13,
+          "p50_ms": 0.025888,
+          "p95_ms": 0.026400000000000003,
+          "population_std_ms": 0.0002951522122606689
+        },
+        "output_head": {
+          "mean_ms": 0.08799776923076924,
+          "n": 13,
+          "p50_ms": 0.087937,
+          "p95_ms": 0.088992,
+          "population_std_ms": 0.00042456876834413246
+        },
+        "router": {
+          "mean_ms": 0.19125576923076928,
+          "n": 13,
+          "p50_ms": 0.18793900000000005,
+          "p95_ms": 0.23312300000000005,
+          "population_std_ms": 0.012158267147240543
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_6(6)": 13,
+        "execute_context_2(2049)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 6.9378406923076925,
+        "n": 13,
+        "p50_ms": 6.920646,
+        "p95_ms": 7.1769359999999995,
+        "population_std_ms": 0.17404824161814114
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 6.471842692307691,
+        "n": 13,
+        "p50_ms": 6.4540469999999965,
+        "p95_ms": 6.708258000000003,
+        "population_std_ms": 0.17518788350506015
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.465998000000001,
+        "n": 13,
+        "p50_ms": 0.4667860000000026,
+        "p95_ms": 0.4705280000000025,
+        "population_std_ms": 0.0037250578518997
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b6-r2-20260723-20260723T093020293408Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b6-r2/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784799236965538368.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp2-b8-r1.json

@@ -0,0 +1,191 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp2-b8-r1",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 1.4162915000000003,
+      "collective": 0.5046254615384616,
+      "linear_norm_rope": 0.933206192307692,
+      "moe": 2.681186538461537,
+      "other": 0.023443884615384616,
+      "output_head": 0.08826711538461537,
+      "router": 0.19981553846153838
+    },
+    "ranks": 2,
+    "slowest_rank_execute_mean_ms": 5.967337846153845,
+    "slowest_rank_kernel_busy_mean_ms": 5.849403615384613
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 1.4268944615384618,
+          "n": 13,
+          "p50_ms": 1.4273899999999997,
+          "p95_ms": 1.4289299999999996,
+          "population_std_ms": 0.0013227596802130133
+        },
+        "collective": {
+          "mean_ms": 0.4871666153846154,
+          "n": 13,
+          "p50_ms": 0.4847079999999998,
+          "p95_ms": 0.5161030000000001,
+          "population_std_ms": 0.009071831835526453
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9357503846153842,
+          "n": 13,
+          "p50_ms": 0.9351129999999993,
+          "p95_ms": 0.9375759999999999,
+          "population_std_ms": 0.0012191190727774577
+        },
+        "moe": {
+          "mean_ms": 2.6808982307692295,
+          "n": 13,
+          "p50_ms": 2.729251,
+          "p95_ms": 2.8241379999999983,
+          "population_std_ms": 0.11820978926868417
+        },
+        "other": {
+          "mean_ms": 0.023988000000000002,
+          "n": 13,
+          "p50_ms": 0.023935,
+          "p95_ms": 0.024639999999999995,
+          "population_std_ms": 0.0002731643630321658
+        },
+        "output_head": {
+          "mean_ms": 0.08829546153846153,
+          "n": 13,
+          "p50_ms": 0.088256,
+          "p95_ms": 0.08921599999999999,
+          "population_std_ms": 0.00048060836693389233
+        },
+        "router": {
+          "mean_ms": 0.2012756923076922,
+          "n": 13,
+          "p50_ms": 0.20156699999999994,
+          "p95_ms": 0.20582599999999993,
+          "population_std_ms": 0.0022949889352713113
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_1(2048)_generation_1(1)": 1,
+        "execute_context_3(4098)_generation_5(5)": 1,
+        "execute_context_4(8190)_generation_2(2)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.966980846153846,
+        "n": 13,
+        "p50_ms": 6.004943,
+        "p95_ms": 6.112047,
+        "population_std_ms": 0.11913819132925879
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 5.844268846153844,
+        "n": 13,
+        "p50_ms": 5.885293999999999,
+        "p95_ms": 5.987602999999998,
+        "population_std_ms": 0.11849233575983938
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.12271200000000157,
+        "n": 13,
+        "p50_ms": 0.12307100000000126,
+        "p95_ms": 0.12728500000000142,
+        "population_std_ms": 0.0027851573357023283
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b8-r1-20260723-20260723T091715999468Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b8-r1/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798455024962606.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 1.405688538461539,
+          "n": 13,
+          "p50_ms": 1.4057520000000006,
+          "p95_ms": 1.4114940000000005,
+          "population_std_ms": 0.0022431369386294517
+        },
+        "collective": {
+          "mean_ms": 0.5220843076923077,
+          "n": 13,
+          "p50_ms": 0.5193970000000001,
+          "p95_ms": 0.5542680000000001,
+          "population_std_ms": 0.010336781774023671
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9306619999999997,
+          "n": 13,
+          "p50_ms": 0.9303679999999996,
+          "p95_ms": 0.9333509999999994,
+          "population_std_ms": 0.0014067406955851874
+        },
+        "moe": {
+          "mean_ms": 2.6814748461538445,
+          "n": 13,
+          "p50_ms": 2.7311729999999965,
+          "p95_ms": 2.8229829999999962,
+          "population_std_ms": 0.11917174241136594
+        },
+        "other": {
+          "mean_ms": 0.02289976923076923,
+          "n": 13,
+          "p50_ms": 0.022943999999999996,
+          "p95_ms": 0.02352,
+          "population_std_ms": 0.0002890651121343674
+        },
+        "output_head": {
+          "mean_ms": 0.08823876923076922,
+          "n": 13,
+          "p50_ms": 0.088193,
+          "p95_ms": 0.088768,
+          "population_std_ms": 0.0003492319383240336
+        },
+        "router": {
+          "mean_ms": 0.19835538461538452,
+          "n": 13,
+          "p50_ms": 0.1982099999999999,
+          "p95_ms": 0.2010899999999999,
+          "population_std_ms": 0.0022037652233764197
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_1(2048)_generation_1(1)": 1,
+        "execute_context_3(4098)_generation_5(5)": 1,
+        "execute_context_4(8190)_generation_2(2)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.967337846153845,
+        "n": 13,
+        "p50_ms": 6.0104049999999996,
+        "p95_ms": 6.1097969999999995,
+        "population_std_ms": 0.11987574925913026
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 5.849403615384613,
+        "n": 13,
+        "p50_ms": 5.889734999999996,
+        "p95_ms": 5.993167999999996,
+        "population_std_ms": 0.11980136802451119
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.11793423076923243,
+        "n": 13,
+        "p50_ms": 0.11715500000000123,
+        "p95_ms": 0.12245800000000262,
+        "population_std_ms": 0.002861606035183301
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b8-r1-20260723-20260723T091715999468Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b8-r1/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798455030360286.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp2-b8-r2.json

@@ -0,0 +1,191 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp2-b8-r2",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 1.400657307692308,
+      "collective": 0.5005616923076923,
+      "linear_norm_rope": 0.9316651538461536,
+      "moe": 2.69281696153846,
+      "other": 0.023672576923076925,
+      "output_head": 0.08830773076923076,
+      "router": 0.1908271153846154
+    },
+    "ranks": 2,
+    "slowest_rank_execute_mean_ms": 6.309095692307692,
+    "slowest_rank_kernel_busy_mean_ms": 5.828548538461537
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 1.4095765384615386,
+          "n": 13,
+          "p50_ms": 1.4091810000000007,
+          "p95_ms": 1.4131269999999998,
+          "population_std_ms": 0.0017758118236840018
+        },
+        "collective": {
+          "mean_ms": 0.4864672307692308,
+          "n": 13,
+          "p50_ms": 0.4871060000000002,
+          "p95_ms": 0.5083250000000001,
+          "population_std_ms": 0.007854287590999577
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9326894615384612,
+          "n": 13,
+          "p50_ms": 0.9327419999999998,
+          "p95_ms": 0.9343809999999997,
+          "population_std_ms": 0.001099693356936376
+        },
+        "moe": {
+          "mean_ms": 2.692551615384614,
+          "n": 13,
+          "p50_ms": 2.7793569999999983,
+          "p95_ms": 2.871725999999997,
+          "population_std_ms": 0.17050402067063422
+        },
+        "other": {
+          "mean_ms": 0.024125384615384618,
+          "n": 13,
+          "p50_ms": 0.024064000000000002,
+          "p95_ms": 0.024864000000000008,
+          "population_std_ms": 0.00029315092948108124
+        },
+        "output_head": {
+          "mean_ms": 0.08821176923076922,
+          "n": 13,
+          "p50_ms": 0.08828799999999999,
+          "p95_ms": 0.088672,
+          "population_std_ms": 0.0004720397887150791
+        },
+        "router": {
+          "mean_ms": 0.19492653846153846,
+          "n": 13,
+          "p50_ms": 0.19584,
+          "p95_ms": 0.19798399999999997,
+          "population_std_ms": 0.002327761308652399
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_2(2051)_generation_6(6)": 1,
+        "execute_context_2(4096)_generation_1(1)": 1,
+        "execute_context_4(8189)_generation_3(3)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 6.309095692307692,
+        "n": 13,
+        "p50_ms": 6.398511,
+        "p95_ms": 6.483886999999999,
+        "population_std_ms": 0.1703797844741088
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 5.828548538461537,
+        "n": 13,
+        "p50_ms": 5.915967999999998,
+        "p95_ms": 6.002267999999997,
+        "population_std_ms": 0.17012899737805992
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4805471538461554,
+        "n": 13,
+        "p50_ms": 0.4803160000000002,
+        "p95_ms": 0.4859510000000009,
+        "population_std_ms": 0.0025392862972511096
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b8-r2-20260723-20260723T093023994242Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b8-r2/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784799245841164026.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 1.3917380769230774,
+          "n": 13,
+          "p50_ms": 1.3920280000000005,
+          "p95_ms": 1.3950360000000008,
+          "population_std_ms": 0.001887833412693738
+        },
+        "collective": {
+          "mean_ms": 0.5146561538461538,
+          "n": 13,
+          "p50_ms": 0.5156739999999999,
+          "p95_ms": 0.5248290000000001,
+          "population_std_ms": 0.00624208518964558
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.9306408461538459,
+          "n": 13,
+          "p50_ms": 0.9303399999999994,
+          "p95_ms": 0.9336909999999994,
+          "population_std_ms": 0.0015690834093706387
+        },
+        "moe": {
+          "mean_ms": 2.693082307692306,
+          "n": 13,
+          "p50_ms": 2.778146,
+          "p95_ms": 2.867843999999997,
+          "population_std_ms": 0.17102703603069247
+        },
+        "other": {
+          "mean_ms": 0.02321976923076923,
+          "n": 13,
+          "p50_ms": 0.023139,
+          "p95_ms": 0.023648000000000002,
+          "population_std_ms": 0.00028077369859458957
+        },
+        "output_head": {
+          "mean_ms": 0.08840369230769231,
+          "n": 13,
+          "p50_ms": 0.08832,
+          "p95_ms": 0.089408,
+          "population_std_ms": 0.0004100955582104095
+        },
+        "router": {
+          "mean_ms": 0.18672769230769232,
+          "n": 13,
+          "p50_ms": 0.18675,
+          "p95_ms": 0.189218,
+          "population_std_ms": 0.0011293141134617489
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_2(2051)_generation_6(6)": 1,
+        "execute_context_2(4096)_generation_1(1)": 1,
+        "execute_context_4(8189)_generation_3(3)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 6.307694615384615,
+        "n": 13,
+        "p50_ms": 6.399717,
+        "p95_ms": 6.488933,
+        "population_std_ms": 0.17066632055317688
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 5.828468538461537,
+        "n": 13,
+        "p50_ms": 5.9137319999999995,
+        "p95_ms": 6.007808999999997,
+        "population_std_ms": 0.17195541813398352
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4792260769230788,
+        "n": 13,
+        "p50_ms": 0.4783710000000063,
+        "p95_ms": 0.4893030000000005,
+        "population_std_ms": 0.00435339587097986
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp2-b8-r2-20260723-20260723T093023994242Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp2-b8-r2/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784799245841819751.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp4-b2-r1.json

@@ -0,0 +1,345 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp4-b2-r1",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.72195021875,
+      "collective": 0.553751203125,
+      "linear_norm_rope": 0.8375226249999999,
+      "moe": 1.7999987343749986,
+      "other": 0.020717078125,
+      "output_head": 0.0456011875,
+      "router": 0.18689503124999995
+    },
+    "ranks": 4,
+    "slowest_rank_execute_mean_ms": 4.6262738125,
+    "slowest_rank_kernel_busy_mean_ms": 4.1718804999999985
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.721057875,
+          "n": 16,
+          "p50_ms": 0.711759,
+          "p95_ms": 0.7497570000000005,
+          "population_std_ms": 0.014212609814329528
+        },
+        "collective": {
+          "mean_ms": 0.50711625,
+          "n": 16,
+          "p50_ms": 0.5068694999999999,
+          "p95_ms": 0.513504,
+          "population_std_ms": 0.004382496085280639
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8348696874999997,
+          "n": 16,
+          "p50_ms": 0.8344809999999998,
+          "p95_ms": 0.838978,
+          "population_std_ms": 0.001855606192014971
+        },
+        "moe": {
+          "mean_ms": 1.8552162499999982,
+          "n": 16,
+          "p50_ms": 1.8067699999999989,
+          "p95_ms": 1.9758739999999988,
+          "population_std_ms": 0.09932662894681066
+        },
+        "other": {
+          "mean_ms": 0.020664125,
+          "n": 16,
+          "p50_ms": 0.0206075,
+          "p95_ms": 0.02112,
+          "population_std_ms": 0.00022492384127744221
+        },
+        "output_head": {
+          "mean_ms": 0.0455501875,
+          "n": 16,
+          "p50_ms": 0.045616500000000004,
+          "p95_ms": 0.046176,
+          "population_std_ms": 0.0003880140491576959
+        },
+        "router": {
+          "mean_ms": 0.18695149999999994,
+          "n": 16,
+          "p50_ms": 0.18694599999999995,
+          "p95_ms": 0.18726899999999994,
+          "population_std_ms": 0.00018246506514945761
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.6235766875,
+        "n": 16,
+        "p50_ms": 4.5675045,
+        "p95_ms": 4.776305000000001,
+        "population_std_ms": 0.11774099993349754
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.171425874999998,
+        "n": 16,
+        "p50_ms": 4.111105499999998,
+        "p95_ms": 4.327753999999999,
+        "population_std_ms": 0.11827515014357118
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4521508125000021,
+        "n": 16,
+        "p50_ms": 0.4516910000000025,
+        "p95_ms": 0.4587190000000012,
+        "population_std_ms": 0.003134758388192225
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-batch-tp4-b2-r1-20260723-20260723T085748513290Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r1/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784797289133609908.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7231355625,
+          "n": 16,
+          "p50_ms": 0.714801,
+          "p95_ms": 0.7511429999999999,
+          "population_std_ms": 0.01670948629135843
+        },
+        "collective": {
+          "mean_ms": 0.5030436875000001,
+          "n": 16,
+          "p50_ms": 0.5025615000000001,
+          "p95_ms": 0.5099899999999999,
+          "population_std_ms": 0.003726454090666345
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8384581250000002,
+          "n": 16,
+          "p50_ms": 0.8374080000000004,
+          "p95_ms": 0.8426070000000001,
+          "population_std_ms": 0.0022757014818238397
+        },
+        "moe": {
+          "mean_ms": 1.8538525624999982,
+          "n": 16,
+          "p50_ms": 1.8066204999999975,
+          "p95_ms": 1.980685999999996,
+          "population_std_ms": 0.0992721840478037
+        },
+        "other": {
+          "mean_ms": 0.02081575,
+          "n": 16,
+          "p50_ms": 0.020768000000000002,
+          "p95_ms": 0.021249000000000004,
+          "population_std_ms": 0.00024472216389203553
+        },
+        "output_head": {
+          "mean_ms": 0.0455743125,
+          "n": 16,
+          "p50_ms": 0.045568,
+          "p95_ms": 0.046335999999999995,
+          "population_std_ms": 0.00038872157753815117
+        },
+        "router": {
+          "mean_ms": 0.18700049999999993,
+          "n": 16,
+          "p50_ms": 0.18708599999999992,
+          "p95_ms": 0.18729799999999988,
+          "population_std_ms": 0.0002492310674855524
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.624022875,
+        "n": 16,
+        "p50_ms": 4.5670969999999995,
+        "p95_ms": 4.776058,
+        "population_std_ms": 0.11785700086167718
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.1718804999999985,
+        "n": 16,
+        "p50_ms": 4.113653999999998,
+        "p95_ms": 4.327779999999995,
+        "population_std_ms": 0.11872818612749433
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4521423750000019,
+        "n": 16,
+        "p50_ms": 0.452637500000002,
+        "p95_ms": 0.4561850000000023,
+        "population_std_ms": 0.002606323557115182
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-batch-tp4-b2-r1-20260723-20260723T085748513290Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r1/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784797289127939045.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7228365624999997,
+          "n": 16,
+          "p50_ms": 0.7140979999999998,
+          "p95_ms": 0.7481270000000001,
+          "population_std_ms": 0.014826216615208781
+        },
+        "collective": {
+          "mean_ms": 0.6974953125000001,
+          "n": 16,
+          "p50_ms": 0.6933765000000001,
+          "p95_ms": 0.7296940000000001,
+          "population_std_ms": 0.019150322023136917
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8396519374999999,
+          "n": 16,
+          "p50_ms": 0.8394079999999997,
+          "p95_ms": 0.8423959999999997,
+          "population_std_ms": 0.0015175569786975635
+        },
+        "moe": {
+          "mean_ms": 1.6409965000000004,
+          "n": 16,
+          "p50_ms": 1.6039915000000007,
+          "p95_ms": 1.749915000000001,
+          "population_std_ms": 0.08420929796569343
+        },
+        "other": {
+          "mean_ms": 0.0206121875,
+          "n": 16,
+          "p50_ms": 0.020608,
+          "p95_ms": 0.021438,
+          "population_std_ms": 0.0003990141317594523
+        },
+        "output_head": {
+          "mean_ms": 0.0457240625,
+          "n": 16,
+          "p50_ms": 0.04584,
+          "p95_ms": 0.046816,
+          "population_std_ms": 0.0005658698468674145
+        },
+        "router": {
+          "mean_ms": 0.18694024999999992,
+          "n": 16,
+          "p50_ms": 0.18695999999999993,
+          "p95_ms": 0.18755099999999994,
+          "population_std_ms": 0.00021861224233788088
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.6262738125,
+        "n": 16,
+        "p50_ms": 4.569678,
+        "p95_ms": 4.780622,
+        "population_std_ms": 0.11869307020905795
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.1542568125,
+        "n": 16,
+        "p50_ms": 4.100302,
+        "p95_ms": 4.307204000000001,
+        "population_std_ms": 0.11747190161545976
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.47201699999999974,
+        "n": 16,
+        "p50_ms": 0.47202699999999975,
+        "p95_ms": 0.47818999999999967,
+        "population_std_ms": 0.0027272721719696818
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-batch-tp4-b2-r1-20260723-20260723T085748513290Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r1/traces/profile/dp0_pp0_tp2_dcp0_ep2_rank2.1784797289123687444.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.720770875,
+          "n": 16,
+          "p50_ms": 0.7128364999999999,
+          "p95_ms": 0.7470090000000001,
+          "population_std_ms": 0.01543277700170565
+        },
+        "collective": {
+          "mean_ms": 0.5073495625,
+          "n": 16,
+          "p50_ms": 0.5075825,
+          "p95_ms": 0.514207,
+          "population_std_ms": 0.004872701393589966
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8371107499999998,
+          "n": 16,
+          "p50_ms": 0.8368309999999997,
+          "p95_ms": 0.8408340000000005,
+          "population_std_ms": 0.0019278550742989576
+        },
+        "moe": {
+          "mean_ms": 1.849929624999998,
+          "n": 16,
+          "p50_ms": 1.8029354999999985,
+          "p95_ms": 1.9727399999999982,
+          "population_std_ms": 0.09810616143359367
+        },
+        "other": {
+          "mean_ms": 0.02077625,
+          "n": 16,
+          "p50_ms": 0.020671500000000002,
+          "p95_ms": 0.021408,
+          "population_std_ms": 0.00033703143399392235
+        },
+        "output_head": {
+          "mean_ms": 0.0455561875,
+          "n": 16,
+          "p50_ms": 0.045392,
+          "p95_ms": 0.046880000000000005,
+          "population_std_ms": 0.0004960692011642635
+        },
+        "router": {
+          "mean_ms": 0.18668787499999998,
+          "n": 16,
+          "p50_ms": 0.18664099999999995,
+          "p95_ms": 0.18707300000000002,
+          "population_std_ms": 0.00021251878593432893
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.623365625,
+        "n": 16,
+        "p50_ms": 4.5671315,
+        "p95_ms": 4.775658999999999,
+        "population_std_ms": 0.11841808903936239
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.168181124999998,
+        "n": 16,
+        "p50_ms": 4.109987499999998,
+        "p95_ms": 4.320808999999999,
+        "population_std_ms": 0.1170474600631484
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.45518450000000205,
+        "n": 16,
+        "p50_ms": 0.45484250000000115,
+        "p95_ms": 0.46047800000000283,
+        "population_std_ms": 0.0026128110101962454
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-batch-tp4-b2-r1-20260723-20260723T085748513290Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r1/traces/profile/dp0_pp0_tp3_dcp0_ep3_rank3.1784797289132564461.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp4-b2-r2.json

@@ -0,0 +1,345 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp4-b2-r2",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.7204233125,
+      "collective": 0.560326375,
+      "linear_norm_rope": 0.8352021093750001,
+      "moe": 1.7914327656249986,
+      "other": 0.020750828125000002,
+      "output_head": 0.04556265625,
+      "router": 0.18667273437499995
+    },
+    "ranks": 4,
+    "slowest_rank_execute_mean_ms": 4.6200790625,
+    "slowest_rank_kernel_busy_mean_ms": 4.168571874999999
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7155257500000002,
+          "n": 16,
+          "p50_ms": 0.7066910000000002,
+          "p95_ms": 0.7447440000000003,
+          "population_std_ms": 0.01558176035265282
+        },
+        "collective": {
+          "mean_ms": 0.5219253125,
+          "n": 16,
+          "p50_ms": 0.5189115,
+          "p95_ms": 0.5383740000000001,
+          "population_std_ms": 0.008178805388921035
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8318194374999998,
+          "n": 16,
+          "p50_ms": 0.8315799999999995,
+          "p95_ms": 0.8361279999999999,
+          "population_std_ms": 0.001956002842813193
+        },
+        "moe": {
+          "mean_ms": 1.8462991249999985,
+          "n": 16,
+          "p50_ms": 1.8014489999999976,
+          "p95_ms": 1.9688019999999977,
+          "population_std_ms": 0.098837188149802
+        },
+        "other": {
+          "mean_ms": 0.020839937500000003,
+          "n": 16,
+          "p50_ms": 0.020830500000000002,
+          "p95_ms": 0.021311,
+          "population_std_ms": 0.00026370875714270555
+        },
+        "output_head": {
+          "mean_ms": 0.0455260625,
+          "n": 16,
+          "p50_ms": 0.045296,
+          "p95_ms": 0.047200000000000006,
+          "population_std_ms": 0.0005808763281402957
+        },
+        "router": {
+          "mean_ms": 0.1866362499999999,
+          "n": 16,
+          "p50_ms": 0.18664149999999985,
+          "p95_ms": 0.1869089999999999,
+          "population_std_ms": 0.0001586574533389501
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.6200790625,
+        "n": 16,
+        "p50_ms": 4.560737,
+        "p95_ms": 4.785777,
+        "population_std_ms": 0.1217774651913628
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.168571874999999,
+        "n": 16,
+        "p50_ms": 4.106530999999998,
+        "p95_ms": 4.335109999999998,
+        "population_std_ms": 0.12256936973397699
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4515071875000016,
+        "n": 16,
+        "p50_ms": 0.45221700000000054,
+        "p95_ms": 0.4565129999999993,
+        "population_std_ms": 0.002988645119839536
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-batch-tp4-b2-r2-20260723-20260723T090307191004Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r2/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784797602726202234.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7221352500000001,
+          "n": 16,
+          "p50_ms": 0.7126265000000003,
+          "p95_ms": 0.7521960000000003,
+          "population_std_ms": 0.015924308283642965
+        },
+        "collective": {
+          "mean_ms": 0.5009256875,
+          "n": 16,
+          "p50_ms": 0.5015225,
+          "p95_ms": 0.5122899999999999,
+          "population_std_ms": 0.0051018101287526214
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8378076250000003,
+          "n": 16,
+          "p50_ms": 0.8377155000000002,
+          "p95_ms": 0.8417550000000004,
+          "population_std_ms": 0.0019228740167716244
+        },
+        "moe": {
+          "mean_ms": 1.851001874999998,
+          "n": 16,
+          "p50_ms": 1.8053344999999985,
+          "p95_ms": 1.9718849999999972,
+          "population_std_ms": 0.09850954145403519
+        },
+        "other": {
+          "mean_ms": 0.0207508125,
+          "n": 16,
+          "p50_ms": 0.0207035,
+          "p95_ms": 0.021217,
+          "population_std_ms": 0.00025922799297867157
+        },
+        "output_head": {
+          "mean_ms": 0.0454381875,
+          "n": 16,
+          "p50_ms": 0.045216,
+          "p95_ms": 0.046560000000000004,
+          "population_std_ms": 0.0004811869723337822
+        },
+        "router": {
+          "mean_ms": 0.1865970625,
+          "n": 16,
+          "p50_ms": 0.1865795,
+          "p95_ms": 0.18706799999999996,
+          "population_std_ms": 0.00018487748265743356
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.6191813125,
+        "n": 16,
+        "p50_ms": 4.5577369999999995,
+        "p95_ms": 4.785498,
+        "population_std_ms": 0.12056056733428579
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.164656499999998,
+        "n": 16,
+        "p50_ms": 4.105473,
+        "p95_ms": 4.326126999999998,
+        "population_std_ms": 0.11950220623590964
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4545248125000018,
+        "n": 16,
+        "p50_ms": 0.45509500000000225,
+        "p95_ms": 0.45937100000000175,
+        "population_std_ms": 0.0031089089770119727
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-batch-tp4-b2-r2-20260723-20260723T090307191004Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r2/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784797602726796969.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7246866249999998,
+          "n": 16,
+          "p50_ms": 0.7141314999999997,
+          "p95_ms": 0.7533139999999997,
+          "population_std_ms": 0.016680379828840117
+        },
+        "collective": {
+          "mean_ms": 0.71273025,
+          "n": 16,
+          "p50_ms": 0.7052324999999999,
+          "p95_ms": 0.7373889999999999,
+          "population_std_ms": 0.01757743828214172
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8347086875000002,
+          "n": 16,
+          "p50_ms": 0.8344474999999999,
+          "p95_ms": 0.8372729999999998,
+          "population_std_ms": 0.0011941147724752546
+        },
+        "moe": {
+          "mean_ms": 1.6199748125000002,
+          "n": 16,
+          "p50_ms": 1.5786544999999998,
+          "p95_ms": 1.7316220000000002,
+          "population_std_ms": 0.08667323273307787
+        },
+        "other": {
+          "mean_ms": 0.0206221875,
+          "n": 16,
+          "p50_ms": 0.0206095,
+          "p95_ms": 0.021182,
+          "population_std_ms": 0.0003110010487823955
+        },
+        "output_head": {
+          "mean_ms": 0.0457541875,
+          "n": 16,
+          "p50_ms": 0.0456,
+          "p95_ms": 0.04736,
+          "population_std_ms": 0.0005682590978979142
+        },
+        "router": {
+          "mean_ms": 0.18685312499999993,
+          "n": 16,
+          "p50_ms": 0.1868349999999999,
+          "p95_ms": 0.18742899999999987,
+          "population_std_ms": 0.00024282758157795687
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.6185679375,
+        "n": 16,
+        "p50_ms": 4.559101999999999,
+        "p95_ms": 4.787310000000001,
+        "population_std_ms": 0.12219376749832045
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.145329875,
+        "n": 16,
+        "p50_ms": 4.0855665,
+        "p95_ms": 4.30906,
+        "population_std_ms": 0.12076811704868254
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4732380625000001,
+        "n": 16,
+        "p50_ms": 0.47285649999999935,
+        "p95_ms": 0.47825000000000095,
+        "population_std_ms": 0.0030751450874057767
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-batch-tp4-b2-r2-20260723-20260723T090307191004Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r2/traces/profile/dp0_pp0_tp2_dcp0_ep2_rank2.1784797602710411107.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7193456250000001,
+          "n": 16,
+          "p50_ms": 0.7110120000000001,
+          "p95_ms": 0.7488160000000005,
+          "population_std_ms": 0.0156480688619834
+        },
+        "collective": {
+          "mean_ms": 0.5057242500000001,
+          "n": 16,
+          "p50_ms": 0.5041895000000001,
+          "p95_ms": 0.5194939999999999,
+          "population_std_ms": 0.00615956476242434
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8364726875,
+          "n": 16,
+          "p50_ms": 0.8363539999999998,
+          "p95_ms": 0.84147,
+          "population_std_ms": 0.0026977789873977565
+        },
+        "moe": {
+          "mean_ms": 1.8484552499999982,
+          "n": 16,
+          "p50_ms": 1.8005024999999995,
+          "p95_ms": 1.9741669999999973,
+          "population_std_ms": 0.09999423839808207
+        },
+        "other": {
+          "mean_ms": 0.020790375,
+          "n": 16,
+          "p50_ms": 0.020800000000000003,
+          "p95_ms": 0.021249999999999998,
+          "population_std_ms": 0.000248934699017634
+        },
+        "output_head": {
+          "mean_ms": 0.0455321875,
+          "n": 16,
+          "p50_ms": 0.045424,
+          "p95_ms": 0.046304000000000005,
+          "population_std_ms": 0.00037697981026011215
+        },
+        "router": {
+          "mean_ms": 0.18660449999999998,
+          "n": 16,
+          "p50_ms": 0.1866215,
+          "p95_ms": 0.18684699999999987,
+          "population_std_ms": 0.00013768759929635568
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_2(2)": 16
+      },
+      "execute_wall": {
+        "mean_ms": 4.617897375,
+        "n": 16,
+        "p50_ms": 4.558091500000001,
+        "p95_ms": 4.783308,
+        "population_std_ms": 0.12193403131912922
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.162924874999998,
+        "n": 16,
+        "p50_ms": 4.102588,
+        "p95_ms": 4.328206999999998,
+        "population_std_ms": 0.12156658812810923
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4549725000000015,
+        "n": 16,
+        "p50_ms": 0.45506600000000175,
+        "p95_ms": 0.4593380000000007,
+        "population_std_ms": 0.002668184377999502
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_2(2)",
+      "steps": 16,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-batch-tp4-b2-r2-20260723-20260723T090307191004Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b2-r2/traces/profile/dp0_pp0_tp3_dcp0_ep3_rank3.1784797602728757539.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp4-b4-r1.json

@@ -0,0 +1,353 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp4-b4-r1",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.8023426607142858,
+      "collective": 0.5361581428571428,
+      "linear_norm_rope": 0.8392915178571427,
+      "moe": 1.9595749285714295,
+      "other": 0.021800214285714284,
+      "output_head": 0.04583946428571428,
+      "router": 0.1894944999999999
+    },
+    "ranks": 4,
+    "slowest_rank_execute_mean_ms": 4.516427428571428,
+    "slowest_rank_kernel_busy_mean_ms": 4.397215785714287
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.8026924285714286,
+          "n": 14,
+          "p50_ms": 0.8022965,
+          "p95_ms": 0.810657,
+          "population_std_ms": 0.004568636975764919
+        },
+        "collective": {
+          "mean_ms": 0.5177565714285713,
+          "n": 14,
+          "p50_ms": 0.5184174999999998,
+          "p95_ms": 0.5240059999999997,
+          "population_std_ms": 0.0039342607349554136
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8483352857142855,
+          "n": 14,
+          "p50_ms": 0.8486384999999999,
+          "p95_ms": 0.8519969999999999,
+          "population_std_ms": 0.0017219747562680734
+        },
+        "moe": {
+          "mean_ms": 1.9629181428571427,
+          "n": 14,
+          "p50_ms": 1.9689039999999998,
+          "p95_ms": 2.131421,
+          "population_std_ms": 0.11758295795361873
+        },
+        "other": {
+          "mean_ms": 0.022141571428571427,
+          "n": 14,
+          "p50_ms": 0.022127,
+          "p95_ms": 0.022622999999999997,
+          "population_std_ms": 0.00024374275499436835
+        },
+        "output_head": {
+          "mean_ms": 0.045906214285714286,
+          "n": 14,
+          "p50_ms": 0.045871999999999996,
+          "p95_ms": 0.046592,
+          "population_std_ms": 0.00048383456124285047
+        },
+        "router": {
+          "mean_ms": 0.18891014285714272,
+          "n": 14,
+          "p50_ms": 0.18883199999999986,
+          "p95_ms": 0.18931099999999992,
+          "population_std_ms": 0.00019712608624317712
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_4(4)": 14,
+        "execute_context_3(6144)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.515594,
+        "n": 14,
+        "p50_ms": 4.512055999999999,
+        "p95_ms": 4.8260879999999995,
+        "population_std_ms": 0.14829775976248402
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.388660357142856,
+        "n": 14,
+        "p50_ms": 4.3929335,
+        "p95_ms": 4.560478,
+        "population_std_ms": 0.12189088321563175
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.12693364285714326,
+        "n": 14,
+        "p50_ms": 0.10895650000000145,
+        "p95_ms": 0.3681599999999996,
+        "population_std_ms": 0.06698859556415654
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b4-r1-20260723-20260723T091059346688Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r1/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798099190414139.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7969025714285717,
+          "n": 14,
+          "p50_ms": 0.7975640000000002,
+          "p95_ms": 0.803777,
+          "population_std_ms": 0.004672418671833448
+        },
+        "collective": {
+          "mean_ms": 0.5435259285714286,
+          "n": 14,
+          "p50_ms": 0.5245414999999998,
+          "p95_ms": 0.7792909999999997,
+          "population_std_ms": 0.0654428389790491
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8351954285714284,
+          "n": 14,
+          "p50_ms": 0.8352339999999999,
+          "p95_ms": 0.8379900000000003,
+          "population_std_ms": 0.0011727439566058622
+        },
+        "moe": {
+          "mean_ms": 1.9629085714285726,
+          "n": 14,
+          "p50_ms": 1.9686130000000002,
+          "p95_ms": 2.1275570000000004,
+          "population_std_ms": 0.11811909377144619
+        },
+        "other": {
+          "mean_ms": 0.02171192857142857,
+          "n": 14,
+          "p50_ms": 0.021582999999999998,
+          "p95_ms": 0.023136000000000004,
+          "population_std_ms": 0.0004568249843501685
+        },
+        "output_head": {
+          "mean_ms": 0.04598635714285714,
+          "n": 14,
+          "p50_ms": 0.0461605,
+          "p95_ms": 0.046688,
+          "population_std_ms": 0.00048825660805167076
+        },
+        "router": {
+          "mean_ms": 0.18940242857142844,
+          "n": 14,
+          "p50_ms": 0.18939349999999988,
+          "p95_ms": 0.19011299999999992,
+          "population_std_ms": 0.0003183344409996063
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_4(4)": 14,
+        "execute_context_3(6144)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.515569785714286,
+        "n": 14,
+        "p50_ms": 4.509453000000001,
+        "p95_ms": 4.822798,
+        "population_std_ms": 0.1475542723441973
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.3956332142857155,
+        "n": 14,
+        "p50_ms": 4.3885055,
+        "p95_ms": 4.704691000000001,
+        "population_std_ms": 0.1480161569835144
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.11993657142857048,
+        "n": 14,
+        "p50_ms": 0.12018349999999911,
+        "p95_ms": 0.12278300000000009,
+        "population_std_ms": 0.0023027362206822326
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b4-r1-20260723-20260723T091059346688Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r1/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798099190728578.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.8076340714285717,
+          "n": 14,
+          "p50_ms": 0.8078890000000003,
+          "p95_ms": 0.811872,
+          "population_std_ms": 0.003509697513711885
+        },
+        "collective": {
+          "mean_ms": 0.538438357142857,
+          "n": 14,
+          "p50_ms": 0.5197394999999998,
+          "p95_ms": 0.7775990000000002,
+          "population_std_ms": 0.06647859896624862
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8366736428571427,
+          "n": 14,
+          "p50_ms": 0.8371874999999998,
+          "p95_ms": 0.8391049999999995,
+          "population_std_ms": 0.002070122274067762
+        },
+        "moe": {
+          "mean_ms": 1.9572025000000013,
+          "n": 14,
+          "p50_ms": 1.9622045000000017,
+          "p95_ms": 2.1186610000000012,
+          "population_std_ms": 0.11785303854665174
+        },
+        "other": {
+          "mean_ms": 0.021651357142857146,
+          "n": 14,
+          "p50_ms": 0.0216015,
+          "p95_ms": 0.022528000000000003,
+          "population_std_ms": 0.0003619630224095226
+        },
+        "output_head": {
+          "mean_ms": 0.04572821428571428,
+          "n": 14,
+          "p50_ms": 0.045616000000000004,
+          "p95_ms": 0.046816,
+          "population_std_ms": 0.0004743333936877599
+        },
+        "router": {
+          "mean_ms": 0.18988764285714277,
+          "n": 14,
+          "p50_ms": 0.1899184999999999,
+          "p95_ms": 0.1904639999999999,
+          "population_std_ms": 0.00036004535570296044
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_4(4)": 14,
+        "execute_context_3(6144)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.516208928571428,
+        "n": 14,
+        "p50_ms": 4.5094615000000005,
+        "p95_ms": 4.827749,
+        "population_std_ms": 0.14783990641591432
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.397215785714287,
+        "n": 14,
+        "p50_ms": 4.391749500000001,
+        "p95_ms": 4.706615000000002,
+        "population_std_ms": 0.14841367943313954
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.11899314285714155,
+        "n": 14,
+        "p50_ms": 0.11911499999999897,
+        "p95_ms": 0.12231999999999754,
+        "population_std_ms": 0.0022138607149742996
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b4-r1-20260723-20260723T091059346688Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r1/traces/profile/dp0_pp0_tp2_dcp0_ep2_rank2.1784798099190414025.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.8021415714285716,
+          "n": 14,
+          "p50_ms": 0.8019065000000002,
+          "p95_ms": 0.8102380000000002,
+          "population_std_ms": 0.004568393789542204
+        },
+        "collective": {
+          "mean_ms": 0.5449117142857143,
+          "n": 14,
+          "p50_ms": 0.5278735,
+          "p95_ms": 0.781215,
+          "population_std_ms": 0.0655966698484312
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8369617142857141,
+          "n": 14,
+          "p50_ms": 0.8372430000000001,
+          "p95_ms": 0.8394259999999998,
+          "population_std_ms": 0.0013668606590375937
+        },
+        "moe": {
+          "mean_ms": 1.9552705000000008,
+          "n": 14,
+          "p50_ms": 1.9632605,
+          "p95_ms": 2.1177540000000006,
+          "population_std_ms": 0.11724808995809416
+        },
+        "other": {
+          "mean_ms": 0.021696,
+          "n": 14,
+          "p50_ms": 0.021648,
+          "p95_ms": 0.02288,
+          "population_std_ms": 0.0004689002330682176
+        },
+        "output_head": {
+          "mean_ms": 0.045737071428571426,
+          "n": 14,
+          "p50_ms": 0.0457755,
+          "p95_ms": 0.046496,
+          "population_std_ms": 0.00039792039122807263
+        },
+        "router": {
+          "mean_ms": 0.18977778571428564,
+          "n": 14,
+          "p50_ms": 0.1896624999999999,
+          "p95_ms": 0.1909739999999999,
+          "population_std_ms": 0.00041644501927821606
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_4(4)": 14,
+        "execute_context_3(6144)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.516427428571428,
+        "n": 14,
+        "p50_ms": 4.508496,
+        "p95_ms": 4.82816,
+        "population_std_ms": 0.14821148953231575
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.3964963571428575,
+        "n": 14,
+        "p50_ms": 4.3874035,
+        "p95_ms": 4.705174,
+        "population_std_ms": 0.14715990628788167
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.11993107142857092,
+        "n": 14,
+        "p50_ms": 0.11941749999999951,
+        "p95_ms": 0.1248560000000003,
+        "population_std_ms": 0.002664138817283219
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b4-r1-20260723-20260723T091059346688Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r1/traces/profile/dp0_pp0_tp3_dcp0_ep3_rank3.1784798099191829059.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp4-b4-r2.json

@@ -0,0 +1,353 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp4-b4-r2",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.801226767857143,
+      "collective": 0.6577525357142857,
+      "linear_norm_rope": 0.8391043928571427,
+      "moe": 1.9621030178571435,
+      "other": 0.021777,
+      "output_head": 0.045781142857142854,
+      "router": 0.18946453571428562
+    },
+    "ranks": 4,
+    "slowest_rank_execute_mean_ms": 4.726592071428572,
+    "slowest_rank_kernel_busy_mean_ms": 4.601115500000001
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.8019157857142859,
+          "n": 14,
+          "p50_ms": 0.8018870000000005,
+          "p95_ms": 0.8074610000000003,
+          "population_std_ms": 0.0036870609313142864
+        },
+        "collective": {
+          "mean_ms": 0.697091642857143,
+          "n": 14,
+          "p50_ms": 0.521808,
+          "p95_ms": 1.635147000000001,
+          "population_std_ms": 0.3581717548612294
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8477593571428571,
+          "n": 14,
+          "p50_ms": 0.847532,
+          "p95_ms": 0.8513850000000003,
+          "population_std_ms": 0.0016779822750273288
+        },
+        "moe": {
+          "mean_ms": 1.9648232142857136,
+          "n": 14,
+          "p50_ms": 1.9675199999999988,
+          "p95_ms": 2.1313579999999983,
+          "population_std_ms": 0.11544109651690553
+        },
+        "other": {
+          "mean_ms": 0.022008857142857143,
+          "n": 14,
+          "p50_ms": 0.022,
+          "p95_ms": 0.022559999999999997,
+          "population_std_ms": 0.0003089169692288703
+        },
+        "output_head": {
+          "mean_ms": 0.045711999999999996,
+          "n": 14,
+          "p50_ms": 0.045792,
+          "p95_ms": 0.046144,
+          "population_std_ms": 0.0003498834499821745
+        },
+        "router": {
+          "mean_ms": 0.18900892857142845,
+          "n": 14,
+          "p50_ms": 0.1890704999999999,
+          "p95_ms": 0.18972699999999987,
+          "population_std_ms": 0.0003452439022500138
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_4(4)": 14,
+        "execute_context_1(2048)_generation_1(1)": 1,
+        "execute_context_2(4096)_generation_2(2)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.726289,
+        "n": 14,
+        "p50_ms": 4.587623499999999,
+        "p95_ms": 6.343786,
+        "population_std_ms": 0.5295863492605742
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.568319785714285,
+        "n": 14,
+        "p50_ms": 4.476973999999999,
+        "p95_ms": 5.5797930000000004,
+        "population_std_ms": 0.3971815245092024
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.15796921428571484,
+        "n": 14,
+        "p50_ms": 0.11138250000000127,
+        "p95_ms": 0.7639929999999993,
+        "population_std_ms": 0.1680896761222489
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b4-r2-20260723-20260723T092328819959Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r2/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798849005052042.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.7960197142857144,
+          "n": 14,
+          "p50_ms": 0.7975315000000003,
+          "p95_ms": 0.8016960000000003,
+          "population_std_ms": 0.005350704885854841
+        },
+        "collective": {
+          "mean_ms": 0.5697055714285714,
+          "n": 14,
+          "p50_ms": 0.527951,
+          "p95_ms": 1.1241249999999998,
+          "population_std_ms": 0.15379550779651446
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8354554285714284,
+          "n": 14,
+          "p50_ms": 0.8356854999999999,
+          "p95_ms": 0.8373729999999999,
+          "population_std_ms": 0.0013868995078585018
+        },
+        "moe": {
+          "mean_ms": 1.9646907142857157,
+          "n": 14,
+          "p50_ms": 1.9679745000000008,
+          "p95_ms": 2.128705000000001,
+          "population_std_ms": 0.11638566052410722
+        },
+        "other": {
+          "mean_ms": 0.021556214285714286,
+          "n": 14,
+          "p50_ms": 0.021551,
+          "p95_ms": 0.022076000000000002,
+          "population_std_ms": 0.000275473923726115
+        },
+        "output_head": {
+          "mean_ms": 0.045609142857142855,
+          "n": 14,
+          "p50_ms": 0.045504,
+          "p95_ms": 0.046592,
+          "population_std_ms": 0.00037677557873977814
+        },
+        "router": {
+          "mean_ms": 0.1894397142857142,
+          "n": 14,
+          "p50_ms": 0.18942449999999988,
+          "p95_ms": 0.19078299999999992,
+          "population_std_ms": 0.0005227123530983776
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_4(4)": 14,
+        "execute_context_1(2048)_generation_1(1)": 1,
+        "execute_context_2(4096)_generation_2(2)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.726592071428572,
+        "n": 14,
+        "p50_ms": 4.587085,
+        "p95_ms": 6.349746,
+        "population_std_ms": 0.5316654164241782
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.422476500000001,
+        "n": 14,
+        "p50_ms": 4.384860500000001,
+        "p95_ms": 5.0564160000000005,
+        "population_std_ms": 0.21186941717878632
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.3041155714285702,
+        "n": 14,
+        "p50_ms": 0.12178199999999872,
+        "p95_ms": 1.2933299999999992,
+        "population_std_ms": 0.3691441767122035
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b4-r2-20260723-20260723T092328819959Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r2/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798849005338612.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.8045258571428573,
+          "n": 14,
+          "p50_ms": 0.8049890000000001,
+          "p95_ms": 0.8098900000000004,
+          "population_std_ms": 0.003910280750920532
+        },
+        "collective": {
+          "mean_ms": 0.7413481428571427,
+          "n": 14,
+          "p50_ms": 0.523762,
+          "p95_ms": 2.205729,
+          "population_std_ms": 0.47534174936067347
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8360086428571426,
+          "n": 14,
+          "p50_ms": 0.8361784999999999,
+          "p95_ms": 0.837317,
+          "population_std_ms": 0.0010630517127015252
+        },
+        "moe": {
+          "mean_ms": 1.9618675000000008,
+          "n": 14,
+          "p50_ms": 1.9661850000000007,
+          "p95_ms": 2.119519000000001,
+          "population_std_ms": 0.1151975200022925
+        },
+        "other": {
+          "mean_ms": 0.02169592857142857,
+          "n": 14,
+          "p50_ms": 0.021664500000000003,
+          "p95_ms": 0.022047999999999998,
+          "population_std_ms": 0.00024448267957514905
+        },
+        "output_head": {
+          "mean_ms": 0.04594742857142858,
+          "n": 14,
+          "p50_ms": 0.045984000000000004,
+          "p95_ms": 0.046784,
+          "population_std_ms": 0.0004970469529827017
+        },
+        "router": {
+          "mean_ms": 0.18972199999999992,
+          "n": 14,
+          "p50_ms": 0.1897264999999999,
+          "p95_ms": 0.1903659999999999,
+          "population_std_ms": 0.00040084571310453965
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_4(4)": 14,
+        "execute_context_1(2048)_generation_1(1)": 1,
+        "execute_context_2(4096)_generation_2(2)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.725667642857142,
+        "n": 14,
+        "p50_ms": 4.586822,
+        "p95_ms": 6.3431750000000005,
+        "population_std_ms": 0.5294299685093403
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.601115500000001,
+        "n": 14,
+        "p50_ms": 4.466799,
+        "p95_ms": 6.1463090000000005,
+        "population_std_ms": 0.5125165009938357
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.12455214285714243,
+        "n": 14,
+        "p50_ms": 0.11967999999999934,
+        "p95_ms": 0.19686599999999999,
+        "population_std_ms": 0.020164973440020648
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b4-r2-20260723-20260723T092328819959Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r2/traces/profile/dp0_pp0_tp2_dcp0_ep2_rank2.1784798849004512081.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.8024457142857144,
+          "n": 14,
+          "p50_ms": 0.8021285000000002,
+          "p95_ms": 0.8093020000000001,
+          "population_std_ms": 0.005437250978581053
+        },
+        "collective": {
+          "mean_ms": 0.6228647857142856,
+          "n": 14,
+          "p50_ms": 0.52369,
+          "p95_ms": 1.3911020000000003,
+          "population_std_ms": 0.2490648031124713
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8371941428571427,
+          "n": 14,
+          "p50_ms": 0.8375324999999998,
+          "p95_ms": 0.8382489999999997,
+          "population_std_ms": 0.00103632529760148
+        },
+        "moe": {
+          "mean_ms": 1.9570306428571438,
+          "n": 14,
+          "p50_ms": 1.9613605000000005,
+          "p95_ms": 2.1156430000000004,
+          "population_std_ms": 0.11617949366611671
+        },
+        "other": {
+          "mean_ms": 0.021847000000000002,
+          "n": 14,
+          "p50_ms": 0.021792,
+          "p95_ms": 0.022240000000000003,
+          "population_std_ms": 0.0002032686470096729
+        },
+        "output_head": {
+          "mean_ms": 0.045856,
+          "n": 14,
+          "p50_ms": 0.04592,
+          "p95_ms": 0.046272,
+          "population_std_ms": 0.00036146388873183906
+        },
+        "router": {
+          "mean_ms": 0.18968749999999987,
+          "n": 14,
+          "p50_ms": 0.18976349999999986,
+          "p95_ms": 0.1904969999999999,
+          "population_std_ms": 0.0004998125005582514
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_4(4)": 14,
+        "execute_context_1(2048)_generation_1(1)": 1,
+        "execute_context_2(4096)_generation_2(2)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.726361142857143,
+        "n": 14,
+        "p50_ms": 4.58896,
+        "p95_ms": 6.346271,
+        "population_std_ms": 0.5297092372787245
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.476925785714287,
+        "n": 14,
+        "p50_ms": 4.454530500000001,
+        "p95_ms": 5.177472,
+        "population_std_ms": 0.28127072648585827
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.2494353571428564,
+        "n": 14,
+        "p50_ms": 0.12384599999999946,
+        "p95_ms": 1.887349999999997,
+        "population_std_ms": 0.4542784406329925
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_4(4)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b4-r2-20260723-20260723T092328819959Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b4-r2/traces/profile/dp0_pp0_tp3_dcp0_ep3_rank3.1784798849009637378.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp4-b6-r1.json

@@ -0,0 +1,357 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp4-b6-r1",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.926535269230769,
+      "collective": 0.5587644230769231,
+      "linear_norm_rope": 0.8382221538461538,
+      "moe": 2.378592788461537,
+      "other": 0.02582126923076923,
+      "output_head": 0.048266576923076926,
+      "router": 0.1886060961538461
+    },
+    "ranks": 4,
+    "slowest_rank_execute_mean_ms": 5.43262976923077,
+    "slowest_rank_kernel_busy_mean_ms": 4.971441538461537
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9201209999999999,
+          "n": 13,
+          "p50_ms": 0.920411,
+          "p95_ms": 0.9225939999999999,
+          "population_std_ms": 0.0019136943731553082
+        },
+        "collective": {
+          "mean_ms": 0.5660879230769231,
+          "n": 13,
+          "p50_ms": 0.5659190000000001,
+          "p95_ms": 0.5763190000000002,
+          "population_std_ms": 0.005072051101743609
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8359634615384616,
+          "n": 13,
+          "p50_ms": 0.8359020000000003,
+          "p95_ms": 0.8375410000000001,
+          "population_std_ms": 0.0010538181579662785
+        },
+        "moe": {
+          "mean_ms": 2.3794084615384605,
+          "n": 13,
+          "p50_ms": 2.3725099999999975,
+          "p95_ms": 2.4436449999999983,
+          "population_std_ms": 0.038306336006573204
+        },
+        "other": {
+          "mean_ms": 0.025853076923076923,
+          "n": 13,
+          "p50_ms": 0.025792,
+          "p95_ms": 0.026593,
+          "population_std_ms": 0.0003076432653223938
+        },
+        "output_head": {
+          "mean_ms": 0.04835438461538462,
+          "n": 13,
+          "p50_ms": 0.048193,
+          "p95_ms": 0.048991999999999994,
+          "population_std_ms": 0.0003663435836776379
+        },
+        "router": {
+          "mean_ms": 0.18904192307692302,
+          "n": 13,
+          "p50_ms": 0.1891829999999999,
+          "p95_ms": 0.1920339999999999,
+          "population_std_ms": 0.0016883016710719592
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_6(6)": 13,
+        "execute_context_2(2049)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.431982538461539,
+        "n": 13,
+        "p50_ms": 5.424736,
+        "p95_ms": 5.495328000000001,
+        "population_std_ms": 0.03904621042109629
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.96483023076923,
+        "n": 13,
+        "p50_ms": 4.958829999999999,
+        "p95_ms": 5.028786999999998,
+        "population_std_ms": 0.0381875548699665
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.46715230769230887,
+        "n": 13,
+        "p50_ms": 0.46785100000000135,
+        "p95_ms": 0.4726150000000029,
+        "population_std_ms": 0.0034291495555826275
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r1-20260723-20260723T091102857920Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r1/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798079116726333.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9270714615384612,
+          "n": 13,
+          "p50_ms": 0.9273969999999994,
+          "p95_ms": 0.9299869999999996,
+          "population_std_ms": 0.0017639532793125299
+        },
+        "collective": {
+          "mean_ms": 0.5647607692307692,
+          "n": 13,
+          "p50_ms": 0.5641320000000001,
+          "p95_ms": 0.5718079999999998,
+          "population_std_ms": 0.004140332932615274
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8313579230769229,
+          "n": 13,
+          "p50_ms": 0.8313590000000001,
+          "p95_ms": 0.8333470000000002,
+          "population_std_ms": 0.0011534636442897947
+        },
+        "moe": {
+          "mean_ms": 2.3681258461538452,
+          "n": 13,
+          "p50_ms": 2.3586600000000004,
+          "p95_ms": 2.4394819999999995,
+          "population_std_ms": 0.04033740793290489
+        },
+        "other": {
+          "mean_ms": 0.02615892307692308,
+          "n": 13,
+          "p50_ms": 0.026175999999999998,
+          "p95_ms": 0.026528000000000003,
+          "population_std_ms": 0.00026144170155812974
+        },
+        "output_head": {
+          "mean_ms": 0.048209615384615385,
+          "n": 13,
+          "p50_ms": 0.048096,
+          "p95_ms": 0.049089,
+          "population_std_ms": 0.0004454711487612834
+        },
+        "router": {
+          "mean_ms": 0.1910713846153846,
+          "n": 13,
+          "p50_ms": 0.19120299999999996,
+          "p95_ms": 0.1940459999999999,
+          "population_std_ms": 0.0014778622574756317
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_6(6)": 13,
+        "execute_context_2(2049)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.430220692307693,
+        "n": 13,
+        "p50_ms": 5.419602,
+        "p95_ms": 5.493810000000001,
+        "population_std_ms": 0.03883185853707986
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.956755923076922,
+        "n": 13,
+        "p50_ms": 4.948898,
+        "p95_ms": 5.020686999999999,
+        "population_std_ms": 0.039974607321805765
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4734647692307705,
+        "n": 13,
+        "p50_ms": 0.47312300000000196,
+        "p95_ms": 0.47953300000000176,
+        "population_std_ms": 0.0027594904533504113
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r1-20260723-20260723T091102857920Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r1/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798079120793210.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9307573846153845,
+          "n": 13,
+          "p50_ms": 0.930942,
+          "p95_ms": 0.9329269999999996,
+          "population_std_ms": 0.0011322629851124393
+        },
+        "collective": {
+          "mean_ms": 0.5509683846153847,
+          "n": 13,
+          "p50_ms": 0.5507769999999997,
+          "p95_ms": 0.559011,
+          "population_std_ms": 0.0047502838868763225
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8407734615384612,
+          "n": 13,
+          "p50_ms": 0.8405719999999995,
+          "p95_ms": 0.8421149999999998,
+          "population_std_ms": 0.0010200678420412281
+        },
+        "moe": {
+          "mean_ms": 2.380922769230769,
+          "n": 13,
+          "p50_ms": 2.3736940000000004,
+          "p95_ms": 2.450815999999999,
+          "population_std_ms": 0.04052726901758431
+        },
+        "other": {
+          "mean_ms": 0.025844076923076924,
+          "n": 13,
+          "p50_ms": 0.025858,
+          "p95_ms": 0.026433,
+          "population_std_ms": 0.0004114569725216555
+        },
+        "output_head": {
+          "mean_ms": 0.04823130769230769,
+          "n": 13,
+          "p50_ms": 0.048128,
+          "p95_ms": 0.048799999999999996,
+          "population_std_ms": 0.0003355393785477529
+        },
+        "router": {
+          "mean_ms": 0.18870923076923074,
+          "n": 13,
+          "p50_ms": 0.18834999999999993,
+          "p95_ms": 0.19122899999999987,
+          "population_std_ms": 0.001473603806155092
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_6(6)": 13,
+        "execute_context_2(2049)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.43262976923077,
+        "n": 13,
+        "p50_ms": 5.423109,
+        "p95_ms": 5.49578,
+        "population_std_ms": 0.039102580987318745
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.966206615384615,
+        "n": 13,
+        "p50_ms": 4.956755999999999,
+        "p95_ms": 5.031173999999998,
+        "population_std_ms": 0.039860749997842636
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.46642315384615496,
+        "n": 13,
+        "p50_ms": 0.4660140000000004,
+        "p95_ms": 0.47152800000000017,
+        "population_std_ms": 0.002432888149003149
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r1-20260723-20260723T091102857920Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r1/traces/profile/dp0_pp0_tp2_dcp0_ep2_rank2.1784798079116114132.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9281912307692306,
+          "n": 13,
+          "p50_ms": 0.9282559999999997,
+          "p95_ms": 0.9303809999999997,
+          "population_std_ms": 0.0012537519785301848
+        },
+        "collective": {
+          "mean_ms": 0.5532406153846153,
+          "n": 13,
+          "p50_ms": 0.5540510000000001,
+          "p95_ms": 0.560127,
+          "population_std_ms": 0.004131487002529642
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8447937692307695,
+          "n": 13,
+          "p50_ms": 0.8444480000000002,
+          "p95_ms": 0.8477320000000004,
+          "population_std_ms": 0.0012637563149865787
+        },
+        "moe": {
+          "mean_ms": 2.385914076923075,
+          "n": 13,
+          "p50_ms": 2.3754019999999985,
+          "p95_ms": 2.4513039999999986,
+          "population_std_ms": 0.03863391885270188
+        },
+        "other": {
+          "mean_ms": 0.025429,
+          "n": 13,
+          "p50_ms": 0.025372999999999996,
+          "p95_ms": 0.02592,
+          "population_std_ms": 0.00022032458572960754
+        },
+        "output_head": {
+          "mean_ms": 0.048271,
+          "n": 13,
+          "p50_ms": 0.048223999999999996,
+          "p95_ms": 0.049121000000000005,
+          "population_std_ms": 0.0004103602544856638
+        },
+        "router": {
+          "mean_ms": 0.18560184615384612,
+          "n": 13,
+          "p50_ms": 0.18550399999999995,
+          "p95_ms": 0.18627299999999997,
+          "population_std_ms": 0.00039899813883387426
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_6(6)": 13,
+        "execute_context_2(2049)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.429620846153846,
+        "n": 13,
+        "p50_ms": 5.421929,
+        "p95_ms": 5.4902809999999995,
+        "population_std_ms": 0.0385510445131288
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.971441538461537,
+        "n": 13,
+        "p50_ms": 4.965375999999999,
+        "p95_ms": 5.033802999999999,
+        "population_std_ms": 0.03898058259148536
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4581793076923092,
+        "n": 13,
+        "p50_ms": 0.45836400000000044,
+        "p95_ms": 0.46242600000000156,
+        "population_std_ms": 0.002535451754152047
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r1-20260723-20260723T091102857920Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r1/traces/profile/dp0_pp0_tp3_dcp0_ep3_rank3.1784798079118504722.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp4-b6-r2.json

@@ -0,0 +1,353 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp4-b6-r2",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.9215755535714284,
+      "collective": 4.099890464285714,
+      "linear_norm_rope": 0.8376546964285715,
+      "moe": 2.383938624999999,
+      "other": 0.02548019642857143,
+      "output_head": 0.04836294642857143,
+      "router": 0.19243757142857137
+    },
+    "ranks": 4,
+    "slowest_rank_execute_mean_ms": 10.1569875,
+    "slowest_rank_kernel_busy_mean_ms": 9.689807857142856
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9138666428571425,
+          "n": 14,
+          "p50_ms": 0.9139704999999996,
+          "p95_ms": 0.9164179999999996,
+          "population_std_ms": 0.0016477326242507631
+        },
+        "collective": {
+          "mean_ms": 5.290839214285713,
+          "n": 14,
+          "p50_ms": 0.5740495000000001,
+          "p95_ms": 28.996437999999984,
+          "population_std_ms": 9.010484784994565
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8356119285714287,
+          "n": 14,
+          "p50_ms": 0.8351870000000001,
+          "p95_ms": 0.8384630000000003,
+          "population_std_ms": 0.0012410830215286587
+        },
+        "moe": {
+          "mean_ms": 2.382082642857142,
+          "n": 14,
+          "p50_ms": 2.368174999999999,
+          "p95_ms": 2.4507140000000005,
+          "population_std_ms": 0.04469279722346926
+        },
+        "other": {
+          "mean_ms": 0.025481714285714285,
+          "n": 14,
+          "p50_ms": 0.025521000000000002,
+          "p95_ms": 0.025824,
+          "population_std_ms": 0.00017456289434365226
+        },
+        "output_head": {
+          "mean_ms": 0.048274285714285714,
+          "n": 14,
+          "p50_ms": 0.048175499999999996,
+          "p95_ms": 0.048896,
+          "population_std_ms": 0.0003860829719431935
+        },
+        "router": {
+          "mean_ms": 0.19008764285714277,
+          "n": 14,
+          "p50_ms": 0.19016149999999996,
+          "p95_ms": 0.1924179999999999,
+          "population_std_ms": 0.001319140120746566
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_6(6)": 14,
+        "execute_context_2(4096)_generation_4(4)": 1,
+        "execute_context_3(6144)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 10.1569875,
+        "n": 14,
+        "p50_ms": 5.495552,
+        "p95_ms": 33.916852,
+        "population_std_ms": 9.023692546910278
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 9.68624407142857,
+        "n": 14,
+        "p50_ms": 5.0233855,
+        "p95_ms": 33.44527299999998,
+        "population_std_ms": 9.023394784605482
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4707434285714301,
+        "n": 14,
+        "p50_ms": 0.47161650000001076,
+        "p95_ms": 0.47796200000000066,
+        "population_std_ms": 0.004213033072914231
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r2-20260723-20260723T092332531885Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r2/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798836346445272.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9236260714285712,
+          "n": 14,
+          "p50_ms": 0.9236309999999999,
+          "p95_ms": 0.9261099999999999,
+          "population_std_ms": 0.0016452857964623058
+        },
+        "collective": {
+          "mean_ms": 5.292963285714286,
+          "n": 14,
+          "p50_ms": 0.5790519999999999,
+          "p95_ms": 28.995478000000002,
+          "population_std_ms": 9.00940907489549
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8306526428571429,
+          "n": 14,
+          "p50_ms": 0.8308655000000003,
+          "p95_ms": 0.8323229999999998,
+          "population_std_ms": 0.001215001387838325
+        },
+        "moe": {
+          "mean_ms": 2.372463214285713,
+          "n": 14,
+          "p50_ms": 2.3604929999999986,
+          "p95_ms": 2.4334759999999993,
+          "population_std_ms": 0.04456007217899188
+        },
+        "other": {
+          "mean_ms": 0.025364571428571427,
+          "n": 14,
+          "p50_ms": 0.025328499999999997,
+          "p95_ms": 0.025759999999999998,
+          "population_std_ms": 0.00018273154168487357
+        },
+        "output_head": {
+          "mean_ms": 0.04847557142857143,
+          "n": 14,
+          "p50_ms": 0.048352,
+          "p95_ms": 0.049377000000000004,
+          "population_std_ms": 0.00036324720317195205
+        },
+        "router": {
+          "mean_ms": 0.1902149285714285,
+          "n": 14,
+          "p50_ms": 0.1895669999999999,
+          "p95_ms": 0.19510299999999997,
+          "population_std_ms": 0.002332658767044482
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_6(6)": 14,
+        "execute_context_2(4096)_generation_4(4)": 1,
+        "execute_context_3(6144)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 10.156081571428572,
+        "n": 14,
+        "p50_ms": 5.49357,
+        "p95_ms": 33.919204,
+        "population_std_ms": 9.024698774226101
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 9.683760285714284,
+        "n": 14,
+        "p50_ms": 5.019874,
+        "p95_ms": 33.448195,
+        "population_std_ms": 9.025241580098282
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.47232128571428716,
+        "n": 14,
+        "p50_ms": 0.47191050000000256,
+        "p95_ms": 0.47722900000000035,
+        "population_std_ms": 0.002093886660480702
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r2-20260723-20260723T092332531885Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r2/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798836333019388.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9213676428571428,
+          "n": 14,
+          "p50_ms": 0.921938,
+          "p95_ms": 0.9297530000000004,
+          "population_std_ms": 0.004745679668425435
+        },
+        "collective": {
+          "mean_ms": 5.281759785714286,
+          "n": 14,
+          "p50_ms": 0.5755820000000003,
+          "p95_ms": 28.99219900000001,
+          "population_std_ms": 9.010812440241091
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.840441214285714,
+          "n": 14,
+          "p50_ms": 0.8404519999999995,
+          "p95_ms": 0.8441329999999994,
+          "population_std_ms": 0.0019087839501543548
+        },
+        "moe": {
+          "mean_ms": 2.3862941428571425,
+          "n": 14,
+          "p50_ms": 2.373951,
+          "p95_ms": 2.459937000000001,
+          "population_std_ms": 0.04582171989953023
+        },
+        "other": {
+          "mean_ms": 0.025662357142857143,
+          "n": 14,
+          "p50_ms": 0.0256645,
+          "p95_ms": 0.026143999999999997,
+          "population_std_ms": 0.00029092699308610424
+        },
+        "output_head": {
+          "mean_ms": 0.048436714285714284,
+          "n": 14,
+          "p50_ms": 0.048575999999999994,
+          "p95_ms": 0.048991999999999994,
+          "population_std_ms": 0.0004154401157759286
+        },
+        "router": {
+          "mean_ms": 0.18584599999999996,
+          "n": 14,
+          "p50_ms": 0.185825,
+          "p95_ms": 0.18636899999999995,
+          "population_std_ms": 0.00029020731505990466
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_6(6)": 14,
+        "execute_context_2(4096)_generation_4(4)": 1,
+        "execute_context_3(6144)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 10.156770214285714,
+        "n": 14,
+        "p50_ms": 5.493733,
+        "p95_ms": 33.916464999999995,
+        "population_std_ms": 9.023992692816982
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 9.689807857142856,
+        "n": 14,
+        "p50_ms": 5.033896,
+        "p95_ms": 33.45033900000001,
+        "population_std_ms": 9.02452540636983
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.4669623571428573,
+        "n": 14,
+        "p50_ms": 0.467391000000001,
+        "p95_ms": 0.47341499999999836,
+        "population_std_ms": 0.004137908505636468
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r2-20260723-20260723T092332531885Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r2/traces/profile/dp0_pp0_tp2_dcp0_ep2_rank2.1784798836334934191.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9274418571428571,
+          "n": 14,
+          "p50_ms": 0.9274730000000002,
+          "p95_ms": 0.9312359999999998,
+          "population_std_ms": 0.0016344587071277954
+        },
+        "collective": {
+          "mean_ms": 0.5339995714285714,
+          "n": 14,
+          "p50_ms": 0.5353115,
+          "p95_ms": 0.5461410000000002,
+          "population_std_ms": 0.006859063864846362
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8439130000000005,
+          "n": 14,
+          "p50_ms": 0.8440930000000004,
+          "p95_ms": 0.8451400000000011,
+          "population_std_ms": 0.000779706171763452
+        },
+        "moe": {
+          "mean_ms": 2.3949144999999987,
+          "n": 14,
+          "p50_ms": 2.386371499999999,
+          "p95_ms": 2.457828999999998,
+          "population_std_ms": 0.044398212431438724
+        },
+        "other": {
+          "mean_ms": 0.02541214285714286,
+          "n": 14,
+          "p50_ms": 0.0254395,
+          "p95_ms": 0.025761999999999997,
+          "population_std_ms": 0.00027587053307739895
+        },
+        "output_head": {
+          "mean_ms": 0.048265214285714286,
+          "n": 14,
+          "p50_ms": 0.048063999999999996,
+          "p95_ms": 0.049057,
+          "population_std_ms": 0.00045419413385081205
+        },
+        "router": {
+          "mean_ms": 0.20360171428571425,
+          "n": 14,
+          "p50_ms": 0.20369749999999998,
+          "p95_ms": 0.20981799999999995,
+          "population_std_ms": 0.0021429506170088787
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_6(6)": 14,
+        "execute_context_2(4096)_generation_4(4)": 1,
+        "execute_context_3(6144)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 10.156985785714285,
+        "n": 14,
+        "p50_ms": 5.495433,
+        "p95_ms": 33.916939,
+        "population_std_ms": 9.023746050846789
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.977547999999999,
+        "n": 14,
+        "p50_ms": 4.961732999999999,
+        "p95_ms": 5.047107999999999,
+        "population_std_ms": 0.04773951894694539
+      },
+      "non_kernel_gap": {
+        "mean_ms": 5.179437785714287,
+        "n": 14,
+        "p50_ms": 0.4635420000000008,
+        "p95_ms": 28.877357,
+        "population_std_ms": 9.007091777202437
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r2-20260723-20260723T092332531885Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r2/traces/profile/dp0_pp0_tp3_dcp0_ep3_rank3.1784798836405275119.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp4-b6-r3.json

@@ -0,0 +1,353 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp4-b6-r3",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.9331604285714286,
+      "collective": 3.192824375,
+      "linear_norm_rope": 0.8404031785714285,
+      "moe": 2.399789857142857,
+      "other": 0.025532517857142854,
+      "output_head": 0.048368125,
+      "router": 0.19489794642857136
+    },
+    "ranks": 4,
+    "slowest_rank_execute_mean_ms": 8.633801928571428,
+    "slowest_rank_kernel_busy_mean_ms": 8.521951642857141
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9356066428571428,
+          "n": 14,
+          "p50_ms": 0.9354089999999998,
+          "p95_ms": 0.9385260000000003,
+          "population_std_ms": 0.001676932599682409
+        },
+        "collective": {
+          "mean_ms": 4.066270071428571,
+          "n": 14,
+          "p50_ms": 0.55129,
+          "p95_ms": 16.453366,
+          "population_std_ms": 5.619590739357225
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8387703571428572,
+          "n": 14,
+          "p50_ms": 0.8386610000000003,
+          "p95_ms": 0.8405769999999998,
+          "population_std_ms": 0.001042224242880996
+        },
+        "moe": {
+          "mean_ms": 2.403109357142857,
+          "n": 14,
+          "p50_ms": 2.3945385000000012,
+          "p95_ms": 2.4938269999999996,
+          "population_std_ms": 0.04863547144920812
+        },
+        "other": {
+          "mean_ms": 0.025629785714285713,
+          "n": 14,
+          "p50_ms": 0.025696499999999997,
+          "p95_ms": 0.02624,
+          "population_std_ms": 0.00037226741129221103
+        },
+        "output_head": {
+          "mean_ms": 0.04842978571428571,
+          "n": 14,
+          "p50_ms": 0.048384,
+          "p95_ms": 0.049248,
+          "population_std_ms": 0.00035716626351871235
+        },
+        "router": {
+          "mean_ms": 0.19324292857142852,
+          "n": 14,
+          "p50_ms": 0.19292599999999996,
+          "p95_ms": 0.1981709999999999,
+          "population_std_ms": 0.0022171815591706862
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_6(6)": 14,
+        "execute_context_1(2048)_generation_2(2)": 1,
+        "execute_context_3(6144)_generation_3(3)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 8.630969642857142,
+        "n": 14,
+        "p50_ms": 5.180047500000001,
+        "p95_ms": 20.987705000000002,
+        "population_std_ms": 5.5894495457541336
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 8.511058928571428,
+        "n": 14,
+        "p50_ms": 5.061531499999999,
+        "p95_ms": 20.864821999999997,
+        "population_std_ms": 5.589475958722368
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.11991071428571518,
+        "n": 14,
+        "p50_ms": 0.12010500000000235,
+        "p95_ms": 0.12316100000000052,
+        "population_std_ms": 0.002551737571274248
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r3-20260723-20260723T093751542606Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r3/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784799692262262652.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9304471428571427,
+          "n": 14,
+          "p50_ms": 0.930654,
+          "p95_ms": 0.9319719999999995,
+          "population_std_ms": 0.0009635016389000793
+        },
+        "collective": {
+          "mean_ms": 0.5773663571428572,
+          "n": 14,
+          "p50_ms": 0.5793035000000002,
+          "p95_ms": 0.59031,
+          "population_std_ms": 0.008760785031745086
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8337531428571427,
+          "n": 14,
+          "p50_ms": 0.8337829999999997,
+          "p95_ms": 0.8351629999999995,
+          "population_std_ms": 0.0008970194182593586
+        },
+        "moe": {
+          "mean_ms": 2.3872937857142857,
+          "n": 14,
+          "p50_ms": 2.3779580000000005,
+          "p95_ms": 2.4758159999999987,
+          "population_std_ms": 0.04801953596369253
+        },
+        "other": {
+          "mean_ms": 0.02552642857142857,
+          "n": 14,
+          "p50_ms": 0.025569500000000002,
+          "p95_ms": 0.025981999999999998,
+          "population_std_ms": 0.00035502126984283196
+        },
+        "output_head": {
+          "mean_ms": 0.048183142857142855,
+          "n": 14,
+          "p50_ms": 0.048112,
+          "p95_ms": 0.049249,
+          "population_std_ms": 0.00048348686747460613
+        },
+        "router": {
+          "mean_ms": 0.19290549999999992,
+          "n": 14,
+          "p50_ms": 0.19270699999999996,
+          "p95_ms": 0.1958099999999999,
+          "population_std_ms": 0.0013893215584799379
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_6(6)": 14,
+        "execute_context_1(2048)_generation_2(2)": 1,
+        "execute_context_3(6144)_generation_3(3)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 8.221614142857144,
+        "n": 14,
+        "p50_ms": 5.1788165,
+        "p95_ms": 21.941601,
+        "population_std_ms": 5.439025316289738
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.9954754999999995,
+        "n": 14,
+        "p50_ms": 4.988138500000001,
+        "p95_ms": 5.080260999999998,
+        "population_std_ms": 0.04987686944831562
+      },
+      "non_kernel_gap": {
+        "mean_ms": 3.2261386428571432,
+        "n": 14,
+        "p50_ms": 0.12194000000000038,
+        "p95_ms": 16.981787,
+        "population_std_ms": 5.465271976091736
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r3-20260723-20260723T093751542606Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r3/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784799692310997248.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9339937142857143,
+          "n": 14,
+          "p50_ms": 0.9335875,
+          "p95_ms": 0.9370900000000002,
+          "population_std_ms": 0.001917785904502957
+        },
+        "collective": {
+          "mean_ms": 4.071160642857143,
+          "n": 14,
+          "p50_ms": 0.5595504999999998,
+          "p95_ms": 16.441921000000004,
+          "population_std_ms": 5.617545177819675
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8418829285714284,
+          "n": 14,
+          "p50_ms": 0.8420469999999998,
+          "p95_ms": 0.8443809999999999,
+          "population_std_ms": 0.0015434930914874625
+        },
+        "moe": {
+          "mean_ms": 2.3997612142857148,
+          "n": 14,
+          "p50_ms": 2.3921425000000003,
+          "p95_ms": 2.4903699999999995,
+          "population_std_ms": 0.04851648560788174
+        },
+        "other": {
+          "mean_ms": 0.025574571428571426,
+          "n": 14,
+          "p50_ms": 0.025535999999999996,
+          "p95_ms": 0.026114000000000002,
+          "population_std_ms": 0.0002516030644503687
+        },
+        "output_head": {
+          "mean_ms": 0.04837264285714286,
+          "n": 14,
+          "p50_ms": 0.048512,
+          "p95_ms": 0.048960000000000004,
+          "population_std_ms": 0.00043459432761120634
+        },
+        "router": {
+          "mean_ms": 0.1906739285714285,
+          "n": 14,
+          "p50_ms": 0.19033549999999994,
+          "p95_ms": 0.19376099999999996,
+          "population_std_ms": 0.0011997104093599232
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_6(6)": 14,
+        "execute_context_1(2048)_generation_2(2)": 1,
+        "execute_context_3(6144)_generation_3(3)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 8.630474571428572,
+        "n": 14,
+        "p50_ms": 5.184212499999999,
+        "p95_ms": 20.985098999999998,
+        "population_std_ms": 5.5897660056633045
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 8.511419642857144,
+        "n": 14,
+        "p50_ms": 5.0612965,
+        "p95_ms": 20.866541000000005,
+        "population_std_ms": 5.589603699080663
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.11905492857142821,
+        "n": 14,
+        "p50_ms": 0.1185754999999995,
+        "p95_ms": 0.1234339999999996,
+        "population_std_ms": 0.0022576749881340562
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r3-20260723-20260723T093751542606Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r3/traces/profile/dp0_pp0_tp2_dcp0_ep2_rank2.1784799692265838678.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9325942142857142,
+          "n": 14,
+          "p50_ms": 0.9325534999999999,
+          "p95_ms": 0.9368339999999997,
+          "population_std_ms": 0.0022950169367122264
+        },
+        "collective": {
+          "mean_ms": 4.056500428571428,
+          "n": 14,
+          "p50_ms": 0.5449905,
+          "p95_ms": 16.437329999999996,
+          "population_std_ms": 5.618098886085954
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8472062857142859,
+          "n": 14,
+          "p50_ms": 0.8469400000000002,
+          "p95_ms": 0.8496459999999999,
+          "population_std_ms": 0.001113636605808041
+        },
+        "moe": {
+          "mean_ms": 2.4089950714285697,
+          "n": 14,
+          "p50_ms": 2.400836999999998,
+          "p95_ms": 2.491555999999999,
+          "population_std_ms": 0.04692533473274991
+        },
+        "other": {
+          "mean_ms": 0.025399285714285715,
+          "n": 14,
+          "p50_ms": 0.025391999999999998,
+          "p95_ms": 0.025988,
+          "population_std_ms": 0.00023926925316514007
+        },
+        "output_head": {
+          "mean_ms": 0.04848692857142857,
+          "n": 14,
+          "p50_ms": 0.048528,
+          "p95_ms": 0.04912,
+          "population_std_ms": 0.0004430105552250606
+        },
+        "router": {
+          "mean_ms": 0.20276942857142852,
+          "n": 14,
+          "p50_ms": 0.2021774999999999,
+          "p95_ms": 0.20793499999999998,
+          "population_std_ms": 0.0026834677382150296
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_6(6)": 14,
+        "execute_context_1(2048)_generation_2(2)": 1,
+        "execute_context_3(6144)_generation_3(3)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 8.633801928571428,
+        "n": 14,
+        "p50_ms": 5.179864499999999,
+        "p95_ms": 20.988474,
+        "population_std_ms": 5.589972337133016
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 8.521951642857141,
+        "n": 14,
+        "p50_ms": 5.071154499999998,
+        "p95_ms": 20.875197999999994,
+        "population_std_ms": 5.589281480492545
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.11185028571428772,
+        "n": 14,
+        "p50_ms": 0.11232300000000128,
+        "p95_ms": 0.11633899999999997,
+        "population_std_ms": 0.0023228722192460186
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_6(6)",
+      "steps": 14,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b6-r3-20260723-20260723T093751542606Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b6-r3/traces/profile/dp0_pp0_tp3_dcp0_ep3_rank3.1784799692262484339.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp4-b8-r1.json

@@ -0,0 +1,357 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp4-b8-r1",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.9206188846153844,
+      "collective": 0.6864052307692307,
+      "linear_norm_rope": 0.8407714807692306,
+      "moe": 2.0225017884615384,
+      "other": 0.023575519230769233,
+      "output_head": 0.04601186538461538,
+      "router": 0.19361455769230762
+    },
+    "ranks": 4,
+    "slowest_rank_execute_mean_ms": 4.898132615384616,
+    "slowest_rank_kernel_busy_mean_ms": 4.786881538461537
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9177004615384619,
+          "n": 13,
+          "p50_ms": 0.9157820000000005,
+          "p95_ms": 0.9225340000000003,
+          "population_std_ms": 0.0034723709615584538
+        },
+        "collective": {
+          "mean_ms": 0.7274665384615383,
+          "n": 13,
+          "p50_ms": 0.5518440000000001,
+          "p95_ms": 1.9667529999999998,
+          "population_std_ms": 0.43009803968337174
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8444890769230771,
+          "n": 13,
+          "p50_ms": 0.8439450000000001,
+          "p95_ms": 0.8488609999999996,
+          "population_std_ms": 0.0016826488943849141
+        },
+        "moe": {
+          "mean_ms": 2.0259989230769215,
+          "n": 13,
+          "p50_ms": 1.988923999999999,
+          "p95_ms": 2.1999619999999984,
+          "population_std_ms": 0.12419348290932108
+        },
+        "other": {
+          "mean_ms": 0.023444076923076925,
+          "n": 13,
+          "p50_ms": 0.023455999999999998,
+          "p95_ms": 0.023873,
+          "population_std_ms": 0.00022429083507407156
+        },
+        "output_head": {
+          "mean_ms": 0.046225461538461535,
+          "n": 13,
+          "p50_ms": 0.046369,
+          "p95_ms": 0.046944,
+          "population_std_ms": 0.00042185881987381115
+        },
+        "router": {
+          "mean_ms": 0.20155699999999996,
+          "n": 13,
+          "p50_ms": 0.20153599999999994,
+          "p95_ms": 0.2069459999999999,
+          "population_std_ms": 0.0028213344905722135
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_4(6145)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.898132615384616,
+        "n": 13,
+        "p50_ms": 4.671016,
+        "p95_ms": 6.300266,
+        "population_std_ms": 0.4965030281369486
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.786881538461537,
+        "n": 13,
+        "p50_ms": 4.558992999999998,
+        "p95_ms": 6.188288999999999,
+        "population_std_ms": 0.49662703894922927
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.11125107692307844,
+        "n": 13,
+        "p50_ms": 0.11197700000000044,
+        "p95_ms": 0.11466700000000163,
+        "population_std_ms": 0.0023827902345303826
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b8-r1-20260723-20260723T091720058921Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r1/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784798477249293674.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.920315615384615,
+          "n": 13,
+          "p50_ms": 0.9189889999999995,
+          "p95_ms": 0.9261809999999997,
+          "population_std_ms": 0.004422100328306619
+        },
+        "collective": {
+          "mean_ms": 0.5528313846153846,
+          "n": 13,
+          "p50_ms": 0.5527689999999998,
+          "p95_ms": 0.5607639999999999,
+          "population_std_ms": 0.006298594807127897
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8396410769230767,
+          "n": 13,
+          "p50_ms": 0.8399539999999995,
+          "p95_ms": 0.8417579999999998,
+          "population_std_ms": 0.0011731414408487047
+        },
+        "moe": {
+          "mean_ms": 2.02596976923077,
+          "n": 13,
+          "p50_ms": 1.9839060000000013,
+          "p95_ms": 2.2111380000000014,
+          "population_std_ms": 0.1288343954994783
+        },
+        "other": {
+          "mean_ms": 0.023620923076923078,
+          "n": 13,
+          "p50_ms": 0.023712,
+          "p95_ms": 0.024191,
+          "population_std_ms": 0.00032006452751773154
+        },
+        "output_head": {
+          "mean_ms": 0.045848769230769235,
+          "n": 13,
+          "p50_ms": 0.045729,
+          "p95_ms": 0.046465000000000006,
+          "population_std_ms": 0.0003160560208001501
+        },
+        "router": {
+          "mean_ms": 0.1913852307692307,
+          "n": 13,
+          "p50_ms": 0.1909419999999999,
+          "p95_ms": 0.19360099999999988,
+          "population_std_ms": 0.0011535338449135141
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_4(6145)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.897705538461539,
+        "n": 13,
+        "p50_ms": 4.67211,
+        "p95_ms": 6.299505,
+        "population_std_ms": 0.4965302476577009
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.599612769230769,
+        "n": 13,
+        "p50_ms": 4.552709000000001,
+        "p95_ms": 4.782309000000001,
+        "population_std_ms": 0.13015763047648984
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.2980927692307692,
+        "n": 13,
+        "p50_ms": 0.12020799999999987,
+        "p95_ms": 1.5448980000000008,
+        "population_std_ms": 0.4320880301128383
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b8-r1-20260723-20260723T091720058921Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r1/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784798477252917819.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9192469230769227,
+          "n": 13,
+          "p50_ms": 0.918141,
+          "p95_ms": 0.9271450000000001,
+          "population_std_ms": 0.004336312124402079
+        },
+        "collective": {
+          "mean_ms": 0.7371800769230769,
+          "n": 13,
+          "p50_ms": 0.5602280000000001,
+          "p95_ms": 1.9925419999999985,
+          "population_std_ms": 0.43457656793648614
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.838645769230769,
+          "n": 13,
+          "p50_ms": 0.8387470000000002,
+          "p95_ms": 0.8435129999999993,
+          "population_std_ms": 0.0020585651894104804
+        },
+        "moe": {
+          "mean_ms": 2.016775461538462,
+          "n": 13,
+          "p50_ms": 1.9734350000000007,
+          "p95_ms": 2.202970000000001,
+          "population_std_ms": 0.12585738148372902
+        },
+        "other": {
+          "mean_ms": 0.023712538461538462,
+          "n": 13,
+          "p50_ms": 0.023677999999999998,
+          "p95_ms": 0.024418000000000002,
+          "population_std_ms": 0.00033176465601725676
+        },
+        "output_head": {
+          "mean_ms": 0.04599407692307692,
+          "n": 13,
+          "p50_ms": 0.045984000000000004,
+          "p95_ms": 0.046912,
+          "population_std_ms": 0.00044461590364622505
+        },
+        "router": {
+          "mean_ms": 0.1922016153846153,
+          "n": 13,
+          "p50_ms": 0.19241399999999995,
+          "p95_ms": 0.1942349999999999,
+          "population_std_ms": 0.0013174549204643176
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_4(6145)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.895284846153846,
+        "n": 13,
+        "p50_ms": 4.669829,
+        "p95_ms": 6.300007,
+        "population_std_ms": 0.49721882332999884
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.773756461538461,
+        "n": 13,
+        "p50_ms": 4.548986,
+        "p95_ms": 6.178948,
+        "population_std_ms": 0.49733430942307366
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.12152838461538465,
+        "n": 13,
+        "p50_ms": 0.1210589999999998,
+        "p95_ms": 0.12582700000000013,
+        "population_std_ms": 0.0025302041431304327
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b8-r1-20260723-20260723T091720058921Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r1/traces/profile/dp0_pp0_tp2_dcp0_ep2_rank2.1784798477249455220.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9252125384615381,
+          "n": 13,
+          "p50_ms": 0.9248339999999994,
+          "p95_ms": 0.9334720000000002,
+          "population_std_ms": 0.00510464326509754
+        },
+        "collective": {
+          "mean_ms": 0.728142923076923,
+          "n": 13,
+          "p50_ms": 0.5495350000000001,
+          "p95_ms": 1.9773729999999994,
+          "population_std_ms": 0.43309947849824937
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8403099999999996,
+          "n": 13,
+          "p50_ms": 0.8403849999999997,
+          "p95_ms": 0.8420689999999995,
+          "population_std_ms": 0.0010695695755409923
+        },
+        "moe": {
+          "mean_ms": 2.0212630000000003,
+          "n": 13,
+          "p50_ms": 1.9823120000000007,
+          "p95_ms": 2.2015650000000004,
+          "population_std_ms": 0.12667332402498332
+        },
+        "other": {
+          "mean_ms": 0.02352453846153846,
+          "n": 13,
+          "p50_ms": 0.02352,
+          "p95_ms": 0.023999000000000003,
+          "population_std_ms": 0.0002336379893321535
+        },
+        "output_head": {
+          "mean_ms": 0.04597915384615384,
+          "n": 13,
+          "p50_ms": 0.04592,
+          "p95_ms": 0.046784,
+          "population_std_ms": 0.0004999015050916534
+        },
+        "router": {
+          "mean_ms": 0.18931438461538452,
+          "n": 13,
+          "p50_ms": 0.18934399999999993,
+          "p95_ms": 0.18963499999999994,
+          "population_std_ms": 0.00023338826628654523
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_1(1)": 1,
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_4(6145)_generation_4(4)": 1,
+        "execute_context_4(8191)_generation_1(1)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 4.895507769230769,
+        "n": 13,
+        "p50_ms": 4.665760000000001,
+        "p95_ms": 6.301792000000001,
+        "population_std_ms": 0.49796797169854445
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.773746538461538,
+        "n": 13,
+        "p50_ms": 4.544830999999999,
+        "p95_ms": 6.177703999999999,
+        "population_std_ms": 0.49768554496537193
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.12176123076923144,
+        "n": 13,
+        "p50_ms": 0.12077500000000008,
+        "p95_ms": 0.128610000000001,
+        "population_std_ms": 0.0033113621913244884
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b8-r1-20260723-20260723T091720058921Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r1/traces/profile/dp0_pp0_tp3_dcp0_ep3_rank3.1784798477250211216.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/results/tp4-b8-r2.json

@@ -0,0 +1,357 @@
+{
+  "contract": {
+    "component_time": "sum of CUDA kernel durations inside execute range",
+    "timing": "CUDA graph-on GPU execute annotations",
+    "tp_aggregation": "per-rank; slowest rank mean approximates critical path"
+  },
+  "label": "tp4-b8-r2",
+  "rank_summary": {
+    "component_rank_mean_ms": {
+      "attention": 0.9134566730769228,
+      "collective": 0.5546497884615385,
+      "linear_norm_rope": 0.8387807692307689,
+      "moe": 2.0065153269230764,
+      "other": 0.02364873076923077,
+      "output_head": 0.04593917307692308,
+      "router": 0.19024988461538456
+    },
+    "ranks": 4,
+    "slowest_rank_execute_mean_ms": 5.046865153846154,
+    "slowest_rank_kernel_busy_mean_ms": 4.576482615384614
+  },
+  "ranks": [
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9168304615384618,
+          "n": 13,
+          "p50_ms": 0.9167360000000001,
+          "p95_ms": 0.9226820000000003,
+          "population_std_ms": 0.0042293902200123395
+        },
+        "collective": {
+          "mean_ms": 0.533513923076923,
+          "n": 13,
+          "p50_ms": 0.5343079999999998,
+          "p95_ms": 0.5469470000000001,
+          "population_std_ms": 0.005929160823506614
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8429445384615385,
+          "n": 13,
+          "p50_ms": 0.8424360000000001,
+          "p95_ms": 0.8458869999999998,
+          "population_std_ms": 0.001745678030732511
+        },
+        "moe": {
+          "mean_ms": 2.013382153846152,
+          "n": 13,
+          "p50_ms": 1.9625909999999984,
+          "p95_ms": 2.195520999999998,
+          "population_std_ms": 0.13058695301059595
+        },
+        "other": {
+          "mean_ms": 0.023465076923076922,
+          "n": 13,
+          "p50_ms": 0.023581,
+          "p95_ms": 0.023903,
+          "population_std_ms": 0.00026946969618078375
+        },
+        "output_head": {
+          "mean_ms": 0.045974307692307696,
+          "n": 13,
+          "p50_ms": 0.046048,
+          "p95_ms": 0.046624000000000006,
+          "population_std_ms": 0.00041656145259371096
+        },
+        "router": {
+          "mean_ms": 0.2003721538461538,
+          "n": 13,
+          "p50_ms": 0.199709,
+          "p95_ms": 0.20480099999999998,
+          "population_std_ms": 0.002108014079873938
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_1(4)_generation_7(7)": 1,
+        "execute_context_3(6144)_generation_1(1)": 1,
+        "execute_context_4(8188)_generation_4(4)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.04529823076923,
+        "n": 13,
+        "p50_ms": 4.9983770000000005,
+        "p95_ms": 5.2427600000000005,
+        "population_std_ms": 0.1337420429024321
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.576482615384614,
+        "n": 13,
+        "p50_ms": 4.520321999999998,
+        "p95_ms": 4.770687999999998,
+        "population_std_ms": 0.13507694537418738
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.46881561538461686,
+        "n": 13,
+        "p50_ms": 0.46984200000000076,
+        "p95_ms": 0.4780550000000021,
+        "population_std_ms": 0.004066459044397948
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b8-r2-20260723-20260723T093027863838Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r2/traces/profile/dp0_pp0_tp0_dcp0_ep0_rank0.1784799265976107810.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9115253076923072,
+          "n": 13,
+          "p50_ms": 0.9091299999999993,
+          "p95_ms": 0.9181569999999997,
+          "population_std_ms": 0.004755129738335539
+        },
+        "collective": {
+          "mean_ms": 0.5619680769230769,
+          "n": 13,
+          "p50_ms": 0.5602239999999999,
+          "p95_ms": 0.5761279999999999,
+          "population_std_ms": 0.0068347860634070876
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8369866923076922,
+          "n": 13,
+          "p50_ms": 0.8367649999999999,
+          "p95_ms": 0.8395509999999997,
+          "population_std_ms": 0.0014106170163667622
+        },
+        "moe": {
+          "mean_ms": 2.008448384615385,
+          "n": 13,
+          "p50_ms": 1.955683,
+          "p95_ms": 2.194793000000001,
+          "population_std_ms": 0.13403461779448436
+        },
+        "other": {
+          "mean_ms": 0.023700230769230772,
+          "n": 13,
+          "p50_ms": 0.023742999999999997,
+          "p95_ms": 0.024096,
+          "population_std_ms": 0.00023815901405643158
+        },
+        "output_head": {
+          "mean_ms": 0.045964538461538466,
+          "n": 13,
+          "p50_ms": 0.045856,
+          "p95_ms": 0.046624000000000006,
+          "population_std_ms": 0.0003695759764471495
+        },
+        "router": {
+          "mean_ms": 0.1853764615384615,
+          "n": 13,
+          "p50_ms": 0.18530699999999997,
+          "p95_ms": 0.18560199999999993,
+          "population_std_ms": 0.0001339481044082144
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_1(4)_generation_7(7)": 1,
+        "execute_context_3(6144)_generation_1(1)": 1,
+        "execute_context_4(8188)_generation_4(4)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.045370692307693,
+        "n": 13,
+        "p50_ms": 4.998862,
+        "p95_ms": 5.242511,
+        "population_std_ms": 0.13589867228614172
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.573969692307692,
+        "n": 13,
+        "p50_ms": 4.520643,
+        "p95_ms": 4.767782,
+        "population_std_ms": 0.13563036850452942
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.47140100000000024,
+        "n": 13,
+        "p50_ms": 0.4708439999999987,
+        "p95_ms": 0.48116299999999956,
+        "population_std_ms": 0.0040546694639067024
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b8-r2-20260723-20260723T093027863838Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r2/traces/profile/dp0_pp0_tp1_dcp0_ep1_rank1.1784799265964743771.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9081704615384611,
+          "n": 13,
+          "p50_ms": 0.9067839999999995,
+          "p95_ms": 0.9137029999999994,
+          "population_std_ms": 0.003844601502114216
+        },
+        "collective": {
+          "mean_ms": 0.5607229230769231,
+          "n": 13,
+          "p50_ms": 0.5606410000000001,
+          "p95_ms": 0.5753579999999999,
+          "population_std_ms": 0.005889349049330662
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8375191538461533,
+          "n": 13,
+          "p50_ms": 0.8377699999999999,
+          "p95_ms": 0.8382449999999991,
+          "population_std_ms": 0.0007602619838080075
+        },
+        "moe": {
+          "mean_ms": 2.0034716923076927,
+          "n": 13,
+          "p50_ms": 1.9509769999999995,
+          "p95_ms": 2.1876420000000003,
+          "population_std_ms": 0.1321404819107208
+        },
+        "other": {
+          "mean_ms": 0.023784923076923075,
+          "n": 13,
+          "p50_ms": 0.023744,
+          "p95_ms": 0.024192,
+          "population_std_ms": 0.00026262885820062297
+        },
+        "output_head": {
+          "mean_ms": 0.04589046153846154,
+          "n": 13,
+          "p50_ms": 0.04592,
+          "p95_ms": 0.046464,
+          "population_std_ms": 0.0003319519463334356
+        },
+        "router": {
+          "mean_ms": 0.18966023076923072,
+          "n": 13,
+          "p50_ms": 0.18950599999999995,
+          "p95_ms": 0.19261299999999995,
+          "population_std_ms": 0.0014891042818184978
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_1(4)_generation_7(7)": 1,
+        "execute_context_3(6144)_generation_1(1)": 1,
+        "execute_context_4(8188)_generation_4(4)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.043654153846154,
+        "n": 13,
+        "p50_ms": 4.99831,
+        "p95_ms": 5.233926,
+        "population_std_ms": 0.1368864811675641
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.569219846153846,
+        "n": 13,
+        "p50_ms": 4.526768999999998,
+        "p95_ms": 4.755531999999999,
+        "population_std_ms": 0.13605665130991768
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.47443430769230827,
+        "n": 13,
+        "p50_ms": 0.47430400000000006,
+        "p95_ms": 0.48320900000000044,
+        "population_std_ms": 0.0038414357969708507
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b8-r2-20260723-20260723T093027863838Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r2/traces/profile/dp0_pp0_tp2_dcp0_ep2_rank2.1784799265970389602.pt.trace.json.gz"
+    },
+    {
+      "components": {
+        "attention": {
+          "mean_ms": 0.9173004615384611,
+          "n": 13,
+          "p50_ms": 0.915619999999999,
+          "p95_ms": 0.9253379999999995,
+          "population_std_ms": 0.0052849307922895015
+        },
+        "collective": {
+          "mean_ms": 0.5623942307692308,
+          "n": 13,
+          "p50_ms": 0.5634610000000001,
+          "p95_ms": 0.570651,
+          "population_std_ms": 0.006084772534195447
+        },
+        "linear_norm_rope": {
+          "mean_ms": 0.8376726923076918,
+          "n": 13,
+          "p50_ms": 0.8376949999999989,
+          "p95_ms": 0.8402979999999994,
+          "population_std_ms": 0.0016093297880994772
+        },
+        "moe": {
+          "mean_ms": 2.000759076923077,
+          "n": 13,
+          "p50_ms": 1.9501820000000014,
+          "p95_ms": 2.184468999999999,
+          "population_std_ms": 0.12953482671732042
+        },
+        "other": {
+          "mean_ms": 0.023644692307692308,
+          "n": 13,
+          "p50_ms": 0.023615999999999998,
+          "p95_ms": 0.023968,
+          "population_std_ms": 0.0002102991951465574
+        },
+        "output_head": {
+          "mean_ms": 0.04592738461538462,
+          "n": 13,
+          "p50_ms": 0.045824,
+          "p95_ms": 0.046688,
+          "population_std_ms": 0.0003889230769230783
+        },
+        "router": {
+          "mean_ms": 0.18559069230769226,
+          "n": 13,
+          "p50_ms": 0.18560099999999996,
+          "p95_ms": 0.18578999999999996,
+          "population_std_ms": 0.00010438640098240813
+        }
+      },
+      "execute_annotation_histogram": {
+        "execute_context_0(0)_generation_8(8)": 13,
+        "execute_context_1(4)_generation_7(7)": 1,
+        "execute_context_3(6144)_generation_1(1)": 1,
+        "execute_context_4(8188)_generation_4(4)": 1
+      },
+      "execute_wall": {
+        "mean_ms": 5.046865153846154,
+        "n": 13,
+        "p50_ms": 5.003904,
+        "p95_ms": 5.240352,
+        "population_std_ms": 0.1362175904832164
+      },
+      "gpu_kernel_busy": {
+        "mean_ms": 4.57328923076923,
+        "n": 13,
+        "p50_ms": 4.526548000000001,
+        "p95_ms": 4.767014999999998,
+        "population_std_ms": 0.13627994044452013
+      },
+      "non_kernel_gap": {
+        "mean_ms": 0.47357592307692375,
+        "n": 13,
+        "p50_ms": 0.474012000000001,
+        "p95_ms": 0.4803690000000005,
+        "population_std_ms": 0.0037547593482557514
+      },
+      "selected_execute_annotation": "execute_context_0(0)_generation_8(8)",
+      "steps": 13,
+      "trace": "runs/frontier-decode-batch-grid-v0/fleet-artifacts/decode-grid-tp4-b8-r2-20260723-20260723T093027863838Z/artifacts/runs/frontier-decode-batch-grid-v0/remote-outputs/tp4-b8-r2/traces/profile/dp0_pp0_tp3_dcp0_ep3_rank3.1784799265964152963.pt.trace.json.gz"
+    }
+  ],
+  "schema": "frontier-decode-batch-trace.v1"
+}

runs/frontier-decode-batch-grid-v0/run_bc8_replay.py

@@ -0,0 +1,222 @@
+#!/usr/bin/env python3
+"""Replay the exact BC-8 simulator cells with the whole-layer decode curve."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import subprocess
+import sys
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parent
+REPO = ROOT.parents[1]
+EXPECTED_FRONTIER_COMMIT = "deadc4a321f0baaa534c6ebd17f974123733cdc2"
+CONFIGS = ("tp1_mns16", "tp2_mns16", "tp4_mns16", "tp4_mns32")
+SOURCE_MANIFEST = (
+    REPO / "runs/frontier-collective-joint-v0/counterfactual/joint-r2/manifest.json"
+)
+JOINT_INPUTS = REPO / "runs/frontier-knee-sweep-v0/inputs"
+GRID = ROOT / "results/grid.json"
+WRAPPER = ROOT / "run_frontier_with_whole_decode_curve.py"
+CACHE_ROOT = REPO / "runs/frontier-knee-sweep-v0/cache"
+LOCAL_DEPENDENCY_ROOTS = (
+    REPO / "runs/frontier-collective-joint-v0/python-deps",
+    Path("/home/gahow/.cache/uv/archive-v0/-_kzErLcPO5nASZFX8b9k"),
+    Path("/home/gahow/.cache/uv/archive-v0/FbaBs_QJ9QKEbQ9V_4aIR"),
+    Path("/home/gahow/.cache/uv/archive-v0/fuHsGXD0Lv_UjFC8yI4-7"),
+    Path("/home/gahow/.cache/uv/archive-v0/jFGdqQLpB1eopfm9VxT3j"),
+    Path("/home/gahow/.cache/uv/archive-v0/YWW6ExSJuPVvv4-qYQTin"),
+    Path("/home/gahow/.cache/uv/archive-v0/3_qxZ5Ll-EpVAGZfbksfe"),
+)
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--frontier-checkout", type=Path, required=True)
+    parser.add_argument("--output-root", type=Path, default=ROOT / "replay/bc8")
+    parser.add_argument("--jobs", type=int, default=2)
+    return parser.parse_args()
+
+
+def sha256(path: Path) -> str:
+    return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def write_json(path: Path, payload) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+
+
+def replace_flag(argv: list[str], flag: str, value: str) -> None:
+    index = argv.index(flag)
+    argv[index + 1] = value
+
+
+def validate(frontier: Path) -> None:
+    commit = subprocess.check_output(
+        ["git", "-C", str(frontier), "rev-parse", "HEAD"], text=True
+    ).strip()
+    status = subprocess.check_output(
+        ["git", "-C", str(frontier), "status", "--porcelain"], text=True
+    ).strip()
+    if commit != EXPECTED_FRONTIER_COMMIT or status:
+        raise ValueError(
+            f"Frontier must be clean at {EXPECTED_FRONTIER_COMMIT}, "
+            f"got commit={commit}, dirty={bool(status)}"
+        )
+    required = (
+        SOURCE_MANIFEST,
+        GRID,
+        WRAPPER,
+        CACHE_ROOT,
+        *LOCAL_DEPENDENCY_ROOTS,
+    )
+    missing = [str(path) for path in required if not path.exists()]
+    if missing:
+        raise ValueError(f"missing replay inputs: {missing}")
+
+
+def trace_for(config: str) -> Path:
+    tp = int(config[2])
+    return (
+        JOINT_INPUTS
+        / f"traces-per-gpu-low/tp{tp}/w0-short-fixed-uniform-none/"
+        "rho0p02/public/frontier.csv"
+    )
+
+
+def run_one(
+    config: str,
+    *,
+    frontier: Path,
+    output_root: Path,
+    templates: dict,
+) -> dict:
+    point = output_root / "raw" / config
+    metrics_root = point / "metrics"
+    expected = list(metrics_root.glob("**/system_metrics.json"))
+    if len(expected) == 1 and (point / "usage.json").is_file():
+        return {"config": config, "status": "skipped_complete", "elapsed_s": 0.0}
+
+    argv = list(templates[config]["argv"])
+    argv[0] = sys.executable
+    argv[1] = str(WRAPPER.resolve())
+    replace_flag(argv, "--trace_request_generator_config_trace_file", str(trace_for(config)))
+    replace_flag(argv, "--metrics_config_output_dir", str(metrics_root))
+    replace_flag(argv, "--metrics_config_run_id", f"decode_grid_bc8_{config}")
+    replace_flag(argv, "--metrics_config_cache_dir", str(CACHE_ROOT / "model"))
+    replace_flag(argv, "--vidur_cc_backend_config_cache_dir", str(CACHE_ROOT / "cc"))
+    env = os.environ.copy()
+    env.update(
+        {
+            "CUDA_VISIBLE_DEVICES": "",
+            "PYTHONDONTWRITEBYTECODE": "1",
+            "PYTHONPATH": os.pathsep.join(
+                [str(frontier), *(str(path) for path in LOCAL_DEPENDENCY_ROOTS)]
+            ),
+            "FRONTIER_COLLECTIVE_CURVE": str(
+                (JOINT_INPUTS / "collective-curve-b4-extrapolated.json").resolve()
+            ),
+            "FRONTIER_COLLECTIVE_CURVE_VARIANT": "drop_mean",
+            "FRONTIER_FUSED_MOE_CURVE": str(
+                (JOINT_INPUTS / "fused-moe-curve-b4-extrapolated.json").resolve()
+            ),
+            "FRONTIER_WHOLE_DECODE_GRID": str(GRID.resolve()),
+            "FRONTIER_CURVE_USAGE": str((point / "usage.json").resolve()),
+        }
+    )
+    point.mkdir(parents=True, exist_ok=True)
+    write_json(point / "command.json", argv)
+    started = time.monotonic()
+    with (point / "run.log").open("w") as output:
+        completed = subprocess.run(
+            argv,
+            cwd=frontier,
+            env=env,
+            stdout=output,
+            stderr=subprocess.STDOUT,
+            check=False,
+        )
+    elapsed = time.monotonic() - started
+    metrics = list(metrics_root.glob("**/system_metrics.json"))
+    status = (
+        "completed"
+        if completed.returncode == 0
+        and len(metrics) == 1
+        and (point / "usage.json").is_file()
+        else "failed"
+    )
+    record = {
+        "config": config,
+        "status": status,
+        "returncode": completed.returncode,
+        "elapsed_s": elapsed,
+    }
+    write_json(point / "run-status.json", record)
+    return record
+
+
+def main() -> None:
+    args = parse_args()
+    if args.jobs < 1:
+        raise ValueError("--jobs must be positive")
+    frontier = args.frontier_checkout.resolve()
+    output_root = args.output_root.resolve()
+    validate(frontier)
+    source = json.loads(SOURCE_MANIFEST.read_text())
+    manifest = {
+        "schema": "frontier-decode-grid-bc8-replay.v1",
+        "frontier_checkout": str(frontier),
+        "frontier_commit": EXPECTED_FRONTIER_COMMIT,
+        "configs": list(CONFIGS),
+        "rho_per_gpu": 0.02,
+        "wrapper": str(WRAPPER.resolve()),
+        "wrapper_sha256": sha256(WRAPPER),
+        "whole_decode_grid": str(GRID.resolve()),
+        "whole_decode_grid_sha256": sha256(GRID),
+        "collective_curve_sha256": sha256(
+            JOINT_INPUTS / "collective-curve-b4-extrapolated.json"
+        ),
+        "moe_curve_sha256": sha256(
+            JOINT_INPUTS / "fused-moe-curve-b4-extrapolated.json"
+        ),
+        "traces": {
+            config: {
+                "path": str(trace_for(config).resolve()),
+                "sha256": sha256(trace_for(config)),
+            }
+            for config in CONFIGS
+        },
+    }
+    write_json(output_root / "manifest.json", manifest)
+    results = []
+    with ThreadPoolExecutor(max_workers=args.jobs) as pool:
+        futures = {
+            pool.submit(
+                run_one,
+                config,
+                frontier=frontier,
+                output_root=output_root,
+                templates=source["cells"],
+            ): config
+            for config in CONFIGS
+        }
+        for future in as_completed(futures):
+            result = future.result()
+            results.append(result)
+            print(json.dumps(result, sort_keys=True), flush=True)
+    results.sort(key=lambda row: CONFIGS.index(row["config"]))
+    write_json(output_root / "run-summary.json", results)
+    failures = [row for row in results if row["status"] == "failed"]
+    if failures:
+        raise SystemExit(f"failed replay cells: {failures}")
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-decode-batch-grid-v0/run_decode_profile.sh

@@ -0,0 +1,174 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+TP="${TP:?TP is required}"
+PROFILE_BATCH="${PROFILE_BATCH:?PROFILE_BATCH is required}"
+OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT is required}"
+SERVER_PORT="${SERVER_PORT:?SERVER_PORT is required}"
+VENV_ROOT="${VENV_ROOT:-/home/admin/cpfs/wjh/venvs/vllm-0.20.0-cu129-workload-regime-v2}"
+MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}"
+GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.92}"
+SERVER_READY_ATTEMPTS="${SERVER_READY_ATTEMPTS:-900}"
+ACTIVE_ITERATIONS="${ACTIVE_ITERATIONS:-16}"
+SCRIPT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "${SCRIPT_ROOT}/../.." && pwd)"
+MATERIALIZER="${SCRIPT_ROOT}/materialize_decode_batch.py"
+CLIENT="${PROJECT_ROOT}/runs/frontier-fidelity-envelope-v1/qwen30_exact_trace_client.py"
+SERVER_PID=""
+
+if [[ ! "${PROFILE_BATCH}" =~ ^(2|4|6|8)$ ]]; then
+  echo "ERROR: PROFILE_BATCH must be 2, 4, 6, or 8" >&2
+  exit 1
+fi
+IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?GPU allocation is required}"
+if [[ "${#GPU_IDS[@]}" -ne "${TP}" ]]; then
+  echo "ERROR: TP=${TP}, but CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES}" >&2
+  exit 1
+fi
+
+mkdir -p "${OUTPUT_ROOT}"
+OUTPUT_ROOT="$(cd "${OUTPUT_ROOT}" && pwd)"
+mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" \
+  "${OUTPUT_ROOT}/requests" "${OUTPUT_ROOT}/results" \
+  "${OUTPUT_ROOT}/trace-staging" "${OUTPUT_ROOT}/traces/profile"
+exec > >(tee -a "${OUTPUT_ROOT}/logs/controller.log") 2>&1
+
+cleanup() {
+  if [[ -n "${SERVER_PID}" ]] && kill -0 "${SERVER_PID}" 2>/dev/null; then
+    kill -TERM -- "-${SERVER_PID}" 2>/dev/null || true
+    for _ in $(seq 1 30); do
+      kill -0 "${SERVER_PID}" 2>/dev/null || break
+      sleep 1
+    done
+    kill -KILL -- "-${SERVER_PID}" 2>/dev/null || true
+  fi
+  SERVER_PID=""
+}
+trap cleanup EXIT INT TERM
+
+sha256sum "${BASH_SOURCE[0]}" "${MATERIALIZER}" "${CLIENT}" \
+  "${MODEL_ROOT}/config.json" > "${OUTPUT_ROOT}/provenance/inputs.sha256"
+"${VENV_ROOT}/bin/python" -c \
+  'import torch, transformers, vllm; print(f"torch={torch.__version__}"); print(f"transformers={transformers.__version__}"); print(f"vllm={vllm.__version__}")' \
+  > "${OUTPUT_ROOT}/provenance/runtime.versions"
+nvidia-smi --query-gpu=index,name,uuid,driver_version,memory.total \
+  --format=csv,noheader > "${OUTPUT_ROOT}/provenance/gpus.before.csv"
+ps -eo user,pid,ppid,etimes,pcpu,pmem,args --sort=pid \
+  > "${OUTPUT_ROOT}/provenance/processes.before.txt"
+uptime > "${OUTPUT_ROOT}/provenance/uptime.before.txt"
+env | sort > "${OUTPUT_ROOT}/provenance/environment.txt"
+
+PROFILE_CONFIG="$("${VENV_ROOT}/bin/python" - "${OUTPUT_ROOT}/trace-staging" \
+  "${ACTIVE_ITERATIONS}" <<'PY'
+import json
+import sys
+
+print(json.dumps({
+    "profiler": "torch",
+    "torch_profiler_dir": sys.argv[1],
+    "torch_profiler_with_stack": False,
+    "torch_profiler_record_shapes": True,
+    "torch_profiler_use_gzip": True,
+    "ignore_frontend": True,
+    "wait_iterations": 0,
+    "warmup_iterations": 2,
+    "active_iterations": int(sys.argv[2]),
+}, separators=(",", ":")))
+PY
+)"
+printf '%s\n' "${PROFILE_CONFIG}" \
+  > "${OUTPUT_ROOT}/provenance/profiler-config.json"
+
+export TOKENIZERS_PARALLELISM=false
+export VLLM_USE_V1=1
+export HF_HUB_OFFLINE=1
+export TRANSFORMERS_OFFLINE=1
+export FLASHINFER_WORKSPACE_BASE="${FLASHINFER_WORKSPACE_BASE:-${OUTPUT_ROOT}/flashinfer-workspace}"
+mkdir -p "${FLASHINFER_WORKSPACE_BASE}"
+ulimit -n 65536
+
+"${VENV_ROOT}/bin/python" "${MATERIALIZER}" --model "${MODEL_ROOT}" \
+  --batch "${PROFILE_BATCH}" --input-tokens 2048 --output-tokens 128 \
+  --output "${OUTPUT_ROOT}/requests/b${PROFILE_BATCH}.jsonl"
+
+printf 'LAUNCH host=%s tp=%s batch=%s gpus=%s output=%s\n' \
+  "$(hostname)" "${TP}" "${PROFILE_BATCH}" "${CUDA_VISIBLE_DEVICES}" \
+  "${OUTPUT_ROOT}"
+setsid "${VENV_ROOT}/bin/vllm" serve "${MODEL_ROOT}" \
+  --host 127.0.0.1 --port "${SERVER_PORT}" \
+  --served-model-name qwen30-decode-batch-profile \
+  --tensor-parallel-size "${TP}" \
+  --gpu-memory-utilization "${GPU_MEMORY_UTILIZATION}" \
+  --max-model-len 40960 --max-num-batched-tokens 8192 --max-num-seqs 16 \
+  --no-enable-prefix-caching --enable-chunked-prefill --no-enable-log-requests \
+  --enable-logging-iteration-details --profiler-config "${PROFILE_CONFIG}" \
+  > "${OUTPUT_ROOT}/logs/server.log" 2>&1 &
+SERVER_PID=$!
+
+READY=0
+for _ in $(seq 1 "${SERVER_READY_ATTEMPTS}"); do
+  if curl -fsS --max-time 2 \
+    "http://127.0.0.1:${SERVER_PORT}/v1/models" \
+    > "${OUTPUT_ROOT}/results/models.json" 2>/dev/null; then
+    READY=1
+    break
+  fi
+  if ! kill -0 "${SERVER_PID}" 2>/dev/null; then
+    tail -200 "${OUTPUT_ROOT}/logs/server.log"
+    exit 1
+  fi
+  sleep 3
+done
+if [[ "${READY}" -ne 1 ]]; then
+  echo "ERROR: server readiness timeout" >&2
+  tail -200 "${OUTPUT_ROOT}/logs/server.log"
+  exit 1
+fi
+
+run_client() {
+  local label="$1"
+  "${VENV_ROOT}/bin/python" "${CLIENT}" \
+    --port "${SERVER_PORT}" \
+    --requests-file "${OUTPUT_ROOT}/requests/b${PROFILE_BATCH}.jsonl" \
+    --served-model qwen30-decode-batch-profile \
+    --output "${OUTPUT_ROOT}/results/${label}.json" \
+    --tpot-slo-ms 150 --timeout-seconds 1800
+}
+
+# Exercise the same graph and scheduler path twice before profiling.
+run_client warmup-1
+run_client warmup-2
+
+curl -fsS -X POST "http://127.0.0.1:${SERVER_PORT}/start_profile" \
+  > "${OUTPUT_ROOT}/logs/start-profile.txt"
+run_client profile
+
+deadline=$((SECONDS + 120))
+while (( SECONDS < deadline )); do
+  trace_count="$(find "${OUTPUT_ROOT}/trace-staging" -maxdepth 1 -type f \
+    -name '*.pt.trace.json*' | wc -l)"
+  if (( trace_count >= TP )); then
+    break
+  fi
+  sleep 2
+done
+trace_count="$(find "${OUTPUT_ROOT}/trace-staging" -maxdepth 1 -type f \
+  -name '*.pt.trace.json*' | wc -l)"
+if (( trace_count < TP )); then
+  echo "ERROR: expected ${TP} rank traces, found ${trace_count}" >&2
+  exit 1
+fi
+find "${OUTPUT_ROOT}/trace-staging" -maxdepth 1 -type f \
+  -name '*.pt.trace.json*' -exec mv -t "${OUTPUT_ROOT}/traces/profile" {} +
+curl -fsS -X POST "http://127.0.0.1:${SERVER_PORT}/stop_profile" \
+  > "${OUTPUT_ROOT}/logs/stop-profile.txt"
+
+cleanup
+nvidia-smi --query-gpu=index,name,uuid,driver_version,memory.total \
+  --format=csv,noheader > "${OUTPUT_ROOT}/provenance/gpus.after.csv"
+find "${OUTPUT_ROOT}" -type f \
+  ! -path '*/provenance/artifacts.sha256' -print0 \
+  | sort -z | xargs -0 sha256sum \
+  > "${OUTPUT_ROOT}/provenance/artifacts.sha256"
+echo DECODE_BATCH_PROFILE_COMPLETE

runs/frontier-decode-batch-grid-v0/run_frontier_with_whole_decode_curve.py

@@ -0,0 +1,114 @@
+#!/usr/bin/env python3
+"""Run Frontier with the existing joint repair plus a whole-layer decode curve."""
+
+from __future__ import annotations
+
+import json
+import os
+import runpy
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parent
+REPO = ROOT.parents[1]
+GRID = json.loads(Path(os.environ["FRONTIER_WHOLE_DECODE_GRID"]).read_text())
+WHOLE_CURVE = {
+    str(tp): {
+        str(cell["batch"]): float(cell["median_execute_ms"])
+        for cell in GRID["cells"]
+        if int(cell["tp"]) == tp
+    }
+    for tp in (2, 4)
+}
+
+# Apply the existing serving-path collective/MoE correction first.
+joint = runpy.run_path(
+    str(REPO / "runs/frontier-collective-joint-v0/run_frontier_with_curves.py")
+)
+USAGE = joint["USAGE"]
+_pure_decode_point = joint["_pure_decode_point"]
+
+from frontier.entities import ExecutionTime  # noqa: E402
+from frontier.execution_time_predictor.sklearn_moe_execution_time_predictor import (  # noqa: E402
+    SklearnMoEExecutionTimePredictor,
+)
+
+
+_JOINT_STAGE_PREDICT = SklearnMoEExecutionTimePredictor.predict_stage_execution_time
+
+
+def _whole_decode_stage_time(
+    self,
+    batch,
+    stage_id,
+    cluster_type,
+    num_layers=1,
+    layer_id=0,
+):
+    base = _JOINT_STAGE_PREDICT(
+        self,
+        batch,
+        stage_id,
+        cluster_type,
+        num_layers=num_layers,
+        layer_id=layer_id,
+    )
+    point = _pure_decode_point(self, batch)
+    if point is None or point[0] == "1" or point[1] == "1":
+        return base
+    tp, decode_batch = point
+    if tp not in WHOLE_CURVE or decode_batch not in WHOLE_CURVE[tp]:
+        raise ValueError(
+            "Whole-layer curve has no exact pure-decode point for "
+            f"TP={tp}, batch={decode_batch}; refusing to extrapolate"
+        )
+    layers = int(base.num_layers)
+    if layers <= 0:
+        raise ValueError(f"invalid stage layer count: {layers}")
+    target_ms = WHOLE_CURVE[tp][decode_batch]
+    USAGE[f"whole_decode:tp{tp}-b{decode_batch}:target_ms={target_ms:.9f}"] += 1
+    return ExecutionTime(
+        num_layers_per_pipeline_stage=layers,
+        attention_rope_execution_time=0.0,
+        attention_kv_cache_save_execution_time=0.0,
+        attention_decode_execution_time=0.0,
+        attention_prefill_execution_time=0.0,
+        attention_layer_pre_proj_execution_time=0.0,
+        attention_layer_post_proj_execution_time=0.0,
+        attn_norm_time=0.0,
+        mlp_norm_time=0.0,
+        add_time=0.0,
+        tensor_parallel_communication_time=0.0,
+        pipeline_parallel_communication_time=0.0,
+        expert_parallel_communication_time=0.0,
+        moe_gating_time=0.0,
+        moe_shuffling_time=0.0,
+        schedule_time=base._schedule_time,
+        sampler_e2e_time=base._sampler_e2e_time,
+        prepare_inputs_e2e_time=base._prepare_inputs_e2e_time,
+        process_model_outputs_time=base._process_model_outputs_time,
+        ray_comm_time=base._ray_comm_time,
+        is_moe=True,
+        moe_grouped_gemm_time=target_ms / layers,
+        pp_producer_send_path_runtime_time=(
+            base._pp_producer_send_path_runtime_time
+        ),
+        pp_receiver_head_runtime_time=base._pp_receiver_head_runtime_time,
+        pp_prefill_consumer_active_runtime_time=(
+            base._pp_prefill_consumer_active_runtime_time
+        ),
+        pp_stage_boundary_residual_runtime_time=(
+            base._pp_stage_boundary_residual_runtime_time
+        ),
+    )
+
+
+SklearnMoEExecutionTimePredictor.predict_stage_execution_time = (
+    _whole_decode_stage_time
+)
+
+from frontier.main import main  # noqa: E402
+
+
+if __name__ == "__main__":
+    main()

runs/frontier-fidelity-envelope-v1/run_frontier_qwen30_exact_trace_surface.py

@@ -67,6 +67,11 @@ def parse_args() -> argparse.Namespace:
     parser.add_argument("--kernel-profile-root", type=Path)
     parser.add_argument("--python-deps", type=Path, required=True)
     parser.add_argument("--output-root", type=Path, required=True)
+    parser.add_argument(
+        "--predictor-cache-root",
+        type=Path,
+        help="Reuse a validated predictor cache outside the output root.",
+    )
     parser.add_argument(
         "--trace",
         action="append",
@@ -107,6 +112,11 @@ def parse_args() -> argparse.Namespace:
     )
     parser.add_argument("--resume", action="store_true")
     parser.add_argument("--continue-on-failure", action="store_true")
+    parser.add_argument(
+        "--store-stage-batch-ledger",
+        action="store_true",
+        help="Persist Frontier's per-stage batch/state ledger for diagnosis.",
+    )
     return parser.parse_args()
 
 
@@ -315,6 +325,17 @@ def classify_frontier_failure(stderr: str) -> str:
     return "frontier_error"
 
 
+def configure_stage_batch_ledger(command: list[str], *, enabled: bool) -> list[str]:
+    if not enabled:
+        return command
+    disabled = "--no-metrics_config_store_frontier_stage_batch_ledger"
+    enabled_flag = "--metrics_config_store_frontier_stage_batch_ledger"
+    if command.count(disabled) != 1 or enabled_flag in command:
+        raise ValueError("unexpected stage-batch-ledger command contract")
+    command[command.index(disabled)] = enabled_flag
+    return command
+
+
 def score(path: Path, expected_shapes: list[tuple[int, int]]) -> dict[str, Any]:
     with path.open(newline="") as source:
         rows = list(csv.DictReader(source))
@@ -542,9 +563,10 @@ def main() -> None:
     )
 
     config_results = []
+    predictor_cache_root = args.predictor_cache_root or args.output_root / "cache"
     for config in selected:
         loads = []
-        config_knobs = BASE.knobs(config, paths, args.output_root / "cache")
+        config_knobs = BASE.knobs(config, paths, predictor_cache_root)
         config_knobs["enable_prefix_caching"] = args.prefix_caching
         config_knobs["prediction_max_tokens_per_request"] = 40960
         config_knobs["decode_cuda_graph_mode"] = args.decode_cuda_graph_mode
@@ -592,6 +614,15 @@ def main() -> None:
                     str(args.predictor_training_job_threads),
                 ]
             )
+            command = configure_stage_batch_ledger(
+                command, enabled=args.store_stage_batch_ledger
+            )
+            command.extend(
+                [
+                    "--metrics_config_cache_dir",
+                    str(predictor_cache_root),
+                ]
+            )
             if args.align_real_graph_runtime:
                 command.extend(
                     [
@@ -621,7 +652,7 @@ def main() -> None:
                 )
             fallback_evidence = (
                 collective_fallback_evidence(run_dir)
-                if args.cc_backend == "vidur"
+                if args.cc_backend == "vidur" and config.tp > 1
                 else []
             )
             if completed.returncode == 0 and fallback_evidence:
@@ -803,6 +834,8 @@ def main() -> None:
             "prefix_caching": args.prefix_caching,
             "arrival": "original_trace_timestamp_and_order",
             "input_output": "exact_source_values",
+            "store_stage_batch_ledger": args.store_stage_batch_ledger,
+            "predictor_cache_root": str(predictor_cache_root),
             "ttft_slo": "1000ms + 1000ms * input_tokens / 8000",
             "tpot_slos_ms": TPOT_SLOS_MS,
             "primary_tpot_slo_ms": 150.0,

runs/frontier-simulator-gap-campaign-v0/README.md

@@ -0,0 +1,97 @@
+# Frontier simulator residual-gap campaign
+
+目标：在已经完成的 1h+ production chat-trace fidelity matrix 上，按信息增益和工程可修性依次关闭三个残余误差源。三个实验严格串行；前一项的产物是后一项的 baseline，不允许同时改多个 profile/component。
+
+## 冻结基线
+
+- Frontier：`deadc4a321f0baaa534c6ebd17f974123733cdc2`
+- Profile：`runs/frontier-prefill-kvgrowth-fix-v0/profiles/profile-v5-kvgrowth`
+- Workload：与 `docs/assets/frontier-fidelity/full-matrix.csv` 相同的 7 个 TP×load cells
+- Real：复用现有每 cell 两次 60-min trial；不重新采集，不做逐 cell E2E calibration
+- 指标：TTFT/TPOT/E2E 的 mean/p50/p90/p99；real trial 分别与同一 sim request ID 配对后汇总，不再把两次 real trial pool 成一个分布
+- 有效域：TP1 两个 cell 的 simulated waiting p99 已超过原 1 s subcritical gate，必须标成 `gate-fail diagnostic`，不能继续称为 subcritical evidence
+
+## 串行任务列表
+
+### EXP-1：structured attention-prefill predictor
+
+- [x] 锁定 Frontier/profile/trace provenance
+- [x] 审计训练与运行时 feature contract
+- [x] 冻结 experiment card、事前判据和 mock figure
+- [x] 实现最小 predictor patch：
+  - standard prefill 只训练 `batch_size=1`；多请求继续走既有 mixed predictor
+  - `base(q)` 使用单请求 `KV=0` profile 的单调分段插值
+  - KV growth 使用非负 `KV + q×KV` 项
+- [x] 单元测试：exact anchors、q/KV 单调、非负、pickle round-trip
+- [x] 离线 ablation：RF-all / RF-single / structured-single；held-out context MAPE
+- [x] 7-cell CPU replay
+- [x] trial-aware paired verdict 与旧 v5 baseline 对照
+
+Decision：**profile gate PASS，trace/merge gate FAIL**。TP1/TP2 TTFT mean/p99
+绝对误差改善约 7--10 pp；TP4 三个 cell 的 TTFT mean 绝对误差稳定恶化
+5.3--5.8 pp。该 patch 保留为机制 ablation，不作为全局默认 predictor。
+EXP-2 先执行 entry audit；仅当 structured 分支上的 TP2 chunk residual 仍 ≥10%
+才进入 GPU 三臂测量。
+
+Go/no-go：
+
+- profile gate：held-out context MAPE ≤5%，且所有 TP 的 q/KV 单调检查通过
+- trace gate：TP1 TTFT mean/p99 的绝对偏差各改善 ≥5 pp；TP2/TP4 任一 TTFT/E2E quantile 不得恶化 >5 pp
+- 若 profile gate 不过，不进入 trace replay；若 trace gate 不过，保留 profile diagnosis，回退 patch，不进入 EXP-2
+
+### EXP-2：TP2 base-prefill serving-path profile
+
+- [x] 在 EXP-1 冻结输出上重新确认 chunk #1 residual：TP2 是否仍为孤立点
+- [x] probe `dash1`--`dash4`；只选择 8 张 H20 全部 idle/healthy 的主机
+- [x] 三臂中的 A/C 同形状 profile：
+  - A：当前 standalone microbenchmark
+  - C：真实 serving path extract
+- [x] q8k smoke 与两个 TP rank 的 component contract
+- [x] 注入 q8k ratio，重放 TP2 两个 load cell；其余 profile 冻结
+- [x] paired verdict
+
+Decision：**mechanism gate PASS，global constant injection FAIL**。serving q8k
+execute=`408.19 ms`（real=`410 ms`），MoE=`214.52 ms` vs sim=`171.12 ms`，
+解释 `80.75%` residual。全 prefill-domain constant `1.25366×` 会过校正：
+subcritical TTFT mean `−4.54%→+5.23%` 且 E2E mean
+`+12.70%→+15.55%`。保留的工程方向是 TP2 token/routing-conditioned MoE
+curve，不合入 constant scale。B arm 合并为该 follow-up curve 的真实 routing
+采样，不再为已被 C 直接确认的机制单独占一次 GPU run。
+
+Go/no-go：
+
+- 若 B/C 相对 A 都没有稳定的 ≥10% shift，拒绝“tactic/warmup 或 serving composition”假设，不做 profile 注入
+- 若 C 能解释 real chunk residual，TP2 TTFT mean 目标收敛至 ±10%，且 E2E/TPOT 不恶化 >5 pp
+
+### EXP-3：decode whole-layer residual × batch curve
+
+- [x] 在 EXP-2 冻结输出上确认 TP4 low-load TPOT/E2E 正偏仍存在
+- [x] probe dash1--dash4；补 TP2/TP4 × batch 2/4/6/8 whole-layer grid，
+  b1 复用既有 serving anchor
+- [x] 将 whole-layer time 与 component sum 对齐，定位 TP4/b6 collective tail
+- [x] 稳定性 gate：TP4/b2 fresh-process CV=`0.067%`；TP4/b6 依规则加 r3
+- [x] 重放历史 BC-8 knee，检查 `TP2<TP4<TP1` 排序
+- [x] 审计 7-cell runtime support；因 batch 最大到 b15 且 TP4/b6
+  deterministic mean 不稳定，拒绝 b8 constant extrapolation，不做 global injection
+- [x] 分层判决：BC-8 排序不升级 event telemetry；绝对残差/MNS/high-batch
+  tail 保留 event/distribution-aware follow-up
+
+Decision：**BC-8 engineering fix PASS，global curve merge NO-GO**。稳定 TP4/b2
+median=`4.623 ms`；exact b2 injection 使 TP4 P4 sim TPOT
+`5.864→5.367 ms`，完整 MNS16 排序恢复为真实的 `TP2<TP4<TP1`。
+绝对 residual 仍为 `+0.918--+1.001 ms`，MNS16/32 仍 tie。TP4/b6 的
+5.433/10.157/8.634 ms 跨进程均值由 collective tail 主导；1h trace 又访问
+b9--b15，故当前只接受 scoped TP4/b2 correction，不接受全局 b2--b8 lookup。
+
+Go/no-go：
+
+- low-load TPOT/E2E mean 绝对偏差各改善 ≥5 pp
+- BC-8 topology order 恢复，且原 7-cell TTFT 任一 quantile 不恶化 >5 pp
+
+## 统一输出格式
+
+每个实验最终报告都分为：
+
+1. `Fact`：原始测量与 paired metric
+2. `Interpretation`：支持/反驳哪条机制假设
+3. `Decision`：merge、回退、继续下一实验或停止

runs/frontier-simulator-gap-campaign-v0/results/three-experiment-verdict.md

@@ -0,0 +1,46 @@
+# Frontier residual-gap 三实验判决
+
+## EXP-1 structured attention-prefill
+
+- **Fact：** held-out context MAPE 为 TP1/2/4 `0.84/1.60/3.01%`，profile
+  gate PASS；7-cell 中 TP1/TP2 TTFT 绝对误差改善约 7--10 pp，但 TP4
+  TTFT mean 稳定恶化 5.3--5.8 pp。
+- **Interpretation：** structured feature contract 正确，但旧 TP4 的好结果含
+  error cancellation；单独替换 attention 会暴露其它正偏。
+- **Decision：** 不合入默认 predictor，保留为机制 ablation。
+
+## EXP-2 TP2 prefill serving-path
+
+- **Fact：** q8k TP2 serving execute=`408.19 ms`（real anchor=`410 ms`）；
+  MoE=`214.52 ms` vs sim=`171.12 ms`，解释 `80.75%` residual。全域
+  `1.25366×` 常数注入使 subcritical TTFT mean `-4.54%→+5.23%`，
+  E2E mean `+12.70%→+15.55%`。
+- **Interpretation：** TP2 residual 的主机制是 serving-path MoE composition，
+  但 correction 随 token/routing state 变化，不是常数。
+- **Decision：** 机制 PASS、constant merge FAIL；工程项为 TP2
+  token/routing-conditioned MoE curve。
+
+## EXP-3 decode whole-layer batch grid
+
+- **Fact：** TP4/b2 fresh-process=`4.626/4.620 ms`，CV=`0.067%`，
+  median=`4.623 ms`。BC-8 exact b2 replay 将 TP4 sim TPOT
+  `5.864→5.367 ms`，排序恢复 `TP2<TP4<TP1`；残差仍
+  `+0.918--+1.001 ms`。TP4/b6 三次均值 `5.433/10.157/8.634 ms`，
+  差异来自 collective tail。
+- **Interpretation：** BC-8 topology gap 可由稳定 TP4/b2 whole-layer
+  service correction 工程修复；高 batch 需要分布式 collective-tail 模型，
+  不能当确定性 lookup。
+- **Decision：** scoped TP4/b2 correction PASS；global b2--b8 merge NO-GO。
+  1h trace 会访问 b9--b15，下一步补 b9--b16 与 collective-tail telemetry。
+
+## 总结
+
+三个明显 gap 中：
+
+1. **可直接工程化：** BC-8 的 scoped TP4/b2 whole-layer correction。
+2. **可工程化但需条件曲线：** TP2 prefill 的 token/routing-conditioned MoE。
+3. **不应直接合入：** structured attention 全局替换、prefill 常数 scale、
+   decode 全局 deterministic b2--b8 lookup。
+
+剩余新机制工作集中在 MNS/admission event semantics、TP4 high-batch
+collective-tail distribution，以及 b9--b16 profile support。

runs/frontier-split-rootcause-v0/frozen-inputs/q235-ablation-a1-prov/aituner.commit

@@ -0,0 +1 @@
+10567da5238db57c29a45e09ebd40343a7d1a543

runs/frontier-split-rootcause-v0/frozen-inputs/q235-ablation-a1-prov/artifacts.sha256

@@ -0,0 +1,180 @@
+bfbfb84d0310b312c3f3f0047219d5b615850d8b5b465282d6207ebe7b5ba987  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/analysis/comparison.json
+aa0295fcd1a0465483ca3655aae94de543acf5b39bde2fcfff6ff40a7fb27f2a  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/analysis/comparison.md
+f2c2a94627194431497c751e1ed0e3768f1f7971c3667e1f70563f1d7c5118f3  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/controller.log
+30bdc9da5c8e638d6278b3ae3f812613a1f7ef296439e34d28dcb72ae30b0d1b  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/profiles/qwen235-serving-allreduce.csv
+87a92ee3ba6955b95cf0fcdbf5dd8675a8ec85bf4a0dfd61ae12a75c32679fac  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/profiles/qwen235-serving-allreduce.manifest.json
+13a6cd30307a6dcdd12dc460949ce1e618cbd167cd976ad1b71ad6477869b55d  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/provenance/aituner.commit
+e3f7b627aa262187e7e02b703595f32fb1ced508724484ff515a68fe037d4f03  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/provenance/frontier.commit
+cd3d7ea44155f2d3e801926b35134d9735777256d72ba3f6a8f0cff2770ba09a  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/provenance/input.sha256
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/cc-cache/138b2eb7_prediction_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/cc-cache/479e1bac_prediction_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/cc-cache/5ae40466_model_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/cc-cache/6e9cd646_model_lock.file
+a6ad892511f6689578fca7b77a971dcfa16b11b2af54a2aaa90b98fa29643776  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/cc-cache/all_reduce_138b2eb7_predictions.csv
+f38672b7f93f91f2cf83996389fca9f659fe211df0853044dfa9cafa75b296c2  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/cc-cache/all_reduce_138b2eb7_predictions.pkl
+628f44a6a1c0f7c2ea121df3ef3a388e997581a998b97933d81be2577b0333f8  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/cc-cache/all_reduce_479e1bac_predictions.csv
+7dcc465e6b8e39b157bb7a2838b8385ef98f49eb34cffa9832afc2ab8dc6a785  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/cc-cache/all_reduce_479e1bac_predictions.pkl
+7fe2a18a04257bb047feb3a5abe09a299e24e454f567ec54b213ba5722be5f5f  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/cc-cache/all_reduce_5ae40466.pkl
+125f0eda94a6a39cf3d47ec3c99a9d57180de075ec49d85348eecbd19010a5aa  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/cc-cache/all_reduce_6e9cd646.pkl
+c56228d3654b51ea42c73f0883d58c67c19b1024c14f1677c63cb882e4dc3334  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/frontier_surface.json
+cf54e197cfcb6b480ccbd658f73dad0a90597bd58399d1c07f25e3987ca02227  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns128/eval/command.json
+7796df845a96bdb9603baf372118466e27200fc10ee83b26b469d4e2c089308f  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/op_precision_metadata.csv
+01ed41e9babf2eb71286e93d8f4986dda9820f92212c975ed9411a95a5f8758f  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/request_metrics.csv
+ab0ee72a7e2fbb3bf87c1653a9b1805b7805fd0f9ae3676ce69a0d9b112a4545  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/system_metrics.json
+84a609818002bcfb648a236a65b1d40dcd3c964360cb488dfb7549eddc546533  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns128/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns128/eval/stderr.log
+469ad61c0082534fa069a3ddc38510b4dc2180d016518834d3ac69f31c3ef7e5  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns128/eval/stdout.log
+080edcceed5f26160cd073a33c87aa8399cb0fe7cc9227bacb442aa1f90c5a30  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns64/eval/command.json
+db4e6b4632fc16a368407e458d26fd7a0adc9ae930755ac7784b899da60ee9fc  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/op_precision_metadata.csv
+01ed41e9babf2eb71286e93d8f4986dda9820f92212c975ed9411a95a5f8758f  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/request_metrics.csv
+ab0ee72a7e2fbb3bf87c1653a9b1805b7805fd0f9ae3676ce69a0d9b112a4545  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/system_metrics.json
+eb987dc5b2222740cef7577350190033cccee41946638aa5e9a5fd20c6be8b45  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns64/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns64/eval/stderr.log
+62312ef79e6302bf7be0c6ba276211065950b46eb4b4aef0a031358a804e17ce  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp4_ep1_mns64/eval/stdout.log
+89b4fc5eb8ccd1d5dc58b778352acaec5d496dded0181990fd3bce30c557ab48  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns128/eval/command.json
+bcc034a01b617f26aa4d7d760082fc1baa1e62b1b63ba50e11c9e234124d2c37  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/op_precision_metadata.csv
+51a848c4c12eacabbc72a0f6a8b5c1863e8b624b316b7ab55341e3fe5a3a753b  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/request_metrics.csv
+758424010e61780cbcbb320d68caa22c2e35c067654f9c4ef98217ff90e07b30  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/system_metrics.json
+463057bcb54084ab86eb35fc4f33ad62d1b3ebd05b6b04a1510ce8df6e686807  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns128/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns128/eval/stderr.log
+0a0433b5c338632e35516e34b262cf8b714cea73488ba9e3459f4881a0fe327e  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns128/eval/stdout.log
+2a69949a89e4dcedbede612b8beab834d63eb8683d97975bc779764304b1a627  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns64/eval/command.json
+88def2ba5bd00b375e0d907976bdd45c36faf53ad7980d8a1eb7e46057ccc8ec  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/op_precision_metadata.csv
+51a848c4c12eacabbc72a0f6a8b5c1863e8b624b316b7ab55341e3fe5a3a753b  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/request_metrics.csv
+758424010e61780cbcbb320d68caa22c2e35c067654f9c4ef98217ff90e07b30  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/system_metrics.json
+6ed18c855e41284d27a8aeca93799ba55745e7527344671edfed449693b29758  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns64/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns64/eval/stderr.log
+30bc99671a0bedac259b8be1d980a2981a48439e2d04a742d7baa4cc82638b39  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-pd/runs/tp8_ep8_mns64/eval/stdout.log
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/cc-cache/138b2eb7_prediction_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/cc-cache/479e1bac_prediction_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/cc-cache/5ae40466_model_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/cc-cache/6e9cd646_model_lock.file
+a6ad892511f6689578fca7b77a971dcfa16b11b2af54a2aaa90b98fa29643776  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/cc-cache/all_reduce_138b2eb7_predictions.csv
+f38672b7f93f91f2cf83996389fca9f659fe211df0853044dfa9cafa75b296c2  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/cc-cache/all_reduce_138b2eb7_predictions.pkl
+628f44a6a1c0f7c2ea121df3ef3a388e997581a998b97933d81be2577b0333f8  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/cc-cache/all_reduce_479e1bac_predictions.csv
+7dcc465e6b8e39b157bb7a2838b8385ef98f49eb34cffa9832afc2ab8dc6a785  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/cc-cache/all_reduce_479e1bac_predictions.pkl
+7fe2a18a04257bb047feb3a5abe09a299e24e454f567ec54b213ba5722be5f5f  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/cc-cache/all_reduce_5ae40466.pkl
+125f0eda94a6a39cf3d47ec3c99a9d57180de075ec49d85348eecbd19010a5aa  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/cc-cache/all_reduce_6e9cd646.pkl
+e83e951102f893a83c7ef4ebbae6e6f998a1a43a8d9f7a6ed165f5db96609980  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/frontier_surface.json
+e3f2699e0b9e182ab440acc690ddd8aaf0c954c5a83b7e1f444b3d494fcb2379  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns128/eval/command.json
+4a28f5aa74e7f9909545b1d1bc00a66098d85f7fff6b72fbb98d11d4a3f786c0  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/op_precision_metadata.csv
+0030204dc1047a02d601f7c1d57eee9fc2234080afd4e499c016b902a8da7573  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/request_metrics.csv
+d5b01e9e1689106d8174b1246da2c0c7a35ab7b83b1f8f2f3919753ac320e1e3  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/system_metrics.json
+599bb360437de9ebb16c1b29958c27b2c7f8e95eb047f9b8f26cb5709003755a  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns128/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns128/eval/stderr.log
+386b21c7695ae4159fb60f5c8772fd91512319f1511769e9b675955b7f26a223  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns128/eval/stdout.log
+3db0c671d763a7db9565b151464b96060455bafe7f065b35719e27c7d45b12b5  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns64/eval/command.json
+dfe34eef677ee24edd43ef01008aa30ed02fd7f251876149a55b9436130f179b  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/op_precision_metadata.csv
+0030204dc1047a02d601f7c1d57eee9fc2234080afd4e499c016b902a8da7573  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/request_metrics.csv
+d5b01e9e1689106d8174b1246da2c0c7a35ab7b83b1f8f2f3919753ac320e1e3  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/system_metrics.json
+9cceef591ceae33b03dddd558387bf4e1853c2a3e74913dd7839fb47dacfcb9c  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns64/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns64/eval/stderr.log
+1d5494b859450660d3f06050a91af27072ec4d15f4fe4bfb159f32f2e6769b89  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp4_ep1_mns64/eval/stdout.log
+439fe52e1d3c10c9c5cb4db20308e62da26a44866085d44768244d82ff0d22d1  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns128/eval/command.json
+9c1211fc6032dfa395933dbd2977115c5494a37b814c335dd2ed1ce70716c7ce  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/op_precision_metadata.csv
+d3ebd523ddf95f021d98c043d290670b20d8c0ffc2fdbca27fc5740ae46981f5  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/request_metrics.csv
+264cd59b0b0b9797f5054f8527e13412a9c9b5fd3a4608d3f81b228a6c08d0bc  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/system_metrics.json
+80adf7553fe587b542d0c43acf6850412199503a0eaa17ebbb04ac2870326059  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns128/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns128/eval/stderr.log
+3b86967c56b8dbd8db2edbbbbae367864ef41c703ee51fc4636811ca43f5c1ac  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns128/eval/stdout.log
+0ee66ca4a8363be762616e69329287c5a74993f36283ed5a470a4b012ec9a000  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns64/eval/command.json
+98b3db100b934f7b3473fa340f1c368f510f55071f7f79d458264a6d3aac5209  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/op_precision_metadata.csv
+d3ebd523ddf95f021d98c043d290670b20d8c0ffc2fdbca27fc5740ae46981f5  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/request_metrics.csv
+264cd59b0b0b9797f5054f8527e13412a9c9b5fd3a4608d3f81b228a6c08d0bc  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/system_metrics.json
+6b2fad699b75bf96907cf6408da1cb5c34ba7429da791df8f5c55ab24a949240  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns64/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns64/eval/stderr.log
+05475a151dca4d87d060c2714d7bd8837964d77b1e4ba335f1363b38a74e3a01  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/fixed-po/runs/tp8_ep8_mns64/eval/stdout.log
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/cc-cache/138b2eb7_prediction_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/cc-cache/479e1bac_prediction_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/cc-cache/5ae40466_model_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/cc-cache/6e9cd646_model_lock.file
+a6ad892511f6689578fca7b77a971dcfa16b11b2af54a2aaa90b98fa29643776  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/cc-cache/all_reduce_138b2eb7_predictions.csv
+f38672b7f93f91f2cf83996389fca9f659fe211df0853044dfa9cafa75b296c2  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/cc-cache/all_reduce_138b2eb7_predictions.pkl
+628f44a6a1c0f7c2ea121df3ef3a388e997581a998b97933d81be2577b0333f8  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/cc-cache/all_reduce_479e1bac_predictions.csv
+7dcc465e6b8e39b157bb7a2838b8385ef98f49eb34cffa9832afc2ab8dc6a785  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/cc-cache/all_reduce_479e1bac_predictions.pkl
+7fe2a18a04257bb047feb3a5abe09a299e24e454f567ec54b213ba5722be5f5f  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/cc-cache/all_reduce_5ae40466.pkl
+125f0eda94a6a39cf3d47ec3c99a9d57180de075ec49d85348eecbd19010a5aa  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/cc-cache/all_reduce_6e9cd646.pkl
+42420125d3a8ce4aed8dd1c859d39ff840b4e77a52ad5752de73d00a0aed5c3b  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/frontier_surface.json
+a222ec73d311c5d4e57b0ed3ed2054cb83ec31ff78031eb2b8bde6031472f9ca  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns128/eval/command.json
+9ff45df909e4f94da9435e85d893c0582fbb57ecc28ab15dddadce26604c0f1b  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/op_precision_metadata.csv
+12984c1c3ac1ab1d8fb35e7bcbacb147d827cdba43e934fdc757ee8ea7baf6fb  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/request_metrics.csv
+343388af242d66c2e8e46a4462dd23862d6437725c80961588fa6cdc328acdec  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/system_metrics.json
+b64186eaaa0f2fa566d181b40ed42cd9303b8131ce6fd7c4ae3d77855533a645  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns128/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns128/eval/stderr.log
+ece0b457d6c6fb30280d003e209912b918d06abc5e0781296ce29c6855542ed5  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns128/eval/stdout.log
+837819c6bcb470bc6067a3e5b24713166e4e3b1164090793690874597ea60e03  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns64/eval/command.json
+8db5d447bfc3506e4cf76a511a5a4760c567e2303240f878212aad44ff6f7cdc  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/op_precision_metadata.csv
+d244788351f45f3473a8a2705b232e1a951485b5a24e09029331f42eb47bbafc  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/request_metrics.csv
+5795a7d8f48ea0cde14f5c724d5a77d0903398a75151a95c39ad99c2ba2bfb68  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/system_metrics.json
+5baba11faf64f5c9e4ebbd123aa745c46d73c3f0f81cdd0a16249d212020d443  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns64/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns64/eval/stderr.log
+4d70795d321dff067aca5e9ead098726df129d556ceca8eb437d63cefa0c900c  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp4_ep1_mns64/eval/stdout.log
+40b8d7eb4a946406f37532bb308514bc4982088d872374d078635ce6ef764001  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns128/eval/command.json
+658539186152881399746213af4e3b041b5589cf3a8559346a109d444149767b  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/op_precision_metadata.csv
+00e4ed87800ce6054cf4818a4bac9b5db72c5e6dff52ff57d3b7b2489e97ffda  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/request_metrics.csv
+8947f894df28ac9f9cec7be09e79767a685f2ef96835e495d78eadcf5092633a  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/system_metrics.json
+04134b8170da075e36d33766ea9d520dd4db57abcf188116e05aaee1d125f8f3  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns128/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns128/eval/stderr.log
+bf6e1295c814c455d1c0b9f02ba0669636b58475efe8873907c6f53677e75fce  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns128/eval/stdout.log
+8719be2b20a1d0b35fc5bf23f398801737725527e6ca502a6f343566741517ca  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns64/eval/command.json
+5280da88ab96a0da6e54dc6dcf17d783822b8a49387a696e79830dfd7e81d2d2  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/op_precision_metadata.csv
+20adfc333507bbe98f607b1075b0f790e4651ad3185f96906009f6177839de96  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/request_metrics.csv
+1b8a2c5be9bb1c5ab551768a9bbe1c95a6c3f93c9f2b1cb6e1a53a86d3ffb239  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/system_metrics.json
+683afcccfbcfe232313965d985217bbc8acce32426bc31df749c6b8177cee8a9  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns64/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns64/eval/stderr.log
+02b34698d18bb18b46e4792b6208c6277249399e80e5f32c42a112f2fdc4a0e2  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-pd/runs/tp8_ep8_mns64/eval/stdout.log
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/cc-cache/138b2eb7_prediction_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/cc-cache/479e1bac_prediction_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/cc-cache/5ae40466_model_lock.file
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/cc-cache/6e9cd646_model_lock.file
+a6ad892511f6689578fca7b77a971dcfa16b11b2af54a2aaa90b98fa29643776  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/cc-cache/all_reduce_138b2eb7_predictions.csv
+f38672b7f93f91f2cf83996389fca9f659fe211df0853044dfa9cafa75b296c2  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/cc-cache/all_reduce_138b2eb7_predictions.pkl
+628f44a6a1c0f7c2ea121df3ef3a388e997581a998b97933d81be2577b0333f8  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/cc-cache/all_reduce_479e1bac_predictions.csv
+7dcc465e6b8e39b157bb7a2838b8385ef98f49eb34cffa9832afc2ab8dc6a785  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/cc-cache/all_reduce_479e1bac_predictions.pkl
+7fe2a18a04257bb047feb3a5abe09a299e24e454f567ec54b213ba5722be5f5f  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/cc-cache/all_reduce_5ae40466.pkl
+125f0eda94a6a39cf3d47ec3c99a9d57180de075ec49d85348eecbd19010a5aa  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/cc-cache/all_reduce_6e9cd646.pkl
+aafb5ba279ba8d0f693e12778851c563de4702ffba51e908324ad0e613468d3c  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/frontier_surface.json
+1ae2c8f734d6bde7740d06953f21c69af00b6307bcb8fc15b3a6a9415e6d29d8  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns128/eval/command.json
+b8f251e1ae5034352f35d61cbfdcad77c48ff978452b43092102ad785dc59930  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/op_precision_metadata.csv
+c164e64b1f0b2eb009185b07247d7ce3abbe4baf377ad62ef071a4767395b704  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/request_metrics.csv
+3b65bd4a043ab4c395b988fa94172123e851638b1f250ca8761f8725d1cf8cf5  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns128_eval/system_metrics.json
+2b61dc88137c817f96e7dcd45c65f451b8031647e56c6f8e554c9b91aee11982  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns128/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns128/eval/stderr.log
+0fb063af9bf0cc706e42f12ac1883a309bb4e7ca41924dac8b9ae8d9d580b223  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns128/eval/stdout.log
+08e6f686731ff6ceda8577c088ade825182777cff2528e207b444f2fe575f7ef  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns64/eval/command.json
+2f870b68c0820e241e43ebe435d301c88e2abf68bf9e29ff22c08baf39aeb574  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/op_precision_metadata.csv
+9349cb9f1ab478343b1b8d079cc175b50328b7c0bffb37018b3a7dc3e208c915  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/request_metrics.csv
+141593e073d0247455c73de04da5d82e918e3bf4128c225153a3e85faf63420c  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp4_ep1_mns64_eval/system_metrics.json
+c6faee6e21ba387608c5387e1a05eae3158a6edda0c530c38d4d8664b5e34a3c  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns64/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns64/eval/stderr.log
+cd8ee4fb98de672f9b7550dc39e0bf3e80a44c41c65f80c2f5192a9083d1b0cb  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp4_ep1_mns64/eval/stdout.log
+961af5f501fcc9606d0ba69924aa50b06e8030dc4239091216780e036702c95d  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns128/eval/command.json
+d9330a115bd2bf15f84986c44fdb35aecf67b8f8138807ed7c2256f36e1a6543  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/op_precision_metadata.csv
+6c6f40eba2795ef41f027c7412134694a91baa484933afc9b300803412d376cf  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/request_metrics.csv
+d7a7c8fe3fdcfb5d33589cf4ac1706e9fc011c95302180d6d7a1b874f6fd19f4  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns128/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns128_eval/system_metrics.json
+8aed33f0fc90d82c63f0a3c17dc0ef37ffd555046a4155f6333fca8ca4250df0  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns128/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns128/eval/stderr.log
+756930da43bef6ec443ebe6d6360ad15069a3a43cc0c0c6ef07d2795b83cfdb2  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns128/eval/stdout.log
+3dad31841fddfff0dc4c8dec00f78f9ffc75de82fc94eec50a354ee64902aa6d  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns64/eval/command.json
+b0437765d7f3c27102ea54833ff86d9c14c4f2b76185e770cf1276a53edb5669  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/config.json
+9d2e51495987994f2ef7f8308141f893d96a96b1a6268b29e6799643265bcb02  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/op_precision_metadata.csv
+015d5b1b48dbe1bb0de1a8a3265a1292e75af1f58d5ac076486e0b85fb133968  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/request_metrics.csv
+c0cb828fcaf067076c82a68eeb4c180e5e38d6f0f43dfcf666b0a4515b78a70c  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns64/eval/metrics/qwen3_235b_a22b/online_serving/qwen235_v020_tp8_ep8_mns64_eval/system_metrics.json
+8e4de34f2714e2b0cc1b7a7cc7e657662c3f99231adcea922dca73503a1d9ca2  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns64/eval/result.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns64/eval/stderr.log
+336c646cf9d88d4059f6a4584dab312a2e55401a808627fa8011280c7eb39c3a  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/sim/trace-po/runs/tp8_ep8_mns64/eval/stdout.log

runs/frontier-split-rootcause-v0/frozen-inputs/q235-ablation-a1-prov/frontier.commit

@@ -0,0 +1 @@
+6e8e0d845bceff11b0b62cb29df3a1a93411fdd4

runs/frontier-split-rootcause-v0/frozen-inputs/q235-ablation-a1-prov/input.sha256

@@ -0,0 +1,4 @@
+8eadf8b39becfcbe103c6328ca616902689e9694cc6cb6c1f08609f0e758ec41  /home/admin/cpfs/wjh/aituner/aituner-q235-collective-20260719/runs/frontier-fidelity-envelope-v1/run_qwen235_collective_profile_ablation.sh
+30bdc9da5c8e638d6278b3ae3f812613a1f7ef296439e34d28dcb72ae30b0d1b  /home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1/profiles/qwen235-serving-allreduce.csv
+5aa8657c5271abec6fca78a9f8fa2cfc0696f24c61033a850d7289f53f48632e  /home/admin/cpfs/wjh/aituner/qwen235-v020-profiles-20260719-r2/frozen/manifest.json
+99befe893683e3a638ea28048d00fa6c7b3d6a209921fe19ad51e78dfbc39459  /home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/provenance/runtime-contract.json

runs/frontier-split-rootcause-v0/frozen-inputs/q235-ablation-a1/comparison.json

@@ -0,0 +1,799 @@
+{
+  "cases": {
+    "fixed-pd": {
+      "comparison": {
+        "e2e_mean_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp4_ep1_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.3067524317507717,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "e2e_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp4_ep1_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.3456375801249614,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "tpot_mean_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp4_ep1_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.33025510615656906,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "tpot_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp4_ep1_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.37228890038064755,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "ttft_mean_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 1.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": true
+        },
+        "ttft_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 1.0,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [],
+          "selected_real_regret": 0.002088257708717345,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        }
+      },
+      "real": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 5729.31677040774,
+          "e2e_p90_ms": 5871.668232022785,
+          "request_samples": 387,
+          "tpot_mean_ms": 21.044200828709027,
+          "tpot_p90_ms": 21.59612646244247,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 363.00995954954925,
+          "ttft_p90_ms": 369.09779999405146
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 5769.497427325912,
+          "e2e_p90_ms": 5888.945628073998,
+          "request_samples": 387,
+          "tpot_mean_ms": 21.187139407074238,
+          "tpot_p90_ms": 21.653656356547977,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 366.7432916322386,
+          "ttft_p90_ms": 369.99153497163206
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 7486.7986220007915,
+          "e2e_p90_ms": 7901.13743103575,
+          "request_samples": 387,
+          "tpot_mean_ms": 27.994155607374484,
+          "tpot_p90_ms": 29.63612463562658,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 348.25438050341427,
+          "ttft_p90_ms": 365.8816539682448
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 7484.121415326708,
+          "e2e_p90_ms": 7908.651847974397,
+          "request_samples": 387,
+          "tpot_mean_ms": 27.978544004773987,
+          "tpot_p90_ms": 29.650389521346227,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 349.5577253122725,
+          "ttft_p90_ms": 365.11919100303203
+        }
+      },
+      "sim": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 19261.684938205322,
+          "e2e_p50_ms": 19681.421068277567,
+          "e2e_p90_ms": 19712.678973231064,
+          "e2e_p95_ms": 19717.18006692948,
+          "tpot_mean_ms": 74.22292738142716,
+          "tpot_p50_ms": 75.8976524911963,
+          "tpot_p90_ms": 75.8976524912086,
+          "tpot_p95_ms": 75.89765249120967,
+          "ttft_mean_ms": 334.8384559413977,
+          "ttft_p50_ms": 334.75193988430976,
+          "ttft_p90_ms": 359.7779859328796,
+          "ttft_p95_ms": 363.2786816744229
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 19261.684938205322,
+          "e2e_p50_ms": 19681.421068277567,
+          "e2e_p90_ms": 19712.678973231064,
+          "e2e_p95_ms": 19717.18006692948,
+          "tpot_mean_ms": 74.22292738142716,
+          "tpot_p50_ms": 75.8976524911963,
+          "tpot_p90_ms": 75.8976524912086,
+          "tpot_p95_ms": 75.89765249120967,
+          "ttft_mean_ms": 334.8384559413977,
+          "ttft_p50_ms": 334.75193988430976,
+          "ttft_p90_ms": 359.7779859328796,
+          "ttft_p95_ms": 363.2786816744229
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 14056.464119438264,
+          "e2e_p50_ms": 14650.653983203298,
+          "e2e_p90_ms": 14674.997135818061,
+          "e2e_p95_ms": 14677.859760866291,
+          "tpot_mean_ms": 54.15667269956665,
+          "tpot_p50_ms": 56.52350191837838,
+          "tpot_p90_ms": 56.52350850045728,
+          "tpot_p95_ms": 56.523546100039816,
+          "ttft_mean_ms": 246.51258104876885,
+          "ttft_p50_ms": 246.7781280760306,
+          "ttft_p90_ms": 263.0087113349084,
+          "ttft_p95_ms": 264.77576248352364
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 14056.464119438264,
+          "e2e_p50_ms": 14650.653983203298,
+          "e2e_p90_ms": 14674.997135818061,
+          "e2e_p95_ms": 14677.859760866291,
+          "tpot_mean_ms": 54.15667269956665,
+          "tpot_p50_ms": 56.52350191837838,
+          "tpot_p90_ms": 56.52350850045728,
+          "tpot_p95_ms": 56.523546100039816,
+          "ttft_mean_ms": 246.51258104876885,
+          "ttft_p50_ms": 246.7781280760306,
+          "ttft_p90_ms": 263.0087113349084,
+          "ttft_p95_ms": 264.77576248352364
+        }
+      }
+    },
+    "fixed-po": {
+      "comparison": {
+        "e2e_mean_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 1.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": true
+        },
+        "e2e_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 1.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": true
+        },
+        "ttft_mean_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 1.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": true
+        },
+        "ttft_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 1.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": true
+        }
+      },
+      "real": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 345.40159726161653,
+          "e2e_p90_ms": 346.7378600034863,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 345.35322551680514,
+          "ttft_p90_ms": 346.6896569589153
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 345.8011649264437,
+          "e2e_p90_ms": 347.9283060878515,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 345.75379653310387,
+          "ttft_p90_ms": 347.88418107200414
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 326.2389304512842,
+          "e2e_p90_ms": 341.0220619989559,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 326.19059710668336,
+          "ttft_p90_ms": 340.98013502079993
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 326.89427194042366,
+          "e2e_p90_ms": 341.91650594584644,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 326.846390089937,
+          "ttft_p90_ms": 341.87235799618065
+        }
+      },
+      "sim": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 303.2842141052072,
+          "e2e_p50_ms": 303.28421410520434,
+          "e2e_p90_ms": 303.28421410521855,
+          "e2e_p95_ms": 303.28421410521855,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 303.2842141052072,
+          "ttft_p50_ms": 303.28421410520434,
+          "ttft_p90_ms": 303.28421410521855,
+          "ttft_p95_ms": 303.28421410521855
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 303.2842141052072,
+          "e2e_p50_ms": 303.28421410520434,
+          "e2e_p90_ms": 303.28421410521855,
+          "e2e_p95_ms": 303.28421410521855,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 303.2842141052072,
+          "ttft_p50_ms": 303.28421410520434,
+          "ttft_p90_ms": 303.28421410521855,
+          "ttft_p95_ms": 303.28421410521855
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 224.0004422359394,
+          "e2e_p50_ms": 224.00044223590854,
+          "e2e_p90_ms": 224.00044223607017,
+          "e2e_p95_ms": 224.00044223607017,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 224.0004422359394,
+          "ttft_p50_ms": 224.00044223590854,
+          "ttft_p90_ms": 224.00044223607017,
+          "ttft_p95_ms": 224.00044223607017
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 224.0004422359394,
+          "e2e_p50_ms": 224.00044223590854,
+          "e2e_p90_ms": 224.00044223607017,
+          "e2e_p95_ms": 224.00044223607017,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 224.0004422359394,
+          "ttft_p50_ms": 224.00044223590854,
+          "ttft_p90_ms": 224.00044223607017,
+          "ttft_p95_ms": 224.00044223607017
+        }
+      }
+    },
+    "trace-pd": {
+      "comparison": {
+        "e2e_mean_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.8333333333333334,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [
+            [
+              "tp8_ep8_mns64",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.006154214824712234,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "e2e_p90_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.8333333333333334,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [
+            [
+              "tp8_ep8_mns64",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.062164651568153984,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "tpot_mean_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns64",
+          "winner_match": true
+        },
+        "tpot_p90_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns64",
+          "winner_match": true
+        },
+        "ttft_mean_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ]
+          ],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": true
+        },
+        "ttft_p90_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ]
+          ],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": true
+        }
+      },
+      "real": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 203026.22048668287,
+          "e2e_p90_ms": 344183.7428959552,
+          "request_samples": 387,
+          "tpot_mean_ms": 58.09653520572589,
+          "tpot_p90_ms": 75.82137218925735,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 26487.547554662306,
+          "ttft_p90_ms": 93044.83354603872
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 198593.25371425177,
+          "e2e_p90_ms": 332386.17597706616,
+          "request_samples": 387,
+          "tpot_mean_ms": 53.60856802543236,
+          "tpot_p90_ms": 64.87020396877685,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 31557.98602907994,
+          "ttft_p90_ms": 112666.82129900437
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 182457.35910624202,
+          "e2e_p90_ms": 318000.89644105174,
+          "request_samples": 387,
+          "tpot_mean_ms": 61.88514020253078,
+          "tpot_p90_ms": 78.6175951982538,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 1141.539354895564,
+          "ttft_p90_ms": 2616.6939180111513
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 181341.34550937495,
+          "e2e_p90_ms": 299389.4552709535,
+          "request_samples": 387,
+          "tpot_mean_ms": 45.89371812653489,
+          "tpot_p90_ms": 51.37046630467056,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 37215.26084531421,
+          "ttft_p90_ms": 132333.42359995004
+        }
+      },
+      "sim": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 350485.31604335306,
+          "e2e_p50_ms": 329886.2419457557,
+          "e2e_p90_ms": 618683.6003279199,
+          "e2e_p95_ms": 719028.7082294676,
+          "tpot_mean_ms": 92.88846066050185,
+          "tpot_p50_ms": 91.14780565610876,
+          "tpot_p90_ms": 104.48481783284998,
+          "tpot_p95_ms": 117.7639165127043,
+          "ttft_mean_ms": 51997.2508135414,
+          "ttft_p50_ms": 915.3872825012286,
+          "ttft_p90_ms": 175964.46056759413,
+          "ttft_p95_ms": 254194.45368846075
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 350205.0884431818,
+          "e2e_p50_ms": 345577.391630324,
+          "e2e_p90_ms": 612408.2764512596,
+          "e2e_p95_ms": 709831.5675796239,
+          "tpot_mean_ms": 86.77595832694777,
+          "tpot_p50_ms": 87.01966249041537,
+          "tpot_p90_ms": 95.27552612769624,
+          "tpot_p95_ms": 98.22236811591964,
+          "ttft_mean_ms": 64276.51238724817,
+          "ttft_p50_ms": 2913.567965463905,
+          "ttft_p90_ms": 231348.6370051864,
+          "ttft_p95_ms": 250924.94176651328
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 197537.45276613487,
+          "e2e_p50_ms": 184538.43666142275,
+          "e2e_p90_ms": 356508.87615749374,
+          "e2e_p95_ms": 426676.71753314673,
+          "tpot_mean_ms": 62.78094439594437,
+          "tpot_p50_ms": 61.98322714624318,
+          "tpot_p90_ms": 73.38270919472093,
+          "tpot_p95_ms": 79.82017533221833,
+          "ttft_mean_ms": 496.4562724801789,
+          "ttft_p50_ms": 258.18551545088656,
+          "ttft_p90_ms": 1185.2339118913733,
+          "ttft_p95_ms": 1563.6528594879735
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 230232.00122833299,
+          "e2e_p50_ms": 225231.54242404614,
+          "e2e_p90_ms": 399325.5400221139,
+          "e2e_p95_ms": 451116.17307457025,
+          "tpot_mean_ms": 56.01681518539994,
+          "tpot_p50_ms": 56.94928232211729,
+          "tpot_p90_ms": 60.38839798597023,
+          "tpot_p95_ms": 62.67163079918898,
+          "ttft_mean_ms": 46908.98176860714,
+          "ttft_p50_ms": 1623.170360660838,
+          "ttft_p90_ms": 166051.82069777363,
+          "ttft_p95_ms": 181013.0283352888
+        }
+      }
+    },
+    "trace-po": {
+      "comparison": {
+        "e2e_mean_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp4_ep1_mns128"
+            ],
+            [
+              "tp8_ep8_mns64",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.0137482658615673,
+          "sim_winner": "tp8_ep8_mns64",
+          "winner_match": false
+        },
+        "e2e_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 1.0,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [],
+          "selected_real_regret": 0.0025194410024500513,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "ttft_mean_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp4_ep1_mns128"
+            ],
+            [
+              "tp8_ep8_mns64",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.013750984200462835,
+          "sim_winner": "tp8_ep8_mns64",
+          "winner_match": false
+        },
+        "ttft_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 1.0,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [],
+          "selected_real_regret": 0.002607797240047338,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        }
+      },
+      "real": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 1083.6389664896267,
+          "e2e_p90_ms": 3156.966591021046,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 1083.561832460368,
+          "ttft_p90_ms": 3156.9307829486206
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 1102.0341126990652,
+          "e2e_p90_ms": 3162.989347940311,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 1101.9558890756384,
+          "ttft_p90_ms": 3162.9534389358014
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 1029.581229927413,
+          "e2e_p90_ms": 2612.0267439400777,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 1029.4865175674956,
+          "ttft_p90_ms": 2611.9744409807026
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 1043.7361864025345,
+          "e2e_p90_ms": 2605.4624350508675,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 1043.6429704051557,
+          "ttft_p90_ms": 2605.180658050813
+        }
+      },
+      "sim": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 698.2455408721717,
+          "e2e_p50_ms": 208.8217362068434,
+          "e2e_p90_ms": 2102.42229741975,
+          "e2e_p95_ms": 2868.6579248910534,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 698.2455408721717,
+          "ttft_p50_ms": 208.8217362068434,
+          "ttft_p90_ms": 2102.42229741975,
+          "ttft_p95_ms": 2868.6579248910534
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 698.0132356513792,
+          "e2e_p50_ms": 208.8217362068434,
+          "e2e_p90_ms": 2102.42229741975,
+          "e2e_p95_ms": 2868.6579248910534,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 698.0132356513792,
+          "ttft_p50_ms": 208.8217362068434,
+          "ttft_p90_ms": 2102.42229741975,
+          "ttft_p95_ms": 2868.6579248910534
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 487.42679595918156,
+          "e2e_p50_ms": 253.39288448911512,
+          "e2e_p90_ms": 1264.1466431938486,
+          "e2e_p95_ms": 1656.1837032209396,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 487.42679595918156,
+          "ttft_p50_ms": 253.39288448911512,
+          "ttft_p90_ms": 1264.1466431938486,
+          "ttft_p95_ms": 1656.1837032209396
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 487.3699604726311,
+          "e2e_p50_ms": 253.41330144894124,
+          "e2e_p90_ms": 1264.1466431938486,
+          "e2e_p95_ms": 1656.1837032209396,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 487.3699604726311,
+          "ttft_p50_ms": 253.41330144894124,
+          "ttft_p90_ms": 1264.1466431938486,
+          "ttft_p95_ms": 1656.1837032209396
+        }
+      }
+    }
+  },
+  "real_root": "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1",
+  "schema": "qwen235-v020-simulator-real-comparison-v1",
+  "sim_root": "/home/admin/cpfs/wjh/aituner/qwen235-collective-profile-ablation-20260719-r1"
+}

runs/frontier-split-rootcause-v0/frozen-inputs/q235-ablation-a1/comparison.md

@@ -0,0 +1,24 @@
+# Qwen235 vLLM 0.20 Frontier vs real
+
+| case | metric | Frontier winner | real winner | match | regret | pair agreement |
+|---|---|---|---|---:|---:|---:|
+| fixed-pd | ttft_mean_ms | tp8_ep8_mns128 | tp8_ep8_mns128 | yes | 0.0% | 100.0% |
+| fixed-pd | ttft_p90_ms | tp8_ep8_mns128 | tp8_ep8_mns64 | no | 0.2% | 100.0% |
+| fixed-pd | tpot_mean_ms | tp8_ep8_mns128 | tp4_ep1_mns128 | no | 33.0% | 0.0% |
+| fixed-pd | tpot_p90_ms | tp8_ep8_mns128 | tp4_ep1_mns128 | no | 37.2% | 0.0% |
+| fixed-pd | e2e_mean_ms | tp8_ep8_mns128 | tp4_ep1_mns128 | no | 30.7% | 0.0% |
+| fixed-pd | e2e_p90_ms | tp8_ep8_mns128 | tp4_ep1_mns128 | no | 34.6% | 0.0% |
+| fixed-po | ttft_mean_ms | tp8_ep8_mns128 | tp8_ep8_mns128 | yes | 0.0% | 100.0% |
+| fixed-po | ttft_p90_ms | tp8_ep8_mns128 | tp8_ep8_mns128 | yes | 0.0% | 100.0% |
+| fixed-po | e2e_mean_ms | tp8_ep8_mns128 | tp8_ep8_mns128 | yes | 0.0% | 100.0% |
+| fixed-po | e2e_p90_ms | tp8_ep8_mns128 | tp8_ep8_mns128 | yes | 0.0% | 100.0% |
+| trace-pd | ttft_mean_ms | tp8_ep8_mns128 | tp8_ep8_mns128 | yes | 0.0% | 66.7% |
+| trace-pd | ttft_p90_ms | tp8_ep8_mns128 | tp8_ep8_mns128 | yes | 0.0% | 66.7% |
+| trace-pd | tpot_mean_ms | tp8_ep8_mns64 | tp8_ep8_mns64 | yes | 0.0% | 66.7% |
+| trace-pd | tpot_p90_ms | tp8_ep8_mns64 | tp8_ep8_mns64 | yes | 0.0% | 66.7% |
+| trace-pd | e2e_mean_ms | tp8_ep8_mns128 | tp8_ep8_mns64 | no | 0.6% | 83.3% |
+| trace-pd | e2e_p90_ms | tp8_ep8_mns128 | tp8_ep8_mns64 | no | 6.2% | 83.3% |
+| trace-po | ttft_mean_ms | tp8_ep8_mns64 | tp8_ep8_mns128 | no | 1.4% | 66.7% |
+| trace-po | ttft_p90_ms | tp8_ep8_mns128 | tp8_ep8_mns64 | no | 0.3% | 100.0% |
+| trace-po | e2e_mean_ms | tp8_ep8_mns64 | tp8_ep8_mns128 | no | 1.4% | 66.7% |
+| trace-po | e2e_p90_ms | tp8_ep8_mns128 | tp8_ep8_mns64 | no | 0.3% | 100.0% |

runs/frontier-split-rootcause-v0/frozen-inputs/q235-fourcase-a0/comparison.json

@@ -0,0 +1,952 @@
+{
+  "cases": {
+    "fixed-pd": {
+      "comparison": {
+        "e2e_mean_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp4_ep1_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.3067524317507717,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "e2e_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp4_ep1_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.3456375801249614,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "tpot_mean_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp4_ep1_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.33025510615656906,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "tpot_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp4_ep1_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.37228890038064755,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "ttft_mean_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.04237011756982145,
+          "sim_winner": "tp4_ep1_mns128",
+          "winner_match": false
+        },
+        "ttft_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 1.0,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [],
+          "selected_real_regret": 0.002088257708717345,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        }
+      },
+      "real": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 5729.31677040774,
+          "e2e_p90_ms": 5871.668232022785,
+          "request_samples": 387,
+          "tpot_mean_ms": 21.044200828709027,
+          "tpot_p90_ms": 21.59612646244247,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 363.00995954954925,
+          "ttft_p90_ms": 369.09779999405146
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 5769.497427325912,
+          "e2e_p90_ms": 5888.945628073998,
+          "request_samples": 387,
+          "tpot_mean_ms": 21.187139407074238,
+          "tpot_p90_ms": 21.653656356547977,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp4_ep1_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 366.7432916322386,
+          "ttft_p90_ms": 369.99153497163206
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 7486.7986220007915,
+          "e2e_p90_ms": 7901.13743103575,
+          "request_samples": 387,
+          "tpot_mean_ms": 27.994155607374484,
+          "tpot_p90_ms": 29.63612463562658,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 348.25438050341427,
+          "ttft_p90_ms": 365.8816539682448
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 7484.121415326708,
+          "e2e_p90_ms": 7908.651847974397,
+          "request_samples": 387,
+          "tpot_mean_ms": 27.978544004773987,
+          "tpot_p90_ms": 29.650389521346227,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-pd/real/tp8_ep8_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 349.5577253122725,
+          "ttft_p90_ms": 365.11919100303203
+        }
+      },
+      "sim": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 22734.095381397616,
+          "e2e_p50_ms": 23441.595611495926,
+          "e2e_p90_ms": 23484.306701188103,
+          "e2e_p95_ms": 23489.511452611594,
+          "tpot_mean_ms": 87.77349881741111,
+          "tpot_p50_ms": 90.6067822139126,
+          "tpot_p90_ms": 90.60678221391389,
+          "tpot_p95_ms": 90.60678221391389,
+          "ttft_mean_ms": 351.8531829577845,
+          "ttft_p50_ms": 351.44748294051453,
+          "ttft_p90_ms": 381.14669406792245,
+          "ttft_p95_ms": 385.3185272085611
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 22734.095381397616,
+          "e2e_p50_ms": 23441.595611495926,
+          "e2e_p90_ms": 23484.306701188103,
+          "e2e_p95_ms": 23489.511452611594,
+          "tpot_mean_ms": 87.77349881741111,
+          "tpot_p50_ms": 90.6067822139126,
+          "tpot_p90_ms": 90.60678221391389,
+          "tpot_p95_ms": 90.60678221391389,
+          "ttft_mean_ms": 351.8531829577845,
+          "ttft_p50_ms": 351.44748294051453,
+          "ttft_p90_ms": 381.14669406792245,
+          "ttft_p95_ms": 385.3185272085611
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 16059.899811799207,
+          "e2e_p50_ms": 17256.45308650381,
+          "e2e_p90_ms": 17280.553823012277,
+          "e2e_p95_ms": 17282.746891876577,
+          "tpot_mean_ms": 61.58781555229474,
+          "tpot_p50_ms": 66.32324900760649,
+          "tpot_p90_ms": 66.3246446898248,
+          "tpot_p95_ms": 66.3246446898248,
+          "ttft_mean_ms": 355.00684596404665,
+          "ttft_p50_ms": 355.16695480338,
+          "ttft_p90_ms": 368.8582878708644,
+          "ttft_p95_ms": 370.503089520021
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 16059.899811799207,
+          "e2e_p50_ms": 17256.45308650381,
+          "e2e_p90_ms": 17280.553823012277,
+          "e2e_p95_ms": 17282.746891876577,
+          "tpot_mean_ms": 61.58781555229474,
+          "tpot_p50_ms": 66.32324900760649,
+          "tpot_p90_ms": 66.3246446898248,
+          "tpot_p95_ms": 66.3246446898248,
+          "ttft_mean_ms": 355.00684596404665,
+          "ttft_p50_ms": 355.16695480338,
+          "ttft_p90_ms": 368.8582878708644,
+          "ttft_p95_ms": 370.503089520021
+        }
+      }
+    },
+    "fixed-po": {
+      "comparison": {
+        "e2e_mean_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.05873813644442971,
+          "sim_winner": "tp4_ep1_mns128",
+          "winner_match": false
+        },
+        "e2e_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.016760786592592503,
+          "sim_winner": "tp4_ep1_mns128",
+          "winner_match": false
+        },
+        "ttft_mean_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.058746722254088946,
+          "sim_winner": "tp4_ep1_mns128",
+          "winner_match": false
+        },
+        "ttft_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.016744441542810407,
+          "sim_winner": "tp4_ep1_mns128",
+          "winner_match": false
+        }
+      },
+      "real": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 345.40159726161653,
+          "e2e_p90_ms": 346.7378600034863,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 345.35322551680514,
+          "ttft_p90_ms": 346.6896569589153
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 345.8011649264437,
+          "e2e_p90_ms": 347.9283060878515,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp4_ep1_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 345.75379653310387,
+          "ttft_p90_ms": 347.88418107200414
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 326.2389304512842,
+          "e2e_p90_ms": 341.0220619989559,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 326.19059710668336,
+          "ttft_p90_ms": 340.98013502079993
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 326.89427194042366,
+          "e2e_p90_ms": 341.91650594584644,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/fixed-po/real/tp8_ep8_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 326.846390089937,
+          "ttft_p90_ms": 341.87235799618065
+        }
+      },
+      "sim": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 313.84871569626137,
+          "e2e_p50_ms": 313.848715696261,
+          "e2e_p90_ms": 313.8487156962668,
+          "e2e_p95_ms": 313.8487156962668,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 313.84871569626137,
+          "ttft_p50_ms": 313.848715696261,
+          "ttft_p90_ms": 313.8487156962668,
+          "ttft_p95_ms": 313.8487156962668
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 313.84871569626137,
+          "e2e_p50_ms": 313.848715696261,
+          "e2e_p90_ms": 313.8487156962668,
+          "e2e_p95_ms": 313.8487156962668,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 313.84871569626137,
+          "ttft_p50_ms": 313.848715696261,
+          "ttft_p90_ms": 313.8487156962668,
+          "ttft_p95_ms": 313.8487156962668
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 334.2963928223074,
+          "e2e_p50_ms": 334.2963928222744,
+          "e2e_p90_ms": 334.2963928224325,
+          "e2e_p95_ms": 334.2963928224325,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 334.2963928223074,
+          "ttft_p50_ms": 334.2963928222744,
+          "ttft_p90_ms": 334.2963928224325,
+          "ttft_p95_ms": 334.2963928224325
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 334.2963928223074,
+          "e2e_p50_ms": 334.2963928222744,
+          "e2e_p90_ms": 334.2963928224325,
+          "e2e_p95_ms": 334.2963928224325,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 334.2963928223074,
+          "ttft_p50_ms": 334.2963928222744,
+          "ttft_p90_ms": 334.2963928224325,
+          "ttft_p95_ms": 334.2963928224325
+        }
+      }
+    },
+    "trace-pd": {
+      "comparison": {
+        "e2e_mean_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.8333333333333334,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [
+            [
+              "tp8_ep8_mns64",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.006154214824712234,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "e2e_p90_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp4_ep1_mns128"
+            ],
+            [
+              "tp8_ep8_mns64",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.062164651568153984,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": false
+        },
+        "tpot_mean_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns64",
+          "winner_match": true
+        },
+        "tpot_p90_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns64",
+          "winner_match": true
+        },
+        "ttft_mean_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ]
+          ],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": true
+        },
+        "ttft_p90_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.6666666666666666,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ]
+          ],
+          "selected_real_regret": 0.0,
+          "sim_winner": "tp8_ep8_mns128",
+          "winner_match": true
+        }
+      },
+      "real": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 203026.22048668287,
+          "e2e_p90_ms": 344183.7428959552,
+          "request_samples": 387,
+          "tpot_mean_ms": 58.09653520572589,
+          "tpot_p90_ms": 75.82137218925735,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 26487.547554662306,
+          "ttft_p90_ms": 93044.83354603872
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 198593.25371425177,
+          "e2e_p90_ms": 332386.17597706616,
+          "request_samples": 387,
+          "tpot_mean_ms": 53.60856802543236,
+          "tpot_p90_ms": 64.87020396877685,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp4_ep1_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 31557.98602907994,
+          "ttft_p90_ms": 112666.82129900437
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 182457.35910624202,
+          "e2e_p90_ms": 318000.89644105174,
+          "request_samples": 387,
+          "tpot_mean_ms": 61.88514020253078,
+          "tpot_p90_ms": 78.6175951982538,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 1141.539354895564,
+          "ttft_p90_ms": 2616.6939180111513
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 181341.34550937495,
+          "e2e_p90_ms": 299389.4552709535,
+          "request_samples": 387,
+          "tpot_mean_ms": 45.89371812653489,
+          "tpot_p90_ms": 51.37046630467056,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-pd/real/tp8_ep8_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 37215.26084531421,
+          "ttft_p90_ms": 132333.42359995004
+        }
+      },
+      "sim": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 385157.8895230749,
+          "e2e_p50_ms": 367024.8243163692,
+          "e2e_p90_ms": 673951.1520207045,
+          "e2e_p95_ms": 785718.9717121953,
+          "tpot_mean_ms": 100.38179934319204,
+          "tpot_p50_ms": 100.42433178975269,
+          "tpot_p90_ms": 113.67517808832463,
+          "tpot_p95_ms": 121.63558651348922,
+          "ttft_mean_ms": 59067.296381594686,
+          "ttft_p50_ms": 908.4160127742393,
+          "ttft_p90_ms": 200789.2407497713,
+          "ttft_p95_ms": 289881.2920798696
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 380604.1804371099,
+          "e2e_p50_ms": 378158.20614125655,
+          "e2e_p90_ms": 678448.1689084907,
+          "e2e_p95_ms": 763969.1713963022,
+          "tpot_mean_ms": 93.54579486627527,
+          "tpot_p50_ms": 92.86650455014843,
+          "tpot_p90_ms": 102.66261969117096,
+          "tpot_p95_ms": 112.87824516729198,
+          "ttft_mean_ms": 72259.33174260205,
+          "ttft_p50_ms": 2897.238841988653,
+          "ttft_p90_ms": 265395.14114254655,
+          "ttft_p95_ms": 273163.7753733101
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 174939.1805624549,
+          "e2e_p50_ms": 166723.75338099696,
+          "e2e_p90_ms": 317439.7347771481,
+          "e2e_p95_ms": 371471.8509246631,
+          "tpot_mean_ms": 58.37204729277776,
+          "tpot_p50_ms": 54.29394782230334,
+          "tpot_p90_ms": 75.91440592454555,
+          "tpot_p95_ms": 84.58610544605006,
+          "ttft_mean_ms": 945.9090659083846,
+          "ttft_p50_ms": 669.9533543429972,
+          "ttft_p90_ms": 2348.607298432925,
+          "ttft_p95_ms": 2781.46390848989
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 208352.4497977196,
+          "e2e_p50_ms": 207573.22184026084,
+          "e2e_p90_ms": 353236.8601573669,
+          "e2e_p95_ms": 402705.2991150841,
+          "tpot_mean_ms": 52.06723645713691,
+          "tpot_p50_ms": 51.64653531173072,
+          "tpot_p90_ms": 60.374786839964116,
+          "tpot_p95_ms": 62.94158800335912,
+          "ttft_mean_ms": 43679.663681651284,
+          "ttft_p50_ms": 2772.0384805371,
+          "ttft_p90_ms": 151027.16878398336,
+          "ttft_p95_ms": 165014.86227423933
+        }
+      }
+    },
+    "trace-po": {
+      "comparison": {
+        "e2e_mean_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp4_ep1_mns128"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp8_ep8_mns64",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.07037121566091509,
+          "sim_winner": "tp4_ep1_mns64",
+          "winner_match": false
+        },
+        "e2e_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.21167227304868486,
+          "sim_winner": "tp4_ep1_mns128",
+          "winner_match": false
+        },
+        "ttft_mean_ms": {
+          "informative_pairs": 6,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp8_ep8_mns128",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp4_ep1_mns128"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp8_ep8_mns64",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.07039370625209918,
+          "sim_winner": "tp4_ep1_mns64",
+          "winner_match": false
+        },
+        "ttft_p90_ms": {
+          "informative_pairs": 4,
+          "pair_direction_agreement": 0.0,
+          "real_winner": "tp8_ep8_mns64",
+          "reversals": [
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns64",
+              "tp8_ep8_mns128"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns64"
+            ],
+            [
+              "tp4_ep1_mns128",
+              "tp8_ep8_mns128"
+            ]
+          ],
+          "selected_real_regret": 0.2117895828808376,
+          "sim_winner": "tp4_ep1_mns128",
+          "winner_match": false
+        }
+      },
+      "real": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 1083.6389664896267,
+          "e2e_p90_ms": 3156.966591021046,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 1083.561832460368,
+          "ttft_p90_ms": 3156.9307829486206
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 1102.0341126990652,
+          "e2e_p90_ms": 3162.989347940311,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp4_ep1_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 1101.9558890756384,
+          "ttft_p90_ms": 3162.9534389358014
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 1029.581229927413,
+          "e2e_p90_ms": 2612.0267439400777,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns128/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns128/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns128/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 1029.4865175674956,
+          "ttft_p90_ms": 2611.9744409807026
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 1043.7361864025345,
+          "e2e_p90_ms": 2605.4624350508675,
+          "request_samples": 387,
+          "tpot_mean_ms": null,
+          "tpot_p90_ms": null,
+          "trials": [
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns64/trial1/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns64/trial2/results/result.json",
+            "/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/real/trace-po/real/tp8_ep8_mns64/trial3/results/result.json"
+          ],
+          "ttft_mean_ms": 1043.6429704051557,
+          "ttft_p90_ms": 2605.180658050813
+        }
+      },
+      "sim": {
+        "tp4_ep1_mns128": {
+          "e2e_mean_ms": 691.7873605232029,
+          "e2e_p50_ms": 211.81272326305088,
+          "e2e_p90_ms": 2068.417017674378,
+          "e2e_p95_ms": 2843.8890088954663,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 691.7873605232029,
+          "ttft_p50_ms": 211.81272326305088,
+          "ttft_p90_ms": 2068.417017674378,
+          "ttft_p95_ms": 2843.8890088954663
+        },
+        "tp4_ep1_mns64": {
+          "e2e_mean_ms": 691.5722223601863,
+          "e2e_p50_ms": 210.89579132517144,
+          "e2e_p90_ms": 2068.417017674378,
+          "e2e_p95_ms": 2843.8890088954663,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 691.5722223601863,
+          "ttft_p50_ms": 210.89579132517144,
+          "ttft_p90_ms": 2068.417017674378,
+          "ttft_p95_ms": 2843.8890088954663
+        },
+        "tp8_ep8_mns128": {
+          "e2e_mean_ms": 928.7998403823643,
+          "e2e_p50_ms": 593.0899264387222,
+          "e2e_p90_ms": 2341.4108333476474,
+          "e2e_p95_ms": 2898.493085937801,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 928.7998403823643,
+          "ttft_p50_ms": 593.0899264387222,
+          "ttft_p90_ms": 2341.4108333476474,
+          "ttft_p95_ms": 2898.493085937801
+        },
+        "tp8_ep8_mns64": {
+          "e2e_mean_ms": 928.7182564423873,
+          "e2e_p50_ms": 593.0899264387222,
+          "e2e_p90_ms": 2341.4108333476474,
+          "e2e_p95_ms": 2898.493085937801,
+          "tpot_mean_ms": null,
+          "tpot_p50_ms": null,
+          "tpot_p90_ms": null,
+          "tpot_p95_ms": null,
+          "ttft_mean_ms": 928.7182564423873,
+          "ttft_p50_ms": 593.0899264387222,
+          "ttft_p90_ms": 2341.4108333476474,
+          "ttft_p95_ms": 2898.493085937801
+        }
+      }
+    }
+  },
+  "schema": "qwen235-v020-simulator-real-comparison-v1"
+}

runs/frontier-split-rootcause-v0/frozen-inputs/q235-fourcase-a0/comparison.md

@@ -0,0 +1,24 @@
+# Qwen235 vLLM 0.20 Frontier vs real
+
+| case | metric | Frontier winner | real winner | match | regret | pair agreement |
+|---|---|---|---|---:|---:|---:|
+| fixed-pd | ttft_mean_ms | tp4_ep1_mns128 | tp8_ep8_mns128 | no | 4.2% | 0.0% |
+| fixed-pd | ttft_p90_ms | tp8_ep8_mns128 | tp8_ep8_mns64 | no | 0.2% | 100.0% |
+| fixed-pd | tpot_mean_ms | tp8_ep8_mns128 | tp4_ep1_mns128 | no | 33.0% | 0.0% |
+| fixed-pd | tpot_p90_ms | tp8_ep8_mns128 | tp4_ep1_mns128 | no | 37.2% | 0.0% |
+| fixed-pd | e2e_mean_ms | tp8_ep8_mns128 | tp4_ep1_mns128 | no | 30.7% | 0.0% |
+| fixed-pd | e2e_p90_ms | tp8_ep8_mns128 | tp4_ep1_mns128 | no | 34.6% | 0.0% |
+| fixed-po | ttft_mean_ms | tp4_ep1_mns128 | tp8_ep8_mns128 | no | 5.9% | 0.0% |
+| fixed-po | ttft_p90_ms | tp4_ep1_mns128 | tp8_ep8_mns128 | no | 1.7% | 0.0% |
+| fixed-po | e2e_mean_ms | tp4_ep1_mns128 | tp8_ep8_mns128 | no | 5.9% | 0.0% |
+| fixed-po | e2e_p90_ms | tp4_ep1_mns128 | tp8_ep8_mns128 | no | 1.7% | 0.0% |
+| trace-pd | ttft_mean_ms | tp8_ep8_mns128 | tp8_ep8_mns128 | yes | 0.0% | 66.7% |
+| trace-pd | ttft_p90_ms | tp8_ep8_mns128 | tp8_ep8_mns128 | yes | 0.0% | 66.7% |
+| trace-pd | tpot_mean_ms | tp8_ep8_mns64 | tp8_ep8_mns64 | yes | 0.0% | 66.7% |
+| trace-pd | tpot_p90_ms | tp8_ep8_mns64 | tp8_ep8_mns64 | yes | 0.0% | 66.7% |
+| trace-pd | e2e_mean_ms | tp8_ep8_mns128 | tp8_ep8_mns64 | no | 0.6% | 83.3% |
+| trace-pd | e2e_p90_ms | tp8_ep8_mns128 | tp8_ep8_mns64 | no | 6.2% | 66.7% |
+| trace-po | ttft_mean_ms | tp4_ep1_mns64 | tp8_ep8_mns128 | no | 7.0% | 0.0% |
+| trace-po | ttft_p90_ms | tp4_ep1_mns128 | tp8_ep8_mns64 | no | 21.2% | 0.0% |
+| trace-po | e2e_mean_ms | tp4_ep1_mns64 | tp8_ep8_mns128 | no | 7.0% | 0.0% |
+| trace-po | e2e_p90_ms | tp4_ep1_mns128 | tp8_ep8_mns64 | no | 21.2% | 0.0% |

runs/frontier-split-rootcause-v0/frozen-inputs/q235-state-diag/state-diagnosis-exact.md

@@ -0,0 +1,54 @@
+# Qwen235 Fixed-PD state diagnosis
+
+**Verdict:** Exact real state composition flips Frontier to the correct TP4 winner; the ranking error is caused by closed-loop state/composition mismatch, not a conditional stage reversal.
+
+| Config | Real Running proxy mean | Sim decode batch mean | Proxy coverage |
+|---|---:|---:|---:|
+| tp4_ep1_mns64 | 4.667 | 13.537 | 100.0% |
+| tp8_ep8_mns64 | 12.238 | 18.938 | 100.0% |
+
+| Config | Real exact decode batch mean | Real token-weighted iteration ms | Matched simulator ms | Exact coverage |
+|---|---:|---:|---:|---:|
+| tp4_ep1_mns64 | 3.890 | 18.4710 | 34.2575 | 100.0% |
+| tp8_ep8_mns64 | 10.921 | 27.5887 | 45.1603 | 100.0% |
+
+## Frontier internal component contrast over its own executed composition
+
+Decode-token-weighted over both pure-decode and mixed prefill/decode steps.
+
+| Component | TP8 - TP4 (ms/decoded token step) |
+|---|---:|
+| moe_compute | -32.4544 |
+| total | -20.0663 |
+| attention | -7.4619 |
+| tp_dp_communication | -6.5711 |
+| ep_communication | +6.2354 |
+| moe_routing | -0.2862 |
+| dense_mlp_compute | -0.1231 |
+| pipeline_communication | +0.0000 |
+| runtime_overhead | +0.0000 |
+
+## Frontier component contrast at exact real token composition
+
+Positive means TP8 slower; negative means Frontier gives TP8 an advantage.
+
+| Component | TP8 - TP4 (ms/step) |
+|---|---:|
+| total | +10.9028 |
+| ep_communication | +6.5355 |
+| tp_dp_communication | -6.4718 |
+| attention | -2.1304 |
+| moe_compute | -1.9774 |
+| moe_routing | +0.1384 |
+| dense_mlp_compute | -0.0308 |
+| pipeline_communication | +0.0000 |
+| runtime_overhead | +0.0000 |
+
+## Interpretation boundary
+
+- Real observed TPOT contrast: +6.9500 ms/token.
+- Real exact-iteration contrast: +9.1177 ms/decode-token-weighted step.
+- vLLM 10-second Running is active requests, not per-iteration decode batch; context lengths and graph buckets are unavailable in frozen logs.
+- Frontier state outputs have no direct graph bucket/padding/launch-overhead field; graph effects remain folded into predictors.
+- For TP8 shared-domain sync, total is ledger stage_end-start (critical path); op categories are serialized work estimates and are not additive because overlap and lane-summed wait diagnostics are represented separately.
+- Component deltas identify where Frontier creates its own TP8 advantage; without real per-stage measurements they are not yet root-cause proof.

runs/frontier-split-rootcause-v0/frozen-inputs/q235-state-diag/state-diagnosis-optrace.md

@@ -0,0 +1,48 @@
+# Qwen235 Fixed-PD state diagnosis
+
+**Verdict:** State-composition mismatch is insufficient: after reweighting Frontier to measured real decode composition, it still predicts TP8 faster.
+
+| Config | Real Running proxy mean | Sim decode batch mean | Proxy coverage |
+|---|---:|---:|---:|
+| tp4_ep1_mns64 | 4.667 | 13.537 | 100.0% |
+| tp8_ep8_mns64 | 12.238 | 18.938 | 100.0% |
+
+## Frontier internal component contrast over its own executed composition
+
+Decode-token-weighted over both pure-decode and mixed prefill/decode steps.
+
+| Component | TP8 - TP4 (ms/decoded token step) |
+|---|---:|
+| moe_compute | -32.4544 |
+| total | -20.0663 |
+| attention | -7.4619 |
+| tp_dp_communication | -6.5711 |
+| ep_communication | +6.2354 |
+| moe_routing | -0.2862 |
+| dense_mlp_compute | -0.1231 |
+| pipeline_communication | +0.0000 |
+| runtime_overhead | +0.0000 |
+
+## Frontier internal component contrast at real Running proxy
+
+Positive means TP8 slower; negative means Frontier gives TP8 an advantage.
+
+| Component | TP8 - TP4 (ms/step) |
+|---|---:|
+| moe_compute | -16.0581 |
+| total | -8.5517 |
+| ep_communication | +6.8248 |
+| tp_dp_communication | -5.4010 |
+| moe_routing | +0.3109 |
+| attention | -0.2407 |
+| dense_mlp_compute | +0.0018 |
+| pipeline_communication | +0.0000 |
+| runtime_overhead | +0.0000 |
+
+## Interpretation boundary
+
+- Real observed TPOT contrast: +6.9500 ms/token.
+- vLLM 10-second Running is active requests, not per-iteration decode batch; context lengths and graph buckets are unavailable in frozen logs.
+- Frontier state outputs have no direct graph bucket/padding/launch-overhead field; graph effects remain folded into predictors.
+- For TP8 shared-domain sync, total is ledger stage_end-start (critical path); op categories are serialized work estimates and are not additive because overlap and lane-summed wait diagnostics are represented separately.
+- Component deltas identify where Frontier creates its own TP8 advantage; without real per-stage measurements they are not yet root-cause proof.

runs/frontier-split-rootcause-v0/frozen-inputs/q235-state-diag/state-diagnosis.md

@@ -0,0 +1,47 @@
+# Qwen235 Fixed-PD state diagnosis
+
+**Verdict:** Active-batch-count mismatch alone is insufficient: after exact reweighting to each config's real Running histogram, Frontier still predicts TP8 faster.
+
+| Config | Real Running proxy mean | Sim decode batch mean | Proxy coverage |
+|---|---:|---:|---:|
+| tp4_ep1_mns64 | 4.667 | 13.537 | 100.0% |
+| tp8_ep8_mns64 | 12.238 | 18.938 | 100.0% |
+
+## Frontier internal component contrast over its own executed composition
+
+Decode-token-weighted over both pure-decode and mixed prefill/decode steps.
+
+| Component | TP8 - TP4 (ms/decoded token step) |
+|---|---:|
+| total | -73.6733 |
+| moe_compute | -45.2710 |
+| attention | -13.6950 |
+| tp_dp_communication | -12.7244 |
+| moe_routing | -1.7759 |
+| dense_mlp_compute | -0.2890 |
+| ep_communication | +0.0821 |
+| pipeline_communication | +0.0000 |
+| runtime_overhead | +0.0000 |
+
+## Frontier internal component contrast at real Running proxy
+
+Positive means TP8 slower; negative means Frontier gives TP8 an advantage.
+
+| Component | TP8 - TP4 (ms/step) |
+|---|---:|
+| total | -46.0760 |
+| moe_compute | -26.3387 |
+| tp_dp_communication | -12.1532 |
+| attention | -6.2393 |
+| moe_routing | -1.2405 |
+| dense_mlp_compute | -0.1769 |
+| ep_communication | +0.0726 |
+| pipeline_communication | +0.0000 |
+| runtime_overhead | +0.0000 |
+
+## Interpretation boundary
+
+- Real observed TPOT contrast: +6.9500 ms/token.
+- vLLM 10-second Running is active requests, not per-iteration decode batch; context lengths and graph buckets are unavailable in frozen logs.
+- Frontier ledger exposes additive execution components but no direct graph bucket/padding/launch-overhead field; graph effects remain folded into predictors.
+- Component deltas identify where Frontier creates its own TP8 advantage; without real per-stage measurements they are not yet root-cause proof.

runs/frontier-split-rootcause-v0/frozen-inputs/q30-admission-diag/result.json

@@ -0,0 +1,173 @@
+{
+  "boundary": "This identifies where the selection error is created, but the frozen real logs do not attribute service-time overprediction to one operator.",
+  "cells": {
+    "tp2_mns64": {
+      "arrival_rate_requests_per_s": 2.25,
+      "mns": 64,
+      "real": {
+        "e2e_ms": 3354.8816545874283,
+        "periodic_queue": {
+          "running_max": 10,
+          "running_mean": 7.257142857142857,
+          "samples": 35,
+          "waiting_max": 0,
+          "waiting_mean": 0.0
+        },
+        "required_slots_upper_bound": 7.548483722821714,
+        "slot_margin_lower_bound": 56.451516277178285,
+        "tpot_ms": 12.393283087182931,
+        "ttft_ms": 194.56193759030603
+      },
+      "simulator": {
+        "first_scheduling_delay_ms": 38.22329108338011,
+        "queue_free_ttft_ms": 191.02937373774816,
+        "request_execution_ms": 25761.91467886552,
+        "required_slots": 57.96430802744742,
+        "slot_margin": 6.035691972552577,
+        "tpot_ms": 100.27798158873513,
+        "ttft_ms": 229.2526648211283,
+        "ttft_queue_fraction": 0.16672997503956338
+      },
+      "tp": 2,
+      "tpot_overprediction_ratio": 8.091316956395687
+    },
+    "tp4_mns16": {
+      "arrival_rate_requests_per_s": 4.5,
+      "mns": 16,
+      "real": {
+        "e2e_ms": 3252.7993474094965,
+        "periodic_queue": {
+          "running_max": 16,
+          "running_mean": 15.166666666666666,
+          "samples": 18,
+          "waiting_max": 0,
+          "waiting_mean": 0.0
+        },
+        "required_slots_upper_bound": 14.637597063342735,
+        "slot_margin_lower_bound": 1.362402936657265,
+        "tpot_ms": 12.267737219892378,
+        "ttft_ms": 124.49453797538061
+      },
+      "simulator": {
+        "first_scheduling_delay_ms": 64906.25631809643,
+        "queue_free_ttft_ms": 134.74066995065758,
+        "request_execution_ms": 12108.627705569606,
+        "required_slots": 54.488824675063235,
+        "slot_margin": -38.488824675063235,
+        "tpot_ms": 46.95641974752248,
+        "ttft_ms": 65040.99698804708,
+        "ttft_queue_fraction": 0.9979283732385681
+      },
+      "tp": 4,
+      "tpot_overprediction_ratio": 3.8276349505907024
+    },
+    "tp4_mns32": {
+      "arrival_rate_requests_per_s": 4.5,
+      "mns": 32,
+      "real": {
+        "e2e_ms": 3227.015463910283,
+        "periodic_queue": {
+          "running_max": 16,
+          "running_mean": 14.944444444444445,
+          "samples": 18,
+          "waiting_max": 0,
+          "waiting_mean": 0.0
+        },
+        "required_slots_upper_bound": 14.521569587596273,
+        "slot_margin_lower_bound": 17.478430412403725,
+        "tpot_ms": 12.171863019540417,
+        "ttft_ms": 123.15859724408143
+      },
+      "simulator": {
+        "first_scheduling_delay_ms": 27180.676177927482,
+        "queue_free_ttft_ms": 135.62147333612225,
+        "request_execution_ms": 14643.080360640608,
+        "required_slots": 65.89386162288272,
+        "slot_margin": -33.893861622882724,
+        "tpot_ms": 56.89199563649036,
+        "ttft_ms": 27316.2976512636,
+        "ttft_queue_fraction": 0.9950351444010626
+      },
+      "tp": 4,
+      "tpot_overprediction_ratio": 4.6740581573385525
+    },
+    "tp4_mns64": {
+      "arrival_rate_requests_per_s": 4.5,
+      "mns": 64,
+      "real": {
+        "e2e_ms": 3258.202659738964,
+        "periodic_queue": {
+          "running_max": 17,
+          "running_mean": 14.833333333333334,
+          "samples": 18,
+          "waiting_max": 0,
+          "waiting_mean": 0.0
+        },
+        "required_slots_upper_bound": 14.661911968825338,
+        "slot_margin_lower_bound": 49.338088031174664,
+        "tpot_ms": 12.29409478300949,
+        "ttft_ms": 123.17715882596677
+      },
+      "simulator": {
+        "first_scheduling_delay_ms": 6367.273779218055,
+        "queue_free_ttft_ms": 136.7304324624105,
+        "request_execution_ms": 17736.957363112386,
+        "required_slots": 79.81630813400574,
+        "slot_margin": -15.816308134005737,
+        "tpot_ms": 69.02049776725552,
+        "ttft_ms": 6504.004211680465,
+        "ttft_queue_fraction": 0.9789774993969319
+      },
+      "tp": 4,
+      "tpot_overprediction_ratio": 5.614117914776633
+    }
+  },
+  "contrasts": {
+    "observed_sim_ttft_tp4_minus_tp2_ms": 27087.044986442474,
+    "queue_free_sim_ttft_tp4_minus_tp2_ms": -55.40790040162591,
+    "real_ttft_tp4_minus_tp2_ms": -71.4033403462246
+  },
+  "schema": "qwen30-fixed-pd-ttft-admission-diagnosis-v1",
+  "selection": {
+    "agreement": 0.5909090909090909,
+    "concordant_pairs": 39,
+    "discordant_pairs": 27,
+    "informative_pairs": 66,
+    "real_ranking": [
+      "tp4_mns32",
+      "tp4_mns64",
+      "tp4_mns16",
+      "tp2_mns64",
+      "tp2_mns8",
+      "tp2_mns32",
+      "tp2_mns16",
+      "tp1_mns8",
+      "tp1_mns32",
+      "tp1_mns16",
+      "tp1_mns64",
+      "tp4_mns8"
+    ],
+    "real_winner": "tp4_mns32",
+    "selected_config_real_regret": 0.5797674051508896,
+    "sim_ranking": [
+      "tp2_mns64",
+      "tp1_mns64",
+      "tp4_mns64",
+      "tp1_mns32",
+      "tp2_mns32",
+      "tp4_mns32",
+      "tp4_mns16",
+      "tp2_mns16",
+      "tp1_mns16",
+      "tp4_mns8",
+      "tp2_mns8",
+      "tp1_mns8"
+    ],
+    "sim_winner": "tp2_mns64",
+    "tied_pairs": 0,
+    "winner_match": false
+  },
+  "status": "PASS",
+  "verdict": "Frontier overpredicts decode service time, so TP4's higher global arrival rate crosses the MNS admission cap only in simulation. The resulting first-scheduling queue reverses the TP topology ranking; without that queue Frontier itself predicts TP4 faster than TP2, matching real hardware.",
+  "workload": "Qwen3-30B Fixed-PD 4096->256 at 1.125 req/s/GPU"
+}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-admission-diag/result.md

@@ -0,0 +1,18 @@
+# Qwen30 Fixed-PD TTFT admission diagnosis
+
+**Verdict:** Frontier overpredicts decode service time, so TP4's higher global arrival rate crosses the MNS admission cap only in simulation. The resulting first-scheduling queue reverses the TP topology ranking; without that queue Frontier itself predicts TP4 faster than TP2, matching real hardware.
+
+| Config | Sim TPOT / real | Sim required slots / MNS | Real slots upper bound / MNS | Sim first-schedule wait | Real waiting max |
+|---|---:|---:|---:|---:|---:|
+| tp2_mns64 | 8.09x | 58.0/64 | 7.5/64 | 38.2 ms | 0 |
+| tp4_mns16 | 3.83x | 54.5/16 | 14.6/16 | 64906.3 ms | 0 |
+| tp4_mns32 | 4.67x | 65.9/32 | 14.5/32 | 27180.7 ms | 0 |
+| tp4_mns64 | 5.61x | 79.8/64 | 14.7/64 | 6367.3 ms | 0 |
+
+## TP4/MNS32 minus TP2/MNS64 TTFT
+
+- Observed Frontier: +27087.0 ms.
+- Frontier without first-scheduling queue: -55.4 ms.
+- Real hardware: -71.4 ms.
+
+**Boundary:** This identifies where the selection error is created, but the frozen real logs do not attribute service-time overprediction to one operator.

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/analysis-launch.log

@@ -0,0 +1,2 @@
+Q30_ANALYSIS_ECHO source_real=/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6 source_sim=/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim cases=trace-po,fixed-pd,fixed-po requests=129 configs=12 trials=3 metrics=mean,p90(TTFT,TPOT-if-OSL-gt-1,E2E) gpu_cost=0 output=/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/analysis-r6
+Q30_ANALYSIS_COMPLETE

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-pd-comparison.json

@@ -0,0 +1,453 @@
+{
+  "applicable_metrics": [
+    "ttft_ms",
+    "tpot_ms",
+    "e2e_ms"
+  ],
+  "cells": {
+    "tp1_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 2624.2011811364387,
+          "p90": 2624.2011811379484
+        },
+        "tpot_ms": {
+          "mean": 19.350639886513438,
+          "p90": 19.3506398865253
+        },
+        "ttft_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp1_mns16/tp1/result.json"
+    },
+    "tp1_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 2624.2011811364387,
+          "p90": 2624.2011811379484
+        },
+        "tpot_ms": {
+          "mean": 19.350639886513438,
+          "p90": 19.3506398865253
+        },
+        "ttft_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp1_mns32/tp1/result.json"
+    },
+    "tp1_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 2624.2011811364387,
+          "p90": 2624.2011811379484
+        },
+        "tpot_ms": {
+          "mean": 19.350639886513438,
+          "p90": 19.3506398865253
+        },
+        "ttft_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp1_mns64/tp1/result.json"
+    },
+    "tp1_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 2624.2011811364387,
+          "p90": 2624.2011811379484
+        },
+        "tpot_ms": {
+          "mean": 19.350639886513438,
+          "p90": 19.3506398865253
+        },
+        "ttft_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp1_mns8/tp1/result.json"
+    },
+    "tp2_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 2808.3844446026824,
+          "p90": 2817.9486849812747
+        },
+        "tpot_ms": {
+          "mean": 21.06177881019329,
+          "p90": 21.09350467849898
+        },
+        "ttft_ms": {
+          "mean": 133.53853570813533,
+          "p90": 142.1523942165379
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp2_mns16/tp2/result.json"
+    },
+    "tp2_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 2808.3844446026824,
+          "p90": 2817.9486849812747
+        },
+        "tpot_ms": {
+          "mean": 21.06177881019329,
+          "p90": 21.09350467849898
+        },
+        "ttft_ms": {
+          "mean": 133.53853570813533,
+          "p90": 142.1523942165379
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp2_mns32/tp2/result.json"
+    },
+    "tp2_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 2808.3844446026824,
+          "p90": 2817.9486849812747
+        },
+        "tpot_ms": {
+          "mean": 21.06177881019329,
+          "p90": 21.09350467849898
+        },
+        "ttft_ms": {
+          "mean": 133.53853570813533,
+          "p90": 142.1523942165379
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp2_mns64/tp2/result.json"
+    },
+    "tp2_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 2808.3844446026824,
+          "p90": 2817.9486849812747
+        },
+        "tpot_ms": {
+          "mean": 21.06177881019329,
+          "p90": 21.09350467849898
+        },
+        "ttft_ms": {
+          "mean": 133.53853570813533,
+          "p90": 142.1523942165379
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp2_mns8/tp2/result.json"
+    },
+    "tp4_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 3752.684495467103,
+          "p90": 3794.6349827560653
+        },
+        "tpot_ms": {
+          "mean": 28.70502970984996,
+          "p90": 28.949734665706984
+        },
+        "ttft_ms": {
+          "mean": 107.14572231615826,
+          "p90": 118.03470639992497
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp4_mns16/tp4/result.json"
+    },
+    "tp4_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 3752.684495467103,
+          "p90": 3794.6349827560653
+        },
+        "tpot_ms": {
+          "mean": 28.70502970984996,
+          "p90": 28.949734665706984
+        },
+        "ttft_ms": {
+          "mean": 107.14572231615826,
+          "p90": 118.03470639992497
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp4_mns32/tp4/result.json"
+    },
+    "tp4_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 3752.684495467103,
+          "p90": 3794.6349827560653
+        },
+        "tpot_ms": {
+          "mean": 28.70502970984996,
+          "p90": 28.949734665706984
+        },
+        "ttft_ms": {
+          "mean": 107.14572231615826,
+          "p90": 118.03470639992497
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp4_mns64/tp4/result.json"
+    },
+    "tp4_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 3752.684495467103,
+          "p90": 3794.6349827560653
+        },
+        "tpot_ms": {
+          "mean": 28.70502970984996,
+          "p90": 28.949734665706984
+        },
+        "ttft_ms": {
+          "mean": 107.14572231615826,
+          "p90": 118.03470639992497
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd/runs/tp4_mns8/tp4/result.json"
+    }
+  },
+  "prefill_only": false,
+  "real_audit": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/analysis-r6/fixed-pd-real-audit.json",
+  "schema": "qwen30-latency-case-frontier-real-comparison-v1",
+  "selection": {
+    "e2e_ms:mean": {
+      "agreement": 0.0,
+      "concordant_pairs": 0,
+      "discordant_pairs": 48,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns32",
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns16",
+        "tp2_mns16",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp1_mns64",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns8"
+      ],
+      "real_winner": "tp4_mns32",
+      "selected_config_real_regret": 0.26921939465653383,
+      "sim_ranking": [
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8"
+      ],
+      "sim_winner": "tp1_mns16",
+      "tied_pairs": 18,
+      "winner_match": false
+    },
+    "e2e_ms:p90": {
+      "agreement": 0.0,
+      "concordant_pairs": 0,
+      "discordant_pairs": 48,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns16",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp2_mns64",
+        "tp1_mns64",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns8"
+      ],
+      "real_winner": "tp4_mns8",
+      "selected_config_real_regret": 0.25730483004000526,
+      "sim_ranking": [
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8"
+      ],
+      "sim_winner": "tp1_mns16",
+      "tied_pairs": 18,
+      "winner_match": false
+    },
+    "tpot_ms:mean": {
+      "agreement": 0.0,
+      "concordant_pairs": 0,
+      "discordant_pairs": 48,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns32",
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns16",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns8",
+        "tp2_mns64",
+        "tp1_mns64",
+        "tp1_mns32",
+        "tp1_mns16",
+        "tp1_mns8"
+      ],
+      "real_winner": "tp4_mns32",
+      "selected_config_real_regret": 0.13962124906187545,
+      "sim_ranking": [
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8"
+      ],
+      "sim_winner": "tp1_mns16",
+      "tied_pairs": 18,
+      "winner_match": false
+    },
+    "tpot_ms:p90": {
+      "agreement": 0.0,
+      "concordant_pairs": 0,
+      "discordant_pairs": 48,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns8",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns16",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp2_mns8",
+        "tp2_mns64",
+        "tp1_mns16",
+        "tp1_mns64",
+        "tp1_mns32",
+        "tp1_mns8"
+      ],
+      "real_winner": "tp4_mns8",
+      "selected_config_real_regret": 0.13357103696654363,
+      "sim_ranking": [
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8"
+      ],
+      "sim_winner": "tp1_mns16",
+      "tied_pairs": 18,
+      "winner_match": false
+    },
+    "ttft_ms:mean": {
+      "agreement": 1.0,
+      "concordant_pairs": 48,
+      "discordant_pairs": 0,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns16",
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns8",
+        "tp2_mns8",
+        "tp2_mns64",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns8",
+        "tp1_mns64"
+      ],
+      "real_winner": "tp4_mns16",
+      "selected_config_real_regret": 0.0,
+      "sim_ranking": [
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8"
+      ],
+      "sim_winner": "tp4_mns16",
+      "tied_pairs": 18,
+      "winner_match": true
+    },
+    "ttft_ms:p90": {
+      "agreement": 1.0,
+      "concordant_pairs": 48,
+      "discordant_pairs": 0,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns16",
+        "tp4_mns64",
+        "tp4_mns8",
+        "tp4_mns32",
+        "tp2_mns8",
+        "tp2_mns64",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns32",
+        "tp1_mns8",
+        "tp1_mns64",
+        "tp1_mns16"
+      ],
+      "real_winner": "tp4_mns16",
+      "selected_config_real_regret": 0.0,
+      "sim_ranking": [
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8"
+      ],
+      "sim_winner": "tp4_mns16",
+      "tied_pairs": 18,
+      "winner_match": true
+    }
+  },
+  "sim_root": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-pd"
+}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-pd-comparison.md

@@ -0,0 +1,10 @@
+# Qwen30 Frontier vs real latency selection
+
+| Objective | Frontier | Real | Match | Regret | Pairwise |
+|---|---|---|---:|---:|---:|
+| ttft_ms:mean | tp4_mns16 | tp4_mns16 | yes | 0.0% | 100.0% |
+| ttft_ms:p90 | tp4_mns16 | tp4_mns16 | yes | 0.0% | 100.0% |
+| tpot_ms:mean | tp1_mns16 | tp4_mns32 | no | 14.0% | 0.0% |
+| tpot_ms:p90 | tp1_mns16 | tp4_mns8 | no | 13.4% | 0.0% |
+| e2e_ms:mean | tp1_mns16 | tp4_mns32 | no | 26.9% | 0.0% |
+| e2e_ms:p90 | tp1_mns16 | tp4_mns8 | no | 25.7% | 0.0% |

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-pd-comparison.stdout.json

@@ -0,0 +1 @@
+{"e2e_ms:mean": {"agreement": 0.0, "concordant_pairs": 0, "discordant_pairs": 48, "informative_pairs": 48, "real_ranking": ["tp4_mns32", "tp4_mns8", "tp4_mns64", "tp4_mns16", "tp2_mns16", "tp2_mns8", "tp2_mns32", "tp2_mns64", "tp1_mns64", "tp1_mns16", "tp1_mns32", "tp1_mns8"], "real_winner": "tp4_mns32", "selected_config_real_regret": 0.26921939465653383, "sim_ranking": ["tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8"], "sim_winner": "tp1_mns16", "tied_pairs": 18, "winner_match": false}, "e2e_ms:p90": {"agreement": 0.0, "concordant_pairs": 0, "discordant_pairs": 48, "informative_pairs": 48, "real_ranking": ["tp4_mns8", "tp4_mns64", "tp4_mns32", "tp4_mns16", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp2_mns64", "tp1_mns64", "tp1_mns16", "tp1_mns32", "tp1_mns8"], "real_winner": "tp4_mns8", "selected_config_real_regret": 0.25730483004000526, "sim_ranking": ["tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8"], "sim_winner": "tp1_mns16", "tied_pairs": 18, "winner_match": false}, "tpot_ms:mean": {"agreement": 0.0, "concordant_pairs": 0, "discordant_pairs": 48, "informative_pairs": 48, "real_ranking": ["tp4_mns32", "tp4_mns8", "tp4_mns64", "tp4_mns16", "tp2_mns16", "tp2_mns32", "tp2_mns8", "tp2_mns64", "tp1_mns64", "tp1_mns32", "tp1_mns16", "tp1_mns8"], "real_winner": "tp4_mns32", "selected_config_real_regret": 0.13962124906187545, "sim_ranking": ["tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8"], "sim_winner": "tp1_mns16", "tied_pairs": 18, "winner_match": false}, "tpot_ms:p90": {"agreement": 0.0, "concordant_pairs": 0, "discordant_pairs": 48, "informative_pairs": 48, "real_ranking": ["tp4_mns8", "tp4_mns32", "tp4_mns64", "tp4_mns16", "tp2_mns32", "tp2_mns16", "tp2_mns8", "tp2_mns64", "tp1_mns16", "tp1_mns64", "tp1_mns32", "tp1_mns8"], "real_winner": "tp4_mns8", "selected_config_real_regret": 0.13357103696654363, "sim_ranking": ["tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8"], "sim_winner": "tp1_mns16", "tied_pairs": 18, "winner_match": false}, "ttft_ms:mean": {"agreement": 1.0, "concordant_pairs": 48, "discordant_pairs": 0, "informative_pairs": 48, "real_ranking": ["tp4_mns16", "tp4_mns64", "tp4_mns32", "tp4_mns8", "tp2_mns8", "tp2_mns64", "tp2_mns32", "tp2_mns16", "tp1_mns16", "tp1_mns32", "tp1_mns8", "tp1_mns64"], "real_winner": "tp4_mns16", "selected_config_real_regret": 0.0, "sim_ranking": ["tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8"], "sim_winner": "tp4_mns16", "tied_pairs": 18, "winner_match": true}, "ttft_ms:p90": {"agreement": 1.0, "concordant_pairs": 48, "discordant_pairs": 0, "informative_pairs": 48, "real_ranking": ["tp4_mns16", "tp4_mns64", "tp4_mns8", "tp4_mns32", "tp2_mns8", "tp2_mns64", "tp2_mns32", "tp2_mns16", "tp1_mns32", "tp1_mns8", "tp1_mns64", "tp1_mns16"], "real_winner": "tp4_mns16", "selected_config_real_regret": 0.0, "sim_ranking": ["tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8"], "sim_winner": "tp4_mns16", "tied_pairs": 18, "winner_match": true}}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-pd-real-audit.md

@@ -0,0 +1,12 @@
+# Qwen30 real latency case audit
+
+Prefill-only: `False`.
+
+| Objective | Real winner | Value (ms) |
+|---|---|---:|
+| ttft_ms:pooled_mean_ms | tp4_mns16 | 63.04 |
+| ttft_ms:pooled_p90_ms | tp4_mns16 | 66.42 |
+| tpot_ms:pooled_mean_ms | tp4_mns32 | 4.25 |
+| tpot_ms:pooled_p90_ms | tp4_mns8 | 4.27 |
+| e2e_ms:pooled_mean_ms | tp4_mns32 | 604.41 |
+| e2e_ms:pooled_p90_ms | tp4_mns8 | 610.01 |

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-pd-real-audit.stdout.json

@@ -0,0 +1 @@
+{"e2e_ms:pooled_mean_ms": {"ranking": ["tp4_mns32", "tp4_mns8", "tp4_mns64", "tp4_mns16", "tp2_mns16", "tp2_mns8", "tp2_mns32", "tp2_mns64", "tp1_mns64", "tp1_mns16", "tp1_mns32", "tp1_mns8"], "winner": "tp4_mns32", "winner_value_ms": 604.4058500071964}, "e2e_ms:pooled_p90_ms": {"ranking": ["tp4_mns8", "tp4_mns64", "tp4_mns32", "tp4_mns16", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp2_mns64", "tp1_mns64", "tp1_mns16", "tp1_mns32", "tp1_mns8"], "winner": "tp4_mns8", "winner_value_ms": 610.0081680342555}, "tpot_ms:pooled_mean_ms": {"ranking": ["tp4_mns32", "tp4_mns8", "tp4_mns64", "tp4_mns16", "tp2_mns16", "tp2_mns32", "tp2_mns8", "tp2_mns64", "tp1_mns64", "tp1_mns32", "tp1_mns16", "tp1_mns8"], "winner": "tp4_mns32", "winner_value_ms": 4.254808547439789}, "tpot_ms:pooled_p90_ms": {"ranking": ["tp4_mns8", "tp4_mns32", "tp4_mns64", "tp4_mns16", "tp2_mns32", "tp2_mns16", "tp2_mns8", "tp2_mns64", "tp1_mns16", "tp1_mns64", "tp1_mns32", "tp1_mns8"], "winner": "tp4_mns8", "winner_value_ms": 4.268226101921182}, "ttft_ms:pooled_mean_ms": {"ranking": ["tp4_mns16", "tp4_mns64", "tp4_mns32", "tp4_mns8", "tp2_mns8", "tp2_mns64", "tp2_mns32", "tp2_mns16", "tp1_mns16", "tp1_mns32", "tp1_mns8", "tp1_mns64"], "winner": "tp4_mns16", "winner_value_ms": 63.03574686938565}, "ttft_ms:pooled_p90_ms": {"ranking": ["tp4_mns16", "tp4_mns64", "tp4_mns8", "tp4_mns32", "tp2_mns8", "tp2_mns64", "tp2_mns32", "tp2_mns16", "tp1_mns32", "tp1_mns8", "tp1_mns64", "tp1_mns16"], "winner": "tp4_mns16", "winner_value_ms": 66.4233659626916}}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-po-comparison.json

@@ -0,0 +1,326 @@
+{
+  "applicable_metrics": [
+    "ttft_ms",
+    "e2e_ms"
+  ],
+  "cells": {
+    "tp1_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        },
+        "ttft_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp1_mns16/tp1/result.json"
+    },
+    "tp1_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        },
+        "ttft_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp1_mns32/tp1/result.json"
+    },
+    "tp1_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        },
+        "ttft_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp1_mns64/tp1/result.json"
+    },
+    "tp1_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        },
+        "ttft_ms": {
+          "mean": 166.66991554923203,
+          "p90": 166.6699155492779
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp1_mns8/tp1/result.json"
+    },
+    "tp2_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 118.55592059231998,
+          "p90": 118.55592059232833
+        },
+        "ttft_ms": {
+          "mean": 118.55592059231998,
+          "p90": 118.55592059232833
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp2_mns16/tp2/result.json"
+    },
+    "tp2_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 118.55592059231998,
+          "p90": 118.55592059232833
+        },
+        "ttft_ms": {
+          "mean": 118.55592059231998,
+          "p90": 118.55592059232833
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp2_mns32/tp2/result.json"
+    },
+    "tp2_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 118.55592059231998,
+          "p90": 118.55592059232833
+        },
+        "ttft_ms": {
+          "mean": 118.55592059231998,
+          "p90": 118.55592059232833
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp2_mns64/tp2/result.json"
+    },
+    "tp2_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 118.55592059231998,
+          "p90": 118.55592059232833
+        },
+        "ttft_ms": {
+          "mean": 118.55592059231998,
+          "p90": 118.55592059232833
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp2_mns8/tp2/result.json"
+    },
+    "tp4_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 86.52545838364681,
+          "p90": 86.52545838364745
+        },
+        "ttft_ms": {
+          "mean": 86.52545838364681,
+          "p90": 86.52545838364745
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp4_mns16/tp4/result.json"
+    },
+    "tp4_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 86.52545838364681,
+          "p90": 86.52545838364745
+        },
+        "ttft_ms": {
+          "mean": 86.52545838364681,
+          "p90": 86.52545838364745
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp4_mns32/tp4/result.json"
+    },
+    "tp4_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 86.52545838364681,
+          "p90": 86.52545838364745
+        },
+        "ttft_ms": {
+          "mean": 86.52545838364681,
+          "p90": 86.52545838364745
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp4_mns64/tp4/result.json"
+    },
+    "tp4_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 86.52545838364681,
+          "p90": 86.52545838364745
+        },
+        "ttft_ms": {
+          "mean": 86.52545838364681,
+          "p90": 86.52545838364745
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po/runs/tp4_mns8/tp4/result.json"
+    }
+  },
+  "prefill_only": true,
+  "real_audit": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/analysis-r6/fixed-po-real-audit.json",
+  "schema": "qwen30-latency-case-frontier-real-comparison-v1",
+  "selection": {
+    "e2e_ms:mean": {
+      "agreement": 1.0,
+      "concordant_pairs": 48,
+      "discordant_pairs": 0,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns8",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns32",
+        "tp1_mns16",
+        "tp1_mns8",
+        "tp1_mns64"
+      ],
+      "real_winner": "tp4_mns8",
+      "selected_config_real_regret": 0.014895639803871355,
+      "sim_ranking": [
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8"
+      ],
+      "sim_winner": "tp4_mns16",
+      "tied_pairs": 18,
+      "winner_match": false
+    },
+    "e2e_ms:p90": {
+      "agreement": 1.0,
+      "concordant_pairs": 48,
+      "discordant_pairs": 0,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns8",
+        "tp1_mns64"
+      ],
+      "real_winner": "tp4_mns8",
+      "selected_config_real_regret": 0.03590282591969048,
+      "sim_ranking": [
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8"
+      ],
+      "sim_winner": "tp4_mns16",
+      "tied_pairs": 18,
+      "winner_match": false
+    },
+    "ttft_ms:mean": {
+      "agreement": 1.0,
+      "concordant_pairs": 48,
+      "discordant_pairs": 0,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns8",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns32",
+        "tp1_mns16",
+        "tp1_mns8",
+        "tp1_mns64"
+      ],
+      "real_winner": "tp4_mns8",
+      "selected_config_real_regret": 0.014940997472395743,
+      "sim_ranking": [
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8"
+      ],
+      "sim_winner": "tp4_mns16",
+      "tied_pairs": 18,
+      "winner_match": false
+    },
+    "ttft_ms:p90": {
+      "agreement": 1.0,
+      "concordant_pairs": 48,
+      "discordant_pairs": 0,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns8",
+        "tp1_mns64"
+      ],
+      "real_winner": "tp4_mns8",
+      "selected_config_real_regret": 0.035908453871728634,
+      "sim_ranking": [
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8"
+      ],
+      "sim_winner": "tp4_mns16",
+      "tied_pairs": 18,
+      "winner_match": false
+    }
+  },
+  "sim_root": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/fixed-po"
+}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-po-comparison.md

@@ -0,0 +1,8 @@
+# Qwen30 Frontier vs real latency selection
+
+| Objective | Frontier | Real | Match | Regret | Pairwise |
+|---|---|---|---:|---:|---:|
+| ttft_ms:mean | tp4_mns16 | tp4_mns8 | no | 1.5% | 100.0% |
+| ttft_ms:p90 | tp4_mns16 | tp4_mns8 | no | 3.6% | 100.0% |
+| e2e_ms:mean | tp4_mns16 | tp4_mns8 | no | 1.5% | 100.0% |
+| e2e_ms:p90 | tp4_mns16 | tp4_mns8 | no | 3.6% | 100.0% |

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-po-comparison.stdout.json

@@ -0,0 +1 @@
+{"e2e_ms:mean": {"agreement": 1.0, "concordant_pairs": 48, "discordant_pairs": 0, "informative_pairs": 48, "real_ranking": ["tp4_mns8", "tp4_mns32", "tp4_mns64", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns32", "tp1_mns16", "tp1_mns8", "tp1_mns64"], "real_winner": "tp4_mns8", "selected_config_real_regret": 0.014895639803871355, "sim_ranking": ["tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8"], "sim_winner": "tp4_mns16", "tied_pairs": 18, "winner_match": false}, "e2e_ms:p90": {"agreement": 1.0, "concordant_pairs": 48, "discordant_pairs": 0, "informative_pairs": 48, "real_ranking": ["tp4_mns8", "tp4_mns64", "tp4_mns32", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns16", "tp1_mns32", "tp1_mns8", "tp1_mns64"], "real_winner": "tp4_mns8", "selected_config_real_regret": 0.03590282591969048, "sim_ranking": ["tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8"], "sim_winner": "tp4_mns16", "tied_pairs": 18, "winner_match": false}, "ttft_ms:mean": {"agreement": 1.0, "concordant_pairs": 48, "discordant_pairs": 0, "informative_pairs": 48, "real_ranking": ["tp4_mns8", "tp4_mns32", "tp4_mns64", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns32", "tp1_mns16", "tp1_mns8", "tp1_mns64"], "real_winner": "tp4_mns8", "selected_config_real_regret": 0.014940997472395743, "sim_ranking": ["tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8"], "sim_winner": "tp4_mns16", "tied_pairs": 18, "winner_match": false}, "ttft_ms:p90": {"agreement": 1.0, "concordant_pairs": 48, "discordant_pairs": 0, "informative_pairs": 48, "real_ranking": ["tp4_mns8", "tp4_mns64", "tp4_mns32", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns16", "tp1_mns32", "tp1_mns8", "tp1_mns64"], "real_winner": "tp4_mns8", "selected_config_real_regret": 0.035908453871728634, "sim_ranking": ["tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8"], "sim_winner": "tp4_mns16", "tied_pairs": 18, "winner_match": false}}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-po-real-audit.json

@@ -0,0 +1,997 @@
+{
+  "applicable_metrics": [
+    "ttft_ms",
+    "e2e_ms"
+  ],
+  "case_root": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po",
+  "configs": {
+    "tp1_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 151.54830888467202,
+          "pooled_p90_ms": 152.33826905023307,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 151.54830888467202,
+          "trial_stddev_of_means_ms": 0.33839071804993504
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 151.50490087229585,
+          "pooled_p90_ms": 152.29528304189444,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 151.50490087229585,
+          "trial_stddev_of_means_ms": 0.3392777547312034
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 151.25221894176806,
+              "p90_ms": 152.03110210131854,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 151.20715750724278,
+              "p90_ms": 151.9814640050754,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns16/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 151.4755454836702,
+              "p90_ms": 152.21454401034862,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 151.4332707151915,
+              "p90_ms": 152.16313605196774,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns16/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 151.9171622285778,
+              "p90_ms": 152.64088998083025,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 151.87427439445327,
+              "p90_ms": 152.5908849434927,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns16/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp1_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 151.4782552960514,
+          "pooled_p90_ms": 152.49883604701608,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 151.4782552960514,
+          "trial_stddev_of_means_ms": 0.5555788512092701
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 151.43368434935846,
+          "pooled_p90_ms": 152.45853806845844,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 151.43368434935846,
+          "trial_stddev_of_means_ms": 0.5560791927635007
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 151.00544339222762,
+              "p90_ms": 151.96295792702585,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 150.96285229624618,
+              "p90_ms": 151.92402398679405,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns32/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 151.33915553880178,
+              "p90_ms": 152.3007940268144,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 151.29099727893436,
+              "p90_ms": 152.25290204398334,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns32/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 152.09016695712484,
+              "p90_ms": 152.93943404685706,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 152.04720347289486,
+              "p90_ms": 152.89701998699456,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns32/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp1_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 152.62579285143815,
+          "pooled_p90_ms": 153.7184970220551,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 152.62579285143815,
+          "trial_stddev_of_means_ms": 0.5522776932799126
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 152.58183204256076,
+          "pooled_p90_ms": 153.6727879429236,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 152.58183204256076,
+          "trial_stddev_of_means_ms": 0.552880728503269
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 152.98813100463835,
+              "p90_ms": 153.7133390083909,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 152.9461515773189,
+              "p90_ms": 153.67183403577656,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns64/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 152.89909483832415,
+              "p90_ms": 154.20521004125476,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 152.85368830727253,
+              "p90_ms": 154.1550400434062,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns64/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 151.99015271135195,
+              "p90_ms": 152.67854696139693,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 151.94565624309084,
+              "p90_ms": 152.6353599037975,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns64/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp1_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 152.22744446955332,
+          "pooled_p90_ms": 152.97825809102505,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 152.22744446955332,
+          "trial_stddev_of_means_ms": 0.15844676271042335
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 152.18195707226792,
+          "pooled_p90_ms": 152.93638303410262,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 152.18195707226792,
+          "trial_stddev_of_means_ms": 0.1592296957231754
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 152.08507488055747,
+              "p90_ms": 152.94339996762574,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 152.04033442473315,
+              "p90_ms": 152.89624000433832,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns8/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 152.3981458541771,
+              "p90_ms": 153.33553205709904,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 152.3543132889597,
+              "p90_ms": 153.2915000570938,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns8/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 152.19911267392538,
+              "p90_ms": 152.6885419152677,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 152.15122350311094,
+              "p90_ms": 152.64206391293555,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp1_mns8/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp2_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 97.51495216234325,
+          "pooled_p90_ms": 98.90919097233564,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 97.51495216234325,
+          "trial_stddev_of_means_ms": 0.41813696851903276
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 97.46812211917545,
+          "pooled_p90_ms": 98.85558602400124,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 97.46812211917545,
+          "trial_stddev_of_means_ms": 0.41503784318372017
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 97.09843800785814,
+              "p90_ms": 98.22496003471315,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 97.05583205674017,
+              "p90_ms": 98.16953900735825,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns16/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 97.93469326682533,
+              "p90_ms": 99.07599200960249,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 97.88585422369255,
+              "p90_ms": 99.02978304307908,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns16/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 97.51172521234628,
+              "p90_ms": 99.18248606845737,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 97.46268007709364,
+              "p90_ms": 99.13825406692922,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns16/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp2_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 97.09498990484747,
+          "pooled_p90_ms": 98.50096399895847,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 97.09498990484747,
+          "trial_stddev_of_means_ms": 0.27051136106210416
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 97.04873380816761,
+          "pooled_p90_ms": 98.45811000559479,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 97.0487338081676,
+          "trial_stddev_of_means_ms": 0.27130252759105145
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 96.8018446240372,
+              "p90_ms": 97.90838800836354,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 96.75326022814035,
+              "p90_ms": 97.85841300617903,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns32/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 97.33497442735373,
+              "p90_ms": 98.68263697717339,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 97.28661816668256,
+              "p90_ms": 98.63975096959621,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns32/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 97.14815066315147,
+              "p90_ms": 98.70580595452338,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 97.1063230296799,
+              "p90_ms": 98.66256301756948,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns32/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp2_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 94.78371690426897,
+          "pooled_p90_ms": 95.95681494101882,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 94.78371690426899,
+          "trial_stddev_of_means_ms": 0.09464944657839716
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 94.73932527026128,
+          "pooled_p90_ms": 95.90691793709993,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 94.73932527026129,
+          "trial_stddev_of_means_ms": 0.09810418936504331
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 94.80111797740899,
+              "p90_ms": 95.66861705388874,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 94.75897805848297,
+              "p90_ms": 95.61934601515532,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns64/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 94.68157430154149,
+              "p90_ms": 95.95681494101882,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 94.63288232525305,
+              "p90_ms": 95.90691793709993,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns64/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 94.86845843385645,
+              "p90_ms": 96.28609393257648,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 94.82611542704782,
+              "p90_ms": 96.24539199285209,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns64/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp2_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 95.65880776170006,
+          "pooled_p90_ms": 97.55863400641829,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 95.65880776170006,
+          "trial_stddev_of_means_ms": 1.2232525642643362
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 95.61450884031319,
+          "pooled_p90_ms": 97.50729007646441,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 95.61450884031319,
+          "trial_stddev_of_means_ms": 1.2245345925842415
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 97.06745062893357,
+              "p90_ms": 99.17883900925517,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 97.02484949686846,
+              "p90_ms": 99.13210896775126,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns8/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 94.86425999920209,
+              "p90_ms": 96.06455289758742,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 94.82164932920259,
+              "p90_ms": 96.02120891213417,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns8/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 95.0447126569645,
+              "p90_ms": 97.03064302448183,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 94.99702769486855,
+              "p90_ms": 96.98293695691973,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp2_mns8/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp4_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 65.28745120113297,
+          "pooled_p90_ms": 70.42863499373198,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 65.28745120113297,
+          "trial_stddev_of_means_ms": 3.4421132551731515
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 65.24474184372936,
+          "pooled_p90_ms": 70.38743107113987,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 65.24474184372936,
+          "trial_stddev_of_means_ms": 3.440733102883723
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 64.19509824892405,
+              "p90_ms": 67.92093603871763,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 64.15208354917203,
+              "p90_ms": 67.87848006933928,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns16/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 69.1431922154039,
+              "p90_ms": 78.7114379927516,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 69.09912444791908,
+              "p90_ms": 78.66681204177439,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns16/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 62.524063139070954,
+              "p90_ms": 63.537837006151676,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 62.48301753409698,
+              "p90_ms": 63.49677708931267,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns16/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp4_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 64.54067908164875,
+          "pooled_p90_ms": 69.61812602821738,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 64.54067908164875,
+          "trial_stddev_of_means_ms": 0.8919869576553885
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 64.4944863910101,
+          "pooled_p90_ms": 69.50995896477252,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 64.49448639101008,
+          "trial_stddev_of_means_ms": 0.8942503909647054
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 65.55543364614046,
+              "p90_ms": 71.84744393453002,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 65.51214804161744,
+              "p90_ms": 71.80331903509796,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns32/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 63.88050908684846,
+              "p90_ms": 66.97378400713205,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 63.83413878312811,
+              "p90_ms": 66.91711395978928,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns32/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 64.18609451195731,
+              "p90_ms": 69.55521402414888,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 64.13717234828474,
+              "p90_ms": 69.50995896477252,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns32/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp4_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 65.03269931623949,
+          "pooled_p90_ms": 68.43663903418928,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 65.03269931623947,
+          "trial_stddev_of_means_ms": 1.1339679108088652
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 64.98623655348516,
+          "pooled_p90_ms": 68.38861701544374,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 64.98623655348516,
+          "trial_stddev_of_means_ms": 1.1341449568113504
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 66.32654335125991,
+              "p90_ms": 70.97568700555712,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 66.28015979965643,
+              "p90_ms": 70.93840499874204,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns64/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 64.56001544761104,
+              "p90_ms": 67.99875595606863,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 64.51422280213225,
+              "p90_ms": 67.94618198182434,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns64/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 64.21153914984748,
+              "p90_ms": 67.55471904762089,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 64.16432705866679,
+              "p90_ms": 67.51206307671964,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns64/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp4_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 64.32922621852013,
+          "pooled_p90_ms": 67.98768497537822,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 64.32922621852013,
+          "trial_stddev_of_means_ms": 1.8052761808674234
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 64.2842707174255,
+          "pooled_p90_ms": 67.94753996655345,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 64.2842707174255,
+          "trial_stddev_of_means_ms": 1.80289690665978
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 62.84650166739183,
+              "p90_ms": 63.89774801209569,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 62.80200301792271,
+              "p90_ms": 63.856702065095305,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns8/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 66.3395124548945,
+              "p90_ms": 70.76063798740506,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 66.29134260132447,
+              "p90_ms": 70.7101090811193,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns8/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 63.801664533274064,
+              "p90_ms": 67.48319591861218,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 63.759466533029325,
+              "p90_ms": 67.4400629941374,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/fixed-po/real/tp4_mns8/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    }
+  },
+  "prefill_only": true,
+  "schema": "qwen30-latency-case-real-audit-v1",
+  "trace_manifests": {
+    "tp1": {
+      "description": "fixed-shape: deterministic token IDs, uniform TP-normalized QPS, no prefix reuse",
+      "first_arrival_s": 0.0,
+      "global_offered_request_rate": 0.215,
+      "last_arrival_s": 595.3488372093024,
+      "output_tokens": [
+        1
+      ],
+      "per_gpu_offered_request_rate": 0.215,
+      "prefix_caching": false,
+      "private_jsonl": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/fixed-po/tp1/private/real_requests.jsonl",
+      "private_jsonl_sha256": "ce7582fc13050078a243d7f6adfc2b2e071dbe87b5f760c7bf111af57718ccfb",
+      "public_csv": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/fixed-po/tp1/public/frontier.csv",
+      "public_csv_sha256": "e6f1ca0354e9d93c60ff18261ae6e08e2e54dbc888e30e244776f9cc8695a8cf",
+      "requests": 129,
+      "row_vector_sha256": "7c8dc7cccd2bd874768c1b06ca40d2327452390577b9907bacb76270b46998fa",
+      "schema": "qwen30-latency-case-v1",
+      "tensor_parallel_size": 1
+    },
+    "tp2": {
+      "description": "fixed-shape: deterministic token IDs, uniform TP-normalized QPS, no prefix reuse",
+      "first_arrival_s": 0.0,
+      "global_offered_request_rate": 0.43,
+      "last_arrival_s": 297.6744186046512,
+      "output_tokens": [
+        1
+      ],
+      "per_gpu_offered_request_rate": 0.215,
+      "prefix_caching": false,
+      "private_jsonl": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/fixed-po/tp2/private/real_requests.jsonl",
+      "private_jsonl_sha256": "3fe3c01ac31cd776b2458cafb818d889b20c5e14948154bbd716cce7200c8d5b",
+      "public_csv": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/fixed-po/tp2/public/frontier.csv",
+      "public_csv_sha256": "78b900d8f47fec3a1c83b748f82cb9623ce3f0cf08265aa20c3ef1511a4a97e0",
+      "requests": 129,
+      "row_vector_sha256": "c9382d9007ea1dc572898836f2d0efb8e2244c4febc1ed59d5aeacfcc5475c0a",
+      "schema": "qwen30-latency-case-v1",
+      "tensor_parallel_size": 2
+    },
+    "tp4": {
+      "description": "fixed-shape: deterministic token IDs, uniform TP-normalized QPS, no prefix reuse",
+      "first_arrival_s": 0.0,
+      "global_offered_request_rate": 0.86,
+      "last_arrival_s": 148.8372093023256,
+      "output_tokens": [
+        1
+      ],
+      "per_gpu_offered_request_rate": 0.215,
+      "prefix_caching": false,
+      "private_jsonl": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/fixed-po/tp4/private/real_requests.jsonl",
+      "private_jsonl_sha256": "af3ba5a7888ea5f283bb360dd36f1f455906f69f5ad82cef84e82018545710db",
+      "public_csv": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/fixed-po/tp4/public/frontier.csv",
+      "public_csv_sha256": "8b1100ccab34d9dac62b7ff1508d36bdf75aa3c3135e6f6642a0d3ac77af2505",
+      "requests": 129,
+      "row_vector_sha256": "a5e4959f28790eaa99a59a1b86c449a4e484967f1e35e47900d1a27fa061607f",
+      "schema": "qwen30-latency-case-v1",
+      "tensor_parallel_size": 4
+    }
+  },
+  "traces_root": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/fixed-po",
+  "winners": {
+    "e2e_ms:pooled_mean_ms": {
+      "ranking": [
+        "tp4_mns8",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns32",
+        "tp1_mns16",
+        "tp1_mns8",
+        "tp1_mns64"
+      ],
+      "winner": "tp4_mns8",
+      "winner_value_ms": 64.32922621852013
+    },
+    "e2e_ms:pooled_p90_ms": {
+      "ranking": [
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns8",
+        "tp1_mns64"
+      ],
+      "winner": "tp4_mns8",
+      "winner_value_ms": 67.98768497537822
+    },
+    "ttft_ms:pooled_mean_ms": {
+      "ranking": [
+        "tp4_mns8",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns32",
+        "tp1_mns16",
+        "tp1_mns8",
+        "tp1_mns64"
+      ],
+      "winner": "tp4_mns8",
+      "winner_value_ms": 64.2842707174255
+    },
+    "ttft_ms:pooled_p90_ms": {
+      "ranking": [
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns8",
+        "tp1_mns64"
+      ],
+      "winner": "tp4_mns8",
+      "winner_value_ms": 67.94753996655345
+    }
+  }
+}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-po-real-audit.md

@@ -0,0 +1,10 @@
+# Qwen30 real latency case audit
+
+Prefill-only: `True`.
+
+| Objective | Real winner | Value (ms) |
+|---|---|---:|
+| ttft_ms:pooled_mean_ms | tp4_mns8 | 64.28 |
+| ttft_ms:pooled_p90_ms | tp4_mns8 | 67.95 |
+| e2e_ms:pooled_mean_ms | tp4_mns8 | 64.33 |
+| e2e_ms:pooled_p90_ms | tp4_mns8 | 67.99 |

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/fixed-po-real-audit.stdout.json

@@ -0,0 +1 @@
+{"e2e_ms:pooled_mean_ms": {"ranking": ["tp4_mns8", "tp4_mns32", "tp4_mns64", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns32", "tp1_mns16", "tp1_mns8", "tp1_mns64"], "winner": "tp4_mns8", "winner_value_ms": 64.32922621852013}, "e2e_ms:pooled_p90_ms": {"ranking": ["tp4_mns8", "tp4_mns64", "tp4_mns32", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns16", "tp1_mns32", "tp1_mns8", "tp1_mns64"], "winner": "tp4_mns8", "winner_value_ms": 67.98768497537822}, "ttft_ms:pooled_mean_ms": {"ranking": ["tp4_mns8", "tp4_mns32", "tp4_mns64", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns32", "tp1_mns16", "tp1_mns8", "tp1_mns64"], "winner": "tp4_mns8", "winner_value_ms": 64.2842707174255}, "ttft_ms:pooled_p90_ms": {"ranking": ["tp4_mns8", "tp4_mns64", "tp4_mns32", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns16", "tp1_mns32", "tp1_mns8", "tp1_mns64"], "winner": "tp4_mns8", "winner_value_ms": 67.94753996655345}}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/trace-po-comparison.json

@@ -0,0 +1,326 @@
+{
+  "applicable_metrics": [
+    "ttft_ms",
+    "e2e_ms"
+  ],
+  "cells": {
+    "tp1_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 340.75522510609915,
+          "p90": 832.752247743656
+        },
+        "ttft_ms": {
+          "mean": 340.75522510609915,
+          "p90": 832.752247743656
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp1_mns16/tp1/result.json"
+    },
+    "tp1_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 340.75598360435777,
+          "p90": 832.752247743656
+        },
+        "ttft_ms": {
+          "mean": 340.75598360435777,
+          "p90": 832.752247743656
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp1_mns32/tp1/result.json"
+    },
+    "tp1_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 340.7569941487134,
+          "p90": 832.752247743656
+        },
+        "ttft_ms": {
+          "mean": 340.7569941487134,
+          "p90": 832.752247743656
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp1_mns64/tp1/result.json"
+    },
+    "tp1_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 340.75522510609915,
+          "p90": 832.752247743656
+        },
+        "ttft_ms": {
+          "mean": 340.75522510609915,
+          "p90": 832.752247743656
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp1_mns8/tp1/result.json"
+    },
+    "tp2_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 222.73961839402438,
+          "p90": 513.3421414172687
+        },
+        "ttft_ms": {
+          "mean": 222.73961839402438,
+          "p90": 513.3421414172687
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp2_mns16/tp2/result.json"
+    },
+    "tp2_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 222.7393083124536,
+          "p90": 513.3421414172687
+        },
+        "ttft_ms": {
+          "mean": 222.7393083124536,
+          "p90": 513.3421414172687
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp2_mns32/tp2/result.json"
+    },
+    "tp2_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 222.73932355314494,
+          "p90": 513.3421414172687
+        },
+        "ttft_ms": {
+          "mean": 222.73932355314494,
+          "p90": 513.3421414172687
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp2_mns64/tp2/result.json"
+    },
+    "tp2_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 222.73961839402438,
+          "p90": 513.3421414172687
+        },
+        "ttft_ms": {
+          "mean": 222.73961839402438,
+          "p90": 513.3421414172687
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp2_mns8/tp2/result.json"
+    },
+    "tp4_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 164.583607484367,
+          "p90": 355.93888195481327
+        },
+        "ttft_ms": {
+          "mean": 164.583607484367,
+          "p90": 355.93888195481327
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp4_mns16/tp4/result.json"
+    },
+    "tp4_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 164.58396574139448,
+          "p90": 355.93888195481327
+        },
+        "ttft_ms": {
+          "mean": 164.58396574139448,
+          "p90": 355.93888195481327
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp4_mns32/tp4/result.json"
+    },
+    "tp4_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 164.58471606968033,
+          "p90": 355.93888195481327
+        },
+        "ttft_ms": {
+          "mean": 164.58471606968033,
+          "p90": 355.93888195481327
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp4_mns64/tp4/result.json"
+    },
+    "tp4_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "mean": 164.583607484367,
+          "p90": 355.93888195481327
+        },
+        "ttft_ms": {
+          "mean": 164.583607484367,
+          "p90": 355.93888195481327
+        }
+      },
+      "path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po/runs/tp4_mns8/tp4/result.json"
+    }
+  },
+  "prefill_only": true,
+  "real_audit": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/analysis-r6/trace-po-real-audit.json",
+  "schema": "qwen30-latency-case-frontier-real-comparison-v1",
+  "selection": {
+    "e2e_ms:mean": {
+      "agreement": 0.8412698412698413,
+      "concordant_pairs": 53,
+      "discordant_pairs": 10,
+      "informative_pairs": 63,
+      "real_ranking": [
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns8",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns16",
+        "tp1_mns64",
+        "tp1_mns32",
+        "tp1_mns8"
+      ],
+      "real_winner": "tp4_mns64",
+      "selected_config_real_regret": 0.0318784105178655,
+      "sim_ranking": [
+        "tp4_mns16",
+        "tp4_mns8",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns16",
+        "tp2_mns8",
+        "tp1_mns16",
+        "tp1_mns8",
+        "tp1_mns32",
+        "tp1_mns64"
+      ],
+      "sim_winner": "tp4_mns16",
+      "tied_pairs": 3,
+      "winner_match": false
+    },
+    "e2e_ms:p90": {
+      "agreement": 1.0,
+      "concordant_pairs": 48,
+      "discordant_pairs": 0,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64"
+      ],
+      "real_winner": "tp4_mns8",
+      "selected_config_real_regret": 0.0033158177113489728,
+      "sim_ranking": [
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8"
+      ],
+      "sim_winner": "tp4_mns16",
+      "tied_pairs": 18,
+      "winner_match": false
+    },
+    "ttft_ms:mean": {
+      "agreement": 0.8412698412698413,
+      "concordant_pairs": 53,
+      "discordant_pairs": 10,
+      "informative_pairs": 63,
+      "real_ranking": [
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns8",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns16",
+        "tp1_mns64",
+        "tp1_mns32",
+        "tp1_mns8"
+      ],
+      "real_winner": "tp4_mns64",
+      "selected_config_real_regret": 0.03187213306057872,
+      "sim_ranking": [
+        "tp4_mns16",
+        "tp4_mns8",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns16",
+        "tp2_mns8",
+        "tp1_mns16",
+        "tp1_mns8",
+        "tp1_mns32",
+        "tp1_mns64"
+      ],
+      "sim_winner": "tp4_mns16",
+      "tied_pairs": 3,
+      "winner_match": false
+    },
+    "ttft_ms:p90": {
+      "agreement": 1.0,
+      "concordant_pairs": 48,
+      "discordant_pairs": 0,
+      "informative_pairs": 48,
+      "real_ranking": [
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64"
+      ],
+      "real_winner": "tp4_mns8",
+      "selected_config_real_regret": 0.0037779133092611917,
+      "sim_ranking": [
+        "tp4_mns16",
+        "tp4_mns32",
+        "tp4_mns64",
+        "tp4_mns8",
+        "tp2_mns16",
+        "tp2_mns32",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64",
+        "tp1_mns8"
+      ],
+      "sim_winner": "tp4_mns16",
+      "tied_pairs": 18,
+      "winner_match": false
+    }
+  },
+  "sim_root": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/sim/trace-po"
+}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/trace-po-comparison.md

@@ -0,0 +1,8 @@
+# Qwen30 Frontier vs real latency selection
+
+| Objective | Frontier | Real | Match | Regret | Pairwise |
+|---|---|---|---:|---:|---:|
+| ttft_ms:mean | tp4_mns16 | tp4_mns64 | no | 3.2% | 84.1% |
+| ttft_ms:p90 | tp4_mns16 | tp4_mns8 | no | 0.4% | 100.0% |
+| e2e_ms:mean | tp4_mns16 | tp4_mns64 | no | 3.2% | 84.1% |
+| e2e_ms:p90 | tp4_mns16 | tp4_mns8 | no | 0.3% | 100.0% |

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/trace-po-comparison.stdout.json

@@ -0,0 +1 @@
+{"e2e_ms:mean": {"agreement": 0.8412698412698413, "concordant_pairs": 53, "discordant_pairs": 10, "informative_pairs": 63, "real_ranking": ["tp4_mns64", "tp4_mns32", "tp4_mns8", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns16", "tp1_mns64", "tp1_mns32", "tp1_mns8"], "real_winner": "tp4_mns64", "selected_config_real_regret": 0.0318784105178655, "sim_ranking": ["tp4_mns16", "tp4_mns8", "tp4_mns32", "tp4_mns64", "tp2_mns32", "tp2_mns64", "tp2_mns16", "tp2_mns8", "tp1_mns16", "tp1_mns8", "tp1_mns32", "tp1_mns64"], "sim_winner": "tp4_mns16", "tied_pairs": 3, "winner_match": false}, "e2e_ms:p90": {"agreement": 1.0, "concordant_pairs": 48, "discordant_pairs": 0, "informative_pairs": 48, "real_ranking": ["tp4_mns8", "tp4_mns64", "tp4_mns32", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64"], "real_winner": "tp4_mns8", "selected_config_real_regret": 0.0033158177113489728, "sim_ranking": ["tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8"], "sim_winner": "tp4_mns16", "tied_pairs": 18, "winner_match": false}, "ttft_ms:mean": {"agreement": 0.8412698412698413, "concordant_pairs": 53, "discordant_pairs": 10, "informative_pairs": 63, "real_ranking": ["tp4_mns64", "tp4_mns32", "tp4_mns8", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns16", "tp1_mns64", "tp1_mns32", "tp1_mns8"], "real_winner": "tp4_mns64", "selected_config_real_regret": 0.03187213306057872, "sim_ranking": ["tp4_mns16", "tp4_mns8", "tp4_mns32", "tp4_mns64", "tp2_mns32", "tp2_mns64", "tp2_mns16", "tp2_mns8", "tp1_mns16", "tp1_mns8", "tp1_mns32", "tp1_mns64"], "sim_winner": "tp4_mns16", "tied_pairs": 3, "winner_match": false}, "ttft_ms:p90": {"agreement": 1.0, "concordant_pairs": 48, "discordant_pairs": 0, "informative_pairs": 48, "real_ranking": ["tp4_mns8", "tp4_mns64", "tp4_mns32", "tp4_mns16", "tp2_mns64", "tp2_mns8", "tp2_mns32", "tp2_mns16", "tp1_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64"], "real_winner": "tp4_mns8", "selected_config_real_regret": 0.0037779133092611917, "sim_ranking": ["tp4_mns16", "tp4_mns32", "tp4_mns64", "tp4_mns8", "tp2_mns16", "tp2_mns32", "tp2_mns64", "tp2_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64", "tp1_mns8"], "sim_winner": "tp4_mns16", "tied_pairs": 18, "winner_match": false}}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/trace-po-real-audit.json

@@ -0,0 +1,997 @@
+{
+  "applicable_metrics": [
+    "ttft_ms",
+    "e2e_ms"
+  ],
+  "case_root": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po",
+  "configs": {
+    "tp1_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 552.8720716136033,
+          "pooled_p90_ms": 1849.5772210881114,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 552.8720716136035,
+          "trial_stddev_of_means_ms": 2.521785864921604
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 552.8045099093855,
+          "pooled_p90_ms": 1849.544870085083,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 552.8045099093856,
+          "trial_stddev_of_means_ms": 2.520195001336989
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 551.4591356410912,
+              "p90_ms": 1849.5772210881114,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 551.3877547012152,
+              "p90_ms": 1849.544870085083,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns16/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 551.3735196675904,
+              "p90_ms": 1854.1427809977904,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 551.311527398711,
+              "p90_ms": 1854.09467399586,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns16/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 555.7835595321286,
+              "p90_ms": 1865.3576440410689,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 555.7142476282306,
+              "p90_ms": 1865.324669983238,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns16/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp1_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 553.9655365312751,
+          "pooled_p90_ms": 1858.5613030008972,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 553.9655365312751,
+          "trial_stddev_of_means_ms": 0.562579474030608
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 553.8995644734638,
+          "pooled_p90_ms": 1858.5308340843767,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 553.8995644734638,
+          "trial_stddev_of_means_ms": 0.5595598051891147
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 554.533127208002,
+              "p90_ms": 1863.7738740071654,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 554.4650907711263,
+              "p90_ms": 1863.7414400000125,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns32/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 553.4081058838412,
+              "p90_ms": 1860.4962331010029,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 553.3461683987488,
+              "p90_ms": 1860.462275100872,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns32/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 553.9553765019821,
+              "p90_ms": 1858.5613030008972,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 553.8874342505163,
+              "p90_ms": 1858.5308340843767,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns32/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp1_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 553.5116215725009,
+          "pooled_p90_ms": 1862.3707490041852,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 553.5116215725008,
+          "trial_stddev_of_means_ms": 2.551946974085908
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 553.4431933351968,
+          "pooled_p90_ms": 1862.3419440118596,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 553.4431933351968,
+          "trial_stddev_of_means_ms": 2.558476651513527
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 551.9927626163446,
+              "p90_ms": 1863.004481070675,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 551.9189011646695,
+              "p90_ms": 1862.9671899834648,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns64/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 556.4578829970312,
+              "p90_ms": 1876.7878300277516,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 556.3969604667392,
+              "p90_ms": 1876.7514559440315,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns64/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 552.0842191041266,
+              "p90_ms": 1862.3707490041852,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 552.0137183741818,
+              "p90_ms": 1862.3419440118596,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns64/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp1_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 554.3623829256859,
+          "pooled_p90_ms": 1849.5380710810423,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 554.3623829256859,
+          "trial_stddev_of_means_ms": 1.2895027720806873
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 554.2973810096947,
+          "pooled_p90_ms": 1849.499023053795,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 554.2973810096947,
+          "trial_stddev_of_means_ms": 1.2879745625238963
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 552.8872783905863,
+              "p90_ms": 1849.5380710810423,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 552.8246594192243,
+              "p90_ms": 1849.499023053795,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns8/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 555.2756262775293,
+              "p90_ms": 1865.9542039968073,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 555.2131765045572,
+              "p90_ms": 1865.9242549911141,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns8/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 554.9242441089419,
+              "p90_ms": 1863.5734919225797,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 554.8543071053027,
+              "p90_ms": 1863.5354900034145,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp1_mns8/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp2_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 354.36655512622264,
+          "pooled_p90_ms": 1137.23775499966,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 354.36655512622264,
+          "trial_stddev_of_means_ms": 1.8755237488897505
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 354.2987044818834,
+          "pooled_p90_ms": 1137.2028539190069,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 354.29870448188336,
+          "trial_stddev_of_means_ms": 1.8738441773930947
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 352.474492817554,
+              "p90_ms": 1146.7004970181733,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 352.4049650534785,
+              "p90_ms": 1146.5505469823256,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns16/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 356.2250907998532,
+              "p90_ms": 1142.300286097452,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 356.1519986188198,
+              "p90_ms": 1142.2789900097996,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns16/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 354.40008176126076,
+              "p90_ms": 1137.23775499966,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 354.3391497733519,
+              "p90_ms": 1137.2028539190069,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns16/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp2_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 353.6182792498718,
+          "pooled_p90_ms": 1133.6436610436067,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 353.6182792498718,
+          "trial_stddev_of_means_ms": 1.6113304304047518
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 353.5506138878872,
+          "pooled_p90_ms": 1133.6100719636306,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 353.5506138878872,
+          "trial_stddev_of_means_ms": 1.6067873099412449
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 355.363455723769,
+              "p90_ms": 1137.4261770397425,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 355.2912532689425,
+              "p90_ms": 1137.4054140178487,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns32/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 353.3043994003358,
+              "p90_ms": 1133.6436610436067,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 353.2365282921684,
+              "p90_ms": 1133.6100719636306,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns32/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 352.1869826255106,
+              "p90_ms": 1137.214289046824,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 352.12406010255074,
+              "p90_ms": 1137.1923619881272,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns32/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp2_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 344.2724434967521,
+          "pooled_p90_ms": 1112.3684010235593,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 344.27244349675215,
+          "trial_stddev_of_means_ms": 1.2293359433030469
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 344.2076703236325,
+          "pooled_p90_ms": 1112.340173101984,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 344.20767032363256,
+          "trial_stddev_of_means_ms": 1.2250537007477627
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 342.85445375793955,
+              "p90_ms": 1112.3684010235593,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 342.79442101431823,
+              "p90_ms": 1112.340173101984,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns64/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 345.03840929702244,
+              "p90_ms": 1119.2332479404286,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 344.96722287271,
+              "p90_ms": 1119.2107619717717,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns64/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 344.92446743529433,
+              "p90_ms": 1121.5684060007334,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 344.86136708386937,
+              "p90_ms": 1121.5376510517672,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns64/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp2_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 347.7476487716525,
+          "pooled_p90_ms": 1112.5693990616128,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 347.74764877165256,
+          "trial_stddev_of_means_ms": 0.717530137997859
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 347.67954972228495,
+          "pooled_p90_ms": 1112.5206090509892,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 347.67954972228495,
+          "trial_stddev_of_means_ms": 0.7137879732940912
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 347.8964949396706,
+              "p90_ms": 1126.5470329672098,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 347.825654325718,
+              "p90_ms": 1126.4982499415055,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns8/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 348.37908198508296,
+              "p90_ms": 1108.5135659668595,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 348.3089811246221,
+              "p90_ms": 1108.4930589422584,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns8/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 346.967369390204,
+              "p90_ms": 1112.5693990616128,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 346.90401371651467,
+              "p90_ms": 1112.5206090509892,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp2_mns8/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp4_mns16": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 230.30740981957652,
+          "pooled_p90_ms": 669.3113599903882,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 230.30740981957652,
+          "trial_stddev_of_means_ms": 2.164263687280578
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 230.23591205587212,
+          "pooled_p90_ms": 669.2857249872759,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 230.2359120558721,
+          "trial_stddev_of_means_ms": 2.1675472527430966
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 230.70611610156737,
+              "p90_ms": 682.0242760004476,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 230.64041384333356,
+              "p90_ms": 681.8402550416067,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns16/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 232.2445988110014,
+              "p90_ms": 669.8487129760906,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 232.17271348053603,
+              "p90_ms": 669.6453359909356,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns16/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 227.9715145461608,
+              "p90_ms": 668.2263769907877,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 227.89460884374674,
+              "p90_ms": 667.8900560364127,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns16/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp4_mns32": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 227.01489424738475,
+          "pooled_p90_ms": 668.1078639812768,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 227.01489424738475,
+          "trial_stddev_of_means_ms": 1.0302265785061928
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 226.9508410934046,
+          "pooled_p90_ms": 668.0838000029325,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 226.9508410934046,
+          "trial_stddev_of_means_ms": 1.0309199236641802
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 227.36105044253256,
+              "p90_ms": 668.9074879977852,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 227.29304120623212,
+              "p90_ms": 668.8842560397461,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns32/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 227.82746249586054,
+              "p90_ms": 668.1078639812768,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 227.7671468305192,
+              "p90_ms": 668.0838000029325,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns32/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 225.8561698037611,
+              "p90_ms": 668.5177159961313,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 225.79233524346247,
+              "p90_ms": 668.1753649609163,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns32/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp4_mns64": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 223.1923911500318,
+          "pooled_p90_ms": 668.0814319988713,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 223.1923911500318,
+          "trial_stddev_of_means_ms": 1.0834054130133073
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 223.12445959072676,
+          "pooled_p90_ms": 668.0591680342332,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 223.1244595907268,
+          "trial_stddev_of_means_ms": 1.078830920161661
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 224.0322316110553,
+              "p90_ms": 668.6453759903088,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 223.96008559961103,
+              "p90_ms": 668.1041569681838,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns64/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 223.57544556978453,
+              "p90_ms": 697.0440980512649,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 223.50675472538455,
+              "p90_ms": 697.0191639848053,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns64/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 221.96949626925553,
+              "p90_ms": 668.0814319988713,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 221.9065384471847,
+              "p90_ms": 668.0591680342332,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns64/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    },
+    "tp4_mns8": {
+      "metrics": {
+        "e2e_ms": {
+          "pooled_mean_ms": 229.07142115104702,
+          "pooled_p90_ms": 667.0993800507858,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 229.07142115104702,
+          "trial_stddev_of_means_ms": 1.6982436653662605
+        },
+        "ttft_ms": {
+          "pooled_mean_ms": 229.00724306236953,
+          "pooled_p90_ms": 666.7667380534112,
+          "pooled_samples": 387,
+          "trial_mean_of_means_ms": 229.00724306236953,
+          "trial_stddev_of_means_ms": 1.6999412520227009
+        }
+      },
+      "trials": [
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 227.73505919955872,
+              "p90_ms": 667.3817320261151,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 227.67112628275225,
+              "p90_ms": 667.0240580569953,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns8/trial1/results/result.json",
+          "trial": 1
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 230.98243522013775,
+              "p90_ms": 667.0993800507858,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 230.92065048909407,
+              "p90_ms": 666.7667380534112,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns8/trial2/results/result.json",
+          "trial": 2
+        },
+        {
+          "metrics": {
+            "e2e_ms": {
+              "mean_ms": 228.4967690334446,
+              "p90_ms": 667.9027479840443,
+              "samples": 129
+            },
+            "ttft_ms": {
+              "mean_ms": 228.42995241526225,
+              "p90_ms": 667.7060039946809,
+              "samples": 129
+            }
+          },
+          "requests": 129,
+          "result_path": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/real-r6/trace-po/real/tp4_mns8/trial3/results/result.json",
+          "trial": 3
+        }
+      ]
+    }
+  },
+  "prefill_only": true,
+  "schema": "qwen30-latency-case-real-audit-v1",
+  "trace_manifests": {
+    "tp1": {
+      "description": "trace-derived: exact input/arrival/session/prefix; output override only",
+      "first_arrival_s": 6.698,
+      "global_offered_request_rate": 0.215,
+      "last_arrival_s": 597.037,
+      "output_tokens": [
+        1
+      ],
+      "per_gpu_offered_request_rate": 0.215,
+      "prefix_caching": true,
+      "private_jsonl": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/trace-po/tp1/private/real_requests.jsonl",
+      "private_jsonl_sha256": "525bf779bbd111bc3c42ac65af5cfed36ce59806acc521ee16d1c5ce7b3a6152",
+      "public_csv": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/trace-po/tp1/public/frontier.csv",
+      "public_csv_sha256": "eccab2e2f50f566eb4ec78291d126fce5fe9cb426eb89993c1b63b18f83dfc29",
+      "requests": 129,
+      "row_vector_sha256": "3df3a57e6a796e76d42f846fbfef59049f2bd804f4d3eda8ecbc9d96b43dc6ce",
+      "schema": "qwen30-latency-case-v1",
+      "tensor_parallel_size": 1
+    },
+    "tp2": {
+      "description": "trace-derived: exact input/arrival/session/prefix; output override only",
+      "first_arrival_s": 3.349,
+      "global_offered_request_rate": 0.43,
+      "last_arrival_s": 298.5185,
+      "output_tokens": [
+        1
+      ],
+      "per_gpu_offered_request_rate": 0.215,
+      "prefix_caching": true,
+      "private_jsonl": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/trace-po/tp2/private/real_requests.jsonl",
+      "private_jsonl_sha256": "61bdb8a7245426a7c283d0f2944657d3eda46b1c04998c88b025a2411fe15340",
+      "public_csv": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/trace-po/tp2/public/frontier.csv",
+      "public_csv_sha256": "26701486cfb655a301b129acd14ab8d78b94ec843b4ceca8e222cee466e40535",
+      "requests": 129,
+      "row_vector_sha256": "f20dd473240fd243d38cb727f7896457e86da9e965b4f23f565aef5f3fc70351",
+      "schema": "qwen30-latency-case-v1",
+      "tensor_parallel_size": 2
+    },
+    "tp4": {
+      "description": "trace-derived: exact input/arrival/session/prefix; output override only",
+      "first_arrival_s": 1.6745,
+      "global_offered_request_rate": 0.86,
+      "last_arrival_s": 149.25925,
+      "output_tokens": [
+        1
+      ],
+      "per_gpu_offered_request_rate": 0.215,
+      "prefix_caching": true,
+      "private_jsonl": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/trace-po/tp4/private/real_requests.jsonl",
+      "private_jsonl_sha256": "c38514fa524fc215cd9097d990a2985532f83ac6314bc46ad89ebfc1bd89b4f8",
+      "public_csv": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/trace-po/tp4/public/frontier.csv",
+      "public_csv_sha256": "f4112a2b0366b559bfced6a0986a01e2b7b39899a650d030c3e680b3dc63c92e",
+      "requests": 129,
+      "row_vector_sha256": "851e796f6d3e87444b58da87e0ef4b7482a13bb6502cb6873fa80523877ab74e",
+      "schema": "qwen30-latency-case-v1",
+      "tensor_parallel_size": 4
+    }
+  },
+  "traces_root": "/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/traces/trace-po",
+  "winners": {
+    "e2e_ms:pooled_mean_ms": {
+      "ranking": [
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns8",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns16",
+        "tp1_mns64",
+        "tp1_mns32",
+        "tp1_mns8"
+      ],
+      "winner": "tp4_mns64",
+      "winner_value_ms": 223.1923911500318
+    },
+    "e2e_ms:pooled_p90_ms": {
+      "ranking": [
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64"
+      ],
+      "winner": "tp4_mns8",
+      "winner_value_ms": 667.0993800507858
+    },
+    "ttft_ms:pooled_mean_ms": {
+      "ranking": [
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns8",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns16",
+        "tp1_mns64",
+        "tp1_mns32",
+        "tp1_mns8"
+      ],
+      "winner": "tp4_mns64",
+      "winner_value_ms": 223.12445959072676
+    },
+    "ttft_ms:pooled_p90_ms": {
+      "ranking": [
+        "tp4_mns8",
+        "tp4_mns64",
+        "tp4_mns32",
+        "tp4_mns16",
+        "tp2_mns64",
+        "tp2_mns8",
+        "tp2_mns32",
+        "tp2_mns16",
+        "tp1_mns8",
+        "tp1_mns16",
+        "tp1_mns32",
+        "tp1_mns64"
+      ],
+      "winner": "tp4_mns8",
+      "winner_value_ms": 666.7667380534112
+    }
+  }
+}

runs/frontier-split-rootcause-v0/frozen-inputs/q30-expansion-lo/trace-po-real-audit.md

@@ -0,0 +1,10 @@
+# Qwen30 real latency case audit
+
+Prefill-only: `True`.
+
+| Objective | Real winner | Value (ms) |
+|---|---|---:|
+| ttft_ms:pooled_mean_ms | tp4_mns64 | 223.12 |
+| ttft_ms:pooled_p90_ms | tp4_mns8 | 666.77 |
+| e2e_ms:pooled_mean_ms | tp4_mns64 | 223.19 |
+| e2e_ms:pooled_p90_ms | tp4_mns8 | 667.10 |