Track simulator fidelity experiment artifacts

runs/frontier-multicase-sufficiency-v0/protocol.md

@@ -0,0 +1,79 @@
+# Frontier multi-case sufficiency protocol v0
+
+## Claim under test
+
+Can Frontier select a low-regret configuration for the frozen candidate grid
+under the objective **maximum SLO-feasible offered request throughput per GPU**?
+
+The allowed conclusion is scoped to the tested model/runtime/hardware/workload
+family. Three matches cannot establish universal simulator fidelity.
+
+## Cases
+
+1. Qwen3-30B-A3B mixed chat, H20, TP x MNS grid (existing data).
+2. Qwen3-235B-A22B prefill-only, H20, TP x MNS x MBT grid.
+3. Qwen3-235B-A22B decode-only, H20, TP/DP/EP x MNS x MBT grid.
+
+For Qwen235B, keep the real trace window, request filtering, SLO, model build,
+runtime flags, and candidate cells recorded by the original trials. Frontier is
+evaluated at the same offered-load anchors; no capacity extrapolation beyond
+the common anchors is allowed.
+
+## Two separate simulator readings
+
+- **Zero-shot/profile-only:** operator profiles and documented hardware/model
+  inputs are allowed; no end-to-end measurement from the evaluated workload is
+  used to scale Frontier.
+- **Calibrated:** every real measurement used to fit a scalar or residual is
+  charged and reported. Calibration is fitted on a declared train fixture and
+  evaluated on a disjoint workload/config holdout.
+
+These readings must never be merged. A per-TP scale fitted from the same
+workload does not count as zero-shot simulator accuracy.
+
+## Ground-truth gate
+
+Before a match decision:
+
+- every result must be a completed primary search without probe failure;
+- capacity is an interval from the largest feasible anchor to the next
+  infeasible anchor, not merely the feasible lower bound;
+- refine probes when the possibly-optimal set induced by those intervals is too
+  broad to distinguish the simulator's selected cell;
+- record the random top-set hit rate and the number of informative (non-tied)
+  pairs.
+
+## Primary metrics and predeclared decision rule
+
+Per case, report:
+
+1. selected-config real regret (point estimate and interval-robust bound);
+2. Kendall tau-b and exact pair-direction accuracy with real ties preserved;
+3. top-set hit and its random-hit baseline;
+4. anchor-level SLO feasibility confusion;
+5. profile/calibration cost and all right-censored cells.
+
+The profile-only simulator is considered sufficient as a **config ranker for
+the tested family** only if every case has:
+
+- worst selected-config real regret <= 5%;
+- Kendall tau-b >= 0.8 on a response surface with enough informative pairs;
+- no unresolved ground-truth interval that can reverse the selected decision;
+- no per-case or per-action end-to-end calibration.
+
+SLO-oracle sufficiency is a stronger claim and additionally requires low false
+feasible/false infeasible rates; ranker success alone does not establish it.
+
+## Current representational gaps to disclose
+
+- The checked-in Frontier Qwen235B model config is BF16 and has no H20
+  Qwen235B operator profiles, while the real runtime uses FP8 weights and FP8 KV.
+- Prefill uses internal vLLM/BLADNN paths and PIECEWISE CUDA graphs.
+- Decode uses TP/DP with EP8, DeepEP/NVSHMEM, EAGLE3, DecodeBenchConnector, and
+  FULL_DECODE_ONLY CUDA graphs.
+- Frontier currently rejects the faithful combination of speculative decoding
+  and decode CUDA-graph modeling unless using a diagnostic opt-in; its public
+  examples disable decode CUDA graphs for speculative decoding.
+
+Therefore an as-is Frontier run and an upgraded/semantically aligned run, if
+implemented, must be reported separately.