80 lines
3.4 KiB
Markdown
80 lines
3.4 KiB
Markdown
# Frontier multi-case sufficiency protocol v0
|
|
|
|
## Claim under test
|
|
|
|
Can Frontier select a low-regret configuration for the frozen candidate grid
|
|
under the objective **maximum SLO-feasible offered request throughput per GPU**?
|
|
|
|
The allowed conclusion is scoped to the tested model/runtime/hardware/workload
|
|
family. Three matches cannot establish universal simulator fidelity.
|
|
|
|
## Cases
|
|
|
|
1. Qwen3-30B-A3B mixed chat, H20, TP x MNS grid (existing data).
|
|
2. Qwen3-235B-A22B prefill-only, H20, TP x MNS x MBT grid.
|
|
3. Qwen3-235B-A22B decode-only, H20, TP/DP/EP x MNS x MBT grid.
|
|
|
|
For Qwen235B, keep the real trace window, request filtering, SLO, model build,
|
|
runtime flags, and candidate cells recorded by the original trials. Frontier is
|
|
evaluated at the same offered-load anchors; no capacity extrapolation beyond
|
|
the common anchors is allowed.
|
|
|
|
## Two separate simulator readings
|
|
|
|
- **Zero-shot/profile-only:** operator profiles and documented hardware/model
|
|
inputs are allowed; no end-to-end measurement from the evaluated workload is
|
|
used to scale Frontier.
|
|
- **Calibrated:** every real measurement used to fit a scalar or residual is
|
|
charged and reported. Calibration is fitted on a declared train fixture and
|
|
evaluated on a disjoint workload/config holdout.
|
|
|
|
These readings must never be merged. A per-TP scale fitted from the same
|
|
workload does not count as zero-shot simulator accuracy.
|
|
|
|
## Ground-truth gate
|
|
|
|
Before a match decision:
|
|
|
|
- every result must be a completed primary search without probe failure;
|
|
- capacity is an interval from the largest feasible anchor to the next
|
|
infeasible anchor, not merely the feasible lower bound;
|
|
- refine probes when the possibly-optimal set induced by those intervals is too
|
|
broad to distinguish the simulator's selected cell;
|
|
- record the random top-set hit rate and the number of informative (non-tied)
|
|
pairs.
|
|
|
|
## Primary metrics and predeclared decision rule
|
|
|
|
Per case, report:
|
|
|
|
1. selected-config real regret (point estimate and interval-robust bound);
|
|
2. Kendall tau-b and exact pair-direction accuracy with real ties preserved;
|
|
3. top-set hit and its random-hit baseline;
|
|
4. anchor-level SLO feasibility confusion;
|
|
5. profile/calibration cost and all right-censored cells.
|
|
|
|
The profile-only simulator is considered sufficient as a **config ranker for
|
|
the tested family** only if every case has:
|
|
|
|
- worst selected-config real regret <= 5%;
|
|
- Kendall tau-b >= 0.8 on a response surface with enough informative pairs;
|
|
- no unresolved ground-truth interval that can reverse the selected decision;
|
|
- no per-case or per-action end-to-end calibration.
|
|
|
|
SLO-oracle sufficiency is a stronger claim and additionally requires low false
|
|
feasible/false infeasible rates; ranker success alone does not establish it.
|
|
|
|
## Current representational gaps to disclose
|
|
|
|
- The checked-in Frontier Qwen235B model config is BF16 and has no H20
|
|
Qwen235B operator profiles, while the real runtime uses FP8 weights and FP8 KV.
|
|
- Prefill uses internal vLLM/BLADNN paths and PIECEWISE CUDA graphs.
|
|
- Decode uses TP/DP with EP8, DeepEP/NVSHMEM, EAGLE3, DecodeBenchConnector, and
|
|
FULL_DECODE_ONLY CUDA graphs.
|
|
- Frontier currently rejects the faithful combination of speculative decoding
|
|
and decode CUDA-graph modeling unless using a diagnostic opt-in; its public
|
|
examples disable decode CUDA graphs for speculative decoding.
|
|
|
|
Therefore an as-is Frontier run and an upgraded/semantically aligned run, if
|
|
implemented, must be reported separately.
|