46 lines
3.4 KiB
Markdown
46 lines
3.4 KiB
Markdown
# Frontier SLO-aligned retrospective experiment
|
|
|
|
## Result
|
|
|
|
After aligning both systems to the same paired-grid objective—maximum offered req/s/GPU with request-level SLO pass rate >= 0.95—the old 30.46% top-1 regret does not reproduce. Frontier nominates `tp2_mns32`, `tp2_mns64` (tie), with real-evaluated regret 0.000%--0.761%.
|
|
|
|
Ranking agreement is high on this fixed candidate grid: Kendall tau-b 0.9668, pairwise exact-sign accuracy 93.94%. This is materially different from the old completed-throughput proxy (tau-b 0.4481, top-1 regret 30.46%).
|
|
|
|
## Per-cell paired-grid capacity
|
|
|
|
| Cell | Real | Frontier | Error | Frontier boundary | Real monotonic violations |
|
|
|---|---:|---:|---:|---|---:|
|
|
| `tp1_mns8` | 2.100000 | 1.716667 | -0.383333 | bracketed | 0 |
|
|
| `tp1_mns16` | 2.350000 | 2.383333 | +0.033333 | bracketed | 0 |
|
|
| `tp1_mns32` | 2.283333 | 2.383333 | +0.100000 | bracketed | 0 |
|
|
| `tp1_mns64` | 2.283333 | 2.383333 | +0.100000 | bracketed | 0 |
|
|
| `tp2_mns8` | 2.275000 | 1.741667 | -0.533333 | bracketed | 0 |
|
|
| `tp2_mns16` | 2.275000 | 2.300000 | +0.025000 | right-censored | 0 |
|
|
| `tp2_mns32` | 3.283333 | 3.750000 | +0.466667 | right-censored | 0 |
|
|
| `tp2_mns64` | 3.258333 | 3.750000 | +0.491667 | right-censored | 0 |
|
|
| `tp4_mns8` | 1.283333 | 1.320833 | +0.037500 | bracketed | 0 |
|
|
| `tp4_mns16` | 2.441667 | 2.500000 | +0.058333 | right-censored | 0 |
|
|
| `tp4_mns32` | 2.441667 | 2.500000 | +0.058333 | right-censored | 0 |
|
|
| `tp4_mns64` | 2.441667 | 2.500000 | +0.058333 | right-censored | 0 |
|
|
|
|
The paired-grid score MAE is 0.1955 req/s/GPU and MAPE is 7.70%. These error aggregates must be read as grid-clipped because 6/12 Frontier cells remain feasible at their highest common anchor.
|
|
|
|
The error is configuration-dependent rather than a uniform scale offset. Mean signed error is -0.2931 req/s/GPU for MNS=8, but rises to +0.2083 and +0.2167 for MNS=32/64. The aligned score therefore still exposes a missing or distorted MNS response.
|
|
|
|
## Anchor-level feasibility
|
|
|
|
Across 92 paired anchors: true-feasible=55, true-infeasible=9, false-feasible=21, false-infeasible=7. Accuracy is 69.57%, but the false-feasible rate among real-infeasible anchors is 70.00%.
|
|
|
|
Therefore the aligned experiment supports a narrower conclusion: Frontier can recover the top configuration family on this frozen, ragged candidate grid, but it is not yet a reliable SLO feasibility oracle. Good top-1 ranking is compatible with many wrong boundary labels.
|
|
|
|
## Validity limits
|
|
|
|
- This is a retrospective single-run analysis; no confidence interval is valid.
|
|
- Real probes within a cell reused one vLLM process and prefix-cache history; Frontier anchors were independent runs. History-dependent real pass-rate values are retained rather than monotonicized; binary feasibility happens to remain monotone on the observed per-cell grids.
|
|
- Right-censored cells expose only a lower bound on Frontier's own capacity. They are valid for paired-grid selection, not for claiming an exact simulator knee.
|
|
- The frozen per-TP calibration and evaluation surface are not fully independent across workload families, so this is not a generalization result.
|
|
|
|
## Required prospective experiment
|
|
|
|
Repeat a small set of boundary cells with one engine restart per anchor, explicit cold-cache state, identical offered-load grid, and repeated trials. Extend the grid until both real and Frontier have at least one feasible and one infeasible point. Only that experiment can estimate boundary displacement and its uncertainty.
|