gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Summarize Frontier selection regret

Browse Source
This commit is contained in:
Gahow Wang
2026-07-19 15:31:16 +08:00
parent 4c8d581a5b
commit 3d3878c5aa
6 changed files with 199 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
runs/frontier-fidelity-envelope-v1/qwen30-latency-expansion-card.md
View File

@@ -1,7 +1,8 @@
# EXP-SIMFID-Q30-LATENCY-EXPANSION: remaining Fixed/Trace × PD/P cases
> Status: prepared for runtime-alignment preflight (2026-07-18). The already
> completed Qwen3-30B-A3B Trace-PD surface is excluded from this card.
> Status: completed (2026-07-18). The low-load Fixed cases below are retained
> as sensitivity evidence; the later 4096-token, 1.125 req/s/GPU Fixed matrix
> is the primary comparison. Trace-PD is excluded from this card.
## Question and fixed boundary
@@ -47,3 +48,45 @@ metric is a coverage failure, not a high-latency cell.
The no-request Fixed runtime preflight is capped at 2 H20-GPUh. Each 36-run
real surface is estimated at 13 nominal / 41 worst-case H20-GPUh, plus CPU-only
Frontier replay. Launch logs record the resolved inputs, paths, and duration.
## Result: exact-selection fidelity is not stable across workload phase/state
All three real surfaces completed 36 valid fresh-server trials (12
TP×MNS cells × 3 trials), each with 129/129 exact-usage requests. The
corresponding 12-cell Frontier replays completed all 129 requests per cell.
The following table compares the simulator argmin with the pooled real argmin;
`regret` is the selected simulator configuration's real relative latency
regret. It deliberately reports exact winner equality rather than treating a
high pairwise-order agreement as success.
| Case | Objective | Frontier winner | Real winner | Match | Real regret |
|---|---|---|---|---:|---:|
| Trace-P (OSL=1) | TTFT mean | TP4/MNS16 | TP4/MNS64 | no | 3.2% |
| Trace-P (OSL=1) | TTFT p90 | TP4/MNS16 | TP4/MNS8 | no | 0.4% |
| Trace-P (OSL=1) | E2E mean | TP4/MNS16 | TP4/MNS64 | no | 3.2% |
| Trace-P (OSL=1) | E2E p90 | TP4/MNS16 | TP4/MNS8 | no | 0.3% |
| Fixed-PD | TTFT mean/p90 | TP4/MNS16 | TP4/MNS16 | yes | 0.0% |
| Fixed-PD | TPOT mean | TP1/MNS16 | TP4/MNS32 | no | 14.0% |
| Fixed-PD | TPOT p90 | TP1/MNS16 | TP4/MNS8 | no | 13.4% |
| Fixed-PD | E2E mean | TP1/MNS16 | TP4/MNS32 | no | 26.9% |
| Fixed-PD | E2E p90 | TP1/MNS16 | TP4/MNS8 | no | 25.7% |
| Fixed-P (OSL=1) | TTFT/E2E mean/p90 | TP4/MNS16 | TP4/MNS8 | no | 1.5--3.6% |
The decisive failure is Fixed-PD: its TPOT and E2E pairwise orders are exactly
reversed on all 48 informative pairs. Thus the graph-aligned profile can
preserve coarse TP ordering yet still choose the wrong configuration for the
serving objective. The prior Trace-PD success is therefore a workload-specific
success, not evidence that Frontier has solved config tuning.
## Superseding primary Fixed workload
The subsequent pressure sweep changed the Fixed shapes to 4096->256 and
4096->1, raised the offered load to 1.125 req/s/GPU, and used 257 requests per
cell. Those Fixed-PD/PO results, together with Trace-PD and Trace-PO, are
reported in the consolidated
[Qwen30/Qwen235 selection-regret report](../../docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md).
Under that primary Fixed-PD workload, Frontier's selected config is 58.0%
slower for real mean TTFT and 56.2% slower for real p90 TTFT than the
real-hardware optimum. The low-load table above must therefore not be quoted
as the final Fixed-PD result.