gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Summarize Frontier selection regret

Browse Source
This commit is contained in:
Gahow Wang
2026-07-19 15:31:16 +08:00
parent 4c8d581a5b
commit 3d3878c5aa
6 changed files with 199 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
runs/frontier-fidelity-envelope-v1/.gitignore vendored
View File

@@ -8,6 +8,7 @@ simulator-f1/
simulator-f1-steady/
simulator-f2/
simulator-t1/
simulator-t1-v4/
fleet-state/
fleet-artifacts/
profiles/profile-v3-batch/

45
runs/frontier-fidelity-envelope-v1/qwen235-v020-portability-card.md
View File

@@ -1,8 +1,8 @@
# EXP-SIMFID-Q235-V020-PORTABILITY: large FP8 MoE boundary case
> Status: compatibility gate prepared. No Qwen235 vLLM 0.20/deadc4a profile or
> real/simulator latency result exists yet; historical vLLM 0.10.2 SLO data is
> explicitly excluded.
> Status: initial compatibility failure diagnosed on 2026-07-18; the vLLM
> 0.20 profile path was subsequently repaired and the four-case matrix
> completed on 2026-07-19. Historical vLLM 0.10.2 SLO data remains excluded.
## Scope
@@ -39,3 +39,42 @@ the two plans above have existing historical memory/topology evidence.
This ordering prevents a profile-runtime mismatch from being misreported as a
simulator tuning failure.
## Initial result: the real plans ran before Frontier had an admissible counterpart
Both real vLLM 0.20 portability gates completed one exact Fixed-P request
(`ISL=2048`, `OSL=1`, prefix off) on H20. These are startup/semantic gates,
not estimates of mean or p90 latency.
| Real plan | Runtime-resolved MoE state | Completed request | TTFT / E2E |
|---|---|---:|---:|
| TP4/EP1, MNS64 | FP8 MoE `TRITON`, 128 experts per rank | 1/1 | 1655.99 / 1656.04 ms |
| TP8/EP8, MNS64 | FP8 MoE `FLASHINFER_CUTLASS`, 16/128 experts per EP rank | 1/1 | 720.54 / 720.58 ms |
The matching Frontier `deadc4a` profile gate failed before any GPU measurement:
1. `Qwen3-235B-A22B.json` exposes `torch_dtype=BF16` but no FP8
`quantization_config` / `[128,128]` block shape, while the checkpoint uses
dynamic block-FP8 weights. Frontier rejects the valid request as a profile
configuration mismatch.
2. Its MoE profiler then imports the removed vLLM symbol
`invoke_fused_moe_kernel`, so it is not ABI-compatible with the installed
community vLLM 0.20 runtime.
This was an admissible intermediate coverage failure. Filling it with
historical vLLM 0.10.2 profiles would have changed both the runtime and the
selected kernels and would not have answered the stated comparison question.
## Final result after profile repair
The later campaign added graph-compatible kernel-only decode profiles and true
mixed attention coverage, then completed all four workloads on the frozen
`{TP4/EP1, TP8/EP8} x MNS{64,128}` surface. All 48 real fresh-server trials
and all 16 Frontier cells completed. The decision-level selection regrets are
maintained in the consolidated
[Qwen30/Qwen235 report](../../docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md).
The decisive counterexample is Fixed-PD: Frontier's selected configuration is
33.0--37.2% slower for real TPOT and 30.7--34.6% slower for real E2E than the
real-hardware optimum. Thus repairing the coverage gate enabled a valid
comparison but did not establish tuning fidelity.

47
runs/frontier-fidelity-envelope-v1/qwen30-latency-expansion-card.md
View File

@@ -1,7 +1,8 @@
# EXP-SIMFID-Q30-LATENCY-EXPANSION: remaining Fixed/Trace × PD/P cases
> Status: prepared for runtime-alignment preflight (2026-07-18). The already
> completed Qwen3-30B-A3B Trace-PD surface is excluded from this card.
> Status: completed (2026-07-18). The low-load Fixed cases below are retained
> as sensitivity evidence; the later 4096-token, 1.125 req/s/GPU Fixed matrix
> is the primary comparison. Trace-PD is excluded from this card.
## Question and fixed boundary
@@ -47,3 +48,45 @@ metric is a coverage failure, not a high-latency cell.
The no-request Fixed runtime preflight is capped at 2 H20-GPUh. Each 36-run
real surface is estimated at 13 nominal / 41 worst-case H20-GPUh, plus CPU-only
Frontier replay. Launch logs record the resolved inputs, paths, and duration.
## Result: exact-selection fidelity is not stable across workload phase/state
All three real surfaces completed 36 valid fresh-server trials (12
TP×MNS cells × 3 trials), each with 129/129 exact-usage requests. The
corresponding 12-cell Frontier replays completed all 129 requests per cell.
The following table compares the simulator argmin with the pooled real argmin;
`regret` is the selected simulator configuration's real relative latency
regret. It deliberately reports exact winner equality rather than treating a
high pairwise-order agreement as success.
| Case | Objective | Frontier winner | Real winner | Match | Real regret |
|---|---|---|---|---:|---:|
| Trace-P (OSL=1) | TTFT mean | TP4/MNS16 | TP4/MNS64 | no | 3.2% |
| Trace-P (OSL=1) | TTFT p90 | TP4/MNS16 | TP4/MNS8 | no | 0.4% |
| Trace-P (OSL=1) | E2E mean | TP4/MNS16 | TP4/MNS64 | no | 3.2% |
| Trace-P (OSL=1) | E2E p90 | TP4/MNS16 | TP4/MNS8 | no | 0.3% |
| Fixed-PD | TTFT mean/p90 | TP4/MNS16 | TP4/MNS16 | yes | 0.0% |
| Fixed-PD | TPOT mean | TP1/MNS16 | TP4/MNS32 | no | 14.0% |
| Fixed-PD | TPOT p90 | TP1/MNS16 | TP4/MNS8 | no | 13.4% |
| Fixed-PD | E2E mean | TP1/MNS16 | TP4/MNS32 | no | 26.9% |
| Fixed-PD | E2E p90 | TP1/MNS16 | TP4/MNS8 | no | 25.7% |
| Fixed-P (OSL=1) | TTFT/E2E mean/p90 | TP4/MNS16 | TP4/MNS8 | no | 1.5--3.6% |
The decisive failure is Fixed-PD: its TPOT and E2E pairwise orders are exactly
reversed on all 48 informative pairs. Thus the graph-aligned profile can
preserve coarse TP ordering yet still choose the wrong configuration for the
serving objective. The prior Trace-PD success is therefore a workload-specific
success, not evidence that Frontier has solved config tuning.
## Superseding primary Fixed workload
The subsequent pressure sweep changed the Fixed shapes to 4096->256 and
4096->1, raised the offered load to 1.125 req/s/GPU, and used 257 requests per
cell. Those Fixed-PD/PO results, together with Trace-PD and Trace-PO, are
reported in the consolidated
[Qwen30/Qwen235 selection-regret report](../../docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md).
Under that primary Fixed-PD workload, Frontier's selected config is 58.0%
slower for real mean TTFT and 56.2% slower for real p90 TTFT than the
real-hardware optimum. The low-load table above must therefore not be quoted
as the final Fixed-PD result.