Summarize Frontier selection regret
This commit is contained in:
22
.gitignore
vendored
22
.gitignore
vendored
@@ -19,3 +19,25 @@ runs/**/*.jsonl
|
||||
.ruff_cache/
|
||||
# Recovered dash1 interaction-run stores (100 MB raw tune logs, kept on disk only)
|
||||
recovered-stores/
|
||||
|
||||
# Local reference material and accidental shell output.
|
||||
/AITuner系统优化与挑战.pdf
|
||||
/16
|
||||
/docs/assets/simulator-fidelity/*.svg
|
||||
|
||||
# Generated experiment state. Protocols, analysis code, compact result tables,
|
||||
# and frozen manifests remain tracked next to these directories.
|
||||
/runs/frontier-phase-factorial-v0/fleet-artifacts*/
|
||||
/runs/frontier-phase-factorial-v0/fleet-state*/
|
||||
/runs/frontier-phase-factorial-v0/invalid-overlap-*/
|
||||
/runs/frontier-phase-factorial-v0/simulator-smoke/
|
||||
/runs/frontier-phase-factorial-v0/simulator-*/cache
|
||||
/runs/frontier-phase-factorial-v0/simulator-*/runs/
|
||||
/runs/frontier-phase-factorial-v0/simulator-*/traces/
|
||||
/runs/frontier-phase-factorial-v0/results/final/qwen30-prefill-ranking.png
|
||||
/runs/frontier-qwen30-vllm020-profile-v1/comparison/
|
||||
/runs/frontier-qwen30-vllm020-profile-v1/fleet-artifacts/
|
||||
/runs/frontier-qwen30-vllm020-profile-v1/fleet-state/
|
||||
/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/
|
||||
/runs/frontier-multicase-sufficiency-v1/fleet-state/
|
||||
/runs/frontier-multicase-sufficiency-v1/frontier-smoke-failure/
|
||||
|
||||
14
docs/simulator-tuning/README.md
Normal file
14
docs/simulator-tuning/README.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Simulator tuning evaluation
|
||||
|
||||
This directory contains decision-level summaries for experiments that compare
|
||||
a serving simulator's selected configuration with the best configuration on
|
||||
real hardware.
|
||||
|
||||
Current report:
|
||||
|
||||
- [Frontier selection regret on Qwen3-30B and Qwen3-235B](frontier-selection-regret-qwen30-qwen235-20260719.md)
|
||||
|
||||
The primary quantity is **real-hardware selection regret**, not simulator
|
||||
absolute-latency error. Raw commands, profiles, traces, and experiment-specific
|
||||
audit records remain under `runs/` or in the immutable remote artifact roots
|
||||
listed by each report.
|
||||
@@ -0,0 +1,75 @@
|
||||
# Frontier selection regret: Qwen3-30B and Qwen3-235B
|
||||
|
||||
> Date: 2026-07-19
|
||||
> Scope: H20, community vLLM 0.20, Frontier piecewise simulation, no SLO gate
|
||||
|
||||
## Question and metric
|
||||
|
||||
For each workload and latency objective, Frontier selects the configuration
|
||||
with the lowest simulated latency. We then look up that configuration on the
|
||||
complete real-hardware surface and compare it with the real-hardware optimum.
|
||||
|
||||
```text
|
||||
selection regret = real_latency(Frontier winner) / real_latency(real winner) - 1
|
||||
```
|
||||
|
||||
Lower is better. `0%` means Frontier selected the real winner. Positive values
|
||||
mean that following Frontier produces slower real serving. Each objective is
|
||||
selected independently; this table does not combine TTFT, TPOT, and E2E into a
|
||||
single score.
|
||||
|
||||
## Qwen3-30B-A3B
|
||||
|
||||
Configuration surface: `TP in {1,2,4} x MNS in {8,16,32,64}`, with
|
||||
`MBT=8192`. Each real cell uses three fresh-server trials.
|
||||
|
||||
| Workload | TTFT mean | TTFT p90 | TPOT mean | TPOT p90 | E2E mean | E2E p90 |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| Trace-PD | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
|
||||
| Fixed-PD, 4096->256, 1.125 req/s/GPU | **58.0%** | **56.2%** | 0.0% | 0.0% | 1.7% | 5.5% |
|
||||
| Trace-PO, OSL=1 | 3.2% | 0.4% | N/A | N/A | 3.2% | 0.3% |
|
||||
| Fixed-PO, 4096->1, 1.125 req/s/GPU | 0.3% | 0.5% | N/A | N/A | 0.3% | 0.5% |
|
||||
|
||||
Interpretation: Frontier is near-optimal for Trace-PD and both prefill-only
|
||||
cases, but the high-pressure Fixed-PD TTFT choice is materially wrong: its
|
||||
selected configuration is 56--58% slower than the real TTFT optimum.
|
||||
|
||||
## Qwen3-235B-A22B-FP8
|
||||
|
||||
Configuration surface: `{TP4/EP1, TP8/EP8} x MNS in {64,128}`, with
|
||||
`MBT=8192`. Each workload has 129 requests per cell and each real cell uses
|
||||
three fresh-server trials.
|
||||
|
||||
| Workload | TTFT mean | TTFT p90 | TPOT mean | TPOT p90 | E2E mean | E2E p90 |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| Trace-PD | 0.0% | 0.0% | 0.0% | 0.0% | 0.6% | 6.2% |
|
||||
| Fixed-PD, 4096->256, 0.2 req/s/GPU | 4.2% | 0.2% | **33.0%** | **37.2%** | **30.7%** | **34.6%** |
|
||||
| Trace-PO, OSL=1 | 7.0% | **21.2%** | N/A | N/A | 7.0% | **21.2%** |
|
||||
| Fixed-PO, 4096->1, 0.2 req/s/GPU | 5.9% | 1.7% | N/A | N/A | 5.9% | 1.7% |
|
||||
|
||||
Interpretation: Trace-PD is mostly near-optimal. Fixed-PD reverses the real
|
||||
decode/E2E preference between the tested parallel configurations and incurs
|
||||
31--37% regret. Trace-PO also has a material p90 failure of 21.2%.
|
||||
|
||||
## Decision
|
||||
|
||||
The tested Frontier stack has **not** solved serving configuration tuning.
|
||||
Its selected configuration can be near-optimal for one workload and materially
|
||||
wrong for another on the same model and hardware. The strongest current
|
||||
counterexamples are Qwen3-30B Fixed-PD TTFT and Qwen3-235B Fixed-PD TPOT/E2E.
|
||||
|
||||
This statement is limited to the two tested MoE models and Frontier. It is not
|
||||
yet evidence about dense models, Vidur/APEX as separately reproduced systems,
|
||||
other hardware, or SLO-constrained tuning.
|
||||
|
||||
## Provenance
|
||||
|
||||
Primary immutable analysis artifacts on `dash0`:
|
||||
|
||||
- Qwen3-30B Trace-PD: `/home/admin/cpfs/wjh/aituner/graph-piecewise-qwen30-20260717/simulator-piecewise-surface-v2/analysis/comparison.json`
|
||||
- Qwen3-30B Fixed-PD/PO: `/home/admin/cpfs/wjh/aituner/qwen30-fixed-pressure-surface-20260719-r1/analysis/`
|
||||
- Qwen3-30B Trace-PO: `/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/analysis-r6/trace-po-comparison.json`
|
||||
- Qwen3-235B four-case matrix: `/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/analysis/comparison.json`
|
||||
|
||||
The Qwen3-235B artifact root includes `provenance/artifacts.sha256`; the final
|
||||
matrix contains 48/48 valid real trials and 16/16 complete simulator cells.
|
||||
@@ -8,6 +8,7 @@ simulator-f1/
|
||||
simulator-f1-steady/
|
||||
simulator-f2/
|
||||
simulator-t1/
|
||||
simulator-t1-v4/
|
||||
fleet-state/
|
||||
fleet-artifacts/
|
||||
profiles/profile-v3-batch/
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# EXP-SIMFID-Q235-V020-PORTABILITY: large FP8 MoE boundary case
|
||||
|
||||
> Status: compatibility gate prepared. No Qwen235 vLLM 0.20/deadc4a profile or
|
||||
> real/simulator latency result exists yet; historical vLLM 0.10.2 SLO data is
|
||||
> explicitly excluded.
|
||||
> Status: initial compatibility failure diagnosed on 2026-07-18; the vLLM
|
||||
> 0.20 profile path was subsequently repaired and the four-case matrix
|
||||
> completed on 2026-07-19. Historical vLLM 0.10.2 SLO data remains excluded.
|
||||
|
||||
## Scope
|
||||
|
||||
@@ -39,3 +39,42 @@ the two plans above have existing historical memory/topology evidence.
|
||||
|
||||
This ordering prevents a profile-runtime mismatch from being misreported as a
|
||||
simulator tuning failure.
|
||||
|
||||
## Initial result: the real plans ran before Frontier had an admissible counterpart
|
||||
|
||||
Both real vLLM 0.20 portability gates completed one exact Fixed-P request
|
||||
(`ISL=2048`, `OSL=1`, prefix off) on H20. These are startup/semantic gates,
|
||||
not estimates of mean or p90 latency.
|
||||
|
||||
| Real plan | Runtime-resolved MoE state | Completed request | TTFT / E2E |
|
||||
|---|---|---:|---:|
|
||||
| TP4/EP1, MNS64 | FP8 MoE `TRITON`, 128 experts per rank | 1/1 | 1655.99 / 1656.04 ms |
|
||||
| TP8/EP8, MNS64 | FP8 MoE `FLASHINFER_CUTLASS`, 16/128 experts per EP rank | 1/1 | 720.54 / 720.58 ms |
|
||||
|
||||
The matching Frontier `deadc4a` profile gate failed before any GPU measurement:
|
||||
|
||||
1. `Qwen3-235B-A22B.json` exposes `torch_dtype=BF16` but no FP8
|
||||
`quantization_config` / `[128,128]` block shape, while the checkpoint uses
|
||||
dynamic block-FP8 weights. Frontier rejects the valid request as a profile
|
||||
configuration mismatch.
|
||||
2. Its MoE profiler then imports the removed vLLM symbol
|
||||
`invoke_fused_moe_kernel`, so it is not ABI-compatible with the installed
|
||||
community vLLM 0.20 runtime.
|
||||
|
||||
This was an admissible intermediate coverage failure. Filling it with
|
||||
historical vLLM 0.10.2 profiles would have changed both the runtime and the
|
||||
selected kernels and would not have answered the stated comparison question.
|
||||
|
||||
## Final result after profile repair
|
||||
|
||||
The later campaign added graph-compatible kernel-only decode profiles and true
|
||||
mixed attention coverage, then completed all four workloads on the frozen
|
||||
`{TP4/EP1, TP8/EP8} x MNS{64,128}` surface. All 48 real fresh-server trials
|
||||
and all 16 Frontier cells completed. The decision-level selection regrets are
|
||||
maintained in the consolidated
|
||||
[Qwen30/Qwen235 report](../../docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md).
|
||||
|
||||
The decisive counterexample is Fixed-PD: Frontier's selected configuration is
|
||||
33.0--37.2% slower for real TPOT and 30.7--34.6% slower for real E2E than the
|
||||
real-hardware optimum. Thus repairing the coverage gate enabled a valid
|
||||
comparison but did not establish tuning fidelity.
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
# EXP-SIMFID-Q30-LATENCY-EXPANSION: remaining Fixed/Trace × PD/P cases
|
||||
|
||||
> Status: prepared for runtime-alignment preflight (2026-07-18). The already
|
||||
> completed Qwen3-30B-A3B Trace-PD surface is excluded from this card.
|
||||
> Status: completed (2026-07-18). The low-load Fixed cases below are retained
|
||||
> as sensitivity evidence; the later 4096-token, 1.125 req/s/GPU Fixed matrix
|
||||
> is the primary comparison. Trace-PD is excluded from this card.
|
||||
|
||||
## Question and fixed boundary
|
||||
|
||||
@@ -47,3 +48,45 @@ metric is a coverage failure, not a high-latency cell.
|
||||
The no-request Fixed runtime preflight is capped at 2 H20-GPUh. Each 36-run
|
||||
real surface is estimated at 13 nominal / 41 worst-case H20-GPUh, plus CPU-only
|
||||
Frontier replay. Launch logs record the resolved inputs, paths, and duration.
|
||||
|
||||
## Result: exact-selection fidelity is not stable across workload phase/state
|
||||
|
||||
All three real surfaces completed 36 valid fresh-server trials (12
|
||||
TP×MNS cells × 3 trials), each with 129/129 exact-usage requests. The
|
||||
corresponding 12-cell Frontier replays completed all 129 requests per cell.
|
||||
The following table compares the simulator argmin with the pooled real argmin;
|
||||
`regret` is the selected simulator configuration's real relative latency
|
||||
regret. It deliberately reports exact winner equality rather than treating a
|
||||
high pairwise-order agreement as success.
|
||||
|
||||
| Case | Objective | Frontier winner | Real winner | Match | Real regret |
|
||||
|---|---|---|---|---:|---:|
|
||||
| Trace-P (OSL=1) | TTFT mean | TP4/MNS16 | TP4/MNS64 | no | 3.2% |
|
||||
| Trace-P (OSL=1) | TTFT p90 | TP4/MNS16 | TP4/MNS8 | no | 0.4% |
|
||||
| Trace-P (OSL=1) | E2E mean | TP4/MNS16 | TP4/MNS64 | no | 3.2% |
|
||||
| Trace-P (OSL=1) | E2E p90 | TP4/MNS16 | TP4/MNS8 | no | 0.3% |
|
||||
| Fixed-PD | TTFT mean/p90 | TP4/MNS16 | TP4/MNS16 | yes | 0.0% |
|
||||
| Fixed-PD | TPOT mean | TP1/MNS16 | TP4/MNS32 | no | 14.0% |
|
||||
| Fixed-PD | TPOT p90 | TP1/MNS16 | TP4/MNS8 | no | 13.4% |
|
||||
| Fixed-PD | E2E mean | TP1/MNS16 | TP4/MNS32 | no | 26.9% |
|
||||
| Fixed-PD | E2E p90 | TP1/MNS16 | TP4/MNS8 | no | 25.7% |
|
||||
| Fixed-P (OSL=1) | TTFT/E2E mean/p90 | TP4/MNS16 | TP4/MNS8 | no | 1.5--3.6% |
|
||||
|
||||
The decisive failure is Fixed-PD: its TPOT and E2E pairwise orders are exactly
|
||||
reversed on all 48 informative pairs. Thus the graph-aligned profile can
|
||||
preserve coarse TP ordering yet still choose the wrong configuration for the
|
||||
serving objective. The prior Trace-PD success is therefore a workload-specific
|
||||
success, not evidence that Frontier has solved config tuning.
|
||||
|
||||
## Superseding primary Fixed workload
|
||||
|
||||
The subsequent pressure sweep changed the Fixed shapes to 4096->256 and
|
||||
4096->1, raised the offered load to 1.125 req/s/GPU, and used 257 requests per
|
||||
cell. Those Fixed-PD/PO results, together with Trace-PD and Trace-PO, are
|
||||
reported in the consolidated
|
||||
[Qwen30/Qwen235 selection-regret report](../../docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md).
|
||||
|
||||
Under that primary Fixed-PD workload, Frontier's selected config is 58.0%
|
||||
slower for real mean TTFT and 56.2% slower for real p90 TTFT than the
|
||||
real-hardware optimum. The low-load table above must therefore not be quoted
|
||||
as the final Fixed-PD result.
|
||||
|
||||
Reference in New Issue
Block a user