From 3d3878c5aaa7ef7772654c37ee400ea88d6e657e Mon Sep 17 00:00:00 2001 From: Gahow Wang Date: Sun, 19 Jul 2026 15:31:16 +0800 Subject: [PATCH] Summarize Frontier selection regret --- .gitignore | 22 ++++++ docs/simulator-tuning/README.md | 14 ++++ ...election-regret-qwen30-qwen235-20260719.md | 75 +++++++++++++++++++ runs/frontier-fidelity-envelope-v1/.gitignore | 1 + .../qwen235-v020-portability-card.md | 45 ++++++++++- .../qwen30-latency-expansion-card.md | 47 +++++++++++- 6 files changed, 199 insertions(+), 5 deletions(-) create mode 100644 docs/simulator-tuning/README.md create mode 100644 docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md diff --git a/.gitignore b/.gitignore index 344874d..b9d8ff4 100644 --- a/.gitignore +++ b/.gitignore @@ -19,3 +19,25 @@ runs/**/*.jsonl .ruff_cache/ # Recovered dash1 interaction-run stores (100 MB raw tune logs, kept on disk only) recovered-stores/ + +# Local reference material and accidental shell output. +/AITuner系统优化与挑战.pdf +/16 +/docs/assets/simulator-fidelity/*.svg + +# Generated experiment state. Protocols, analysis code, compact result tables, +# and frozen manifests remain tracked next to these directories. +/runs/frontier-phase-factorial-v0/fleet-artifacts*/ +/runs/frontier-phase-factorial-v0/fleet-state*/ +/runs/frontier-phase-factorial-v0/invalid-overlap-*/ +/runs/frontier-phase-factorial-v0/simulator-smoke/ +/runs/frontier-phase-factorial-v0/simulator-*/cache +/runs/frontier-phase-factorial-v0/simulator-*/runs/ +/runs/frontier-phase-factorial-v0/simulator-*/traces/ +/runs/frontier-phase-factorial-v0/results/final/qwen30-prefill-ranking.png +/runs/frontier-qwen30-vllm020-profile-v1/comparison/ +/runs/frontier-qwen30-vllm020-profile-v1/fleet-artifacts/ +/runs/frontier-qwen30-vllm020-profile-v1/fleet-state/ +/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/ +/runs/frontier-multicase-sufficiency-v1/fleet-state/ +/runs/frontier-multicase-sufficiency-v1/frontier-smoke-failure/ diff --git a/docs/simulator-tuning/README.md b/docs/simulator-tuning/README.md new file mode 100644 index 0000000..68e03ee --- /dev/null +++ b/docs/simulator-tuning/README.md @@ -0,0 +1,14 @@ +# Simulator tuning evaluation + +This directory contains decision-level summaries for experiments that compare +a serving simulator's selected configuration with the best configuration on +real hardware. + +Current report: + +- [Frontier selection regret on Qwen3-30B and Qwen3-235B](frontier-selection-regret-qwen30-qwen235-20260719.md) + +The primary quantity is **real-hardware selection regret**, not simulator +absolute-latency error. Raw commands, profiles, traces, and experiment-specific +audit records remain under `runs/` or in the immutable remote artifact roots +listed by each report. diff --git a/docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md b/docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md new file mode 100644 index 0000000..c869be6 --- /dev/null +++ b/docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md @@ -0,0 +1,75 @@ +# Frontier selection regret: Qwen3-30B and Qwen3-235B + +> Date: 2026-07-19 +> Scope: H20, community vLLM 0.20, Frontier piecewise simulation, no SLO gate + +## Question and metric + +For each workload and latency objective, Frontier selects the configuration +with the lowest simulated latency. We then look up that configuration on the +complete real-hardware surface and compare it with the real-hardware optimum. + +```text +selection regret = real_latency(Frontier winner) / real_latency(real winner) - 1 +``` + +Lower is better. `0%` means Frontier selected the real winner. Positive values +mean that following Frontier produces slower real serving. Each objective is +selected independently; this table does not combine TTFT, TPOT, and E2E into a +single score. + +## Qwen3-30B-A3B + +Configuration surface: `TP in {1,2,4} x MNS in {8,16,32,64}`, with +`MBT=8192`. Each real cell uses three fresh-server trials. + +| Workload | TTFT mean | TTFT p90 | TPOT mean | TPOT p90 | E2E mean | E2E p90 | +|---|---:|---:|---:|---:|---:|---:| +| Trace-PD | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | +| Fixed-PD, 4096->256, 1.125 req/s/GPU | **58.0%** | **56.2%** | 0.0% | 0.0% | 1.7% | 5.5% | +| Trace-PO, OSL=1 | 3.2% | 0.4% | N/A | N/A | 3.2% | 0.3% | +| Fixed-PO, 4096->1, 1.125 req/s/GPU | 0.3% | 0.5% | N/A | N/A | 0.3% | 0.5% | + +Interpretation: Frontier is near-optimal for Trace-PD and both prefill-only +cases, but the high-pressure Fixed-PD TTFT choice is materially wrong: its +selected configuration is 56--58% slower than the real TTFT optimum. + +## Qwen3-235B-A22B-FP8 + +Configuration surface: `{TP4/EP1, TP8/EP8} x MNS in {64,128}`, with +`MBT=8192`. Each workload has 129 requests per cell and each real cell uses +three fresh-server trials. + +| Workload | TTFT mean | TTFT p90 | TPOT mean | TPOT p90 | E2E mean | E2E p90 | +|---|---:|---:|---:|---:|---:|---:| +| Trace-PD | 0.0% | 0.0% | 0.0% | 0.0% | 0.6% | 6.2% | +| Fixed-PD, 4096->256, 0.2 req/s/GPU | 4.2% | 0.2% | **33.0%** | **37.2%** | **30.7%** | **34.6%** | +| Trace-PO, OSL=1 | 7.0% | **21.2%** | N/A | N/A | 7.0% | **21.2%** | +| Fixed-PO, 4096->1, 0.2 req/s/GPU | 5.9% | 1.7% | N/A | N/A | 5.9% | 1.7% | + +Interpretation: Trace-PD is mostly near-optimal. Fixed-PD reverses the real +decode/E2E preference between the tested parallel configurations and incurs +31--37% regret. Trace-PO also has a material p90 failure of 21.2%. + +## Decision + +The tested Frontier stack has **not** solved serving configuration tuning. +Its selected configuration can be near-optimal for one workload and materially +wrong for another on the same model and hardware. The strongest current +counterexamples are Qwen3-30B Fixed-PD TTFT and Qwen3-235B Fixed-PD TPOT/E2E. + +This statement is limited to the two tested MoE models and Frontier. It is not +yet evidence about dense models, Vidur/APEX as separately reproduced systems, +other hardware, or SLO-constrained tuning. + +## Provenance + +Primary immutable analysis artifacts on `dash0`: + +- Qwen3-30B Trace-PD: `/home/admin/cpfs/wjh/aituner/graph-piecewise-qwen30-20260717/simulator-piecewise-surface-v2/analysis/comparison.json` +- Qwen3-30B Fixed-PD/PO: `/home/admin/cpfs/wjh/aituner/qwen30-fixed-pressure-surface-20260719-r1/analysis/` +- Qwen3-30B Trace-PO: `/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/analysis-r6/trace-po-comparison.json` +- Qwen3-235B four-case matrix: `/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/analysis/comparison.json` + +The Qwen3-235B artifact root includes `provenance/artifacts.sha256`; the final +matrix contains 48/48 valid real trials and 16/16 complete simulator cells. diff --git a/runs/frontier-fidelity-envelope-v1/.gitignore b/runs/frontier-fidelity-envelope-v1/.gitignore index 4de000f..3897191 100644 --- a/runs/frontier-fidelity-envelope-v1/.gitignore +++ b/runs/frontier-fidelity-envelope-v1/.gitignore @@ -8,6 +8,7 @@ simulator-f1/ simulator-f1-steady/ simulator-f2/ simulator-t1/ +simulator-t1-v4/ fleet-state/ fleet-artifacts/ profiles/profile-v3-batch/ diff --git a/runs/frontier-fidelity-envelope-v1/qwen235-v020-portability-card.md b/runs/frontier-fidelity-envelope-v1/qwen235-v020-portability-card.md index 159e7de..5c11256 100644 --- a/runs/frontier-fidelity-envelope-v1/qwen235-v020-portability-card.md +++ b/runs/frontier-fidelity-envelope-v1/qwen235-v020-portability-card.md @@ -1,8 +1,8 @@ # EXP-SIMFID-Q235-V020-PORTABILITY: large FP8 MoE boundary case -> Status: compatibility gate prepared. No Qwen235 vLLM 0.20/deadc4a profile or -> real/simulator latency result exists yet; historical vLLM 0.10.2 SLO data is -> explicitly excluded. +> Status: initial compatibility failure diagnosed on 2026-07-18; the vLLM +> 0.20 profile path was subsequently repaired and the four-case matrix +> completed on 2026-07-19. Historical vLLM 0.10.2 SLO data remains excluded. ## Scope @@ -39,3 +39,42 @@ the two plans above have existing historical memory/topology evidence. This ordering prevents a profile-runtime mismatch from being misreported as a simulator tuning failure. + +## Initial result: the real plans ran before Frontier had an admissible counterpart + +Both real vLLM 0.20 portability gates completed one exact Fixed-P request +(`ISL=2048`, `OSL=1`, prefix off) on H20. These are startup/semantic gates, +not estimates of mean or p90 latency. + +| Real plan | Runtime-resolved MoE state | Completed request | TTFT / E2E | +|---|---|---:|---:| +| TP4/EP1, MNS64 | FP8 MoE `TRITON`, 128 experts per rank | 1/1 | 1655.99 / 1656.04 ms | +| TP8/EP8, MNS64 | FP8 MoE `FLASHINFER_CUTLASS`, 16/128 experts per EP rank | 1/1 | 720.54 / 720.58 ms | + +The matching Frontier `deadc4a` profile gate failed before any GPU measurement: + +1. `Qwen3-235B-A22B.json` exposes `torch_dtype=BF16` but no FP8 + `quantization_config` / `[128,128]` block shape, while the checkpoint uses + dynamic block-FP8 weights. Frontier rejects the valid request as a profile + configuration mismatch. +2. Its MoE profiler then imports the removed vLLM symbol + `invoke_fused_moe_kernel`, so it is not ABI-compatible with the installed + community vLLM 0.20 runtime. + +This was an admissible intermediate coverage failure. Filling it with +historical vLLM 0.10.2 profiles would have changed both the runtime and the +selected kernels and would not have answered the stated comparison question. + +## Final result after profile repair + +The later campaign added graph-compatible kernel-only decode profiles and true +mixed attention coverage, then completed all four workloads on the frozen +`{TP4/EP1, TP8/EP8} x MNS{64,128}` surface. All 48 real fresh-server trials +and all 16 Frontier cells completed. The decision-level selection regrets are +maintained in the consolidated +[Qwen30/Qwen235 report](../../docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md). + +The decisive counterexample is Fixed-PD: Frontier's selected configuration is +33.0--37.2% slower for real TPOT and 30.7--34.6% slower for real E2E than the +real-hardware optimum. Thus repairing the coverage gate enabled a valid +comparison but did not establish tuning fidelity. diff --git a/runs/frontier-fidelity-envelope-v1/qwen30-latency-expansion-card.md b/runs/frontier-fidelity-envelope-v1/qwen30-latency-expansion-card.md index 8ed2b8c..cdccf34 100644 --- a/runs/frontier-fidelity-envelope-v1/qwen30-latency-expansion-card.md +++ b/runs/frontier-fidelity-envelope-v1/qwen30-latency-expansion-card.md @@ -1,7 +1,8 @@ # EXP-SIMFID-Q30-LATENCY-EXPANSION: remaining Fixed/Trace × PD/P cases -> Status: prepared for runtime-alignment preflight (2026-07-18). The already -> completed Qwen3-30B-A3B Trace-PD surface is excluded from this card. +> Status: completed (2026-07-18). The low-load Fixed cases below are retained +> as sensitivity evidence; the later 4096-token, 1.125 req/s/GPU Fixed matrix +> is the primary comparison. Trace-PD is excluded from this card. ## Question and fixed boundary @@ -47,3 +48,45 @@ metric is a coverage failure, not a high-latency cell. The no-request Fixed runtime preflight is capped at 2 H20-GPUh. Each 36-run real surface is estimated at 13 nominal / 41 worst-case H20-GPUh, plus CPU-only Frontier replay. Launch logs record the resolved inputs, paths, and duration. + +## Result: exact-selection fidelity is not stable across workload phase/state + +All three real surfaces completed 36 valid fresh-server trials (12 +TP×MNS cells × 3 trials), each with 129/129 exact-usage requests. The +corresponding 12-cell Frontier replays completed all 129 requests per cell. +The following table compares the simulator argmin with the pooled real argmin; +`regret` is the selected simulator configuration's real relative latency +regret. It deliberately reports exact winner equality rather than treating a +high pairwise-order agreement as success. + +| Case | Objective | Frontier winner | Real winner | Match | Real regret | +|---|---|---|---|---:|---:| +| Trace-P (OSL=1) | TTFT mean | TP4/MNS16 | TP4/MNS64 | no | 3.2% | +| Trace-P (OSL=1) | TTFT p90 | TP4/MNS16 | TP4/MNS8 | no | 0.4% | +| Trace-P (OSL=1) | E2E mean | TP4/MNS16 | TP4/MNS64 | no | 3.2% | +| Trace-P (OSL=1) | E2E p90 | TP4/MNS16 | TP4/MNS8 | no | 0.3% | +| Fixed-PD | TTFT mean/p90 | TP4/MNS16 | TP4/MNS16 | yes | 0.0% | +| Fixed-PD | TPOT mean | TP1/MNS16 | TP4/MNS32 | no | 14.0% | +| Fixed-PD | TPOT p90 | TP1/MNS16 | TP4/MNS8 | no | 13.4% | +| Fixed-PD | E2E mean | TP1/MNS16 | TP4/MNS32 | no | 26.9% | +| Fixed-PD | E2E p90 | TP1/MNS16 | TP4/MNS8 | no | 25.7% | +| Fixed-P (OSL=1) | TTFT/E2E mean/p90 | TP4/MNS16 | TP4/MNS8 | no | 1.5--3.6% | + +The decisive failure is Fixed-PD: its TPOT and E2E pairwise orders are exactly +reversed on all 48 informative pairs. Thus the graph-aligned profile can +preserve coarse TP ordering yet still choose the wrong configuration for the +serving objective. The prior Trace-PD success is therefore a workload-specific +success, not evidence that Frontier has solved config tuning. + +## Superseding primary Fixed workload + +The subsequent pressure sweep changed the Fixed shapes to 4096->256 and +4096->1, raised the offered load to 1.125 req/s/GPU, and used 257 requests per +cell. Those Fixed-PD/PO results, together with Trace-PD and Trace-PO, are +reported in the consolidated +[Qwen30/Qwen235 selection-regret report](../../docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md). + +Under that primary Fixed-PD workload, Frontier's selected config is 58.0% +slower for real mean TTFT and 56.2% slower for real p90 TTFT than the +real-hardware optimum. The low-load table above must therefore not be quoted +as the final Fixed-PD result.