Summarize Frontier selection regret

.gitignore

@@ -19,3 +19,25 @@ runs/**/*.jsonl
 .ruff_cache/
 # Recovered dash1 interaction-run stores (100 MB raw tune logs, kept on disk only)
 recovered-stores/
+
+# Local reference material and accidental shell output.
+/AITuner系统优化与挑战.pdf
+/16
+/docs/assets/simulator-fidelity/*.svg
+
+# Generated experiment state. Protocols, analysis code, compact result tables,
+# and frozen manifests remain tracked next to these directories.
+/runs/frontier-phase-factorial-v0/fleet-artifacts*/
+/runs/frontier-phase-factorial-v0/fleet-state*/
+/runs/frontier-phase-factorial-v0/invalid-overlap-*/
+/runs/frontier-phase-factorial-v0/simulator-smoke/
+/runs/frontier-phase-factorial-v0/simulator-*/cache
+/runs/frontier-phase-factorial-v0/simulator-*/runs/
+/runs/frontier-phase-factorial-v0/simulator-*/traces/
+/runs/frontier-phase-factorial-v0/results/final/qwen30-prefill-ranking.png
+/runs/frontier-qwen30-vllm020-profile-v1/comparison/
+/runs/frontier-qwen30-vllm020-profile-v1/fleet-artifacts/
+/runs/frontier-qwen30-vllm020-profile-v1/fleet-state/
+/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/
+/runs/frontier-multicase-sufficiency-v1/fleet-state/
+/runs/frontier-multicase-sufficiency-v1/frontier-smoke-failure/

docs/simulator-tuning/README.md

@@ -0,0 +1,14 @@
+# Simulator tuning evaluation
+
+This directory contains decision-level summaries for experiments that compare
+a serving simulator's selected configuration with the best configuration on
+real hardware.
+
+Current report:
+
+- [Frontier selection regret on Qwen3-30B and Qwen3-235B](frontier-selection-regret-qwen30-qwen235-20260719.md)
+
+The primary quantity is **real-hardware selection regret**, not simulator
+absolute-latency error. Raw commands, profiles, traces, and experiment-specific
+audit records remain under `runs/` or in the immutable remote artifact roots
+listed by each report.

docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md

@@ -0,0 +1,75 @@
+# Frontier selection regret: Qwen3-30B and Qwen3-235B
+
+> Date: 2026-07-19
+> Scope: H20, community vLLM 0.20, Frontier piecewise simulation, no SLO gate
+
+## Question and metric
+
+For each workload and latency objective, Frontier selects the configuration
+with the lowest simulated latency. We then look up that configuration on the
+complete real-hardware surface and compare it with the real-hardware optimum.
+
+```text
+selection regret = real_latency(Frontier winner) / real_latency(real winner) - 1
+```
+
+Lower is better. `0%` means Frontier selected the real winner. Positive values
+mean that following Frontier produces slower real serving. Each objective is
+selected independently; this table does not combine TTFT, TPOT, and E2E into a
+single score.
+
+## Qwen3-30B-A3B
+
+Configuration surface: `TP in {1,2,4} x MNS in {8,16,32,64}`, with
+`MBT=8192`. Each real cell uses three fresh-server trials.
+
+| Workload | TTFT mean | TTFT p90 | TPOT mean | TPOT p90 | E2E mean | E2E p90 |
+|---|---:|---:|---:|---:|---:|---:|
+| Trace-PD | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
+| Fixed-PD, 4096->256, 1.125 req/s/GPU | **58.0%** | **56.2%** | 0.0% | 0.0% | 1.7% | 5.5% |
+| Trace-PO, OSL=1 | 3.2% | 0.4% | N/A | N/A | 3.2% | 0.3% |
+| Fixed-PO, 4096->1, 1.125 req/s/GPU | 0.3% | 0.5% | N/A | N/A | 0.3% | 0.5% |
+
+Interpretation: Frontier is near-optimal for Trace-PD and both prefill-only
+cases, but the high-pressure Fixed-PD TTFT choice is materially wrong: its
+selected configuration is 56--58% slower than the real TTFT optimum.
+
+## Qwen3-235B-A22B-FP8
+
+Configuration surface: `{TP4/EP1, TP8/EP8} x MNS in {64,128}`, with
+`MBT=8192`. Each workload has 129 requests per cell and each real cell uses
+three fresh-server trials.
+
+| Workload | TTFT mean | TTFT p90 | TPOT mean | TPOT p90 | E2E mean | E2E p90 |
+|---|---:|---:|---:|---:|---:|---:|
+| Trace-PD | 0.0% | 0.0% | 0.0% | 0.0% | 0.6% | 6.2% |
+| Fixed-PD, 4096->256, 0.2 req/s/GPU | 4.2% | 0.2% | **33.0%** | **37.2%** | **30.7%** | **34.6%** |
+| Trace-PO, OSL=1 | 7.0% | **21.2%** | N/A | N/A | 7.0% | **21.2%** |
+| Fixed-PO, 4096->1, 0.2 req/s/GPU | 5.9% | 1.7% | N/A | N/A | 5.9% | 1.7% |
+
+Interpretation: Trace-PD is mostly near-optimal. Fixed-PD reverses the real
+decode/E2E preference between the tested parallel configurations and incurs
+31--37% regret. Trace-PO also has a material p90 failure of 21.2%.
+
+## Decision
+
+The tested Frontier stack has **not** solved serving configuration tuning.
+Its selected configuration can be near-optimal for one workload and materially
+wrong for another on the same model and hardware. The strongest current
+counterexamples are Qwen3-30B Fixed-PD TTFT and Qwen3-235B Fixed-PD TPOT/E2E.
+
+This statement is limited to the two tested MoE models and Frontier. It is not
+yet evidence about dense models, Vidur/APEX as separately reproduced systems,
+other hardware, or SLO-constrained tuning.
+
+## Provenance
+
+Primary immutable analysis artifacts on `dash0`:
+
+- Qwen3-30B Trace-PD: `/home/admin/cpfs/wjh/aituner/graph-piecewise-qwen30-20260717/simulator-piecewise-surface-v2/analysis/comparison.json`
+- Qwen3-30B Fixed-PD/PO: `/home/admin/cpfs/wjh/aituner/qwen30-fixed-pressure-surface-20260719-r1/analysis/`
+- Qwen3-30B Trace-PO: `/home/admin/cpfs/wjh/aituner/qwen30-latency-expansion-20260718-r2/analysis-r6/trace-po-comparison.json`
+- Qwen3-235B four-case matrix: `/home/admin/cpfs/wjh/aituner/qwen235-v020-fourcase-20260719-r1/analysis/comparison.json`
+
+The Qwen3-235B artifact root includes `provenance/artifacts.sha256`; the final
+matrix contains 48/48 valid real trials and 16/16 complete simulator cells.

runs/frontier-fidelity-envelope-v1/.gitignore

@@ -8,6 +8,7 @@ simulator-f1/
 simulator-f1-steady/
 simulator-f2/
 simulator-t1/
+simulator-t1-v4/
 fleet-state/
 fleet-artifacts/
 profiles/profile-v3-batch/

runs/frontier-fidelity-envelope-v1/qwen235-v020-portability-card.md

@@ -1,8 +1,8 @@
 # EXP-SIMFID-Q235-V020-PORTABILITY: large FP8 MoE boundary case
 
-> Status: compatibility gate prepared. No Qwen235 vLLM 0.20/deadc4a profile or
+> Status: initial compatibility failure diagnosed on 2026-07-18; the vLLM
-> real/simulator latency result exists yet; historical vLLM 0.10.2 SLO data is
+> 0.20 profile path was subsequently repaired and the four-case matrix
-> explicitly excluded.
+> completed on 2026-07-19. Historical vLLM 0.10.2 SLO data remains excluded.
 
 ## Scope
 
@@ -39,3 +39,42 @@ the two plans above have existing historical memory/topology evidence.
 
 This ordering prevents a profile-runtime mismatch from being misreported as a
 simulator tuning failure.
+
+## Initial result: the real plans ran before Frontier had an admissible counterpart
+
+Both real vLLM 0.20 portability gates completed one exact Fixed-P request
+(`ISL=2048`, `OSL=1`, prefix off) on H20.  These are startup/semantic gates,
+not estimates of mean or p90 latency.
+
+| Real plan | Runtime-resolved MoE state | Completed request | TTFT / E2E |
+|---|---|---:|---:|
+| TP4/EP1, MNS64 | FP8 MoE `TRITON`, 128 experts per rank | 1/1 | 1655.99 / 1656.04 ms |
+| TP8/EP8, MNS64 | FP8 MoE `FLASHINFER_CUTLASS`, 16/128 experts per EP rank | 1/1 | 720.54 / 720.58 ms |
+
+The matching Frontier `deadc4a` profile gate failed before any GPU measurement:
+
+1. `Qwen3-235B-A22B.json` exposes `torch_dtype=BF16` but no FP8
+   `quantization_config` / `[128,128]` block shape, while the checkpoint uses
+   dynamic block-FP8 weights.  Frontier rejects the valid request as a profile
+   configuration mismatch.
+2. Its MoE profiler then imports the removed vLLM symbol
+   `invoke_fused_moe_kernel`, so it is not ABI-compatible with the installed
+   community vLLM 0.20 runtime.
+
+This was an admissible intermediate coverage failure. Filling it with
+historical vLLM 0.10.2 profiles would have changed both the runtime and the
+selected kernels and would not have answered the stated comparison question.
+
+## Final result after profile repair
+
+The later campaign added graph-compatible kernel-only decode profiles and true
+mixed attention coverage, then completed all four workloads on the frozen
+`{TP4/EP1, TP8/EP8} x MNS{64,128}` surface. All 48 real fresh-server trials
+and all 16 Frontier cells completed. The decision-level selection regrets are
+maintained in the consolidated
+[Qwen30/Qwen235 report](../../docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md).
+
+The decisive counterexample is Fixed-PD: Frontier's selected configuration is
+33.0--37.2% slower for real TPOT and 30.7--34.6% slower for real E2E than the
+real-hardware optimum. Thus repairing the coverage gate enabled a valid
+comparison but did not establish tuning fidelity.

runs/frontier-fidelity-envelope-v1/qwen30-latency-expansion-card.md

@@ -1,7 +1,8 @@
 # EXP-SIMFID-Q30-LATENCY-EXPANSION: remaining Fixed/Trace × PD/P cases
 
-> Status: prepared for runtime-alignment preflight (2026-07-18). The already
+> Status: completed (2026-07-18). The low-load Fixed cases below are retained
-> completed Qwen3-30B-A3B Trace-PD surface is excluded from this card.
+> as sensitivity evidence; the later 4096-token, 1.125 req/s/GPU Fixed matrix
+> is the primary comparison. Trace-PD is excluded from this card.
 
 ## Question and fixed boundary
 
@@ -47,3 +48,45 @@ metric is a coverage failure, not a high-latency cell.
 The no-request Fixed runtime preflight is capped at 2 H20-GPUh. Each 36-run
 real surface is estimated at 13 nominal / 41 worst-case H20-GPUh, plus CPU-only
 Frontier replay. Launch logs record the resolved inputs, paths, and duration.
+
+## Result: exact-selection fidelity is not stable across workload phase/state
+
+All three real surfaces completed 36 valid fresh-server trials (12
+TP×MNS cells × 3 trials), each with 129/129 exact-usage requests.  The
+corresponding 12-cell Frontier replays completed all 129 requests per cell.
+The following table compares the simulator argmin with the pooled real argmin;
+`regret` is the selected simulator configuration's real relative latency
+regret.  It deliberately reports exact winner equality rather than treating a
+high pairwise-order agreement as success.
+
+| Case | Objective | Frontier winner | Real winner | Match | Real regret |
+|---|---|---|---|---:|---:|
+| Trace-P (OSL=1) | TTFT mean | TP4/MNS16 | TP4/MNS64 | no | 3.2% |
+| Trace-P (OSL=1) | TTFT p90 | TP4/MNS16 | TP4/MNS8 | no | 0.4% |
+| Trace-P (OSL=1) | E2E mean | TP4/MNS16 | TP4/MNS64 | no | 3.2% |
+| Trace-P (OSL=1) | E2E p90 | TP4/MNS16 | TP4/MNS8 | no | 0.3% |
+| Fixed-PD | TTFT mean/p90 | TP4/MNS16 | TP4/MNS16 | yes | 0.0% |
+| Fixed-PD | TPOT mean | TP1/MNS16 | TP4/MNS32 | no | 14.0% |
+| Fixed-PD | TPOT p90 | TP1/MNS16 | TP4/MNS8 | no | 13.4% |
+| Fixed-PD | E2E mean | TP1/MNS16 | TP4/MNS32 | no | 26.9% |
+| Fixed-PD | E2E p90 | TP1/MNS16 | TP4/MNS8 | no | 25.7% |
+| Fixed-P (OSL=1) | TTFT/E2E mean/p90 | TP4/MNS16 | TP4/MNS8 | no | 1.5--3.6% |
+
+The decisive failure is Fixed-PD: its TPOT and E2E pairwise orders are exactly
+reversed on all 48 informative pairs.  Thus the graph-aligned profile can
+preserve coarse TP ordering yet still choose the wrong configuration for the
+serving objective.  The prior Trace-PD success is therefore a workload-specific
+success, not evidence that Frontier has solved config tuning.
+
+## Superseding primary Fixed workload
+
+The subsequent pressure sweep changed the Fixed shapes to 4096->256 and
+4096->1, raised the offered load to 1.125 req/s/GPU, and used 257 requests per
+cell. Those Fixed-PD/PO results, together with Trace-PD and Trace-PO, are
+reported in the consolidated
+[Qwen30/Qwen235 selection-regret report](../../docs/simulator-tuning/frontier-selection-regret-qwen30-qwen235-20260719.md).
+
+Under that primary Fixed-PD workload, Frontier's selected config is 58.0%
+slower for real mean TTFT and 56.2% slower for real p90 TTFT than the
+real-hardware optimum. The low-load table above must therefore not be quoted
+as the final Fixed-PD result.