gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1126d9be7df2ee3d30811266144fea78f95ce574
aituner/runs/simulator-tuning-latency-matrix-v0/existing-evidence-audit.md

16 KiB
Raw Blame History

Existing simulator evidence audit for the no-SLO latency matrix

Scope: local-artifact audit on 2026-07-17. This document does not run a simulator or a server. It separates (1) historical evidence about SLO-feasible capacity selection from (2) the new matrix's claim about same-trace request-level mean/p90 TTFT, TPOT, and E2E latency selection. These are different objectives; a historical top-set match is not silently promoted into a no-SLO latency-tuning result.

Executive verdict

The checked-in evidence already rules out the broad claim that Frontier (or a Vidur-class simulator) has generally solved serving-config tuning: a fully covered Qwen3-30B prefill-only surface has no real/simulator top-set overlap, with 12.5% worst regret and τ-b=-1.0. Conversely, two Qwen3-235B surfaces show that an extensively aligned, patched profile can select the capacity optimum in a particular envelope. Neither observation answers the new paper question by itself, because every completed selection result optimizes SLO-feasible offered throughput/GPU rather than a fixed trace's request-level mean/p90 latency.

The appropriate new claim before the latency matrix is therefore:

Existing results establish both a capability envelope and a counterexample for SLO/capacity tuning. They motivate, but do not replace, the planned non-MoE-hybrid/MoE × Fixed-P/Fixed-PD/Trace-P/Trace-PD latency-selection matrix.

Objective boundary (non-negotiable)

All completed comparisons below rank a config by

maximum tested offered request rate / allocated GPU
subject to a joint request SLO pass-rate threshold (normally >= 95%).

The new matrix instead holds one trace fixed for every candidate and independently ranks mean/p90 TTFT, TPOT (when OSL>1), and E2E. It has no SLO feasibility gate and no capacity search. The consequences are material:

  • A capacity top-set hit can arise from a large topology margin even when request-level latency residuals and within-family ordering are wrong.
  • A simulator that makes every config SLO-infeasible (or ties every config) has not selected a config, even if its set mechanically contains the real best config.
  • Historical runs may contain per-request latency observations, but their anchors differ by config because each is a capacity search. They cannot be re-labelled as a same-trace latency ranking without a new fixed-trace analysis contract.

Thus all historical regret and correlation numbers in this document are capacity/SLO diagnostics only. They must not appear in a result table as mean/p90 TTFT, TPOT, or E2E selection evidence.

Completed selection comparisons

Coverage distinguishes complete config-level rankings from request/anchor coverage. A completed simulator run is not semantic coverage when it has no finite request metrics or no discriminative ordering.

Case / condition Model, runtime, workload Real / simulator coverage Historical selection result What it can establish What it cannot establish for the new matrix
Q30-mixed, old profile-only Qwen3-30B-A3B; community vLLM 0.20.0/CUDA 12.9; chat_w20260311_1000 replay, time scale 0.1, input 0–8192, output overridden to 128, max concurrency 64; TP{1,2,4}×MNS{8,16,32,64}. 12/12 real configs; 92 real anchors. Frontier replayed the same frozen cohort for all 92 anchors. Real best TP2/MNS32; sim top {TP4/MNS32,TP4/MNS64}; miss; worst regret 25.63%; τ-b=0.0000; 37/0/55 SLO agree/false-feasible/false-infeasible. A profile-only simulator can select the wrong TP family under a mixed serving capacity objective. It is not trace-faithful under the new definition: time is scaled, outputs are overridden, and the score is SLO capacity, not latency. It proves no mean/p90 latency claim.
Q30-mixed, vLLM-0.20 same-stack profile-only Same Qwen30/mixed 12-cell surface; BF16 H20 profiles re-collected on vLLM source 88d34…; no serving E2E scale. All 92/92 simulator probes completed (two CPU shards, no crash), but all 12 configs are SLO-infeasible. No actionable selection: all 12 tie; worst tie-break regret 60.91%; τ-b=0.0000; exact pair sign 7.58%. Same-stack isolated operator provenance alone did not recover the historical capacity ordering. “All tied” is not a top-set success and says nothing directly about fixed-trace latency ranking. The model/workload is also MoE mixed, not the dense branch.
Q30-mixed, frozen per-TP E2E calibration Same Qwen30/mixed surface; per-TP execution-time scales fitted on separate coder_200 serving workloads then frozen. 12/12 configs and 92 anchors; request-level SLO labels still have 21 false-feasible and 7 false-infeasible anchors. Sim top {TP2/MNS32,TP2/MNS64} intersects real best {TP2/MNS32}; worst regret 0.76%; τ-b=0.9668. An action-conditioned real-serving calibration can make this capacity surface near-optimal. The per-TP E2E scale is prohibited by the new matrix's information boundary. It is an upper-bound/diagnostic, not zero-shot simulator evidence and not latency evidence.
Q30 Fixed-P-like prefill-only, base Frontier Qwen3-30B-A3B; community vLLM 0.20.0+cu129 BF16/FA3/default CUDA graph; fixed ISL=2048, OSL=1, uniform QPS, prefix off; 12 configs TP{1,2,4}×MNS{8,16,32,64}, MBT=8192. 12/12 valid real configs; 96 real/sim SLO anchor decisions, with all requested request metrics present in the accepted comparison. Real top is all TP4 configs (8 req/s/GPU); sim top is all TP1/TP2 configs (8 vs real 7); no overlap; worst regret 12.50%; τ-b=-1.0000; 0/32 real non-tied directions correct. A strong counterexample: removing decode, true-mixed batch, prefix reuse, and initial-KV state is not sufficient for capacity-ranking fidelity. It is closest only to the planned MoE Fixed-P case. It still ranks SLO capacity, not mean/p90 TTFT/E2E on one 600-s trace, so it cannot fill that matrix cell.
Q30 Fixed-P-like, A2 measured-collective fix Same Q30 prefill workload/configs; measured Vidur collective estimator replaces the >100k-element analytical fallback. 12/12 config scores exist. Anchor grid: 60 shared real/sim labels, 36 real-only, 0 sim-only; all simulator top scores tie. Sim top includes all 12 configs; best tie-break regret 0 but worst=12.50%; τ-b=N/A (32 simulator-only ties); 0/32 non-tied directions correct. The collective profile is consumed after the patch; its small change does not make the capacity choice usable. Set overlap is accidental/non-discriminative. This is a simulator-only ablation against historical SLO ground truth, not a new latency result.
Q30 Fixed-P-like, A3 batch-composition rows Same as A2, plus pure-prefill attention rows for MBT=8192-reachable compositions. Same as A2: 12/12 config scores; 60 shared + 36 real-only anchor labels; no missing simulator config score. Same non-selection as A2: all 12 tie; worst regret 12.50%; τ-b=N/A; 0/32 non-tied directions correct. Adding this static batch-profile closure did not resolve the historical capacity ranking. It does not prove scheduler-state root cause, and cannot substitute for no-SLO request-latency ranking.
Q235 prefill-only, best-effort aligned profile Qwen3-235B-A22B-FP8; community vLLM 0.10.2, eager, FP8 weights/BF16 KV; length-stratified fixed 64-request cohort from thinking_w20260327_1000, OSL=1, prefix/speculative off; TP{4,8}×MNS{64,128}×MBT{8192,16384}. 8/8 rankable real configs; refined comparisons report 33 config-load labels with 6 false-infeasible labels, not a missing simulator surface. Exact top-set match: TP4/MBT16K/MNS64 or 128; worst regret 0; Spearman ρ=0.9487; all 20 comparable real non-tie pairs correct. With same-stack FP8/MoE profiles, measured KV capacity and explicit compatibility patches, Frontier can select one prefill-only capacity surface. The workload is a length-stratified cohort, not Fixed-P or Trace-P; it is a different MoE model/runtime/config surface and SLO objective. It does not generalize to dense, decode, prefix reuse, or latency metrics.
Q235 Fixed-PD-like mixed (T0) Qwen3-235B-A22B-FP8; community vLLM 0.10.2 eager, FP8/BF16-KV; fixed ISL=2048, OSL=128, uniform QPS, prefix/spec/CUDA graph off; same 8 config surface. 8/8 real capacity boundaries closed (68 fresh-server anchors); simulator 64/64 config×rate cells have 64 finite, non-negative, shape-exact request records; 34 measured labels with 10 false-infeasible, 0 false-feasible. Exact TP4 top-set match; worst regret 0; τ-b=0.8944; 16/20 real non-tied directions correct. The most complete historical profile-closed mixed-capacity success. It demonstrates that a large TP4-vs-TP8 margin can tolerate sizeable absolute and within-TP residuals. The primary 150-ms TPOT SLO was a disclosed post-pilot sensitivity (40-ms primary had no feasible capacity). It is not a fixed-trace mean/p90 TTFT/TPOT/E2E selection result, and it misses the TP8 MNS×MBT interaction.

The capability envelope is not a global solution

The two Q235 successes are real positive evidence, but must be reported with their alignment cost and failure modes:

  • Q235 prefill used FP8/MoE serving-plan, TP/EP-aware cache-key and critical-lane patches, same-stack operator/collective profiles, and real KV capacity. It was not stock Frontier.
  • Q235 mixed required an attention profile closure (prefill + standard decode + true mixed) with 1,104 rows. The accepted real ground truth alone cost 36.26 H20-GPU-hours. The simulator still flattened the TP8 MNS×MBT checkerboard: 10/34 labels were false-infeasible even though its global top set was correct.
  • The Q30 prefill counterexample used the same 12-config type of surface as the planned MoE Fixed-P branch and reverses the entire topology order. A successful Q235 capacity ranking therefore cannot be offered as evidence that a simulator already solves this new tuning problem.

Cases that are diagnostics or plans, not valid selection evidence

Artifact/case Status and observed coverage Why it is useful Why it cannot enter a new-matrix verdict
Historical internal-runtime Q235 prefill 8/8 real cells, but different serving/runtime contract from the aligned community-vLLM Frontier profile. Records a real response surface and demonstrates stack sensitivity. No fair real/simulator contract; excluded by simulator-fidelity.md.
Historical Q235 decode-only Only 7/8 valid real cells; capacity brackets overlap such that all eight may be optimal. Frontier lacks equivalent initial-KV/EP8 execution semantics. Identifies initial-KV, EAGLE3, DeepEP/NVSHMEM and decode-graph state as contract gaps. No identifiable real best and no semantically aligned simulator; no hit/regret/τ-b is valid.
Q235 T0 pre-profile-closure smoke Real TP4 test completed; original Frontier crash had no standard decode rows. After minimal closure, only a one/two-request representation smoke completed, with 27–35% TPOT absolute error. Demonstrates that a simulator crash must be treated as coverage failure, not an SLO failure; motivated the full T0 profile closure. A single-config smoke cannot select across configs and cannot support any ranking claim.
Envelope F1 short-prefill pilot Fixed ISL=512/OSL=1, 64 requests. The 512-QPS pilot covers only 0.123 s and is explicitly non-decision-bearing. Reveals the need for a steady-arrival duration contract. No sustained real capacity surface and no paired selection metrics.
Envelope F1 steady, F2 fixed mixed, T1 exact trace Protocol/frozen simulator artifacts exist; no paired valid real/simulator selection comparison is recorded. In particular, T1's simulator artifact is not a completed real trace evaluation. The contracts are a useful starting point for fixed/trace workload materialization. NOT RUN for the purpose of a simulator-selection result. Do not cite them as trace-faithful success/failure.
Q235 protocol T1/T2 The protocol explicitly marks T1 trace-faithful mixed and T2 strict decode-only as not run; only T0 is complete. T1 preserves source request fields; T2 correctly requires an explicit initial-KV contract. No real/sim coverage, hit, regret, or τ-b exists.

What can be reused safely

Reusable input Allowed use in the new matrix Prohibited inference
Q30 prefill accepted artifacts Harness conventions: fresh server, two rounds, request shape/usage validation, request-level record layout, and a known failure case for coverage gates. Treating the old TP4/TP1/2 capacity ordering or 12.5% regret as mean/p90 TTFT/E2E selection evidence.
Q235 T0 artifacts Fixed-PD measurement practice: exact 2048/128 token accounting, separate warmup, finite request metrics, profile identity/coverage checks, and state-leakage exclusion. Reusing its Q235 FP8/vLLM0.10 profile or TP4/TP8 conclusion for Qwen3.6 non-MoE hybrid or Qwen3-30B MoE.
Q235 trace audit Trace manifest structure: exact prompt/token lengths, arrival/session order, source hashes, tokenizer audit, and explicit source-to-runtime block-size translation. Calling a trace-faithful latency cell complete before runtime block/cache counter parity and paired real/simulator results exist.
A2/A3 ablations A negative control for the hypothesis that better static collective or pure-prefill batch rows alone solve the Q30 residual. Claiming the remaining error is uniquely caused by scheduler state; routing, graph/fusion, and collective composition remain alternatives.

Required interpretation in the paper and the next experiment

  1. Say “not solved in general”, supported by the Q30 no-overlap counterexample; do not say “all simulators fail,” because the Q235 envelope has valid capacity-selection successes.
  2. Report the positive cases as conditional capacity results and disclose patches, profiles, real KV capacity, profile cost, and—where used—per-TP E2E calibration. Do not call them zero-cost tuning.
  3. Do not average these cases into an aggregate score. Model, precision, runtime, topology, objective, and workload differ; the average would hide the decision-bearing Q30 reversal.
  4. Run the new matrix exactly as specified in experiment-card.md: for every legal config replay the same fixed or trace-derived request vector, report mean/p90 TTFT/TPOT/E2E, require simulator request-metric coverage, and calculate top-set hit, worst tie-break regret, and τ-b separately for each latency objective. OSL=1 cases report TPOT as N/A, never zero.

Primary evidence paths