aituner/runs/simulator-tuning-latency-matrix-v0/existing-evidence-audit.md

# Existing simulator evidence audit for the no-SLO latency matrix

> **Scope:** local-artifact audit on 2026-07-17. This document does not run a
> simulator or a server. It separates (1) historical evidence about
> **SLO-feasible capacity selection** from (2) the new matrix's claim about
> **same-trace request-level mean/p90 TTFT, TPOT, and E2E latency selection**.
> These are different objectives; a historical top-set match is not silently
> promoted into a no-SLO latency-tuning result.

## Executive verdict

The checked-in evidence already rules out the broad claim that Frontier (or a
Vidur-class simulator) has *generally* solved serving-config tuning: a fully
covered Qwen3-30B prefill-only surface has no real/simulator top-set overlap,
with 12.5% worst regret and `τ-b=-1.0`. Conversely, two Qwen3-235B surfaces
show that an extensively aligned, patched profile can select the capacity
optimum in a particular envelope. Neither observation answers the new paper
question by itself, because every completed selection result optimizes
SLO-feasible offered throughput/GPU rather than a fixed trace's request-level
mean/p90 latency.

The appropriate new claim before the latency matrix is therefore:

> Existing results establish both a capability envelope and a counterexample
> for **SLO/capacity tuning**. They motivate, but do not replace, the planned
> non-MoE-hybrid/MoE × Fixed-P/Fixed-PD/Trace-P/Trace-PD latency-selection
> matrix.

## Objective boundary (non-negotiable)

All completed comparisons below rank a config by

```text
maximum tested offered request rate / allocated GPU
subject to a joint request SLO pass-rate threshold (normally >= 95%).
```

The new matrix instead holds one trace fixed for every candidate and independently
ranks `mean`/`p90` TTFT, TPOT (when `OSL>1`), and E2E. It has no SLO feasibility
gate and no capacity search. The consequences are material:

- A capacity top-set hit can arise from a large topology margin even when
  request-level latency residuals and within-family ordering are wrong.
- A simulator that makes every config SLO-infeasible (or ties every config) has
  not selected a config, even if its set mechanically contains the real best
  config.
- Historical runs may contain per-request latency observations, but their
  anchors differ by config because each is a capacity search. They cannot be
  re-labelled as a same-trace latency ranking without a new fixed-trace
  analysis contract.

Thus **all historical regret and correlation numbers in this document are
capacity/SLO diagnostics only**. They must not appear in a result table as
mean/p90 TTFT, TPOT, or E2E selection evidence.

## Completed selection comparisons

`Coverage` distinguishes complete config-level rankings from request/anchor
coverage. A completed simulator run is not semantic coverage when it has no
finite request metrics or no discriminative ordering.

| Case / condition | Model, runtime, workload | Real / simulator coverage | Historical selection result | What it can establish | What it cannot establish for the new matrix |
|---|---|---|---|---|---|
| **Q30-mixed, old profile-only** | Qwen3-30B-A3B; community vLLM 0.20.0/CUDA 12.9; `chat_w20260311_1000` replay, time scale 0.1, input 0–8192, output overridden to 128, max concurrency 64; `TP{1,2,4}×MNS{8,16,32,64}`. | 12/12 real configs; 92 real anchors. Frontier replayed the same frozen cohort for all 92 anchors. | Real best `TP2/MNS32`; sim top `{TP4/MNS32,TP4/MNS64}`; miss; worst regret **25.63%**; `τ-b=0.0000`; 37/0/55 SLO agree/false-feasible/false-infeasible. | A profile-only simulator can select the wrong TP family under a mixed serving capacity objective. | It is not trace-faithful under the new definition: time is scaled, outputs are overridden, and the score is SLO capacity, not latency. It proves no mean/p90 latency claim. |
| **Q30-mixed, vLLM-0.20 same-stack profile-only** | Same Qwen30/mixed 12-cell surface; BF16 H20 profiles re-collected on vLLM source `88d34…`; no serving E2E scale. | All **92/92** simulator probes completed (two CPU shards, no crash), but all 12 configs are SLO-infeasible. | No actionable selection: all 12 tie; worst tie-break regret **60.91%**; `τ-b=0.0000`; exact pair sign 7.58%. | Same-stack isolated operator provenance alone did not recover the historical capacity ordering. | “All tied” is not a top-set success and says nothing directly about fixed-trace latency ranking. The model/workload is also MoE mixed, not the dense branch. |
| **Q30-mixed, frozen per-TP E2E calibration** | Same Qwen30/mixed surface; per-TP execution-time scales fitted on separate `coder_200` serving workloads then frozen. | 12/12 configs and 92 anchors; request-level SLO labels still have 21 false-feasible and 7 false-infeasible anchors. | Sim top `{TP2/MNS32,TP2/MNS64}` intersects real best `{TP2/MNS32}`; worst regret **0.76%**; `τ-b=0.9668`. | An action-conditioned real-serving calibration can make this *capacity* surface near-optimal. | The per-TP E2E scale is prohibited by the new matrix's information boundary. It is an upper-bound/diagnostic, not zero-shot simulator evidence and not latency evidence. |
| **Q30 Fixed-P-like prefill-only, base Frontier** | Qwen3-30B-A3B; community vLLM 0.20.0+cu129 BF16/FA3/default CUDA graph; fixed ISL=2048, OSL=1, uniform QPS, prefix off; 12 configs `TP{1,2,4}×MNS{8,16,32,64}`, MBT=8192. | 12/12 valid real configs; 96 real/sim SLO anchor decisions, with all requested request metrics present in the accepted comparison. | Real top is all TP4 configs (8 req/s/GPU); sim top is all TP1/TP2 configs (8 vs real 7); **no overlap**; worst regret **12.50%**; `τ-b=-1.0000`; 0/32 real non-tied directions correct. | A strong counterexample: removing decode, true-mixed batch, prefix reuse, and initial-KV state is not sufficient for capacity-ranking fidelity. | It is closest only to the planned MoE Fixed-P case. It still ranks SLO capacity, not mean/p90 TTFT/E2E on one 600-s trace, so it cannot fill that matrix cell. |
| **Q30 Fixed-P-like, A2 measured-collective fix** | Same Q30 prefill workload/configs; measured Vidur collective estimator replaces the `>100k`-element analytical fallback. | 12/12 config scores exist. Anchor grid: 60 shared real/sim labels, 36 real-only, 0 sim-only; all simulator top scores tie. | Sim top includes all 12 configs; best tie-break regret 0 but **worst=12.50%**; `τ-b=N/A` (32 simulator-only ties); 0/32 non-tied directions correct. | The collective profile is consumed after the patch; its small change does not make the capacity choice usable. | Set overlap is accidental/non-discriminative. This is a simulator-only ablation against historical SLO ground truth, not a new latency result. |
| **Q30 Fixed-P-like, A3 batch-composition rows** | Same as A2, plus pure-prefill attention rows for MBT=8192-reachable compositions. | Same as A2: 12/12 config scores; 60 shared + 36 real-only anchor labels; no missing simulator config score. | Same non-selection as A2: all 12 tie; worst regret **12.50%**; `τ-b=N/A`; 0/32 non-tied directions correct. | Adding this static batch-profile closure did not resolve the historical capacity ranking. | It does not prove scheduler-state root cause, and cannot substitute for no-SLO request-latency ranking. |
| **Q235 prefill-only, best-effort aligned profile** | Qwen3-235B-A22B-FP8; community vLLM 0.10.2, eager, FP8 weights/BF16 KV; length-stratified fixed 64-request cohort from `thinking_w20260327_1000`, OSL=1, prefix/speculative off; `TP{4,8}×MNS{64,128}×MBT{8192,16384}`. | 8/8 rankable real configs; refined comparisons report 33 config-load labels with 6 false-infeasible labels, not a missing simulator surface. | Exact top-set match: TP4/MBT16K/MNS64 or 128; worst regret **0**; Spearman `ρ=0.9487`; all 20 comparable real non-tie pairs correct. | With same-stack FP8/MoE profiles, measured KV capacity and explicit compatibility patches, Frontier can select one prefill-only capacity surface. | The workload is a length-stratified cohort, not Fixed-P or Trace-P; it is a different MoE model/runtime/config surface and SLO objective. It does not generalize to dense, decode, prefix reuse, or latency metrics. |
| **Q235 Fixed-PD-like mixed (T0)** | Qwen3-235B-A22B-FP8; community vLLM 0.10.2 eager, FP8/BF16-KV; fixed ISL=2048, OSL=128, uniform QPS, prefix/spec/CUDA graph off; same 8 config surface. | **8/8** real capacity boundaries closed (68 fresh-server anchors); simulator **64/64** config×rate cells have 64 finite, non-negative, shape-exact request records; 34 measured labels with 10 false-infeasible, 0 false-feasible. | Exact TP4 top-set match; worst regret **0**; `τ-b=0.8944`; 16/20 real non-tied directions correct. | The most complete historical profile-closed mixed-capacity success. It demonstrates that a large TP4-vs-TP8 margin can tolerate sizeable absolute and within-TP residuals. | The primary 150-ms TPOT SLO was a disclosed post-pilot sensitivity (40-ms primary had no feasible capacity). It is not a fixed-trace mean/p90 TTFT/TPOT/E2E selection result, and it misses the TP8 MNS×MBT interaction. |

### The capability envelope is not a global solution

The two Q235 successes are real positive evidence, but must be reported with
their alignment cost and failure modes:

- Q235 prefill used FP8/MoE serving-plan, TP/EP-aware cache-key and
  critical-lane patches, same-stack operator/collective profiles, and real KV
  capacity. It was not stock Frontier.
- Q235 mixed required an attention profile closure (prefill + standard decode +
  true mixed) with 1,104 rows. The accepted real ground truth alone cost
  36.26 H20-GPU-hours. The simulator still flattened the TP8 MNS×MBT
  checkerboard: 10/34 labels were false-infeasible even though its global top
  set was correct.
- The Q30 prefill counterexample used the same 12-config type of surface as
  the planned MoE Fixed-P branch and reverses the entire topology order. A
  successful Q235 capacity ranking therefore cannot be offered as evidence
  that a simulator already solves this new tuning problem.

## Cases that are diagnostics or plans, not valid selection evidence

| Artifact/case | Status and observed coverage | Why it is useful | Why it cannot enter a new-matrix verdict |
|---|---|---|---|
| Historical internal-runtime Q235 prefill | 8/8 real cells, but different serving/runtime contract from the aligned community-vLLM Frontier profile. | Records a real response surface and demonstrates stack sensitivity. | No fair real/simulator contract; excluded by `simulator-fidelity.md`. |
| Historical Q235 decode-only | Only 7/8 valid real cells; capacity brackets overlap such that all eight may be optimal. Frontier lacks equivalent initial-KV/EP8 execution semantics. | Identifies initial-KV, EAGLE3, DeepEP/NVSHMEM and decode-graph state as contract gaps. | No identifiable real best and no semantically aligned simulator; no hit/regret/`τ-b` is valid. |
| Q235 T0 pre-profile-closure smoke | Real TP4 test completed; original Frontier crash had no standard decode rows. After minimal closure, only a one/two-request representation smoke completed, with 27–35% TPOT absolute error. | Demonstrates that a simulator crash must be treated as coverage failure, not an SLO failure; motivated the full T0 profile closure. | A single-config smoke cannot select across configs and cannot support any ranking claim. |
| Envelope F1 short-prefill pilot | Fixed ISL=512/OSL=1, 64 requests. The 512-QPS pilot covers only 0.123 s and is explicitly non-decision-bearing. | Reveals the need for a steady-arrival duration contract. | No sustained real capacity surface and no paired selection metrics. |
| Envelope F1 steady, F2 fixed mixed, T1 exact trace | Protocol/frozen simulator artifacts exist; no paired valid real/simulator selection comparison is recorded. In particular, T1's simulator artifact is not a completed real trace evaluation. | The contracts are a useful starting point for fixed/trace workload materialization. | `NOT RUN` for the purpose of a simulator-selection result. Do not cite them as trace-faithful success/failure. |
| Q235 protocol T1/T2 | The protocol explicitly marks T1 trace-faithful mixed and T2 strict decode-only as not run; only T0 is complete. | T1 preserves source request fields; T2 correctly requires an explicit initial-KV contract. | No real/sim coverage, hit, regret, or `τ-b` exists. |

## What can be reused safely

| Reusable input | Allowed use in the new matrix | Prohibited inference |
|---|---|---|
| Q30 prefill accepted artifacts | Harness conventions: fresh server, two rounds, request shape/usage validation, request-level record layout, and a known failure case for coverage gates. | Treating the old TP4/TP1/2 capacity ordering or 12.5% regret as mean/p90 TTFT/E2E selection evidence. |
| Q235 T0 artifacts | Fixed-PD measurement practice: exact 2048/128 token accounting, separate warmup, finite request metrics, profile identity/coverage checks, and state-leakage exclusion. | Reusing its Q235 FP8/vLLM0.10 profile or TP4/TP8 conclusion for Qwen3.6 non-MoE hybrid or Qwen3-30B MoE. |
| Q235 trace audit | Trace manifest structure: exact prompt/token lengths, arrival/session order, source hashes, tokenizer audit, and explicit source-to-runtime block-size translation. | Calling a trace-faithful latency cell complete before runtime block/cache counter parity and paired real/simulator results exist. |
| A2/A3 ablations | A negative control for the hypothesis that better static collective or pure-prefill batch rows alone solve the Q30 residual. | Claiming the remaining error is uniquely caused by scheduler state; routing, graph/fusion, and collective composition remain alternatives. |

## Required interpretation in the paper and the next experiment

1. Say **“not solved in general”**, supported by the Q30 no-overlap
   counterexample; do not say “all simulators fail,” because the Q235 envelope
   has valid capacity-selection successes.
2. Report the positive cases as *conditional capacity results* and disclose
   patches, profiles, real KV capacity, profile cost, and—where used—per-TP
   E2E calibration. Do not call them zero-cost tuning.
3. Do not average these cases into an aggregate score. Model, precision,
   runtime, topology, objective, and workload differ; the average would hide
   the decision-bearing Q30 reversal.
4. Run the new matrix exactly as specified in
   [experiment-card.md](experiment-card.md): for every legal config replay the
   same fixed or trace-derived request vector, report mean/p90 TTFT/TPOT/E2E,
   require simulator request-metric coverage, and calculate top-set hit,
   worst tie-break regret, and `τ-b` separately for each latency objective.
   `OSL=1` cases report TPOT as `N/A`, never zero.

## Primary evidence paths

- Overall completed-case synthesis and exclusions:
  [simulator-fidelity.md](../../simulator-fidelity.md).
- Q30 prefill counterexample: [experiment card](../frontier-phase-factorial-v0/experiment-card.md),
  [accepted comparison](../frontier-phase-factorial-v0/results/final/comparison.json).
- Q30 measured-collective/batch ablations:
  [envelope card](../frontier-fidelity-envelope-v1/experiment-card.md),
  [A2 comparison](../frontier-fidelity-envelope-v1/results/a2/comparison.json),
  [A3 comparison](../frontier-fidelity-envelope-v1/results/a3/comparison.json).
- Q235 prefill alignment and exclusions:
  [findings](../frontier-multicase-sufficiency-v0/findings.md),
  [refined comparison](../frontier-multicase-sufficiency-v0/best_effort/fixed_cohort_evidence/v2_refined_comparison.json).
- Q235 fixed mixed T0: [protocol](../frontier-multicase-sufficiency-v1/protocol.md),
  [smoke report](../frontier-multicase-sufficiency-v1/t0-smoke-report.md),
  [final comparison](../frontier-multicase-sufficiency-v1/results/t0-final/comparison.json).