# Qwen3-30B-A3B Trace-PD `u0p01` latency-selection comparison

> **Status:** the 36-run real surface is valid. The initially reported Frontier
> surface is **invalidated for simulator fidelity/selection** by a post-hoc
> prefix-cache trace-contract audit (2026-07-17): its simulator-facing block
> metadata includes non-cacheable partial final blocks. The raw CPU artifacts
> are retained for diagnosis, but are not Frontier evidence. This remains the
> first MoE case of `EXP-SIM-TUNING-LATENCY-MATRIX-V0`, not a claim about the
> remaining three workload classes.

## Question and decision

Can the frozen Frontier surface select the same low-latency Qwen3-30B-A3B
configuration as a fresh community-vLLM deployment when both replay exactly
the same trace-derived prefill+decode request vector? The objective contains
no SLO or capacity threshold: lower mean/p90 TTFT, TPOT, and E2E is better.

The result is a bounded Trace-PD result only. It cannot establish a claim
about Fixed-P, Fixed-PD, Trace-P, other trace rates, another model, or a
modified Frontier.

## Frozen contract

| Item | Value |
|---|---|
| Hardware | `dash0`, H20, one replica; fresh vLLM server per config/trial |
| Model / engine | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`; community `vLLM 0.20.0+cu129`, BF16 |
| Config surface | `TP ∈ {1,2,4} × MNS ∈ {8,16,32,64}`; `DP=PP=EP=1`, `MBT=8192`, GPU-memory utilization `0.92`, chunked prefill and prefix caching on |
| Trace | `trace-exact-v1`, anchor `u0p01`; **129 requests**, original input/output length, order and session are preserved. The first Frontier materialization also copied every 16-token runtime identity, including a final partial block; that is not a legal vLLM prefix-cache-hit representation and invalidates this simulator attempt. For a config of tensor parallelism `TP`, every arrival is transformed as `t′ = t / TP`; the TP1/2/4 traces therefore end at 597.037/298.519/149.259 s. |
| Offered load | Base rate is `129 / 600 = 0.215 req/s`. TP1/2/4 replay at 0.215/0.430/0.860 req/s globally, respectively, so all three topologies receive 0.215 req/s/GPU. This is a **per-GPU-normalized trace**, not an unchanged-wall-clock trace. |
| Simulator baseline | The existing same-global-rate T1 surface is excluded: it evaluates TP2/TP4 at lower per-GPU load. A new Frontier surface must use the identical TP-normalized arrival vector for each topology before coverage or latency can be reported. |
| Trials | Three fresh-server trials per config; independent warmup excluded from the measured trace |

## Validity gates

1. Each real trial must complete all 129 requests, preserve exact prompt and
   completion-token usage, and have the TP-normalized row-vector digest.
2. A simulator metric is valid only if its cell has all 129 finite request
   metrics with matching request ID, prompt/output lengths and transformed
   arrival vector. A simulator `scheduler_stall` is a coverage result, never
   an infeasible-latency label.
3. TTFT, TPOT and E2E are kept in milliseconds. `TPOT` is recorded only for
   outputs longer than one token (all requests in this Trace-PD anchor
   qualify).
4. A failed real run remains an infeasible/safety observation; it is not
   silently converted to infinite latency or omitted from ranking.

## Analysis and decision rule

For every `config × trial`, retain raw per-request records and compute mean
and nearest-rank p90 for TTFT, TPOT and E2E. Report pooled 387-request values
alongside the three per-trial values. A selection-stability bootstrap is
deferred; it is not needed to evaluate Frontier's coverage gate.

The primary gate is Frontier decision-valid coverage on all 12 cells. If any
real-valid candidate lacks a legal simulator metric, Frontier receives no
full-surface selection, top-set overlap, regret, or Kendall-tau score. We
will still show all valid per-cell latency comparisons and the complete real
surface, so any coverage failure and its ground truth remain auditable.

The simulator's pre-existing SLO fields do not influence request generation,
scheduling, or this no-SLO analysis; they are excluded from scoring.

## Post-hoc Frontier trace-contract audit (2026-07-17)

The CPU attempt initially passed only a syntactic trace check (row count,
IDs, lengths, and SHA). It did **not** validate the semantic invariant needed
by vLLM prefix caching: only completed 16-token blocks may be cache hits.

- The trace exporter emitted `ceil(ISL / 16)` identities. Thus 122/129 prompts
  contained one partial final block; the public trace has 36,443 identities,
  while a full-block Frontier/vLLM cache adapter must expose 36,321.
- Exact community-vLLM 0.20 source (`vllm.v1.core.kv_cache_manager`) documents
  that computed prefix blocks "must be full" and bounds hits by
  `prompt_length - 1`. Frontier's fixture contract only requires a non-empty
  `block_hash_ids` field and does not validate this full-block invariant.
- Frontier's prefix manager converts every hit to `len(blocks) * 16`. For the
  final TP2/MNS16 state, requests 89/111/117 had `(ISL, IDs)` of
  `(722,46)`, `(678,43)`, and `(706,45)`. If all supplied identities hit, it
  obtains 736/688/720 computed tokens and hence `num_new_tokens` of
  -14/-10/-14. The scheduler preserves zero-or-negative-token requests in its
  waiting queue; no event remains, so the simulator reports
  `Sequential simulation ended with non-empty scheduler state`.

This proves a trace-adapter/Frontier prefix-cache semantic mismatch, not a
GPU capacity failure: a targeted CPU hook found no failed KV allocation and
no PP-admission deferral, only the three non-admitted requests with a full
8192-token batch budget. The correct next input is a **separate Frontier
adapter** that exports only `floor(ISL / 16)` complete prompt-block identities;
it must not modify the private real trace or its input/output/arrival vector.

## Invalid raw Frontier attempt (2026-07-17; retained for diagnosis only)

The TP-normalized surface completed in 156 seconds of CPU-only wall time on
`dash0`, with Frontier commit `deadc4a321f0baaa534c6ebd17f974123733cdc2`.
All twelve input traces passed the 129-request, ID/shape, and TP-specific
trace-SHA checks.  Only two cells reached valid request metrics; the other ten
ended in `scheduler_stall`.

| Config | Status | mean / p90 TTFT (ms) | mean / p90 TPOT (ms) | mean / p90 E2E (ms) |
|---|---|---:|---:|---:|
| TP1/MNS32 | complete | 440032 / 972628 | 145.8 / 151.5 | 936301 / 1645895 |
| TP1/MNS64 | complete | 174335 / 510962 | 163.6 / 178.9 | 724328 / 1285374 |
| remaining 10 configs | `scheduler_stall` | N/A | N/A | N/A |

The apparent **2/12 (16.7%)** coverage is not a Frontier coverage result:
both the ten stalls and the two completed cells consumed the invalid partial
block metadata. In particular, the two completed cells may receive false
prefix hits. No latency, ranking, coverage, top-set overlap, regret, or
Kendall-tau statement about Frontier may be derived from this attempt.

The legacy surface runner writes an offered-rate summary assuming a fixed
600-second window. That field is not used here because TP2/TP4 have compressed
arrival clocks. The materialized trace manifests are authoritative: they
record global 0.215/0.430/0.860 req/s and the matched 0.215 req/s/GPU rate.

## Corrected-adapter CPU liveness probe (2026-07-17)

One deliberately narrow CPU-only rerun tested the causal fix before launching
another surface: TP2/MNS16, the same 129-request Trace-PD input, MBT=8192,
prefix caching and chunked prefill enabled, and the original Frontier commit.
It changed only the simulator-facing prefix projection from
`ceil(ISL / 16)` to `floor(ISL / 16)`; the private real replay remained bytewise
unchanged.  The materialized trace records 36,321 complete blocks (36,443 raw
runtime identities; 122 prompts with a discarded partial final block), global
arrival rate 0.430 req/s, and the required 0.215 req/s/GPU.

The run completed **129/129** requests in 105.8 CPU wall-clock seconds with no
stall.  This is a direct liveness validation of the diagnosis above: the
partial-block adapter, rather than KV capacity, caused the old deadlock.

It is **not** a valid latency or selection result.  Even after the repair,
Frontier reports p50 TTFT=946,973 ms, p50 TPOT=96.37 ms, p50 E2E=1,305,128 ms,
and 112.2 generated decode tokens/s.  The trace supplies 460,490 output tokens
over a 298.5-s TP2 arrival horizon (about 1,543 output tokens/s), so the
simulated service rate is below the offered decode work and its queue must
grow.  The corresponding real TP2/MNS16 replay has mean/p90 TPOT 14.1/16.1 ms
and mean/p90 TTFT 34,443/77,922 ms.  Thus the post-fix discrepancy is a
decode-service-model / runtime-semantics question, not residual evidence from
the invalid cache metadata.  The probe artifact is
`/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/frontier-contract-probe-v1`.

## Real result (2026-07-17)

All **36/36** `config × trial` replays completed with exit code zero. Each
trial passed all validity gates: 129/129 successful requests, exact requested
and observed input/output token usage, the TP-specific private-trace SHA and
normalized arrival-vector SHA, and HTTP routing to the explicit alias
`qwen3-30b-exact-trace`. The cache-populating smoke and earlier failed launch
attempts are excluded from this table. Values pool 387 requests/config (three
trials); p90 is nearest-rank.

| Config | mean / p90 TTFT (ms) | mean / p90 TPOT (ms) | mean / p90 E2E (ms) |
|---|---:|---:|---:|
| TP1/MNS8 | 88479.9 / 168185.5 | 14.3 / 16.3 | 137996.4 / 222136.0 |
| TP1/MNS16 | 17378.2 / 42739.8 | 20.6 / 25.3 | 87355.2 / 156177.2 |
| TP1/MNS32 | 620.5 / 1931.5 | 23.8 / 30.0 | 81051.2 / 162908.6 |
| TP1/MNS64 | 619.8 / 1956.7 | 24.1 / 30.4 | 82253.7 / 165355.3 |
| TP2/MNS8 | 95920.1 / 183878.8 | 9.2 / 10.2 | 128296.9 / 215848.3 |
| TP2/MNS16 | 34443.0 / 77922.4 | 14.1 / 16.1 | 82936.7 / 137726.4 |
| TP2/MNS32 | 1050.6 / 2521.8 | 18.0 / 21.3 | 62267.5 / 117239.3 |
| TP2/MNS64 | 375.8 / 1148.6 | 18.2 / 21.6 | 62353.6 / 118392.3 |
| TP4/MNS8 | 102605.9 / 193359.7 | **6.8 / 7.2** | 126405.4 / 209092.4 |
| TP4/MNS16 | 35042.6 / 74281.3 | 8.8 / 9.6 | 65779.1 / 110374.3 |
| TP4/MNS32 | 6296.2 / 19159.9 | 12.2 / 13.8 | 47944.5 / 83825.7 |
| TP4/MNS64 | **246.0 / 685.5** | 13.2 / 15.4 | **44985.1 / 83763.6** |

Thus real-vLLM chooses TP4/MNS64 for mean/p90 TTFT and E2E, and TP4/MNS8
for mean/p90 TPOT. These are different single-metric objectives, not a
claim that one configuration simultaneously optimizes all three.

## Correct conclusion and next gate

This run establishes two bounded findings, neither of which is a full Frontier
selection verdict: (1) the initial simulator-facing Trace-PD prefix contract
was invalid, and (2) its corrected TP2/MNS16 liveness probe estimates a much
smaller decode service rate than the real system. The valid real surface remains
a frozen ground truth, but all raw Frontier numbers above are excluded from the
research claim. We therefore retract the prior bounded counterexample and do
not yet know whether Frontier selects the correct config on this case.

Before a CPU-only Frontier rerun, the comparison must additionally record
these remaining alignment gaps:

1. real vLLM uses `FULL_AND_PIECEWISE` CUDA graphs, whereas the raw Frontier
   command explicitly used `--decode_cuda_graph_mode none`; Frontier exposes
   `full_decode_only` and `piecewise`, not the identical combined mode. More
   importantly, Frontier chooses kernel-only predictor families whenever this
   mode is non-`none`, so a graph-compatible rerun also requires corresponding
   kernel-only profile coverage rather than merely changing the CLI flag;
2. Frontier used one explicit KV-block count per TP, while real vLLM's graph
   capture changes the count with MNS (the mismatch is small but measurable);
3. Frontier skipped CPU-overhead modeling, and its MoE CSV has only
   `standalone_legacy` gating rows, so the code warned and fell back rather
   than training the requested `prefill_hot` pseudo-model;
4. the runner's rate *metadata* incorrectly fixed the TP2/TP4 numerator to a
   600-second window. Arrival timestamps supplied to the simulator were
   correct, so this did not alter the raw execution, but the metadata must be
   fixed before reporting a rerun.

The next admissible Frontier result uses complete-block prefix metadata,
per-`TP×MNS` real KV capacity, correct TP-normalized rate metadata, and an
explicit CUDA-graph compatibility decision. Only a complete, semantically
valid surface may receive the selection metrics. Any remaining mismatch after
that CPU rerun is then evidence about Frontier's scheduler/profile fidelity,
not this adapter.

## Cost and output

The measured replay horizons are 597/299/149 s at TP1/2/4. Including five
minutes of per-launch server start/warmup, 36 launches consume about **13
H20-GPUh nominally**; applying the launcher's historical 12--35 minute
fresh-server envelope yields an upper bound of **41 H20-GPUh**. A separate
CPU-only simulator rerun is mandatory before this GPU stage; no profile
collection is included.

The remote, experiment-specific output root is
`/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717`.
It contains the prompt-free real audit JSON/Markdown, commands, environment,
GPU/trace/model hashes, raw prompt-free request records, and cache inventory.
Private prompt text remains on `dash0` and is never copied into the repository.
