aituner/runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md

# EXP-SIMFID-Q30-GRAPH-PIECEWISE：graph-compatible kernel-only profile 是否修正 Frontier trace replay？

> **状态：** completed（2026-07-17）。本卡是已纠正 prefix-trace contract 后的最小判别实验；不复用此前 `decode_cuda_graph_mode=none` 的数值作 fidelity verdict。

## Purpose and hypotheses

- **Parent claim：** Frontier 是否已经足以为 Qwen3-30B-A3B 的真实 trace serving surface 选择 config。
- **Question：** 旧 Frontier replay 低估 decode service rate，是否主要是 simulator 使用 `none` 而真机使用 `FULL_AND_PIECEWISE`、并且没有向 Frontier 提供独立 `KERNEL_ONLY` profile family？
- **G1 (graph-family omission)：** 用同一 vLLM 0.20/FA3/FlashInfer-CUTLASS stack 的 `RecordFunctionTracer` kernel-only measurements，加真实 capture buckets 和 Frontier `piecewise`，会显著缩小 TP2/MNS16 的 TPOT/service-rate gap，并至少改变一个 config 的 latency ranking。
- **G2 (remaining composition error)：** 即使 graph family 对齐，TPOT、TTFT 或 E2E ranking 仍与真机不一致；则 graph omission 只是必要修正，不是 simulator 已解决 tuning 的证据。

## Controlled setup

| Item | Frozen choice |
|---|---|
| model/runtime/hardware | Qwen3-30B-A3B BF16; community vLLM 0.20.0 (`88d34c…`); dash0 NVIDIA H20 |
| simulator | Frontier `deadc4a321f0baaa534c6ebd17f974123733cdc2`; no local source patch |
| workload | exact 129-request Trace-PD public projection; exact ISL/OSL/arrival order; TP-normalized arrival time and complete 16-token prefix blocks |
| surface | TP in {1,2,4}; MNS in {8,16,32,64}; MBT=8192; prefix/chunked prefill on |
| real graph contract | observed vLLM capture sizes: MNS8=[1,2,4,8,16], MNS16=[1,2,4,8,16,24,32], MNS32=[1,2,4,8,16,24,32,40,48,56,64], MNS64=[1,2,4,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128] |
| profile intervention | CUDA-event profile stays frozen for prefill/mixed batches. New `KERNEL_ONLY` linear, FA3 decode + KV-update, MoE, and router rows use Frontier's actual `RecordFunctionTracer` semantics; no relabeling of CUDA-event numbers. |
| exact capacity | per-cell real observed KV block count and capture list; Frontier CPU-overhead model remains disabled on both old/new simulator runs because the intervention is GPU-kernel family only. |

Frontier source inspection fixes the semantic boundary: `piecewise` emits `PIECEWISE` whenever a capture hits, but the MONOLITHIC predictor selects `KERNEL_ONLY` only when `num_prefill_tokens == 0`. Hence new profile coverage is pure decode only; captured mixed/prefill work continues to consume the existing CUDA-event family.

## Measurement and decision rule

- **Primary outputs：** per-config mean/p90 TTFT, TPOT, E2E; ranking for each metric; TP2/MNS16 per-request TPOT gap against the already frozen three-trial real audit.
- **Validity gates：** every kernel CSV hash matches its manifest; every row says `KERNEL_ONLY`; every TP/capture-bucket/KV-context required by the runner is present; command records `piecewise`, per-cell blocks and capture sizes; each simulator cell completes all 129 requests.
- **Decision:** G1 is supported only if the graph-aligned TP2/MNS16 TPOT median moves toward real **and** full-surface rank/error evidence improves. A single-cell timing improvement does not establish tuning sufficiency. If G2 holds, update the research claim to “Frontier has not solved tuning under trace-faithful MoE serving after graph-family alignment,” then profile stage/state composition rather than add arbitrary kernel rows.

## Result (completed 2026-07-17)

### Validity gates

- The fresh profile family is exclusive `KERNEL_ONLY`: linear=57, attention=456, MoE=57 rows. Attention covers every TP in {1,2,4}, all MNS64 capture buckets, and the eight decode KV contexts; the final manifest records Frontier `deadc4a…`.
- All three TP traces contain the same 129 ISL/OSL requests and prompt semantic vector. Their offered rates are 0.215/0.430/0.860 req/s for TP=1/2/4, while the per-GPU rate is fixed. Prefix IDs use `floor(ISL/16)`: 36,321 legal complete blocks, not the invalid 36,443 `ceil` projection.
- All 12 Frontier cells completed 129/129 requests. No result has a Frontier error or scheduler stall. Commands record `piecewise`, the observed capture list, the real KV-block count, frozen CUDA-event prefill/mixed CSVs, and the new kernel-only CSVs.

### Frontier versus real vLLM selection

| Target | Frontier winner | Real winner | Pairwise order agreement |
|---|---|---|---:|
| TTFT mean / p90 | TP4/MNS64 / TP4/MNS64 | TP4/MNS64 / TP4/MNS64 | 56/66 (84.8%) / 56/66 (84.8%) |
| TPOT mean / p90 | TP4/MNS8 / TP4/MNS8 | TP4/MNS8 / TP4/MNS8 | 64/66 (97.0%) / 64/66 (97.0%) |
| E2E mean / p90 | TP4/MNS64 / TP4/MNS64 | TP4/MNS64 / TP4/MNS64 | 61/66 (92.4%) / 55/66 (83.3%) |

Thus, for this one Qwen3-30B-A3B Trace-PD surface, graph-aligned Frontier selects the same argmin config for all six reported latency objectives. This is evidence for *selection fidelity on this surface*, not a general claim about dense models, fixed-shape traffic, prefill-only traffic, another serving engine, or SLO capacity.

### Absolute latency is still unusable

The same simulator is not calibrated for absolute latency. Across all 12 cells, sim/real ratios are: TTFT mean 8.13--510.88x (median 38.29x), TPOT mean 4.06--5.72x (median 4.86x), and E2E mean 4.66--13.99x (median 8.24x). At the TP2/MNS16 anchor, Frontier reports mean/p90 TTFT=606,885/1,194,098 ms, TPOT=64.42/66.54 ms, E2E=828,082/1,404,520 ms; real vLLM reports 34,443/77,922 ms, 14.10/16.15 ms, and 82,937/137,726 ms, respectively.

The TPOT error is service-time error, not merely queueing: Frontier's `tpot_computation` equals its TPOT at the anchor. The large TTFT/E2E error then amplifies through the simulated queue. The profile rows themselves are measured kernel rows, so the next diagnosis must inspect Frontier's layer/operation composition and batching-state model rather than relabeling or adding arbitrary graph kernels.

### Decision

G1 is only partially supported: `piecewise` plus a true kernel-only decode family improves the prior `none` smoke's 96.37 ms TPOT median to 65.60 ms, but leaves a large absolute gap. G2 is rejected for the *argmin selection* question on this surface because all six winners match. The defensible claim is therefore:

> Frontier has solved config selection for this evaluated MoE trace surface, despite failing absolute latency prediction; it has **not** yet been shown to solve the broader tuning problem.

The next discriminating tests are the remaining planned dense/MoE × fixed-shape/trace-faithful × prefill+decode/prefill-only surfaces. In parallel, diagnose the decode service-time composition error; do not use Frontier's current absolute values for SLO, cost, or capacity claims.

Artifacts: `/home/admin/cpfs/wjh/aituner/graph-piecewise-qwen30-20260717/full/frozen-kernel-only/manifest.json` and `/home/admin/cpfs/wjh/aituner/graph-piecewise-qwen30-20260717/simulator-piecewise-surface-v2/analysis/{comparison.json,comparison.md}`.

## Cost and provenance

- **GPU cost:** three 1-GPU FA3 decode profile shards, plus one 1-GPU linear shard and one 1-GPU MoE/router shard; expected 1.5--3.0 H20-GPU-hours, hard cap 4.0 GPU-hours.
- **CPU cost:** 12 exact-trace simulations, expected 20--45 CPU minutes; a one-cell TP2/MNS16 smoke precedes the full surface.
- **Calibration separation:** kernel microprofiles are independent measurements, never fitted to trace E2E latency. The frozen real trace audit is evaluation only.