Align Frontier piecewise graph profiles

2026-07-17 23:22:42 +08:00
parent 47355a9411
commit bdc357dc6c
10 changed files with 804 additions and 65 deletions
--- a/runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md
+++ b/runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md
@@ -0,0 +1,40 @@
+# EXP-SIMFID-Q30-GRAPH-PIECEWISE：graph-compatible kernel-only profile 是否修正 Frontier trace replay？
+
+> **状态：** approved and running（2026-07-17）。本卡是已纠正 prefix-trace contract 后的最小判别实验；不复用此前 `decode_cuda_graph_mode=none` 的数值作 fidelity verdict。
+
+## Purpose and hypotheses
+
+- **Parent claim：** Frontier 是否已经足以为 Qwen3-30B-A3B 的真实 trace serving surface 选择 config。
+- **Question：** 旧 Frontier replay 低估 decode service rate，是否主要是 simulator 使用 `none` 而真机使用 `FULL_AND_PIECEWISE`、并且没有向 Frontier 提供独立 `KERNEL_ONLY` profile family？
+- **G1 (graph-family omission)：** 用同一 vLLM 0.20/FA3/FlashInfer-CUTLASS stack 的 `RecordFunctionTracer` kernel-only measurements，加真实 capture buckets 和 Frontier `piecewise`，会显著缩小 TP2/MNS16 的 TPOT/service-rate gap，并至少改变一个 config 的 latency ranking。
+- **G2 (remaining composition error)：** 即使 graph family 对齐，TPOT、TTFT 或 E2E ranking 仍与真机不一致；则 graph omission 只是必要修正，不是 simulator 已解决 tuning 的证据。
+
+## Controlled setup
+
+| Item | Frozen choice |
+|---|---|
+| model/runtime/hardware | Qwen3-30B-A3B BF16; community vLLM 0.20.0 (`88d34c…`); dash0 NVIDIA H20 |
+| simulator | Frontier `deadc4a321f0baaa534c6ebd17f974123733cdc2`; no local source patch |
+| workload | exact 129-request Trace-PD public projection; exact ISL/OSL/arrival order; TP-normalized arrival time and complete 16-token prefix blocks |
+| surface | TP in {1,2,4}; MNS in {8,16,32,64}; MBT=8192; prefix/chunked prefill on |
+| real graph contract | observed vLLM capture sizes: MNS8=[1,2,4,8,16], MNS16=[1,2,4,8,16,24,32], MNS32=[1,2,4,8,16,24,32,40,48,56,64], MNS64=[1,2,4,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128] |
+| profile intervention | CUDA-event profile stays frozen for prefill/mixed batches. New `KERNEL_ONLY` linear, FA3 decode + KV-update, MoE, and router rows use Frontier's actual `RecordFunctionTracer` semantics; no relabeling of CUDA-event numbers. |
+| exact capacity | per-cell real observed KV block count and capture list; Frontier CPU-overhead model remains disabled on both old/new simulator runs because the intervention is GPU-kernel family only. |
+
+Frontier source inspection fixes the semantic boundary: `piecewise` emits `PIECEWISE` whenever a capture hits, but the MONOLITHIC predictor selects `KERNEL_ONLY` only when `num_prefill_tokens == 0`. Hence new profile coverage is pure decode only; captured mixed/prefill work continues to consume the existing CUDA-event family.
+
+## Measurement and decision rule
+
+- **Primary outputs：** per-config mean/p90 TTFT, TPOT, E2E; ranking for each metric; TP2/MNS16 per-request TPOT gap against the already frozen three-trial real audit.
+- **Validity gates：** every kernel CSV hash matches its manifest; every row says `KERNEL_ONLY`; every TP/capture-bucket/KV-context required by the runner is present; command records `piecewise`, per-cell blocks and capture sizes; each simulator cell completes all 129 requests.
+- **Decision:** G1 is supported only if the graph-aligned TP2/MNS16 TPOT median moves toward real **and** full-surface rank/error evidence improves. A single-cell timing improvement does not establish tuning sufficiency. If G2 holds, update the research claim to “Frontier has not solved tuning under trace-faithful MoE serving after graph-family alignment,” then profile stage/state composition rather than add arbitrary kernel rows.
+
+## Expected figure
+
+`graph-piecewise-profile-prototype.svg` is deliberately schematic. The final figure uses the same axes and adds real data only after the profile and replay validity gates pass.
+
+## Cost and provenance
+
+- **GPU cost:** three 1-GPU FA3 decode profile shards, plus one 1-GPU linear shard and one 1-GPU MoE/router shard; expected 1.5--3.0 H20-GPU-hours, hard cap 4.0 GPU-hours.
+- **CPU cost:** 12 exact-trace simulations, expected 20--45 CPU minutes; a one-cell TP2/MNS16 smoke precedes the full surface.
+- **Calibration separation:** kernel microprofiles are independent measurements, never fitted to trace E2E latency. The frozen real trace audit is evaluation only.
--- a/runs/frontier-fidelity-envelope-v1/graph-piecewise-profile-prototype.svg
+++ b/runs/frontier-fidelity-envelope-v1/graph-piecewise-profile-prototype.svg
@@ -0,0 +1,28 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="960" height="510" viewBox="0 0 960 510">
+  <rect width="960" height="510" fill="#fff"/>
+  <text x="40" y="38" font-family="sans-serif" font-size="20" font-weight="bold">SCHEMATIC — no measured data</text>
+  <text x="40" y="64" font-family="sans-serif" font-size="14">Does graph-compatible KERNEL_ONLY profiling make Frontier select the real trace-serving configuration?</text>
+  <g transform="translate(55 105)" font-family="sans-serif">
+    <text x="130" y="-15" font-size="16" font-weight="bold">A. TP2/MNS16 TPOT prediction</text>
+    <line x1="55" y1="250" x2="390" y2="250" stroke="#333"/>
+    <line x1="55" y1="250" x2="55" y2="20" stroke="#333"/>
+    <text x="0" y="25" font-size="12">latency</text><text x="185" y="285" font-size="12">measurement family</text>
+    <rect x="90" y="80" width="55" height="170" fill="#d55e00" opacity=".75"/>
+    <rect x="205" y="175" width="55" height="75" fill="#0072b2" opacity=".75"/>
+    <rect x="320" y="170" width="55" height="80" fill="#009e73" opacity=".75"/>
+    <text x="73" y="310" font-size="12">none</text><text x="181" y="310" font-size="12">piecewise</text><text x="315" y="310" font-size="12">real</text>
+    <text x="76" y="328" font-size="11">old sim</text><text x="180" y="328" font-size="11">G1: moves closer</text>
+  </g>
+  <g transform="translate(525 105)" font-family="sans-serif">
+    <text x="75" y="-15" font-size="16" font-weight="bold">B. Full 12-cell ranking agreement</text>
+    <line x1="55" y1="250" x2="385" y2="250" stroke="#333"/>
+    <line x1="55" y1="250" x2="55" y2="20" stroke="#333"/>
+    <text x="-3" y="25" font-size="12">rank error</text><text x="155" y="285" font-size="12">simulator variant</text>
+    <polyline points="92,62 205,170 320,178" fill="none" stroke="#0072b2" stroke-width="4"/>
+    <circle cx="92" cy="62" r="6" fill="#0072b2"/><circle cx="205" cy="170" r="6" fill="#0072b2"/><circle cx="320" cy="178" r="6" fill="#0072b2"/>
+    <line x1="55" y1="178" x2="385" y2="178" stroke="#009e73" stroke-dasharray="6 5"/>
+    <text x="72" y="310" font-size="12">none</text><text x="175" y="310" font-size="12">piecewise</text><text x="305" y="310" font-size="12">real rank</text>
+    <text x="76" y="328" font-size="11">G2: stays wrong</text><text x="170" y="328" font-size="11">G1: error falls</text>
+  </g>
+  <text x="42" y="480" font-family="sans-serif" font-size="12">Final figure reports mean/p90 TTFT, TPOT, E2E for the identical 129-request trace, not an SLO-derived proxy.</text>
+</svg>
--- a/runs/frontier-fidelity-envelope-v1/run_frontier_qwen30_exact_trace_surface.py
+++ b/runs/frontier-fidelity-envelope-v1/run_frontier_qwen30_exact_trace_surface.py
@@ -19,6 +19,27 @@ from typing import Any
 TARGET_PASS_RATE = 0.95
 TPOT_SLOS_MS = (50.0, 100.0, 150.0, 180.0)
 WINDOW_SECONDS = 600.0
+GRAPH_CAPTURE_SIZES_BY_MNS = {
+    8: (1, 2, 4, 8, 16),
+    16: (1, 2, 4, 8, 16, 24, 32),
+    32: (1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64),
+    64: (1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128),
+}
+REAL_NUM_BLOCKS_BY_CONFIG = {
+    (1, 8): 20137,
+    (1, 16): 20128,
+    (1, 32): 20108,
+    (1, 64): 20069,
+    (2, 8): 76639,
+    (2, 16): 76620,
+    (2, 32): 76583,
+    (2, 64): 76505,
+    (4, 8): 191930,
+    (4, 16): 191882,
+    (4, 32): 191786,
+    (4, 64): 191589,
+}
+KERNEL_DECODE_KV_CONTEXTS = (128, 1024, 2048, 4096, 8192, 16384, 32768, 40960)
 BASE_RUNNER = (
    Path(__file__).resolve().parents[1]
    / "frontier-phase-factorial-v0/run_frontier_qwen30_prefill_surface.py"
@@ -43,6 +64,7 @@ def parse_args() -> argparse.Namespace:
    parser.add_argument("--frontier-source", type=Path, required=True)
    parser.add_argument("--replayserve-root", type=Path, required=True)
    parser.add_argument("--profile-root", type=Path, required=True)
+    parser.add_argument("--kernel-profile-root", type=Path)
    parser.add_argument("--python-deps", type=Path, required=True)
    parser.add_argument("--output-root", type=Path, required=True)
    parser.add_argument(
@@ -68,6 +90,21 @@ def parse_args() -> argparse.Namespace:
    parser.add_argument("--allreduce-csv", type=Path)
    parser.add_argument("--timeout-seconds", type=float, default=1800.0)
    parser.add_argument("--predictor-training-job-threads", type=int, default=1)
+    parser.add_argument(
+        "--decode-cuda-graph-mode",
+        choices=("none", "full_decode_only", "piecewise"),
+        default="none",
+    )
+    parser.add_argument(
+        "--align-real-graph-runtime",
+        action="store_true",
+        help="Use real observed capture lists and per-(TP,MNS) KV blocks.",
+    )
+    parser.add_argument(
+        "--fresh-predictor-cache",
+        action="store_true",
+        help="Disable Frontier predictor cache reuse for this profile family.",
+    )
    parser.add_argument("--resume", action="store_true")
    parser.add_argument("--continue-on-failure", action="store_true")
    return parser.parse_args()
@@ -241,15 +278,99 @@ def score(path: Path, expected_shapes: list[tuple[int, int]]) -> dict[str, Any]:
        }
    ttfts = [float(row["ttft_ms"]) for row in request_metrics]
    tpots = [float(row["tpot_ms"]) for row in request_metrics if row["tpot_ms"] is not None]
+    e2es = [float(row["e2e_ms"]) for row in request_metrics]
    return {
+        "ttft_mean_ms": sum(ttfts) / len(ttfts),
        "ttft_p50_ms": percentile(ttfts, 0.50),
+        "ttft_p90_ms": percentile(ttfts, 0.90),
        "ttft_p95_ms": percentile(ttfts, 0.95),
+        "tpot_mean_ms": sum(tpots) / len(tpots),
        "tpot_p50_ms": percentile(tpots, 0.50),
+        "tpot_p90_ms": percentile(tpots, 0.90),
        "tpot_p95_ms": percentile(tpots, 0.95),
+        "e2e_mean_ms": sum(e2es) / len(e2es),
+        "e2e_p50_ms": percentile(e2es, 0.50),
+        "e2e_p90_ms": percentile(e2es, 0.90),
+        "e2e_p95_ms": percentile(e2es, 0.95),
        "slos": slos,
    }


+def kernel_profile_paths(root: Path) -> dict[str, Path]:
+    paths = {
+        "linear": root / "linear_op.csv",
+        "attention": root / "attention.csv",
+        "moe": root / "moe.csv",
+        "manifest": root / "manifest.json",
+    }
+    missing = [str(path) for path in paths.values() if not path.is_file()]
+    if missing:
+        raise FileNotFoundError(missing)
+    return paths
+
+
+def validate_kernel_profile(paths: dict[str, Path]) -> dict[str, Any]:
+    manifest = json.loads(paths["manifest"].read_text())
+    outputs = manifest.get("outputs", {})
+    for filename, name in (
+        ("linear_op.csv", "linear"),
+        ("attention.csv", "attention"),
+        ("moe.csv", "moe"),
+    ):
+        if outputs.get(filename) != BASE.sha256(paths[name]):
+            raise ValueError(f"kernel-only profile hash mismatch for {filename}")
+
+    with paths["linear"].open(newline="") as source:
+        linear_rows = list(csv.DictReader(source))
+    with paths["attention"].open(newline="") as source:
+        attention_rows = list(csv.DictReader(source))
+    with paths["moe"].open(newline="") as source:
+        moe_rows = list(csv.DictReader(source))
+    for label, rows in (("linear", linear_rows), ("attention", attention_rows), ("moe", moe_rows)):
+        if not rows or {row.get("measurement_type") for row in rows} != {"KERNEL_ONLY"}:
+            raise ValueError(f"{label} lacks an exclusive KERNEL_ONLY measurement family")
+
+    required_buckets = set(GRAPH_CAPTURE_SIZES_BY_MNS[64])
+    coverage: dict[str, Any] = {}
+    for tp in (1, 2, 4):
+        linear_tokens = {
+            int(float(row["num_tokens"]))
+            for row in linear_rows
+            if int(float(row["num_tensor_parallel_workers"])) == tp
+        }
+        moe_tokens = {
+            int(float(row["num_tokens"]))
+            for row in moe_rows
+            if int(float(row["num_tensor_parallel_workers"])) == tp
+        }
+        attention_pairs = {
+            (int(float(row["batch_size"])), int(float(row["kv_cache_size"])))
+            for row in attention_rows
+            if int(float(row["num_tensor_parallel_workers"])) == tp
+            and row["is_prefill"].lower() == "false"
+            and row.get("is_true_mixed_batch", "").lower() != "true"
+        }
+        missing_linear = required_buckets - linear_tokens
+        missing_moe = required_buckets - moe_tokens
+        missing_attention = {
+            (bucket, kv)
+            for bucket in required_buckets
+            for kv in KERNEL_DECODE_KV_CONTEXTS
+            if (bucket, kv) not in attention_pairs
+        }
+        if missing_linear or missing_moe or missing_attention:
+            raise ValueError(
+                f"kernel-only profile coverage TP{tp}: linear={sorted(missing_linear)}, "
+                f"moe={sorted(missing_moe)}, attention={sorted(missing_attention)}"
+            )
+        coverage[str(tp)] = {
+            "linear_tokens": sorted(linear_tokens),
+            "moe_tokens": sorted(moe_tokens),
+            "attention_decode_pairs": len(attention_pairs),
+        }
+    return {"manifest": manifest, "coverage": coverage}
+
+
 def main() -> None:
    args = parse_args()
    if args.predictor_training_job_threads <= 0:
@@ -264,6 +385,10 @@ def main() -> None:
        setattr(args, name, getattr(args, name).resolve())
    if args.allreduce_csv is not None:
        args.allreduce_csv = args.allreduce_csv.resolve()
+    if args.kernel_profile_root is not None:
+        args.kernel_profile_root = args.kernel_profile_root.resolve()
+        if args.decode_cuda_graph_mode == "none":
+            raise ValueError("--kernel-profile-root requires a non-none graph mode")
    traces = [
        parse_trace(
            specification,
@@ -288,6 +413,11 @@ def main() -> None:
            raise ValueError(f"unknown configs: {wanted - {config.name for config in selected}}")
    paths = BASE.profile_paths(args.profile_root)
    coverage = BASE.validate_profile(paths)
+    kernel_paths = None
+    kernel_coverage = None
+    if args.kernel_profile_root is not None:
+        kernel_paths = kernel_profile_paths(args.kernel_profile_root)
+        kernel_coverage = validate_kernel_profile(kernel_paths)
    builder = BASE.load_module(
        "qwen30_exact_trace_frontier_builder",
        args.replayserve_root / "tools/run_frontier_sweep.py",
@@ -320,6 +450,18 @@ def main() -> None:
        config_knobs = BASE.knobs(config, paths, args.output_root / "cache")
        config_knobs["enable_prefix_caching"] = args.prefix_caching
        config_knobs["prediction_max_tokens_per_request"] = 40960
+        config_knobs["decode_cuda_graph_mode"] = args.decode_cuda_graph_mode
+        config_knobs["no_cache"] = args.fresh_predictor_cache
+        if args.align_real_graph_runtime:
+            config_knobs["num_blocks"] = REAL_NUM_BLOCKS_BY_CONFIG[(config.tp, config.mns)]
+        if kernel_paths is not None:
+            config_knobs.update(
+                {
+                    "linear_op_kernel_only_input_file": str(kernel_paths["linear"]),
+                    "atten_kernel_only_input_file": str(kernel_paths["attention"]),
+                    "moe_kernel_only_input_file": str(kernel_paths["moe"]),
+                }
+            )
        for trace in traces:
            run_dir = args.output_root / "runs" / config.name / trace["label"]
            result_path = run_dir / "result.json"
@@ -340,6 +482,13 @@ def main() -> None:
                    str(args.predictor_training_job_threads),
                ]
            )
+            if args.align_real_graph_runtime:
+                command.extend(
+                    [
+                        "--cudagraph_capture_sizes",
+                        *(str(size) for size in GRAPH_CAPTURE_SIZES_BY_MNS[config.mns]),
+                    ]
+                )
            command = BASE.configure_cc_command(
                command,
                backend=args.cc_backend,
@@ -514,6 +663,9 @@ def main() -> None:
            "primary_tpot_slo_ms": 150.0,
            "target_pass_rate": TARGET_PASS_RATE,
            "predictor_training_job_threads": args.predictor_training_job_threads,
+            "decode_cuda_graph_mode": args.decode_cuda_graph_mode,
+            "align_real_graph_runtime": args.align_real_graph_runtime,
+            "fresh_predictor_cache": args.fresh_predictor_cache,
        },
        "frontier": {
            "source": str(args.frontier_source),
@@ -530,6 +682,30 @@ def main() -> None:
            "coverage": coverage,
            "sha256": {name: BASE.sha256(path) for name, path in paths.items()},
        },
+        "kernel_only_profiles": (
+            None
+            if kernel_paths is None
+            else {
+                "root": str(args.kernel_profile_root),
+                "coverage": kernel_coverage,
+                "sha256": {
+                    name: BASE.sha256(path) for name, path in kernel_paths.items()
+                },
+            }
+        ),
+        "runtime_alignment": {
+            "capture_sizes_by_mns": (
+                GRAPH_CAPTURE_SIZES_BY_MNS if args.align_real_graph_runtime else None
+            ),
+            "num_blocks_by_config": (
+                {
+                    f"tp{tp}_mns{mns}": blocks
+                    for (tp, mns), blocks in REAL_NUM_BLOCKS_BY_CONFIG.items()
+                }
+                if args.align_real_graph_runtime
+                else None
+            ),
+        },
        "collective": {
            "backend": args.cc_backend,
            "allreduce_csv": str(args.allreduce_csv) if args.allreduce_csv else None,
--- a/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_attention.sh
+++ b/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_attention.sh
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+TP="${TP:?TP must be 1, 2, or 4}"
+case "${TP}" in
+  1|2|4) ;;
+  *) echo "invalid TP=${TP}" >&2; exit 1 ;;
+esac
+
+OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT must be set}"
+RUN_DIR="$(pwd -P)"
+PROFILE_DIR="${PROFILE_DIR:-${RUN_DIR%/runs/frontier-fidelity-envelope-v1}/runs/frontier-qwen30-vllm020-profile-v1}"
+VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}"
+VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}"
+FRONTIER_SOURCE="${FRONTIER_SOURCE:-/home/admin/cpfs/wjh/aituner/frontier-t1-dash0-deadc4a}"
+MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}"
+CAPTURE_BUCKETS="${CAPTURE_BUCKETS:-1 2 4 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128}"
+KV_CONTEXTS="${KV_CONTEXTS:-128 1024 2048 4096 8192 16384 32768 40960}"
+WARMUP_ITERS="${WARMUP_ITERS:-3}"
+REPEATS="${REPEATS:-5}"
+
+mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" "${OUTPUT_ROOT}/raw"
+exec > >(tee -a "${OUTPUT_ROOT}/logs/attention-tp${TP}.log") 2>&1
+
+IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?a fleet-allocated GPU is required}"
+if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then
+  echo "expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2
+  exit 1
+fi
+
+BATCH_SPECS=()
+for bucket in ${CAPTURE_BUCKETS}; do
+  prefix=""
+  if [[ "${bucket}" -ne 1 ]]; then
+    prefix="${bucket}"
+  fi
+  for context in ${KV_CONTEXTS}; do
+    BATCH_SPECS+=("${prefix}q1s${context}")
+  done
+done
+
+echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} role=FA3-decode-kernel-only tp=${TP} buckets='${CAPTURE_BUCKETS}' kv_contexts='${KV_CONTEXTS}' method=Frontier-RecordFunctionTracer output=${OUTPUT_ROOT} expected_wall=10-35m expected_gpu_cap=1.0_H20h"
+date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ"
+nvidia-smi --query-gpu=index,name,driver_version,memory.used,utilization.gpu --format=csv,noheader
+test "$(git -C "${FRONTIER_SOURCE}" rev-parse HEAD)" = "deadc4a321f0baaa534c6ebd17f974123733cdc2"
+test "$(git -C "${VLLM_SOURCE}" rev-parse HEAD)" = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1"
+test -f "${MODEL_ROOT}/config.json"
+git rev-parse HEAD > "${OUTPUT_ROOT}/provenance/aituner.commit"
+git -C "${FRONTIER_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/frontier.commit"
+git -C "${VLLM_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/vllm.commit"
+printf '%s\n' "${BATCH_SPECS[@]}" > "${OUTPUT_ROOT}/provenance/batch-specs.txt"
+sha256sum "${PROFILE_DIR}/profile_vllm020_flashattn.py" "${RUN_DIR}/run_graph_kernel_only_attention.sh" > "${OUTPUT_ROOT}/provenance/source.sha256"
+
+timeout --signal=TERM --kill-after=30s 2400 \
+  "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/profile_vllm020_flashattn.py" \
+  --vllm-source "${VLLM_SOURCE}" \
+  --frontier-source "${FRONTIER_SOURCE}" \
+  --model "${MODEL_ROOT}" \
+  --output "${OUTPUT_ROOT}/raw/attention-tp${TP}.json" \
+  --tp "${TP}" \
+  --batch-specs "${BATCH_SPECS[@]}" \
+  --warmup-iters "${WARMUP_ITERS}" \
+  --repeats "${REPEATS}" \
+  --profile-kv-update \
+  --profile-method record_function
+
+test -s "${OUTPUT_ROOT}/raw/attention-tp${TP}.json"
+sha256sum "${OUTPUT_ROOT}/raw/attention-tp${TP}.json" "${OUTPUT_ROOT}/provenance"/* > "${OUTPUT_ROOT}/artifacts.sha256"
+date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ"
+echo "GRAPH_KERNEL_ONLY_ATTENTION_COMPLETE tp=${TP} rows=${#BATCH_SPECS[@]}"
--- a/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_linear.sh
+++ b/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_linear.sh
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT must be set}"
+RUN_DIR="$(pwd -P)"
+PROFILE_DIR="${PROFILE_DIR:-${RUN_DIR%/runs/frontier-fidelity-envelope-v1}/runs/frontier-qwen30-vllm020-profile-v1}"
+VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}"
+VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}"
+FRONTIER_SOURCE="${FRONTIER_SOURCE:-/home/admin/cpfs/wjh/aituner/frontier-t1-dash0-deadc4a}"
+MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}"
+TOKENS=(1 2 4 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128)
+export MODEL_ROOT
+
+mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" "${OUTPUT_ROOT}/profiles"
+exec > >(tee -a "${OUTPUT_ROOT}/logs/linear.log") 2>&1
+IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?a fleet-allocated GPU is required}"
+if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then
+  echo "expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2
+  exit 1
+fi
+
+echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} role=linear-kernel-only tp=1,2,4 tokens='${TOKENS[*]}' method=Frontier-RecordFunctionTracer output=${OUTPUT_ROOT} expected_wall=15-35m expected_gpu_cap=1.0_H20h"
+date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ"
+test "$(git -C "${FRONTIER_SOURCE}" rev-parse HEAD)" = "deadc4a321f0baaa534c6ebd17f974123733cdc2"
+test "$(git -C "${VLLM_SOURCE}" rev-parse HEAD)" = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1"
+git rev-parse HEAD > "${OUTPUT_ROOT}/provenance/aituner.commit"
+git -C "${FRONTIER_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/frontier.commit"
+sha256sum "${PROFILE_DIR}/frontier_vllm020_compat.py" "${RUN_DIR}/run_graph_kernel_only_linear.sh" > "${OUTPUT_ROOT}/provenance/source.sha256"
+
+cd "${FRONTIER_SOURCE}"
+timeout --signal=TERM --kill-after=30s 2400 \
+  "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/frontier_vllm020_compat.py" \
+  --disable_ray --num_gpus 1 --output_dir "${OUTPUT_ROOT}/profiles" \
+  --device h20 --models qwen3-a3b-30b-moe \
+  --num_tensor_parallel_workers 1 2 4 --max_tokens 128 \
+  --num_tokens_list "${TOKENS[@]}" --profile_method record_function \
+  --precision BF16 --is_moe --yes
+
+find "${OUTPUT_ROOT}/profiles" -name linear_op_kernel_only.csv -type f -size +0c -print -quit > "${OUTPUT_ROOT}/provenance/linear-path.txt"
+test -s "${OUTPUT_ROOT}/provenance/linear-path.txt"
+date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ"
+echo "GRAPH_KERNEL_ONLY_LINEAR_COMPLETE"
--- a/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_moe.sh
+++ b/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_moe.sh
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT must be set}"
+RUN_DIR="$(pwd -P)"
+PROFILE_DIR="${PROFILE_DIR:-${RUN_DIR%/runs/frontier-fidelity-envelope-v1}/runs/frontier-qwen30-vllm020-profile-v1}"
+VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}"
+VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}"
+FRONTIER_SOURCE="${FRONTIER_SOURCE:-/home/admin/cpfs/wjh/aituner/frontier-t1-dash0-deadc4a}"
+MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}"
+TOKENS="${TOKENS:-1 2 4 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128}"
+
+mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" "${OUTPUT_ROOT}/raw"
+exec > >(tee -a "${OUTPUT_ROOT}/logs/moe-router.log") 2>&1
+IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?a fleet-allocated GPU is required}"
+if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then
+  echo "expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2
+  exit 1
+fi
+
+echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} role=MoE+router-kernel-only tp=1,2,4 tokens='${TOKENS}' method=Frontier-RecordFunctionTracer output=${OUTPUT_ROOT} expected_wall=10-30m expected_gpu_cap=1.0_H20h"
+date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ"
+test "$(git -C "${FRONTIER_SOURCE}" rev-parse HEAD)" = "deadc4a321f0baaa534c6ebd17f974123733cdc2"
+test "$(git -C "${VLLM_SOURCE}" rev-parse HEAD)" = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1"
+git rev-parse HEAD > "${OUTPUT_ROOT}/provenance/aituner.commit"
+git -C "${FRONTIER_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/frontier.commit"
+printf '%s\n' ${TOKENS} > "${OUTPUT_ROOT}/provenance/tokens.txt"
+sha256sum "${PROFILE_DIR}/profile_vllm020_moe.py" "${PROFILE_DIR}/profile_vllm020_router.py" "${RUN_DIR}/run_graph_kernel_only_moe.sh" > "${OUTPUT_ROOT}/provenance/source.sha256"
+
+timeout --signal=TERM --kill-after=30s 2400 \
+  "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/profile_vllm020_moe.py" \
+  --vllm-source "${VLLM_SOURCE}" --frontier-source "${FRONTIER_SOURCE}" \
+  --model "${MODEL_ROOT}" --output "${OUTPUT_ROOT}/raw/moe.json" \
+  --tp 1 2 4 --num-tokens ${TOKENS} --routing-modes uniform_random_logits \
+  --warmup-iters 3 --repeats 5 --profile-method record_function --check-reference
+
+timeout --signal=TERM --kill-after=30s 1800 \
+  "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/profile_vllm020_router.py" \
+  --vllm-source "${VLLM_SOURCE}" --frontier-source "${FRONTIER_SOURCE}" \
+  --model "${MODEL_ROOT}" --output "${OUTPUT_ROOT}/raw/router.json" \
+  --num-tokens ${TOKENS} --warmup-iters 3 --repeats 5 --profile-method record_function
+
+test -s "${OUTPUT_ROOT}/raw/moe.json"
+test -s "${OUTPUT_ROOT}/raw/router.json"
+sha256sum "${OUTPUT_ROOT}/raw"/*.json "${OUTPUT_ROOT}/provenance"/* > "${OUTPUT_ROOT}/artifacts.sha256"
+date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ"
+echo "GRAPH_KERNEL_ONLY_MOE_COMPLETE"