Align Frontier piecewise graph profiles

runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md

@@ -0,0 +1,40 @@
+# EXP-SIMFID-Q30-GRAPH-PIECEWISE：graph-compatible kernel-only profile 是否修正 Frontier trace replay？
+
+> **状态：** approved and running（2026-07-17）。本卡是已纠正 prefix-trace contract 后的最小判别实验；不复用此前 `decode_cuda_graph_mode=none` 的数值作 fidelity verdict。
+
+## Purpose and hypotheses
+
+- **Parent claim：** Frontier 是否已经足以为 Qwen3-30B-A3B 的真实 trace serving surface 选择 config。
+- **Question：** 旧 Frontier replay 低估 decode service rate，是否主要是 simulator 使用 `none` 而真机使用 `FULL_AND_PIECEWISE`、并且没有向 Frontier 提供独立 `KERNEL_ONLY` profile family？
+- **G1 (graph-family omission)：** 用同一 vLLM 0.20/FA3/FlashInfer-CUTLASS stack 的 `RecordFunctionTracer` kernel-only measurements，加真实 capture buckets 和 Frontier `piecewise`，会显著缩小 TP2/MNS16 的 TPOT/service-rate gap，并至少改变一个 config 的 latency ranking。
+- **G2 (remaining composition error)：** 即使 graph family 对齐，TPOT、TTFT 或 E2E ranking 仍与真机不一致；则 graph omission 只是必要修正，不是 simulator 已解决 tuning 的证据。
+
+## Controlled setup
+
+| Item | Frozen choice |
+|---|---|
+| model/runtime/hardware | Qwen3-30B-A3B BF16; community vLLM 0.20.0 (`88d34c…`); dash0 NVIDIA H20 |
+| simulator | Frontier `deadc4a321f0baaa534c6ebd17f974123733cdc2`; no local source patch |
+| workload | exact 129-request Trace-PD public projection; exact ISL/OSL/arrival order; TP-normalized arrival time and complete 16-token prefix blocks |
+| surface | TP in {1,2,4}; MNS in {8,16,32,64}; MBT=8192; prefix/chunked prefill on |
+| real graph contract | observed vLLM capture sizes: MNS8=[1,2,4,8,16], MNS16=[1,2,4,8,16,24,32], MNS32=[1,2,4,8,16,24,32,40,48,56,64], MNS64=[1,2,4,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128] |
+| profile intervention | CUDA-event profile stays frozen for prefill/mixed batches. New `KERNEL_ONLY` linear, FA3 decode + KV-update, MoE, and router rows use Frontier's actual `RecordFunctionTracer` semantics; no relabeling of CUDA-event numbers. |
+| exact capacity | per-cell real observed KV block count and capture list; Frontier CPU-overhead model remains disabled on both old/new simulator runs because the intervention is GPU-kernel family only. |
+
+Frontier source inspection fixes the semantic boundary: `piecewise` emits `PIECEWISE` whenever a capture hits, but the MONOLITHIC predictor selects `KERNEL_ONLY` only when `num_prefill_tokens == 0`. Hence new profile coverage is pure decode only; captured mixed/prefill work continues to consume the existing CUDA-event family.
+
+## Measurement and decision rule
+
+- **Primary outputs：** per-config mean/p90 TTFT, TPOT, E2E; ranking for each metric; TP2/MNS16 per-request TPOT gap against the already frozen three-trial real audit.
+- **Validity gates：** every kernel CSV hash matches its manifest; every row says `KERNEL_ONLY`; every TP/capture-bucket/KV-context required by the runner is present; command records `piecewise`, per-cell blocks and capture sizes; each simulator cell completes all 129 requests.
+- **Decision:** G1 is supported only if the graph-aligned TP2/MNS16 TPOT median moves toward real **and** full-surface rank/error evidence improves. A single-cell timing improvement does not establish tuning sufficiency. If G2 holds, update the research claim to “Frontier has not solved tuning under trace-faithful MoE serving after graph-family alignment,” then profile stage/state composition rather than add arbitrary kernel rows.
+
+## Expected figure
+
+`graph-piecewise-profile-prototype.svg` is deliberately schematic. The final figure uses the same axes and adds real data only after the profile and replay validity gates pass.
+
+## Cost and provenance
+
+- **GPU cost:** three 1-GPU FA3 decode profile shards, plus one 1-GPU linear shard and one 1-GPU MoE/router shard; expected 1.5--3.0 H20-GPU-hours, hard cap 4.0 GPU-hours.
+- **CPU cost:** 12 exact-trace simulations, expected 20--45 CPU minutes; a one-cell TP2/MNS16 smoke precedes the full surface.
+- **Calibration separation:** kernel microprofiles are independent measurements, never fitted to trace E2E latency. The frozen real trace audit is evaluation only.

runs/frontier-fidelity-envelope-v1/graph-piecewise-profile-prototype.svg

@@ -0,0 +1,28 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="960" height="510" viewBox="0 0 960 510">
+  <rect width="960" height="510" fill="#fff"/>
+  <text x="40" y="38" font-family="sans-serif" font-size="20" font-weight="bold">SCHEMATIC — no measured data</text>
+  <text x="40" y="64" font-family="sans-serif" font-size="14">Does graph-compatible KERNEL_ONLY profiling make Frontier select the real trace-serving configuration?</text>
+  <g transform="translate(55 105)" font-family="sans-serif">
+    <text x="130" y="-15" font-size="16" font-weight="bold">A. TP2/MNS16 TPOT prediction</text>
+    <line x1="55" y1="250" x2="390" y2="250" stroke="#333"/>
+    <line x1="55" y1="250" x2="55" y2="20" stroke="#333"/>
+    <text x="0" y="25" font-size="12">latency</text><text x="185" y="285" font-size="12">measurement family</text>
+    <rect x="90" y="80" width="55" height="170" fill="#d55e00" opacity=".75"/>
+    <rect x="205" y="175" width="55" height="75" fill="#0072b2" opacity=".75"/>
+    <rect x="320" y="170" width="55" height="80" fill="#009e73" opacity=".75"/>
+    <text x="73" y="310" font-size="12">none</text><text x="181" y="310" font-size="12">piecewise</text><text x="315" y="310" font-size="12">real</text>
+    <text x="76" y="328" font-size="11">old sim</text><text x="180" y="328" font-size="11">G1: moves closer</text>
+  </g>
+  <g transform="translate(525 105)" font-family="sans-serif">
+    <text x="75" y="-15" font-size="16" font-weight="bold">B. Full 12-cell ranking agreement</text>
+    <line x1="55" y1="250" x2="385" y2="250" stroke="#333"/>
+    <line x1="55" y1="250" x2="55" y2="20" stroke="#333"/>
+    <text x="-3" y="25" font-size="12">rank error</text><text x="155" y="285" font-size="12">simulator variant</text>
+    <polyline points="92,62 205,170 320,178" fill="none" stroke="#0072b2" stroke-width="4"/>
+    <circle cx="92" cy="62" r="6" fill="#0072b2"/><circle cx="205" cy="170" r="6" fill="#0072b2"/><circle cx="320" cy="178" r="6" fill="#0072b2"/>
+    <line x1="55" y1="178" x2="385" y2="178" stroke="#009e73" stroke-dasharray="6 5"/>
+    <text x="72" y="310" font-size="12">none</text><text x="175" y="310" font-size="12">piecewise</text><text x="305" y="310" font-size="12">real rank</text>
+    <text x="76" y="328" font-size="11">G2: stays wrong</text><text x="170" y="328" font-size="11">G1: error falls</text>
+  </g>
+  <text x="42" y="480" font-family="sans-serif" font-size="12">Final figure reports mean/p90 TTFT, TPOT, E2E for the identical 129-request trace, not an SLO-derived proxy.</text>
+</svg>

runs/frontier-fidelity-envelope-v1/run_frontier_qwen30_exact_trace_surface.py

@@ -19,6 +19,27 @@ from typing import Any
 TARGET_PASS_RATE = 0.95
 TPOT_SLOS_MS = (50.0, 100.0, 150.0, 180.0)
 WINDOW_SECONDS = 600.0
+GRAPH_CAPTURE_SIZES_BY_MNS = {
+    8: (1, 2, 4, 8, 16),
+    16: (1, 2, 4, 8, 16, 24, 32),
+    32: (1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64),
+    64: (1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128),
+}
+REAL_NUM_BLOCKS_BY_CONFIG = {
+    (1, 8): 20137,
+    (1, 16): 20128,
+    (1, 32): 20108,
+    (1, 64): 20069,
+    (2, 8): 76639,
+    (2, 16): 76620,
+    (2, 32): 76583,
+    (2, 64): 76505,
+    (4, 8): 191930,
+    (4, 16): 191882,
+    (4, 32): 191786,
+    (4, 64): 191589,
+}
+KERNEL_DECODE_KV_CONTEXTS = (128, 1024, 2048, 4096, 8192, 16384, 32768, 40960)
 BASE_RUNNER = (
     Path(__file__).resolve().parents[1]
     / "frontier-phase-factorial-v0/run_frontier_qwen30_prefill_surface.py"
@@ -43,6 +64,7 @@ def parse_args() -> argparse.Namespace:
     parser.add_argument("--frontier-source", type=Path, required=True)
     parser.add_argument("--replayserve-root", type=Path, required=True)
     parser.add_argument("--profile-root", type=Path, required=True)
+    parser.add_argument("--kernel-profile-root", type=Path)
     parser.add_argument("--python-deps", type=Path, required=True)
     parser.add_argument("--output-root", type=Path, required=True)
     parser.add_argument(
@@ -68,6 +90,21 @@ def parse_args() -> argparse.Namespace:
     parser.add_argument("--allreduce-csv", type=Path)
     parser.add_argument("--timeout-seconds", type=float, default=1800.0)
     parser.add_argument("--predictor-training-job-threads", type=int, default=1)
+    parser.add_argument(
+        "--decode-cuda-graph-mode",
+        choices=("none", "full_decode_only", "piecewise"),
+        default="none",
+    )
+    parser.add_argument(
+        "--align-real-graph-runtime",
+        action="store_true",
+        help="Use real observed capture lists and per-(TP,MNS) KV blocks.",
+    )
+    parser.add_argument(
+        "--fresh-predictor-cache",
+        action="store_true",
+        help="Disable Frontier predictor cache reuse for this profile family.",
+    )
     parser.add_argument("--resume", action="store_true")
     parser.add_argument("--continue-on-failure", action="store_true")
     return parser.parse_args()
@@ -241,15 +278,99 @@ def score(path: Path, expected_shapes: list[tuple[int, int]]) -> dict[str, Any]:
         }
     ttfts = [float(row["ttft_ms"]) for row in request_metrics]
     tpots = [float(row["tpot_ms"]) for row in request_metrics if row["tpot_ms"] is not None]
+    e2es = [float(row["e2e_ms"]) for row in request_metrics]
     return {
+        "ttft_mean_ms": sum(ttfts) / len(ttfts),
         "ttft_p50_ms": percentile(ttfts, 0.50),
+        "ttft_p90_ms": percentile(ttfts, 0.90),
         "ttft_p95_ms": percentile(ttfts, 0.95),
+        "tpot_mean_ms": sum(tpots) / len(tpots),
         "tpot_p50_ms": percentile(tpots, 0.50),
+        "tpot_p90_ms": percentile(tpots, 0.90),
         "tpot_p95_ms": percentile(tpots, 0.95),
+        "e2e_mean_ms": sum(e2es) / len(e2es),
+        "e2e_p50_ms": percentile(e2es, 0.50),
+        "e2e_p90_ms": percentile(e2es, 0.90),
+        "e2e_p95_ms": percentile(e2es, 0.95),
         "slos": slos,
     }
 
 
+def kernel_profile_paths(root: Path) -> dict[str, Path]:
+    paths = {
+        "linear": root / "linear_op.csv",
+        "attention": root / "attention.csv",
+        "moe": root / "moe.csv",
+        "manifest": root / "manifest.json",
+    }
+    missing = [str(path) for path in paths.values() if not path.is_file()]
+    if missing:
+        raise FileNotFoundError(missing)
+    return paths
+
+
+def validate_kernel_profile(paths: dict[str, Path]) -> dict[str, Any]:
+    manifest = json.loads(paths["manifest"].read_text())
+    outputs = manifest.get("outputs", {})
+    for filename, name in (
+        ("linear_op.csv", "linear"),
+        ("attention.csv", "attention"),
+        ("moe.csv", "moe"),
+    ):
+        if outputs.get(filename) != BASE.sha256(paths[name]):
+            raise ValueError(f"kernel-only profile hash mismatch for {filename}")
+
+    with paths["linear"].open(newline="") as source:
+        linear_rows = list(csv.DictReader(source))
+    with paths["attention"].open(newline="") as source:
+        attention_rows = list(csv.DictReader(source))
+    with paths["moe"].open(newline="") as source:
+        moe_rows = list(csv.DictReader(source))
+    for label, rows in (("linear", linear_rows), ("attention", attention_rows), ("moe", moe_rows)):
+        if not rows or {row.get("measurement_type") for row in rows} != {"KERNEL_ONLY"}:
+            raise ValueError(f"{label} lacks an exclusive KERNEL_ONLY measurement family")
+
+    required_buckets = set(GRAPH_CAPTURE_SIZES_BY_MNS[64])
+    coverage: dict[str, Any] = {}
+    for tp in (1, 2, 4):
+        linear_tokens = {
+            int(float(row["num_tokens"]))
+            for row in linear_rows
+            if int(float(row["num_tensor_parallel_workers"])) == tp
+        }
+        moe_tokens = {
+            int(float(row["num_tokens"]))
+            for row in moe_rows
+            if int(float(row["num_tensor_parallel_workers"])) == tp
+        }
+        attention_pairs = {
+            (int(float(row["batch_size"])), int(float(row["kv_cache_size"])))
+            for row in attention_rows
+            if int(float(row["num_tensor_parallel_workers"])) == tp
+            and row["is_prefill"].lower() == "false"
+            and row.get("is_true_mixed_batch", "").lower() != "true"
+        }
+        missing_linear = required_buckets - linear_tokens
+        missing_moe = required_buckets - moe_tokens
+        missing_attention = {
+            (bucket, kv)
+            for bucket in required_buckets
+            for kv in KERNEL_DECODE_KV_CONTEXTS
+            if (bucket, kv) not in attention_pairs
+        }
+        if missing_linear or missing_moe or missing_attention:
+            raise ValueError(
+                f"kernel-only profile coverage TP{tp}: linear={sorted(missing_linear)}, "
+                f"moe={sorted(missing_moe)}, attention={sorted(missing_attention)}"
+            )
+        coverage[str(tp)] = {
+            "linear_tokens": sorted(linear_tokens),
+            "moe_tokens": sorted(moe_tokens),
+            "attention_decode_pairs": len(attention_pairs),
+        }
+    return {"manifest": manifest, "coverage": coverage}
+
+
 def main() -> None:
     args = parse_args()
     if args.predictor_training_job_threads <= 0:
@@ -264,6 +385,10 @@ def main() -> None:
         setattr(args, name, getattr(args, name).resolve())
     if args.allreduce_csv is not None:
         args.allreduce_csv = args.allreduce_csv.resolve()
+    if args.kernel_profile_root is not None:
+        args.kernel_profile_root = args.kernel_profile_root.resolve()
+        if args.decode_cuda_graph_mode == "none":
+            raise ValueError("--kernel-profile-root requires a non-none graph mode")
     traces = [
         parse_trace(
             specification,
@@ -288,6 +413,11 @@ def main() -> None:
             raise ValueError(f"unknown configs: {wanted - {config.name for config in selected}}")
     paths = BASE.profile_paths(args.profile_root)
     coverage = BASE.validate_profile(paths)
+    kernel_paths = None
+    kernel_coverage = None
+    if args.kernel_profile_root is not None:
+        kernel_paths = kernel_profile_paths(args.kernel_profile_root)
+        kernel_coverage = validate_kernel_profile(kernel_paths)
     builder = BASE.load_module(
         "qwen30_exact_trace_frontier_builder",
         args.replayserve_root / "tools/run_frontier_sweep.py",
@@ -320,6 +450,18 @@ def main() -> None:
         config_knobs = BASE.knobs(config, paths, args.output_root / "cache")
         config_knobs["enable_prefix_caching"] = args.prefix_caching
         config_knobs["prediction_max_tokens_per_request"] = 40960
+        config_knobs["decode_cuda_graph_mode"] = args.decode_cuda_graph_mode
+        config_knobs["no_cache"] = args.fresh_predictor_cache
+        if args.align_real_graph_runtime:
+            config_knobs["num_blocks"] = REAL_NUM_BLOCKS_BY_CONFIG[(config.tp, config.mns)]
+        if kernel_paths is not None:
+            config_knobs.update(
+                {
+                    "linear_op_kernel_only_input_file": str(kernel_paths["linear"]),
+                    "atten_kernel_only_input_file": str(kernel_paths["attention"]),
+                    "moe_kernel_only_input_file": str(kernel_paths["moe"]),
+                }
+            )
         for trace in traces:
             run_dir = args.output_root / "runs" / config.name / trace["label"]
             result_path = run_dir / "result.json"
@@ -340,6 +482,13 @@ def main() -> None:
                     str(args.predictor_training_job_threads),
                 ]
             )
+            if args.align_real_graph_runtime:
+                command.extend(
+                    [
+                        "--cudagraph_capture_sizes",
+                        *(str(size) for size in GRAPH_CAPTURE_SIZES_BY_MNS[config.mns]),
+                    ]
+                )
             command = BASE.configure_cc_command(
                 command,
                 backend=args.cc_backend,
@@ -514,6 +663,9 @@ def main() -> None:
             "primary_tpot_slo_ms": 150.0,
             "target_pass_rate": TARGET_PASS_RATE,
             "predictor_training_job_threads": args.predictor_training_job_threads,
+            "decode_cuda_graph_mode": args.decode_cuda_graph_mode,
+            "align_real_graph_runtime": args.align_real_graph_runtime,
+            "fresh_predictor_cache": args.fresh_predictor_cache,
         },
         "frontier": {
             "source": str(args.frontier_source),
@@ -530,6 +682,30 @@ def main() -> None:
             "coverage": coverage,
             "sha256": {name: BASE.sha256(path) for name, path in paths.items()},
         },
+        "kernel_only_profiles": (
+            None
+            if kernel_paths is None
+            else {
+                "root": str(args.kernel_profile_root),
+                "coverage": kernel_coverage,
+                "sha256": {
+                    name: BASE.sha256(path) for name, path in kernel_paths.items()
+                },
+            }
+        ),
+        "runtime_alignment": {
+            "capture_sizes_by_mns": (
+                GRAPH_CAPTURE_SIZES_BY_MNS if args.align_real_graph_runtime else None
+            ),
+            "num_blocks_by_config": (
+                {
+                    f"tp{tp}_mns{mns}": blocks
+                    for (tp, mns), blocks in REAL_NUM_BLOCKS_BY_CONFIG.items()
+                }
+                if args.align_real_graph_runtime
+                else None
+            ),
+        },
         "collective": {
             "backend": args.cc_backend,
             "allreduce_csv": str(args.allreduce_csv) if args.allreduce_csv else None,

runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_attention.sh

@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+TP="${TP:?TP must be 1, 2, or 4}"
+case "${TP}" in
+  1|2|4) ;;
+  *) echo "invalid TP=${TP}" >&2; exit 1 ;;
+esac
+
+OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT must be set}"
+RUN_DIR="$(pwd -P)"
+PROFILE_DIR="${PROFILE_DIR:-${RUN_DIR%/runs/frontier-fidelity-envelope-v1}/runs/frontier-qwen30-vllm020-profile-v1}"
+VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}"
+VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}"
+FRONTIER_SOURCE="${FRONTIER_SOURCE:-/home/admin/cpfs/wjh/aituner/frontier-t1-dash0-deadc4a}"
+MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}"
+CAPTURE_BUCKETS="${CAPTURE_BUCKETS:-1 2 4 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128}"
+KV_CONTEXTS="${KV_CONTEXTS:-128 1024 2048 4096 8192 16384 32768 40960}"
+WARMUP_ITERS="${WARMUP_ITERS:-3}"
+REPEATS="${REPEATS:-5}"
+
+mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" "${OUTPUT_ROOT}/raw"
+exec > >(tee -a "${OUTPUT_ROOT}/logs/attention-tp${TP}.log") 2>&1
+
+IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?a fleet-allocated GPU is required}"
+if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then
+  echo "expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2
+  exit 1
+fi
+
+BATCH_SPECS=()
+for bucket in ${CAPTURE_BUCKETS}; do
+  prefix=""
+  if [[ "${bucket}" -ne 1 ]]; then
+    prefix="${bucket}"
+  fi
+  for context in ${KV_CONTEXTS}; do
+    BATCH_SPECS+=("${prefix}q1s${context}")
+  done
+done
+
+echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} role=FA3-decode-kernel-only tp=${TP} buckets='${CAPTURE_BUCKETS}' kv_contexts='${KV_CONTEXTS}' method=Frontier-RecordFunctionTracer output=${OUTPUT_ROOT} expected_wall=10-35m expected_gpu_cap=1.0_H20h"
+date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ"
+nvidia-smi --query-gpu=index,name,driver_version,memory.used,utilization.gpu --format=csv,noheader
+test "$(git -C "${FRONTIER_SOURCE}" rev-parse HEAD)" = "deadc4a321f0baaa534c6ebd17f974123733cdc2"
+test "$(git -C "${VLLM_SOURCE}" rev-parse HEAD)" = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1"
+test -f "${MODEL_ROOT}/config.json"
+git rev-parse HEAD > "${OUTPUT_ROOT}/provenance/aituner.commit"
+git -C "${FRONTIER_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/frontier.commit"
+git -C "${VLLM_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/vllm.commit"
+printf '%s\n' "${BATCH_SPECS[@]}" > "${OUTPUT_ROOT}/provenance/batch-specs.txt"
+sha256sum "${PROFILE_DIR}/profile_vllm020_flashattn.py" "${RUN_DIR}/run_graph_kernel_only_attention.sh" > "${OUTPUT_ROOT}/provenance/source.sha256"
+
+timeout --signal=TERM --kill-after=30s 2400 \
+  "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/profile_vllm020_flashattn.py" \
+  --vllm-source "${VLLM_SOURCE}" \
+  --frontier-source "${FRONTIER_SOURCE}" \
+  --model "${MODEL_ROOT}" \
+  --output "${OUTPUT_ROOT}/raw/attention-tp${TP}.json" \
+  --tp "${TP}" \
+  --batch-specs "${BATCH_SPECS[@]}" \
+  --warmup-iters "${WARMUP_ITERS}" \
+  --repeats "${REPEATS}" \
+  --profile-kv-update \
+  --profile-method record_function
+
+test -s "${OUTPUT_ROOT}/raw/attention-tp${TP}.json"
+sha256sum "${OUTPUT_ROOT}/raw/attention-tp${TP}.json" "${OUTPUT_ROOT}/provenance"/* > "${OUTPUT_ROOT}/artifacts.sha256"
+date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ"
+echo "GRAPH_KERNEL_ONLY_ATTENTION_COMPLETE tp=${TP} rows=${#BATCH_SPECS[@]}"

runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_linear.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT must be set}"
+RUN_DIR="$(pwd -P)"
+PROFILE_DIR="${PROFILE_DIR:-${RUN_DIR%/runs/frontier-fidelity-envelope-v1}/runs/frontier-qwen30-vllm020-profile-v1}"
+VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}"
+VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}"
+FRONTIER_SOURCE="${FRONTIER_SOURCE:-/home/admin/cpfs/wjh/aituner/frontier-t1-dash0-deadc4a}"
+MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}"
+TOKENS=(1 2 4 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128)
+export MODEL_ROOT
+
+mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" "${OUTPUT_ROOT}/profiles"
+exec > >(tee -a "${OUTPUT_ROOT}/logs/linear.log") 2>&1
+IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?a fleet-allocated GPU is required}"
+if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then
+  echo "expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2
+  exit 1
+fi
+
+echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} role=linear-kernel-only tp=1,2,4 tokens='${TOKENS[*]}' method=Frontier-RecordFunctionTracer output=${OUTPUT_ROOT} expected_wall=15-35m expected_gpu_cap=1.0_H20h"
+date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ"
+test "$(git -C "${FRONTIER_SOURCE}" rev-parse HEAD)" = "deadc4a321f0baaa534c6ebd17f974123733cdc2"
+test "$(git -C "${VLLM_SOURCE}" rev-parse HEAD)" = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1"
+git rev-parse HEAD > "${OUTPUT_ROOT}/provenance/aituner.commit"
+git -C "${FRONTIER_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/frontier.commit"
+sha256sum "${PROFILE_DIR}/frontier_vllm020_compat.py" "${RUN_DIR}/run_graph_kernel_only_linear.sh" > "${OUTPUT_ROOT}/provenance/source.sha256"
+
+cd "${FRONTIER_SOURCE}"
+timeout --signal=TERM --kill-after=30s 2400 \
+  "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/frontier_vllm020_compat.py" \
+  --disable_ray --num_gpus 1 --output_dir "${OUTPUT_ROOT}/profiles" \
+  --device h20 --models qwen3-a3b-30b-moe \
+  --num_tensor_parallel_workers 1 2 4 --max_tokens 128 \
+  --num_tokens_list "${TOKENS[@]}" --profile_method record_function \
+  --precision BF16 --is_moe --yes
+
+find "${OUTPUT_ROOT}/profiles" -name linear_op_kernel_only.csv -type f -size +0c -print -quit > "${OUTPUT_ROOT}/provenance/linear-path.txt"
+test -s "${OUTPUT_ROOT}/provenance/linear-path.txt"
+date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ"
+echo "GRAPH_KERNEL_ONLY_LINEAR_COMPLETE"

runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_moe.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT must be set}"
+RUN_DIR="$(pwd -P)"
+PROFILE_DIR="${PROFILE_DIR:-${RUN_DIR%/runs/frontier-fidelity-envelope-v1}/runs/frontier-qwen30-vllm020-profile-v1}"
+VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}"
+VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}"
+FRONTIER_SOURCE="${FRONTIER_SOURCE:-/home/admin/cpfs/wjh/aituner/frontier-t1-dash0-deadc4a}"
+MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}"
+TOKENS="${TOKENS:-1 2 4 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128}"
+
+mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" "${OUTPUT_ROOT}/raw"
+exec > >(tee -a "${OUTPUT_ROOT}/logs/moe-router.log") 2>&1
+IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?a fleet-allocated GPU is required}"
+if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then
+  echo "expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2
+  exit 1
+fi
+
+echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} role=MoE+router-kernel-only tp=1,2,4 tokens='${TOKENS}' method=Frontier-RecordFunctionTracer output=${OUTPUT_ROOT} expected_wall=10-30m expected_gpu_cap=1.0_H20h"
+date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ"
+test "$(git -C "${FRONTIER_SOURCE}" rev-parse HEAD)" = "deadc4a321f0baaa534c6ebd17f974123733cdc2"
+test "$(git -C "${VLLM_SOURCE}" rev-parse HEAD)" = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1"
+git rev-parse HEAD > "${OUTPUT_ROOT}/provenance/aituner.commit"
+git -C "${FRONTIER_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/frontier.commit"
+printf '%s\n' ${TOKENS} > "${OUTPUT_ROOT}/provenance/tokens.txt"
+sha256sum "${PROFILE_DIR}/profile_vllm020_moe.py" "${PROFILE_DIR}/profile_vllm020_router.py" "${RUN_DIR}/run_graph_kernel_only_moe.sh" > "${OUTPUT_ROOT}/provenance/source.sha256"
+
+timeout --signal=TERM --kill-after=30s 2400 \
+  "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/profile_vllm020_moe.py" \
+  --vllm-source "${VLLM_SOURCE}" --frontier-source "${FRONTIER_SOURCE}" \
+  --model "${MODEL_ROOT}" --output "${OUTPUT_ROOT}/raw/moe.json" \
+  --tp 1 2 4 --num-tokens ${TOKENS} --routing-modes uniform_random_logits \
+  --warmup-iters 3 --repeats 5 --profile-method record_function --check-reference
+
+timeout --signal=TERM --kill-after=30s 1800 \
+  "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/profile_vllm020_router.py" \
+  --vllm-source "${VLLM_SOURCE}" --frontier-source "${FRONTIER_SOURCE}" \
+  --model "${MODEL_ROOT}" --output "${OUTPUT_ROOT}/raw/router.json" \
+  --num-tokens ${TOKENS} --warmup-iters 3 --repeats 5 --profile-method record_function
+
+test -s "${OUTPUT_ROOT}/raw/moe.json"
+test -s "${OUTPUT_ROOT}/raw/router.json"
+sha256sum "${OUTPUT_ROOT}/raw"/*.json "${OUTPUT_ROOT}/provenance"/* > "${OUTPUT_ROOT}/artifacts.sha256"
+date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ"
+echo "GRAPH_KERNEL_ONLY_MOE_COMPLETE"

runs/frontier-qwen30-vllm020-profile-v1/freeze_frontier_profiles.py

@@ -117,8 +117,18 @@ def parse_args() -> argparse.Namespace:
     parser.add_argument("--attention", type=Path, nargs="+", required=True)
     parser.add_argument("--moe", type=Path, required=True)
     parser.add_argument("--router", type=Path, required=True)
-    parser.add_argument("--allreduce", type=Path, nargs=2, required=True)
+    parser.add_argument("--allreduce", type=Path, nargs=2)
+    parser.add_argument("--allreduce-frozen", type=Path)
     parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument(
+        "--measurement-type",
+        choices=("CUDA_EVENT", "KERNEL_ONLY"),
+        default="CUDA_EVENT",
+    )
+    parser.add_argument(
+        "--frontier-commit",
+        default="d9cfeb6d8791fbf2f295dd9744c56a666171776e",
+    )
     return parser.parse_args()
 
 
@@ -196,7 +206,7 @@ def write_csv(path: Path, fieldnames: list[str], rows: list[dict[str, Any]]) ->
 
 
 def freeze_attention(
-    inputs: list[Path], output: Path
+    inputs: list[Path], output: Path, *, measurement_type: str
 ) -> tuple[int, int, list[str]]:
     rows: list[dict[str, Any]] = []
     mixed_rows: list[dict[str, Any]] = []
@@ -208,6 +218,13 @@ def freeze_attention(
             raise ValueError(f"unexpected attention schema in {path}")
         if payload["environment"].get("vllm_version") != "0.20.0":
             raise ValueError(f"unexpected vLLM version in {path}")
+        expected_method = (
+            "record_function" if measurement_type == "KERNEL_ONLY" else "cuda_event"
+        )
+        if payload["environment"].get("profile_method", "cuda_event") != expected_method:
+            raise ValueError(
+                f"attention profile method mismatch in {path}: expected {expected_method}"
+            )
         for raw in payload["rows"]:
             if raw.get("error") is not None:
                 raise ValueError(f"failed attention row in {path}: {raw['error']}")
@@ -321,7 +338,7 @@ def freeze_attention(
                         "profiling_precision": "BF16",
                         "model_arch": "generic",
                         "quant_signature": "none",
-                        "measurement_type": "CUDA_EVENT",
+                        "measurement_type": measurement_type,
                         "is_true_mixed_batch": True,
                         "prefill_seq_lens": json.dumps(prefill_queries),
                         "prefill_kv_cache_sizes": json.dumps(prefill_contexts),
@@ -418,7 +435,7 @@ def freeze_attention(
                     "profiling_precision": "BF16",
                     "model_arch": "generic",
                     "quant_signature": "none",
-                    "measurement_type": "CUDA_EVENT",
+                    "measurement_type": measurement_type,
                     "is_true_mixed_batch": False,
                     "prefill_seq_lens": "",
                     "prefill_kv_cache_sizes": "",
@@ -498,13 +515,23 @@ def load_features(counts: list[int]) -> dict[str, float]:
     }
 
 
-def freeze_moe(moe_path: Path, router_path: Path, output: Path) -> int:
+def freeze_moe(
+    moe_path: Path, router_path: Path, output: Path, *, measurement_type: str
+) -> int:
     moe = load_json(moe_path)
     router = load_json(router_path)
     if moe.get("schema_version") != "qwen30_vllm020_moe_raw.v1":
         raise ValueError(f"unexpected MoE schema in {moe_path}")
     if router.get("schema_version") != "qwen30_vllm020_router_raw.v1":
         raise ValueError(f"unexpected router schema in {router_path}")
+    expected_method = (
+        "record_function" if measurement_type == "KERNEL_ONLY" else "cuda_event"
+    )
+    for payload, label in ((moe, "moe"), (router, "router")):
+        if payload["environment"].get("profile_method", "cuda_event") != expected_method:
+            raise ValueError(
+                f"{label} profile method mismatch in {payload}: expected {expected_method}"
+            )
     router_by_tokens = {int(row["num_tokens"]): row for row in router["rows"]}
     rows: list[dict[str, Any]] = []
     seen_pairs: set[tuple[int, int, str]] = set()
@@ -550,7 +577,7 @@ def freeze_moe(moe_path: Path, router_path: Path, output: Path) -> int:
                 "load_distribution": routing_mode,
                 "seed": 20260716,
                 "moe_grouped_gemm_backend": raw["backend"],
-                "measurement_type": "CUDA_EVENT",
+                "measurement_type": measurement_type,
                 "profiling_precision": "BF16",
                 "model_arch": "generic",
                 "quant_signature": "none",
@@ -565,9 +592,17 @@ def freeze_moe(moe_path: Path, router_path: Path, output: Path) -> int:
         )
         rows.append(row)
 
-    expected = 3 * 12 * 2
+    tokens = {int(row["num_tokens"]) for row in router["rows"]}
-    if len(rows) != expected:
+    modes = {str(row["routing_mode"]) for row in moe["rows"]}
-        raise ValueError(f"expected {expected} MoE rows, got {len(rows)}")
+    expected = {(tp, tokens_value, mode) for tp in (1, 2, 4) for tokens_value in tokens for mode in modes}
+    actual = {
+        (int(row["num_tensor_parallel_workers"]), int(row["num_tokens"]), str(row["load_distribution"]))
+        for row in rows
+    }
+    if actual != expected:
+        raise ValueError(
+            f"MoE TP/token/routing coverage mismatch: missing={expected - actual}, extra={actual - expected}"
+        )
     moe_fields = [
         f"time_stats.{op}.{stat}" for op in MOE_OPS for stat in STAT_NAMES
     ] + list(MOE_METADATA)
@@ -617,7 +652,13 @@ def freeze_allreduce(inputs: list[Path], output: Path) -> int:
 
 def main() -> None:
     args = parse_args()
-    all_inputs = [args.linear, *args.attention, args.moe, args.router, *args.allreduce]
+    if args.allreduce is not None and args.allreduce_frozen is not None:
+        raise SystemExit("provide either --allreduce or --allreduce-frozen, not both")
+    all_inputs = [args.linear, *args.attention, args.moe, args.router]
+    if args.allreduce is not None:
+        all_inputs.extend(args.allreduce)
+    if args.allreduce_frozen is not None:
+        all_inputs.append(args.allreduce_frozen)
     for path in all_inputs:
         if not path.is_file():
             raise SystemExit(f"missing input: {path}")
@@ -627,24 +668,39 @@ def main() -> None:
     shutil.copyfile(args.linear, linear_output)
     with linear_output.open(newline="") as handle:
         linear_rows = list(csv.DictReader(handle))
-    if len(linear_rows) != 36:
+    if not linear_rows:
-        raise ValueError(f"expected 36 linear rows, got {len(linear_rows)}")
+        raise ValueError("linear profile has no rows")
+    if {row.get("measurement_type") for row in linear_rows} != {args.measurement_type}:
+        raise ValueError(
+            f"linear measurement family mismatch: expected {args.measurement_type}"
+        )
 
     attention_rows, mixed_rows, attention_tps = freeze_attention(
-        list(args.attention), args.output
+        list(args.attention), args.output, measurement_type=args.measurement_type
     )
-    moe_rows = freeze_moe(args.moe, args.router, args.output)
+    moe_rows = freeze_moe(
+        args.moe, args.router, args.output, measurement_type=args.measurement_type
+    )
+    allreduce_rows = 0
+    allreduce_source = "not_included"
+    if args.allreduce is not None:
         allreduce_rows = freeze_allreduce(list(args.allreduce), args.output)
+        allreduce_source = "raw_vllm020_measurements"
+    elif args.allreduce_frozen is not None:
+        shutil.copyfile(args.allreduce_frozen, args.output / "allreduce.json")
+        allreduce_rows = len(load_json(args.allreduce_frozen).get("rows", []))
+        allreduce_source = "carried_forward_frozen_measurements"
 
     output_files = [
         linear_output,
         args.output / "attention.csv",
         args.output / "attention_true_mixed_fused.csv",
         args.output / "moe.csv",
-        args.output / "allreduce.json",
     ]
+    if (args.output / "allreduce.json").is_file():
+        output_files.append(args.output / "allreduce.json")
     batch_composition_augmented = len(args.attention) > 3
-    long_context_augmented = any(
+    long_context_augmented = args.measurement_type == "KERNEL_ONLY" or any(
         "long-context" in path.name for path in args.attention
     )
     long_context_coverage: dict[str, Any] = {"included": long_context_augmented}
@@ -659,23 +715,32 @@ def main() -> None:
                 if int(row["num_tensor_parallel_workers"]) == tp
                 and row["is_prefill"].lower() == "false"
             }
+            required_decode = (
+                {128, 1024, 2048, 4096, 8192, 16384, 32768, 40960}
+                if args.measurement_type == "KERNEL_ONLY"
+                else {16384, 32768, 40960}
+            )
+            if not required_decode.issubset(decode_kv):
+                raise ValueError(f"decode KV coverage mismatch for TP{tp}")
+            by_tp[str(tp)] = {
+                "decode_kv_lengths": sorted(decode_kv),
+            }
+            if args.measurement_type != "KERNEL_ONLY":
                 mixed_kv = {
                     int(float(row["decode_avg_kv_cache_size"]))
                     for row in frozen_attention
                     if int(row["num_tensor_parallel_workers"]) == tp
                     and row.get("is_true_mixed_batch", "").lower() == "true"
                 }
-            if not {16384, 32768, 40960}.issubset(decode_kv):
-                raise ValueError(f"long-context decode coverage mismatch for TP{tp}")
                 if not {16384, 32768}.issubset(mixed_kv):
                     raise ValueError(f"long-context mixed coverage mismatch for TP{tp}")
-            by_tp[str(tp)] = {
+                by_tp[str(tp)]["true_mixed_decode_avg_kv_lengths"] = sorted(mixed_kv)
-                "decode_kv_lengths": sorted(decode_kv),
-                "true_mixed_decode_avg_kv_lengths": sorted(mixed_kv),
-            }
         long_context_coverage["by_tp"] = by_tp
     manifest = {
         "schema_version": (
+            "frontier_qwen30_vllm020_kernel_only_profile.v1"
+            if args.measurement_type == "KERNEL_ONLY"
+            else (
                 "frontier_qwen30_vllm020_frozen_profile.v4"
                 if long_context_augmented
             else (
@@ -683,10 +748,12 @@ def main() -> None:
                 if batch_composition_augmented
                 else "frontier_qwen30_vllm020_frozen_profile.v2"
                 )
+            )
         ),
         "profile_id": (
             "qwen3-30b-a3b-bf16-vllm020-h20-tp1-2-4-"
             "fused-mixed-total-conserving"
+            + ("-kernel-only-record-function" if args.measurement_type == "KERNEL_ONLY" else "")
             + ("-pure-prefill-batch-composition" if batch_composition_augmented else "")
             + ("-long-context-decode-mixed" if long_context_augmented else "")
         ),
@@ -696,7 +763,7 @@ def main() -> None:
             "dtype": "bfloat16",
             "vllm_version": "0.20.0",
             "vllm_source_commit": "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1",
-            "frontier_commit": "d9cfeb6d8791fbf2f295dd9744c56a666171776e",
+            "frontier_commit": args.frontier_commit,
             "tensor_parallel_sizes": [1, 2, 4],
         },
         "row_counts": {
@@ -740,8 +807,7 @@ def main() -> None:
                 "expert measurement already includes prepare/finalize so shuffling is zero"
             ),
             "allreduce": (
-                "Frozen exact runtime measurements; base profile-only comparison keeps the "
+                "Frozen exact runtime measurements; source=" + allreduce_source
-                "historical Frontier CC backend fixed to isolate compute profile fidelity"
             ),
         },
         "inputs": {str(path.resolve()): sha256(path) for path in all_inputs},

runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_flashattn.py

@@ -39,6 +39,12 @@ def parse_args() -> argparse.Namespace:
     parser.add_argument("--repeats", type=int, default=5)
     parser.add_argument("--device", default="cuda:0")
     parser.add_argument("--profile-kv-update", action="store_true")
+    parser.add_argument(
+        "--profile-method",
+        choices=("cuda_event", "record_function"),
+        default="cuda_event",
+    )
+    parser.add_argument("--frontier-source", type=Path)
     return parser.parse_args()
 
 
@@ -66,12 +72,17 @@ def main() -> None:
         raise SystemExit(f"expected vLLM source {VLLM_COMMIT}, got {source_head}")
     if not args.model.joinpath("config.json").is_file():
         raise SystemExit(f"missing model config: {args.model / 'config.json'}")
+    if args.profile_method == "record_function" and args.frontier_source is None:
+        raise SystemExit("--frontier-source is required for --profile-method record_function")
 
     bench_dir = args.vllm_source / "benchmarks" / "attention_benchmarks"
     sys.path.insert(0, str(bench_dir))
     import runner  # type: ignore[import-not-found]  # noqa: PLC0415
     from batch_spec import parse_batch_spec  # type: ignore[import-not-found]  # noqa: PLC0415
-    from common import BenchmarkConfig  # type: ignore[import-not-found]  # noqa: PLC0415
+    from common import (  # type: ignore[import-not-found]  # noqa: PLC0415
+        BenchmarkConfig,
+        BenchmarkResult,
+    )
     from vllm.config import (  # noqa: PLC0415
         CacheConfig,
         CompilationConfig,
@@ -90,6 +101,13 @@ def main() -> None:
     from vllm.v1.kv_cache_interface import FullAttentionSpec  # noqa: PLC0415
     from vllm.v1.worker.workspace import init_workspace_manager  # noqa: PLC0415
 
+    record_function_tracer = None
+    if args.profile_method == "record_function":
+        sys.path.insert(0, str(args.frontier_source.resolve()))
+        from frontier.profiling.utils.record_function_tracer import RecordFunctionTracer
+
+        record_function_tracer = RecordFunctionTracer
+
     def create_vllm_config(config: BenchmarkConfig, max_num_blocks: int) -> VllmConfig:
         model_config = ModelConfig(
             model=str(args.model),
@@ -146,6 +164,9 @@ def main() -> None:
 
     runner._create_vllm_config = create_vllm_config
     init_workspace_manager(args.device)
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    if args.profile_method == "record_function":
+        (args.output.parent / "profiler_traces").mkdir(exist_ok=True)
 
     def profile_kv_cache_update(config: BenchmarkConfig) -> dict[str, float]:
         device = torch.device(config.device)
@@ -218,6 +239,137 @@ def main() -> None:
             "std_ms": statistics.pstdev(samples),
         }
 
+    def profile_kernel_only(
+        config: BenchmarkConfig,
+    ) -> tuple[BenchmarkResult, dict[str, float] | None]:
+        """Trace exactly one vLLM FA3 forward/KV-update per annotation.
+
+        `RecordFunctionTracer` is Frontier's actual KERNEL_ONLY collector: it
+        follows CUDA launch correlations and sums kernels under the annotation.
+        The profiling loop therefore contains no CUDA-event value relabeling.
+        """
+        device = torch.device(config.device)
+        torch.accelerator.set_device_index(device)
+        backend_config = runner._get_backend_config(config.backend)
+        requests = parse_batch_spec(config.batch_spec)
+        q_lens = [request.q_len for request in requests]
+        kv_lens = [request.kv_len for request in requests]
+        total_q = sum(q_lens)
+        max_kv = max(kv_lens)
+        max_blocks_per_request = (max_kv + config.block_size - 1) // config.block_size
+        max_num_blocks = len(requests) * max_blocks_per_request
+
+        with runner.log_warnings_and_errors_only():
+            vllm_config = create_vllm_config(config, max_num_blocks)
+            dtype = vllm_config.model_config.dtype
+            with set_current_vllm_config(vllm_config):
+                backend_class, impl, layer = runner._create_backend_impl(
+                    backend_config, config, device, dtype
+                )
+                required_layout = backend_class.get_required_kv_cache_layout()
+                if required_layout is not None:
+                    set_kv_cache_layout(required_layout)
+                    get_kv_cache_layout.cache_clear()
+                common_metadata = runner._build_common_attn_metadata(
+                    q_lens, kv_lens, config.block_size, device
+                )
+                kv_cache_spec = FullAttentionSpec(
+                    block_size=config.block_size,
+                    num_kv_heads=config.num_kv_heads,
+                    head_size=config.head_dim,
+                    dtype=dtype,
+                )
+                builder = runner._create_metadata_builder(
+                    backend_class, kv_cache_spec, vllm_config, device, config.backend
+                )
+                attn_metadata = builder.build(
+                    common_prefix_len=0, common_attn_metadata=common_metadata
+                )
+                quantize_query = config.kv_cache_dtype.startswith("fp8") and getattr(
+                    impl, "supports_quant_query_input", False
+                )
+                q_list, k_list, v_list = runner._create_input_tensors(
+                    config, total_q, device, dtype, quantize_query=quantize_query
+                )
+                cache_list = runner._create_kv_cache(
+                    config, max_num_blocks, backend_class, device, dtype
+                )
+                output = torch.empty(
+                    total_q,
+                    config.num_q_heads,
+                    config.head_dim,
+                    device=device,
+                    dtype=dtype,
+                )
+
+                def run_core() -> None:
+                    for layer_index in range(config.num_layers):
+                        impl.forward(
+                            layer,
+                            q_list[layer_index],
+                            k_list[layer_index],
+                            v_list[layer_index],
+                            cache_list[layer_index],
+                            attn_metadata,
+                            output=output,
+                        )
+
+                for _ in range(config.warmup_iters):
+                    run_core()
+                torch.accelerator.synchronize()
+                core_tracer = record_function_tracer(str(args.output.parent))
+                with core_tracer:
+                    for _ in range(config.repeats):
+                        with torch.profiler.record_function("vidur_attention_core"):
+                            run_core()
+                core_stats = core_tracer.get_operation_time_stats()
+                if "attention_core" not in core_stats:
+                    raise RuntimeError("missing KERNEL_ONLY FlashAttention core stats")
+                core = {
+                    name: float(value) / config.num_layers
+                    for name, value in core_stats["attention_core"].items()
+                }
+
+                kv_stats = None
+                if args.profile_kv_update:
+                    def run_kv_cache_update() -> None:
+                        for layer_index in range(config.num_layers):
+                            impl.do_kv_cache_update(
+                                layer,
+                                k_list[layer_index],
+                                v_list[layer_index],
+                                cache_list[layer_index],
+                                common_metadata.slot_mapping,
+                            )
+
+                    for _ in range(config.warmup_iters):
+                        run_kv_cache_update()
+                    torch.accelerator.synchronize()
+                    kv_tracer = record_function_tracer(str(args.output.parent))
+                    with kv_tracer:
+                        for _ in range(config.repeats):
+                            with torch.profiler.record_function("vidur_attn_kv_cache_save"):
+                                run_kv_cache_update()
+                    kv_time_stats = kv_tracer.get_operation_time_stats()
+                    if "attn_kv_cache_save" not in kv_time_stats:
+                        raise RuntimeError("missing KERNEL_ONLY KV-update stats")
+                    kv_stats = {
+                        f"{name}_ms": float(value) / config.num_layers
+                        for name, value in kv_time_stats["attn_kv_cache_save"].items()
+                    }
+
+        result = BenchmarkResult(
+            config=config,
+            mean_time=core["mean"] / 1000.0,
+            std_time=core["std"] / 1000.0,
+            min_time=core["min"] / 1000.0,
+            max_time=core["max"] / 1000.0,
+            throughput_tokens_per_sec=(
+                total_q / (core["mean"] / 1000.0) if core["mean"] > 0 else 0.0
+            ),
+        )
+        return result, kv_stats
+
     rows: list[dict[str, object]] = []
     for tp in args.tp:
         for batch_spec in args.batch_specs:
@@ -237,12 +389,18 @@ def main() -> None:
                 kv_cache_dtype="auto",
                 use_cuda_graphs=False,
             )
+            if args.profile_method == "record_function":
+                result, kv_stats = profile_kernel_only(config)
+            else:
                 result = runner.run_attention_benchmark(config)
+                kv_stats = (
+                    profile_kv_cache_update(config) if args.profile_kv_update else None
+                )
             row = result.to_dict()
             row["tensor_parallel_size"] = tp
             row["attention_core_excludes_kv_cache_update"] = True
-            if args.profile_kv_update:
+            if kv_stats is not None:
-                row["kv_cache_update_time"] = profile_kv_cache_update(config)
+                row["kv_cache_update_time"] = kv_stats
             rows.append(row)
             print(
                 json.dumps(
@@ -273,10 +431,10 @@ def main() -> None:
             "attention_backend": "FLASH_ATTN",
             "block_size": 16,
             "profile_kv_update": args.profile_kv_update,
+            "profile_method": args.profile_method,
         },
         "rows": rows,
     }
-    args.output.parent.mkdir(parents=True, exist_ok=True)
     args.output.write_text(
         json.dumps(payload, indent=2, sort_keys=True, default=json_default) + "\n"
     )

runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_moe.py

@@ -9,7 +9,7 @@ import math
 import statistics
 import subprocess
 from pathlib import Path
-from typing import Any
+from typing import Any, Callable
 
 import torch
 import vllm
@@ -40,6 +40,12 @@ def parse_args() -> argparse.Namespace:
     parser.add_argument("--repeats", type=int, default=5)
     parser.add_argument("--device", default="cuda:0")
     parser.add_argument("--check-reference", action="store_true")
+    parser.add_argument(
+        "--profile-method",
+        choices=("cuda_event", "record_function"),
+        default="cuda_event",
+    )
+    parser.add_argument("--frontier-source", type=Path)
     return parser.parse_args()
 
 
@@ -59,6 +65,34 @@ def stats_ms(samples: list[float]) -> dict[str, float]:
     }
 
 
+def measure_kernel_only_ms(
+    fn: Callable[[], torch.Tensor],
+    *,
+    warmup_iters: int,
+    repeats: int,
+    trace_root: Path,
+    operation_name: str,
+    record_function_tracer: type,
+) -> tuple[torch.Tensor, dict[str, float]]:
+    """Use Frontier's KERNEL_ONLY contract, not a CUDA-event relabel."""
+    result = None
+    for _ in range(warmup_iters):
+        result = fn()
+    torch.accelerator.synchronize()
+
+    tracer = record_function_tracer(str(trace_root))
+    with tracer:
+        for _ in range(repeats):
+            with torch.profiler.record_function(f"vidur_{operation_name}"):
+                result = fn()
+    stats = tracer.get_operation_time_stats()
+    if operation_name not in stats:
+        raise RuntimeError(f"missing RecordFunctionTracer stats for {operation_name}")
+    if result is None:
+        raise RuntimeError("kernel-only profiler executed no MoE step")
+    return result, {name: float(value) for name, value in stats[operation_name].items()}
+
+
 def routing_inputs(
     mode: str, num_tokens: int, device: torch.device
 ) -> tuple[torch.Tensor, torch.Tensor, dict[str, Any]]:
@@ -145,6 +179,8 @@ def main() -> None:
         raise SystemExit(
             f"model contract mismatch: expected {expected_model}, got {observed_model}"
         )
+    if args.profile_method == "record_function" and args.frontier_source is None:
+        raise SystemExit("--frontier-source is required for --profile-method record_function")
 
     from vllm.config import ParallelConfig, VllmConfig, set_current_vllm_config
     from vllm.model_executor.layers.fused_moe.activation import MoEActivation
@@ -163,10 +199,22 @@ def main() -> None:
     from vllm.utils.math_utils import next_power_of_2
     from vllm.v1.worker.workspace import init_workspace_manager
 
+    record_function_tracer = None
+    if args.profile_method == "record_function":
+        import sys
+
+        sys.path.insert(0, str(args.frontier_source.resolve()))
+        from frontier.profiling.utils.record_function_tracer import RecordFunctionTracer
+
+        record_function_tracer = RecordFunctionTracer
+
     device = torch.device(args.device)
     torch.accelerator.set_device_index(device)
     torch.manual_seed(20260716)
     init_workspace_manager(args.device)
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    if args.profile_method == "record_function":
+        (args.output.parent / "profiler_traces").mkdir(exist_ok=True)
     max_num_tokens = next_power_of_2(max(args.num_tokens))
 
     rows: list[dict[str, Any]] = []
@@ -257,8 +305,8 @@ def main() -> None:
                         routing_mode, num_tokens, device
                     )
 
-                    for _ in range(args.warmup_iters):
+                    def run_kernel() -> torch.Tensor:
-                        output = kernel.apply(
+                        return kernel.apply(
                             hidden_states=hidden,
                             w1=w13_kernel,
                             w2=w2_kernel,
@@ -269,27 +317,30 @@ def main() -> None:
                             expert_map=None,
                             apply_router_weight_on_input=False,
                         )
-                    torch.accelerator.synchronize()
 
+                    if args.profile_method == "record_function":
+                        output, time_ms = measure_kernel_only_ms(
+                            run_kernel,
+                            warmup_iters=args.warmup_iters,
+                            repeats=args.repeats,
+                            trace_root=args.output.parent,
+                            operation_name="moe_grouped_gemm",
+                            record_function_tracer=record_function_tracer,
+                        )
+                    else:
+                        for _ in range(args.warmup_iters):
+                            output = run_kernel()
+                        torch.accelerator.synchronize()
                         samples: list[float] = []
                         for _ in range(args.repeats):
                             start = torch.cuda.Event(enable_timing=True)
                             end = torch.cuda.Event(enable_timing=True)
                             start.record()
-                        output = kernel.apply(
+                            output = run_kernel()
-                            hidden_states=hidden,
-                            w1=w13_kernel,
-                            w2=w2_kernel,
-                            topk_weights=topk_weights,
-                            topk_ids=topk_ids,
-                            activation=MoEActivation.SILU,
-                            global_num_experts=NUM_EXPERTS,
-                            expert_map=None,
-                            apply_router_weight_on_input=False,
-                        )
                             end.record()
                             torch.accelerator.synchronize()
                             samples.append(float(start.elapsed_time(end)))
+                        time_ms = stats_ms(samples)
 
                     if output.shape != hidden.shape or not torch.isfinite(output).all():
                         raise SystemExit(
@@ -316,7 +367,7 @@ def main() -> None:
                         "backend": backend.value,
                         "intermediate_size_per_partition": INTERMEDIATE_DIM // tp,
                         "output_is_reduced": kernel.output_is_reduced(),
-                        "time_ms": stats_ms(samples),
+                        "time_ms": time_ms,
                         "routing_load": load,
                     }
                     rows.append(row)
@@ -350,6 +401,7 @@ def main() -> None:
             "weight_quantization": "none",
             "top_k": TOP_K,
             "norm_topk_prob": True,
+            "profile_method": args.profile_method,
         },
         "measurement_scope": (
             "one TP-local weight shard: vLLM modular MoE prepare+FlashInfer "
@@ -357,7 +409,6 @@ def main() -> None:
         ),
         "rows": rows,
     }
-    args.output.parent.mkdir(parents=True, exist_ok=True)
     args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
 
 

runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_router.py

@@ -31,6 +31,12 @@ def parse_args() -> argparse.Namespace:
     parser.add_argument("--warmup-iters", type=int, default=5)
     parser.add_argument("--repeats", type=int, default=20)
     parser.add_argument("--device", default="cuda:0")
+    parser.add_argument(
+        "--profile-method",
+        choices=("cuda_event", "record_function"),
+        default="cuda_event",
+    )
+    parser.add_argument("--frontier-source", type=Path)
     return parser.parse_args()
 
 
@@ -51,12 +57,34 @@ def stats_ms(samples: list[float]) -> dict[str, float]:
 
 
 def measure_ms(
-    fn: Callable[[], Any], warmup_iters: int, repeats: int
+    fn: Callable[[], Any],
+    warmup_iters: int,
+    repeats: int,
+    *,
+    profile_method: str,
+    trace_root: Path | None = None,
+    operation_name: str | None = None,
+    record_function_tracer: type | None = None,
 ) -> tuple[Any, dict[str, float]]:
     result = None
     for _ in range(warmup_iters):
         result = fn()
     torch.accelerator.synchronize()
+    if profile_method == "record_function":
+        if trace_root is None or operation_name is None or record_function_tracer is None:
+            raise ValueError("record_function profiling requires tracer metadata")
+        tracer = record_function_tracer(str(trace_root))
+        with tracer:
+            for _ in range(repeats):
+                with torch.profiler.record_function(f"vidur_{operation_name}"):
+                    result = fn()
+        stats = tracer.get_operation_time_stats()
+        if operation_name not in stats:
+            raise RuntimeError(f"missing RecordFunctionTracer stats for {operation_name}")
+        return result, {
+            name: float(value) for name, value in stats[operation_name].items()
+        }
+
     samples: list[float] = []
     for _ in range(repeats):
         start = torch.cuda.Event(enable_timing=True)
@@ -76,6 +104,8 @@ def main() -> None:
     source_head = git_head(args.vllm_source)
     if source_head != VLLM_COMMIT:
         raise SystemExit(f"expected vLLM source {VLLM_COMMIT}, got {source_head}")
+    if args.profile_method == "record_function" and args.frontier_source is None:
+        raise SystemExit("--frontier-source is required for --profile-method record_function")
 
     raw_model_config = json.loads(args.model.joinpath("config.json").read_text())
     observed = {
@@ -103,6 +133,15 @@ def main() -> None:
     from vllm.model_executor.layers.fused_moe import fused_topk
     from vllm.model_executor.layers.linear import ReplicatedLinear
 
+    record_function_tracer = None
+    if args.profile_method == "record_function":
+        import sys
+
+        sys.path.insert(0, str(args.frontier_source.resolve()))
+        from frontier.profiling.utils.record_function_tracer import RecordFunctionTracer
+
+        record_function_tracer = RecordFunctionTracer
+
     device = torch.device(args.device)
     torch.accelerator.set_device_index(device)
     torch.manual_seed(20260716)
@@ -113,6 +152,9 @@ def main() -> None:
         skip_tokenizer_init=True,
         generation_config="vllm",
     )
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    if args.profile_method == "record_function":
+        (args.output.parent / "profiler_traces").mkdir(exist_ok=True)
 
     rows: list[dict[str, Any]] = []
     with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener:
@@ -141,12 +183,22 @@ def main() -> None:
                     (num_tokens, HIDDEN_DIM), device=device, dtype=torch.bfloat16
                 ).uniform_(-0.1, 0.1)
                 logits, gate_time = measure_ms(
-                    lambda: gate(hidden)[0], args.warmup_iters, args.repeats
+                    lambda: gate(hidden)[0],
+                    args.warmup_iters,
+                    args.repeats,
+                    profile_method=args.profile_method,
+                    trace_root=args.output.parent,
+                    operation_name="moe_gating_linear",
+                    record_function_tracer=record_function_tracer,
                 )
                 topk_result, topk_time = measure_ms(
                     lambda: fused_topk(hidden, logits, TOP_K, renormalize=True),
                     args.warmup_iters,
                     args.repeats,
+                    profile_method=args.profile_method,
+                    trace_root=args.output.parent,
+                    operation_name="moe_gating_routing_topk",
+                    record_function_tracer=record_function_tracer,
                 )
 
                 def gate_and_topk() -> tuple[
@@ -158,7 +210,13 @@ def main() -> None:
                     )
 
                 combined_result, combined_time = measure_ms(
-                    gate_and_topk, args.warmup_iters, args.repeats
+                    gate_and_topk,
+                    args.warmup_iters,
+                    args.repeats,
+                    profile_method=args.profile_method,
+                    trace_root=args.output.parent,
+                    operation_name="moe_gating_linear_and_routing_topk",
+                    record_function_tracer=record_function_tracer,
                 )
                 topk_weights, topk_ids, _ = topk_result
                 combined_weights, combined_ids, _ = combined_result
@@ -207,6 +265,7 @@ def main() -> None:
             "gate_replication": "replicated_across_tp",
             "top_k": TOP_K,
             "norm_topk_prob": True,
+            "profile_method": args.profile_method,
         },
         "measurement_scope": (
             "vLLM ReplicatedLinear gate and fused_topk; measured separately and "
@@ -214,7 +273,6 @@ def main() -> None:
         ),
         "rows": rows,
     }
-    args.output.parent.mkdir(parents=True, exist_ok=True)
     args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")