diff --git a/runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md b/runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md new file mode 100644 index 0000000..272fe9c --- /dev/null +++ b/runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md @@ -0,0 +1,40 @@ +# EXP-SIMFID-Q30-GRAPH-PIECEWISE：graph-compatible kernel-only profile 是否修正 Frontier trace replay？ + +> **状态：** approved and running（2026-07-17）。本卡是已纠正 prefix-trace contract 后的最小判别实验；不复用此前 `decode_cuda_graph_mode=none` 的数值作 fidelity verdict。 + +## Purpose and hypotheses + +- **Parent claim：** Frontier 是否已经足以为 Qwen3-30B-A3B 的真实 trace serving surface 选择 config。 +- **Question：** 旧 Frontier replay 低估 decode service rate，是否主要是 simulator 使用 `none` 而真机使用 `FULL_AND_PIECEWISE`、并且没有向 Frontier 提供独立 `KERNEL_ONLY` profile family？ +- **G1 (graph-family omission)：** 用同一 vLLM 0.20/FA3/FlashInfer-CUTLASS stack 的 `RecordFunctionTracer` kernel-only measurements，加真实 capture buckets 和 Frontier `piecewise`，会显著缩小 TP2/MNS16 的 TPOT/service-rate gap，并至少改变一个 config 的 latency ranking。 +- **G2 (remaining composition error)：** 即使 graph family 对齐，TPOT、TTFT 或 E2E ranking 仍与真机不一致；则 graph omission 只是必要修正，不是 simulator 已解决 tuning 的证据。 + +## Controlled setup + +| Item | Frozen choice | +|---|---| +| model/runtime/hardware | Qwen3-30B-A3B BF16; community vLLM 0.20.0 (`88d34c…`); dash0 NVIDIA H20 | +| simulator | Frontier `deadc4a321f0baaa534c6ebd17f974123733cdc2`; no local source patch | +| workload | exact 129-request Trace-PD public projection; exact ISL/OSL/arrival order; TP-normalized arrival time and complete 16-token prefix blocks | +| surface | TP in {1,2,4}; MNS in {8,16,32,64}; MBT=8192; prefix/chunked prefill on | +| real graph contract | observed vLLM capture sizes: MNS8=[1,2,4,8,16], MNS16=[1,2,4,8,16,24,32], MNS32=[1,2,4,8,16,24,32,40,48,56,64], MNS64=[1,2,4,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128] | +| profile intervention | CUDA-event profile stays frozen for prefill/mixed batches. New `KERNEL_ONLY` linear, FA3 decode + KV-update, MoE, and router rows use Frontier's actual `RecordFunctionTracer` semantics; no relabeling of CUDA-event numbers. | +| exact capacity | per-cell real observed KV block count and capture list; Frontier CPU-overhead model remains disabled on both old/new simulator runs because the intervention is GPU-kernel family only. | + +Frontier source inspection fixes the semantic boundary: `piecewise` emits `PIECEWISE` whenever a capture hits, but the MONOLITHIC predictor selects `KERNEL_ONLY` only when `num_prefill_tokens == 0`. Hence new profile coverage is pure decode only; captured mixed/prefill work continues to consume the existing CUDA-event family. + +## Measurement and decision rule + +- **Primary outputs：** per-config mean/p90 TTFT, TPOT, E2E; ranking for each metric; TP2/MNS16 per-request TPOT gap against the already frozen three-trial real audit. +- **Validity gates：** every kernel CSV hash matches its manifest; every row says `KERNEL_ONLY`; every TP/capture-bucket/KV-context required by the runner is present; command records `piecewise`, per-cell blocks and capture sizes; each simulator cell completes all 129 requests. +- **Decision:** G1 is supported only if the graph-aligned TP2/MNS16 TPOT median moves toward real **and** full-surface rank/error evidence improves. A single-cell timing improvement does not establish tuning sufficiency. If G2 holds, update the research claim to “Frontier has not solved tuning under trace-faithful MoE serving after graph-family alignment,” then profile stage/state composition rather than add arbitrary kernel rows. + +## Expected figure + +`graph-piecewise-profile-prototype.svg` is deliberately schematic. The final figure uses the same axes and adds real data only after the profile and replay validity gates pass. + +## Cost and provenance + +- **GPU cost:** three 1-GPU FA3 decode profile shards, plus one 1-GPU linear shard and one 1-GPU MoE/router shard; expected 1.5--3.0 H20-GPU-hours, hard cap 4.0 GPU-hours. +- **CPU cost:** 12 exact-trace simulations, expected 20--45 CPU minutes; a one-cell TP2/MNS16 smoke precedes the full surface. +- **Calibration separation:** kernel microprofiles are independent measurements, never fitted to trace E2E latency. The frozen real trace audit is evaluation only. diff --git a/runs/frontier-fidelity-envelope-v1/graph-piecewise-profile-prototype.svg b/runs/frontier-fidelity-envelope-v1/graph-piecewise-profile-prototype.svg new file mode 100644 index 0000000..3713255 --- /dev/null +++ b/runs/frontier-fidelity-envelope-v1/graph-piecewise-profile-prototype.svg @@ -0,0 +1,28 @@ + + + SCHEMATIC — no measured data + Does graph-compatible KERNEL_ONLY profiling make Frontier select the real trace-serving configuration? + + A. TP2/MNS16 TPOT prediction + + + latencymeasurement family + + + + nonepiecewisereal + old simG1: moves closer + + + B. Full 12-cell ranking agreement + + + rank errorsimulator variant + + + + nonepiecewisereal rank + G2: stays wrongG1: error falls + + Final figure reports mean/p90 TTFT, TPOT, E2E for the identical 129-request trace, not an SLO-derived proxy. + diff --git a/runs/frontier-fidelity-envelope-v1/run_frontier_qwen30_exact_trace_surface.py b/runs/frontier-fidelity-envelope-v1/run_frontier_qwen30_exact_trace_surface.py index 8afd928..454d0b2 100644 --- a/runs/frontier-fidelity-envelope-v1/run_frontier_qwen30_exact_trace_surface.py +++ b/runs/frontier-fidelity-envelope-v1/run_frontier_qwen30_exact_trace_surface.py @@ -19,6 +19,27 @@ from typing import Any TARGET_PASS_RATE = 0.95 TPOT_SLOS_MS = (50.0, 100.0, 150.0, 180.0) WINDOW_SECONDS = 600.0 +GRAPH_CAPTURE_SIZES_BY_MNS = { + 8: (1, 2, 4, 8, 16), + 16: (1, 2, 4, 8, 16, 24, 32), + 32: (1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64), + 64: (1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128), +} +REAL_NUM_BLOCKS_BY_CONFIG = { + (1, 8): 20137, + (1, 16): 20128, + (1, 32): 20108, + (1, 64): 20069, + (2, 8): 76639, + (2, 16): 76620, + (2, 32): 76583, + (2, 64): 76505, + (4, 8): 191930, + (4, 16): 191882, + (4, 32): 191786, + (4, 64): 191589, +} +KERNEL_DECODE_KV_CONTEXTS = (128, 1024, 2048, 4096, 8192, 16384, 32768, 40960) BASE_RUNNER = ( Path(__file__).resolve().parents[1] / "frontier-phase-factorial-v0/run_frontier_qwen30_prefill_surface.py" @@ -43,6 +64,7 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--frontier-source", type=Path, required=True) parser.add_argument("--replayserve-root", type=Path, required=True) parser.add_argument("--profile-root", type=Path, required=True) + parser.add_argument("--kernel-profile-root", type=Path) parser.add_argument("--python-deps", type=Path, required=True) parser.add_argument("--output-root", type=Path, required=True) parser.add_argument( @@ -68,6 +90,21 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--allreduce-csv", type=Path) parser.add_argument("--timeout-seconds", type=float, default=1800.0) parser.add_argument("--predictor-training-job-threads", type=int, default=1) + parser.add_argument( + "--decode-cuda-graph-mode", + choices=("none", "full_decode_only", "piecewise"), + default="none", + ) + parser.add_argument( + "--align-real-graph-runtime", + action="store_true", + help="Use real observed capture lists and per-(TP,MNS) KV blocks.", + ) + parser.add_argument( + "--fresh-predictor-cache", + action="store_true", + help="Disable Frontier predictor cache reuse for this profile family.", + ) parser.add_argument("--resume", action="store_true") parser.add_argument("--continue-on-failure", action="store_true") return parser.parse_args() @@ -241,15 +278,99 @@ def score(path: Path, expected_shapes: list[tuple[int, int]]) -> dict[str, Any]: } ttfts = [float(row["ttft_ms"]) for row in request_metrics] tpots = [float(row["tpot_ms"]) for row in request_metrics if row["tpot_ms"] is not None] + e2es = [float(row["e2e_ms"]) for row in request_metrics] return { + "ttft_mean_ms": sum(ttfts) / len(ttfts), "ttft_p50_ms": percentile(ttfts, 0.50), + "ttft_p90_ms": percentile(ttfts, 0.90), "ttft_p95_ms": percentile(ttfts, 0.95), + "tpot_mean_ms": sum(tpots) / len(tpots), "tpot_p50_ms": percentile(tpots, 0.50), + "tpot_p90_ms": percentile(tpots, 0.90), "tpot_p95_ms": percentile(tpots, 0.95), + "e2e_mean_ms": sum(e2es) / len(e2es), + "e2e_p50_ms": percentile(e2es, 0.50), + "e2e_p90_ms": percentile(e2es, 0.90), + "e2e_p95_ms": percentile(e2es, 0.95), "slos": slos, } +def kernel_profile_paths(root: Path) -> dict[str, Path]: + paths = { + "linear": root / "linear_op.csv", + "attention": root / "attention.csv", + "moe": root / "moe.csv", + "manifest": root / "manifest.json", + } + missing = [str(path) for path in paths.values() if not path.is_file()] + if missing: + raise FileNotFoundError(missing) + return paths + + +def validate_kernel_profile(paths: dict[str, Path]) -> dict[str, Any]: + manifest = json.loads(paths["manifest"].read_text()) + outputs = manifest.get("outputs", {}) + for filename, name in ( + ("linear_op.csv", "linear"), + ("attention.csv", "attention"), + ("moe.csv", "moe"), + ): + if outputs.get(filename) != BASE.sha256(paths[name]): + raise ValueError(f"kernel-only profile hash mismatch for {filename}") + + with paths["linear"].open(newline="") as source: + linear_rows = list(csv.DictReader(source)) + with paths["attention"].open(newline="") as source: + attention_rows = list(csv.DictReader(source)) + with paths["moe"].open(newline="") as source: + moe_rows = list(csv.DictReader(source)) + for label, rows in (("linear", linear_rows), ("attention", attention_rows), ("moe", moe_rows)): + if not rows or {row.get("measurement_type") for row in rows} != {"KERNEL_ONLY"}: + raise ValueError(f"{label} lacks an exclusive KERNEL_ONLY measurement family") + + required_buckets = set(GRAPH_CAPTURE_SIZES_BY_MNS[64]) + coverage: dict[str, Any] = {} + for tp in (1, 2, 4): + linear_tokens = { + int(float(row["num_tokens"])) + for row in linear_rows + if int(float(row["num_tensor_parallel_workers"])) == tp + } + moe_tokens = { + int(float(row["num_tokens"])) + for row in moe_rows + if int(float(row["num_tensor_parallel_workers"])) == tp + } + attention_pairs = { + (int(float(row["batch_size"])), int(float(row["kv_cache_size"]))) + for row in attention_rows + if int(float(row["num_tensor_parallel_workers"])) == tp + and row["is_prefill"].lower() == "false" + and row.get("is_true_mixed_batch", "").lower() != "true" + } + missing_linear = required_buckets - linear_tokens + missing_moe = required_buckets - moe_tokens + missing_attention = { + (bucket, kv) + for bucket in required_buckets + for kv in KERNEL_DECODE_KV_CONTEXTS + if (bucket, kv) not in attention_pairs + } + if missing_linear or missing_moe or missing_attention: + raise ValueError( + f"kernel-only profile coverage TP{tp}: linear={sorted(missing_linear)}, " + f"moe={sorted(missing_moe)}, attention={sorted(missing_attention)}" + ) + coverage[str(tp)] = { + "linear_tokens": sorted(linear_tokens), + "moe_tokens": sorted(moe_tokens), + "attention_decode_pairs": len(attention_pairs), + } + return {"manifest": manifest, "coverage": coverage} + + def main() -> None: args = parse_args() if args.predictor_training_job_threads <= 0: @@ -264,6 +385,10 @@ def main() -> None: setattr(args, name, getattr(args, name).resolve()) if args.allreduce_csv is not None: args.allreduce_csv = args.allreduce_csv.resolve() + if args.kernel_profile_root is not None: + args.kernel_profile_root = args.kernel_profile_root.resolve() + if args.decode_cuda_graph_mode == "none": + raise ValueError("--kernel-profile-root requires a non-none graph mode") traces = [ parse_trace( specification, @@ -288,6 +413,11 @@ def main() -> None: raise ValueError(f"unknown configs: {wanted - {config.name for config in selected}}") paths = BASE.profile_paths(args.profile_root) coverage = BASE.validate_profile(paths) + kernel_paths = None + kernel_coverage = None + if args.kernel_profile_root is not None: + kernel_paths = kernel_profile_paths(args.kernel_profile_root) + kernel_coverage = validate_kernel_profile(kernel_paths) builder = BASE.load_module( "qwen30_exact_trace_frontier_builder", args.replayserve_root / "tools/run_frontier_sweep.py", @@ -320,6 +450,18 @@ def main() -> None: config_knobs = BASE.knobs(config, paths, args.output_root / "cache") config_knobs["enable_prefix_caching"] = args.prefix_caching config_knobs["prediction_max_tokens_per_request"] = 40960 + config_knobs["decode_cuda_graph_mode"] = args.decode_cuda_graph_mode + config_knobs["no_cache"] = args.fresh_predictor_cache + if args.align_real_graph_runtime: + config_knobs["num_blocks"] = REAL_NUM_BLOCKS_BY_CONFIG[(config.tp, config.mns)] + if kernel_paths is not None: + config_knobs.update( + { + "linear_op_kernel_only_input_file": str(kernel_paths["linear"]), + "atten_kernel_only_input_file": str(kernel_paths["attention"]), + "moe_kernel_only_input_file": str(kernel_paths["moe"]), + } + ) for trace in traces: run_dir = args.output_root / "runs" / config.name / trace["label"] result_path = run_dir / "result.json" @@ -340,6 +482,13 @@ def main() -> None: str(args.predictor_training_job_threads), ] ) + if args.align_real_graph_runtime: + command.extend( + [ + "--cudagraph_capture_sizes", + *(str(size) for size in GRAPH_CAPTURE_SIZES_BY_MNS[config.mns]), + ] + ) command = BASE.configure_cc_command( command, backend=args.cc_backend, @@ -514,6 +663,9 @@ def main() -> None: "primary_tpot_slo_ms": 150.0, "target_pass_rate": TARGET_PASS_RATE, "predictor_training_job_threads": args.predictor_training_job_threads, + "decode_cuda_graph_mode": args.decode_cuda_graph_mode, + "align_real_graph_runtime": args.align_real_graph_runtime, + "fresh_predictor_cache": args.fresh_predictor_cache, }, "frontier": { "source": str(args.frontier_source), @@ -530,6 +682,30 @@ def main() -> None: "coverage": coverage, "sha256": {name: BASE.sha256(path) for name, path in paths.items()}, }, + "kernel_only_profiles": ( + None + if kernel_paths is None + else { + "root": str(args.kernel_profile_root), + "coverage": kernel_coverage, + "sha256": { + name: BASE.sha256(path) for name, path in kernel_paths.items() + }, + } + ), + "runtime_alignment": { + "capture_sizes_by_mns": ( + GRAPH_CAPTURE_SIZES_BY_MNS if args.align_real_graph_runtime else None + ), + "num_blocks_by_config": ( + { + f"tp{tp}_mns{mns}": blocks + for (tp, mns), blocks in REAL_NUM_BLOCKS_BY_CONFIG.items() + } + if args.align_real_graph_runtime + else None + ), + }, "collective": { "backend": args.cc_backend, "allreduce_csv": str(args.allreduce_csv) if args.allreduce_csv else None, diff --git a/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_attention.sh b/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_attention.sh new file mode 100644 index 0000000..2d1ff9e --- /dev/null +++ b/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_attention.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash + +set -euo pipefail + +TP="${TP:?TP must be 1, 2, or 4}" +case "${TP}" in + 1|2|4) ;; + *) echo "invalid TP=${TP}" >&2; exit 1 ;; +esac + +OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT must be set}" +RUN_DIR="$(pwd -P)" +PROFILE_DIR="${PROFILE_DIR:-${RUN_DIR%/runs/frontier-fidelity-envelope-v1}/runs/frontier-qwen30-vllm020-profile-v1}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}" +VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}" +FRONTIER_SOURCE="${FRONTIER_SOURCE:-/home/admin/cpfs/wjh/aituner/frontier-t1-dash0-deadc4a}" +MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}" +CAPTURE_BUCKETS="${CAPTURE_BUCKETS:-1 2 4 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128}" +KV_CONTEXTS="${KV_CONTEXTS:-128 1024 2048 4096 8192 16384 32768 40960}" +WARMUP_ITERS="${WARMUP_ITERS:-3}" +REPEATS="${REPEATS:-5}" + +mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" "${OUTPUT_ROOT}/raw" +exec > >(tee -a "${OUTPUT_ROOT}/logs/attention-tp${TP}.log") 2>&1 + +IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?a fleet-allocated GPU is required}" +if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then + echo "expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2 + exit 1 +fi + +BATCH_SPECS=() +for bucket in ${CAPTURE_BUCKETS}; do + prefix="" + if [[ "${bucket}" -ne 1 ]]; then + prefix="${bucket}" + fi + for context in ${KV_CONTEXTS}; do + BATCH_SPECS+=("${prefix}q1s${context}") + done +done + +echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} role=FA3-decode-kernel-only tp=${TP} buckets='${CAPTURE_BUCKETS}' kv_contexts='${KV_CONTEXTS}' method=Frontier-RecordFunctionTracer output=${OUTPUT_ROOT} expected_wall=10-35m expected_gpu_cap=1.0_H20h" +date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ" +nvidia-smi --query-gpu=index,name,driver_version,memory.used,utilization.gpu --format=csv,noheader +test "$(git -C "${FRONTIER_SOURCE}" rev-parse HEAD)" = "deadc4a321f0baaa534c6ebd17f974123733cdc2" +test "$(git -C "${VLLM_SOURCE}" rev-parse HEAD)" = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1" +test -f "${MODEL_ROOT}/config.json" +git rev-parse HEAD > "${OUTPUT_ROOT}/provenance/aituner.commit" +git -C "${FRONTIER_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/frontier.commit" +git -C "${VLLM_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/vllm.commit" +printf '%s\n' "${BATCH_SPECS[@]}" > "${OUTPUT_ROOT}/provenance/batch-specs.txt" +sha256sum "${PROFILE_DIR}/profile_vllm020_flashattn.py" "${RUN_DIR}/run_graph_kernel_only_attention.sh" > "${OUTPUT_ROOT}/provenance/source.sha256" + +timeout --signal=TERM --kill-after=30s 2400 \ + "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/profile_vllm020_flashattn.py" \ + --vllm-source "${VLLM_SOURCE}" \ + --frontier-source "${FRONTIER_SOURCE}" \ + --model "${MODEL_ROOT}" \ + --output "${OUTPUT_ROOT}/raw/attention-tp${TP}.json" \ + --tp "${TP}" \ + --batch-specs "${BATCH_SPECS[@]}" \ + --warmup-iters "${WARMUP_ITERS}" \ + --repeats "${REPEATS}" \ + --profile-kv-update \ + --profile-method record_function + +test -s "${OUTPUT_ROOT}/raw/attention-tp${TP}.json" +sha256sum "${OUTPUT_ROOT}/raw/attention-tp${TP}.json" "${OUTPUT_ROOT}/provenance"/* > "${OUTPUT_ROOT}/artifacts.sha256" +date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ" +echo "GRAPH_KERNEL_ONLY_ATTENTION_COMPLETE tp=${TP} rows=${#BATCH_SPECS[@]}" diff --git a/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_linear.sh b/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_linear.sh new file mode 100644 index 0000000..7af7161 --- /dev/null +++ b/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_linear.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +set -euo pipefail + +OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT must be set}" +RUN_DIR="$(pwd -P)" +PROFILE_DIR="${PROFILE_DIR:-${RUN_DIR%/runs/frontier-fidelity-envelope-v1}/runs/frontier-qwen30-vllm020-profile-v1}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}" +VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}" +FRONTIER_SOURCE="${FRONTIER_SOURCE:-/home/admin/cpfs/wjh/aituner/frontier-t1-dash0-deadc4a}" +MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}" +TOKENS=(1 2 4 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128) +export MODEL_ROOT + +mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" "${OUTPUT_ROOT}/profiles" +exec > >(tee -a "${OUTPUT_ROOT}/logs/linear.log") 2>&1 +IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?a fleet-allocated GPU is required}" +if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then + echo "expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2 + exit 1 +fi + +echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} role=linear-kernel-only tp=1,2,4 tokens='${TOKENS[*]}' method=Frontier-RecordFunctionTracer output=${OUTPUT_ROOT} expected_wall=15-35m expected_gpu_cap=1.0_H20h" +date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ" +test "$(git -C "${FRONTIER_SOURCE}" rev-parse HEAD)" = "deadc4a321f0baaa534c6ebd17f974123733cdc2" +test "$(git -C "${VLLM_SOURCE}" rev-parse HEAD)" = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1" +git rev-parse HEAD > "${OUTPUT_ROOT}/provenance/aituner.commit" +git -C "${FRONTIER_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/frontier.commit" +sha256sum "${PROFILE_DIR}/frontier_vllm020_compat.py" "${RUN_DIR}/run_graph_kernel_only_linear.sh" > "${OUTPUT_ROOT}/provenance/source.sha256" + +cd "${FRONTIER_SOURCE}" +timeout --signal=TERM --kill-after=30s 2400 \ + "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/frontier_vllm020_compat.py" \ + --disable_ray --num_gpus 1 --output_dir "${OUTPUT_ROOT}/profiles" \ + --device h20 --models qwen3-a3b-30b-moe \ + --num_tensor_parallel_workers 1 2 4 --max_tokens 128 \ + --num_tokens_list "${TOKENS[@]}" --profile_method record_function \ + --precision BF16 --is_moe --yes + +find "${OUTPUT_ROOT}/profiles" -name linear_op_kernel_only.csv -type f -size +0c -print -quit > "${OUTPUT_ROOT}/provenance/linear-path.txt" +test -s "${OUTPUT_ROOT}/provenance/linear-path.txt" +date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ" +echo "GRAPH_KERNEL_ONLY_LINEAR_COMPLETE" diff --git a/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_moe.sh b/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_moe.sh new file mode 100644 index 0000000..5a36966 --- /dev/null +++ b/runs/frontier-fidelity-envelope-v1/run_graph_kernel_only_moe.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash + +set -euo pipefail + +OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT must be set}" +RUN_DIR="$(pwd -P)" +PROFILE_DIR="${PROFILE_DIR:-${RUN_DIR%/runs/frontier-fidelity-envelope-v1}/runs/frontier-qwen30-vllm020-profile-v1}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}" +VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}" +FRONTIER_SOURCE="${FRONTIER_SOURCE:-/home/admin/cpfs/wjh/aituner/frontier-t1-dash0-deadc4a}" +MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}" +TOKENS="${TOKENS:-1 2 4 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128}" + +mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" "${OUTPUT_ROOT}/raw" +exec > >(tee -a "${OUTPUT_ROOT}/logs/moe-router.log") 2>&1 +IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?a fleet-allocated GPU is required}" +if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then + echo "expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2 + exit 1 +fi + +echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} role=MoE+router-kernel-only tp=1,2,4 tokens='${TOKENS}' method=Frontier-RecordFunctionTracer output=${OUTPUT_ROOT} expected_wall=10-30m expected_gpu_cap=1.0_H20h" +date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ" +test "$(git -C "${FRONTIER_SOURCE}" rev-parse HEAD)" = "deadc4a321f0baaa534c6ebd17f974123733cdc2" +test "$(git -C "${VLLM_SOURCE}" rev-parse HEAD)" = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1" +git rev-parse HEAD > "${OUTPUT_ROOT}/provenance/aituner.commit" +git -C "${FRONTIER_SOURCE}" rev-parse HEAD > "${OUTPUT_ROOT}/provenance/frontier.commit" +printf '%s\n' ${TOKENS} > "${OUTPUT_ROOT}/provenance/tokens.txt" +sha256sum "${PROFILE_DIR}/profile_vllm020_moe.py" "${PROFILE_DIR}/profile_vllm020_router.py" "${RUN_DIR}/run_graph_kernel_only_moe.sh" > "${OUTPUT_ROOT}/provenance/source.sha256" + +timeout --signal=TERM --kill-after=30s 2400 \ + "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/profile_vllm020_moe.py" \ + --vllm-source "${VLLM_SOURCE}" --frontier-source "${FRONTIER_SOURCE}" \ + --model "${MODEL_ROOT}" --output "${OUTPUT_ROOT}/raw/moe.json" \ + --tp 1 2 4 --num-tokens ${TOKENS} --routing-modes uniform_random_logits \ + --warmup-iters 3 --repeats 5 --profile-method record_function --check-reference + +timeout --signal=TERM --kill-after=30s 1800 \ + "${VENV_ROOT}/bin/python" "${PROFILE_DIR}/profile_vllm020_router.py" \ + --vllm-source "${VLLM_SOURCE}" --frontier-source "${FRONTIER_SOURCE}" \ + --model "${MODEL_ROOT}" --output "${OUTPUT_ROOT}/raw/router.json" \ + --num-tokens ${TOKENS} --warmup-iters 3 --repeats 5 --profile-method record_function + +test -s "${OUTPUT_ROOT}/raw/moe.json" +test -s "${OUTPUT_ROOT}/raw/router.json" +sha256sum "${OUTPUT_ROOT}/raw"/*.json "${OUTPUT_ROOT}/provenance"/* > "${OUTPUT_ROOT}/artifacts.sha256" +date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ" +echo "GRAPH_KERNEL_ONLY_MOE_COMPLETE" diff --git a/runs/frontier-qwen30-vllm020-profile-v1/freeze_frontier_profiles.py b/runs/frontier-qwen30-vllm020-profile-v1/freeze_frontier_profiles.py index ade95f0..c6f8a2a 100644 --- a/runs/frontier-qwen30-vllm020-profile-v1/freeze_frontier_profiles.py +++ b/runs/frontier-qwen30-vllm020-profile-v1/freeze_frontier_profiles.py @@ -117,8 +117,18 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--attention", type=Path, nargs="+", required=True) parser.add_argument("--moe", type=Path, required=True) parser.add_argument("--router", type=Path, required=True) - parser.add_argument("--allreduce", type=Path, nargs=2, required=True) + parser.add_argument("--allreduce", type=Path, nargs=2) + parser.add_argument("--allreduce-frozen", type=Path) parser.add_argument("--output", type=Path, required=True) + parser.add_argument( + "--measurement-type", + choices=("CUDA_EVENT", "KERNEL_ONLY"), + default="CUDA_EVENT", + ) + parser.add_argument( + "--frontier-commit", + default="d9cfeb6d8791fbf2f295dd9744c56a666171776e", + ) return parser.parse_args() @@ -196,7 +206,7 @@ def write_csv(path: Path, fieldnames: list[str], rows: list[dict[str, Any]]) -> def freeze_attention( - inputs: list[Path], output: Path + inputs: list[Path], output: Path, *, measurement_type: str ) -> tuple[int, int, list[str]]: rows: list[dict[str, Any]] = [] mixed_rows: list[dict[str, Any]] = [] @@ -208,6 +218,13 @@ def freeze_attention( raise ValueError(f"unexpected attention schema in {path}") if payload["environment"].get("vllm_version") != "0.20.0": raise ValueError(f"unexpected vLLM version in {path}") + expected_method = ( + "record_function" if measurement_type == "KERNEL_ONLY" else "cuda_event" + ) + if payload["environment"].get("profile_method", "cuda_event") != expected_method: + raise ValueError( + f"attention profile method mismatch in {path}: expected {expected_method}" + ) for raw in payload["rows"]: if raw.get("error") is not None: raise ValueError(f"failed attention row in {path}: {raw['error']}") @@ -321,7 +338,7 @@ def freeze_attention( "profiling_precision": "BF16", "model_arch": "generic", "quant_signature": "none", - "measurement_type": "CUDA_EVENT", + "measurement_type": measurement_type, "is_true_mixed_batch": True, "prefill_seq_lens": json.dumps(prefill_queries), "prefill_kv_cache_sizes": json.dumps(prefill_contexts), @@ -418,7 +435,7 @@ def freeze_attention( "profiling_precision": "BF16", "model_arch": "generic", "quant_signature": "none", - "measurement_type": "CUDA_EVENT", + "measurement_type": measurement_type, "is_true_mixed_batch": False, "prefill_seq_lens": "", "prefill_kv_cache_sizes": "", @@ -498,13 +515,23 @@ def load_features(counts: list[int]) -> dict[str, float]: } -def freeze_moe(moe_path: Path, router_path: Path, output: Path) -> int: +def freeze_moe( + moe_path: Path, router_path: Path, output: Path, *, measurement_type: str +) -> int: moe = load_json(moe_path) router = load_json(router_path) if moe.get("schema_version") != "qwen30_vllm020_moe_raw.v1": raise ValueError(f"unexpected MoE schema in {moe_path}") if router.get("schema_version") != "qwen30_vllm020_router_raw.v1": raise ValueError(f"unexpected router schema in {router_path}") + expected_method = ( + "record_function" if measurement_type == "KERNEL_ONLY" else "cuda_event" + ) + for payload, label in ((moe, "moe"), (router, "router")): + if payload["environment"].get("profile_method", "cuda_event") != expected_method: + raise ValueError( + f"{label} profile method mismatch in {payload}: expected {expected_method}" + ) router_by_tokens = {int(row["num_tokens"]): row for row in router["rows"]} rows: list[dict[str, Any]] = [] seen_pairs: set[tuple[int, int, str]] = set() @@ -550,7 +577,7 @@ def freeze_moe(moe_path: Path, router_path: Path, output: Path) -> int: "load_distribution": routing_mode, "seed": 20260716, "moe_grouped_gemm_backend": raw["backend"], - "measurement_type": "CUDA_EVENT", + "measurement_type": measurement_type, "profiling_precision": "BF16", "model_arch": "generic", "quant_signature": "none", @@ -565,9 +592,17 @@ def freeze_moe(moe_path: Path, router_path: Path, output: Path) -> int: ) rows.append(row) - expected = 3 * 12 * 2 - if len(rows) != expected: - raise ValueError(f"expected {expected} MoE rows, got {len(rows)}") + tokens = {int(row["num_tokens"]) for row in router["rows"]} + modes = {str(row["routing_mode"]) for row in moe["rows"]} + expected = {(tp, tokens_value, mode) for tp in (1, 2, 4) for tokens_value in tokens for mode in modes} + actual = { + (int(row["num_tensor_parallel_workers"]), int(row["num_tokens"]), str(row["load_distribution"])) + for row in rows + } + if actual != expected: + raise ValueError( + f"MoE TP/token/routing coverage mismatch: missing={expected - actual}, extra={actual - expected}" + ) moe_fields = [ f"time_stats.{op}.{stat}" for op in MOE_OPS for stat in STAT_NAMES ] + list(MOE_METADATA) @@ -617,7 +652,13 @@ def freeze_allreduce(inputs: list[Path], output: Path) -> int: def main() -> None: args = parse_args() - all_inputs = [args.linear, *args.attention, args.moe, args.router, *args.allreduce] + if args.allreduce is not None and args.allreduce_frozen is not None: + raise SystemExit("provide either --allreduce or --allreduce-frozen, not both") + all_inputs = [args.linear, *args.attention, args.moe, args.router] + if args.allreduce is not None: + all_inputs.extend(args.allreduce) + if args.allreduce_frozen is not None: + all_inputs.append(args.allreduce_frozen) for path in all_inputs: if not path.is_file(): raise SystemExit(f"missing input: {path}") @@ -627,24 +668,39 @@ def main() -> None: shutil.copyfile(args.linear, linear_output) with linear_output.open(newline="") as handle: linear_rows = list(csv.DictReader(handle)) - if len(linear_rows) != 36: - raise ValueError(f"expected 36 linear rows, got {len(linear_rows)}") + if not linear_rows: + raise ValueError("linear profile has no rows") + if {row.get("measurement_type") for row in linear_rows} != {args.measurement_type}: + raise ValueError( + f"linear measurement family mismatch: expected {args.measurement_type}" + ) attention_rows, mixed_rows, attention_tps = freeze_attention( - list(args.attention), args.output + list(args.attention), args.output, measurement_type=args.measurement_type ) - moe_rows = freeze_moe(args.moe, args.router, args.output) - allreduce_rows = freeze_allreduce(list(args.allreduce), args.output) + moe_rows = freeze_moe( + args.moe, args.router, args.output, measurement_type=args.measurement_type + ) + allreduce_rows = 0 + allreduce_source = "not_included" + if args.allreduce is not None: + allreduce_rows = freeze_allreduce(list(args.allreduce), args.output) + allreduce_source = "raw_vllm020_measurements" + elif args.allreduce_frozen is not None: + shutil.copyfile(args.allreduce_frozen, args.output / "allreduce.json") + allreduce_rows = len(load_json(args.allreduce_frozen).get("rows", [])) + allreduce_source = "carried_forward_frozen_measurements" output_files = [ linear_output, args.output / "attention.csv", args.output / "attention_true_mixed_fused.csv", args.output / "moe.csv", - args.output / "allreduce.json", ] + if (args.output / "allreduce.json").is_file(): + output_files.append(args.output / "allreduce.json") batch_composition_augmented = len(args.attention) > 3 - long_context_augmented = any( + long_context_augmented = args.measurement_type == "KERNEL_ONLY" or any( "long-context" in path.name for path in args.attention ) long_context_coverage: dict[str, Any] = {"included": long_context_augmented} @@ -659,34 +715,45 @@ def main() -> None: if int(row["num_tensor_parallel_workers"]) == tp and row["is_prefill"].lower() == "false" } - mixed_kv = { - int(float(row["decode_avg_kv_cache_size"])) - for row in frozen_attention - if int(row["num_tensor_parallel_workers"]) == tp - and row.get("is_true_mixed_batch", "").lower() == "true" - } - if not {16384, 32768, 40960}.issubset(decode_kv): - raise ValueError(f"long-context decode coverage mismatch for TP{tp}") - if not {16384, 32768}.issubset(mixed_kv): - raise ValueError(f"long-context mixed coverage mismatch for TP{tp}") + required_decode = ( + {128, 1024, 2048, 4096, 8192, 16384, 32768, 40960} + if args.measurement_type == "KERNEL_ONLY" + else {16384, 32768, 40960} + ) + if not required_decode.issubset(decode_kv): + raise ValueError(f"decode KV coverage mismatch for TP{tp}") by_tp[str(tp)] = { "decode_kv_lengths": sorted(decode_kv), - "true_mixed_decode_avg_kv_lengths": sorted(mixed_kv), } + if args.measurement_type != "KERNEL_ONLY": + mixed_kv = { + int(float(row["decode_avg_kv_cache_size"])) + for row in frozen_attention + if int(row["num_tensor_parallel_workers"]) == tp + and row.get("is_true_mixed_batch", "").lower() == "true" + } + if not {16384, 32768}.issubset(mixed_kv): + raise ValueError(f"long-context mixed coverage mismatch for TP{tp}") + by_tp[str(tp)]["true_mixed_decode_avg_kv_lengths"] = sorted(mixed_kv) long_context_coverage["by_tp"] = by_tp manifest = { "schema_version": ( - "frontier_qwen30_vllm020_frozen_profile.v4" - if long_context_augmented + "frontier_qwen30_vllm020_kernel_only_profile.v1" + if args.measurement_type == "KERNEL_ONLY" + else ( + "frontier_qwen30_vllm020_frozen_profile.v4" + if long_context_augmented else ( "frontier_qwen30_vllm020_frozen_profile.v3" if batch_composition_augmented else "frontier_qwen30_vllm020_frozen_profile.v2" + ) ) ), "profile_id": ( "qwen3-30b-a3b-bf16-vllm020-h20-tp1-2-4-" "fused-mixed-total-conserving" + + ("-kernel-only-record-function" if args.measurement_type == "KERNEL_ONLY" else "") + ("-pure-prefill-batch-composition" if batch_composition_augmented else "") + ("-long-context-decode-mixed" if long_context_augmented else "") ), @@ -696,7 +763,7 @@ def main() -> None: "dtype": "bfloat16", "vllm_version": "0.20.0", "vllm_source_commit": "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1", - "frontier_commit": "d9cfeb6d8791fbf2f295dd9744c56a666171776e", + "frontier_commit": args.frontier_commit, "tensor_parallel_sizes": [1, 2, 4], }, "row_counts": { @@ -740,8 +807,7 @@ def main() -> None: "expert measurement already includes prepare/finalize so shuffling is zero" ), "allreduce": ( - "Frozen exact runtime measurements; base profile-only comparison keeps the " - "historical Frontier CC backend fixed to isolate compute profile fidelity" + "Frozen exact runtime measurements; source=" + allreduce_source ), }, "inputs": {str(path.resolve()): sha256(path) for path in all_inputs}, diff --git a/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_flashattn.py b/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_flashattn.py index 0e25d35..200762e 100644 --- a/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_flashattn.py +++ b/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_flashattn.py @@ -39,6 +39,12 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--repeats", type=int, default=5) parser.add_argument("--device", default="cuda:0") parser.add_argument("--profile-kv-update", action="store_true") + parser.add_argument( + "--profile-method", + choices=("cuda_event", "record_function"), + default="cuda_event", + ) + parser.add_argument("--frontier-source", type=Path) return parser.parse_args() @@ -66,12 +72,17 @@ def main() -> None: raise SystemExit(f"expected vLLM source {VLLM_COMMIT}, got {source_head}") if not args.model.joinpath("config.json").is_file(): raise SystemExit(f"missing model config: {args.model / 'config.json'}") + if args.profile_method == "record_function" and args.frontier_source is None: + raise SystemExit("--frontier-source is required for --profile-method record_function") bench_dir = args.vllm_source / "benchmarks" / "attention_benchmarks" sys.path.insert(0, str(bench_dir)) import runner # type: ignore[import-not-found] # noqa: PLC0415 from batch_spec import parse_batch_spec # type: ignore[import-not-found] # noqa: PLC0415 - from common import BenchmarkConfig # type: ignore[import-not-found] # noqa: PLC0415 + from common import ( # type: ignore[import-not-found] # noqa: PLC0415 + BenchmarkConfig, + BenchmarkResult, + ) from vllm.config import ( # noqa: PLC0415 CacheConfig, CompilationConfig, @@ -90,6 +101,13 @@ def main() -> None: from vllm.v1.kv_cache_interface import FullAttentionSpec # noqa: PLC0415 from vllm.v1.worker.workspace import init_workspace_manager # noqa: PLC0415 + record_function_tracer = None + if args.profile_method == "record_function": + sys.path.insert(0, str(args.frontier_source.resolve())) + from frontier.profiling.utils.record_function_tracer import RecordFunctionTracer + + record_function_tracer = RecordFunctionTracer + def create_vllm_config(config: BenchmarkConfig, max_num_blocks: int) -> VllmConfig: model_config = ModelConfig( model=str(args.model), @@ -146,6 +164,9 @@ def main() -> None: runner._create_vllm_config = create_vllm_config init_workspace_manager(args.device) + args.output.parent.mkdir(parents=True, exist_ok=True) + if args.profile_method == "record_function": + (args.output.parent / "profiler_traces").mkdir(exist_ok=True) def profile_kv_cache_update(config: BenchmarkConfig) -> dict[str, float]: device = torch.device(config.device) @@ -218,6 +239,137 @@ def main() -> None: "std_ms": statistics.pstdev(samples), } + def profile_kernel_only( + config: BenchmarkConfig, + ) -> tuple[BenchmarkResult, dict[str, float] | None]: + """Trace exactly one vLLM FA3 forward/KV-update per annotation. + + `RecordFunctionTracer` is Frontier's actual KERNEL_ONLY collector: it + follows CUDA launch correlations and sums kernels under the annotation. + The profiling loop therefore contains no CUDA-event value relabeling. + """ + device = torch.device(config.device) + torch.accelerator.set_device_index(device) + backend_config = runner._get_backend_config(config.backend) + requests = parse_batch_spec(config.batch_spec) + q_lens = [request.q_len for request in requests] + kv_lens = [request.kv_len for request in requests] + total_q = sum(q_lens) + max_kv = max(kv_lens) + max_blocks_per_request = (max_kv + config.block_size - 1) // config.block_size + max_num_blocks = len(requests) * max_blocks_per_request + + with runner.log_warnings_and_errors_only(): + vllm_config = create_vllm_config(config, max_num_blocks) + dtype = vllm_config.model_config.dtype + with set_current_vllm_config(vllm_config): + backend_class, impl, layer = runner._create_backend_impl( + backend_config, config, device, dtype + ) + required_layout = backend_class.get_required_kv_cache_layout() + if required_layout is not None: + set_kv_cache_layout(required_layout) + get_kv_cache_layout.cache_clear() + common_metadata = runner._build_common_attn_metadata( + q_lens, kv_lens, config.block_size, device + ) + kv_cache_spec = FullAttentionSpec( + block_size=config.block_size, + num_kv_heads=config.num_kv_heads, + head_size=config.head_dim, + dtype=dtype, + ) + builder = runner._create_metadata_builder( + backend_class, kv_cache_spec, vllm_config, device, config.backend + ) + attn_metadata = builder.build( + common_prefix_len=0, common_attn_metadata=common_metadata + ) + quantize_query = config.kv_cache_dtype.startswith("fp8") and getattr( + impl, "supports_quant_query_input", False + ) + q_list, k_list, v_list = runner._create_input_tensors( + config, total_q, device, dtype, quantize_query=quantize_query + ) + cache_list = runner._create_kv_cache( + config, max_num_blocks, backend_class, device, dtype + ) + output = torch.empty( + total_q, + config.num_q_heads, + config.head_dim, + device=device, + dtype=dtype, + ) + + def run_core() -> None: + for layer_index in range(config.num_layers): + impl.forward( + layer, + q_list[layer_index], + k_list[layer_index], + v_list[layer_index], + cache_list[layer_index], + attn_metadata, + output=output, + ) + + for _ in range(config.warmup_iters): + run_core() + torch.accelerator.synchronize() + core_tracer = record_function_tracer(str(args.output.parent)) + with core_tracer: + for _ in range(config.repeats): + with torch.profiler.record_function("vidur_attention_core"): + run_core() + core_stats = core_tracer.get_operation_time_stats() + if "attention_core" not in core_stats: + raise RuntimeError("missing KERNEL_ONLY FlashAttention core stats") + core = { + name: float(value) / config.num_layers + for name, value in core_stats["attention_core"].items() + } + + kv_stats = None + if args.profile_kv_update: + def run_kv_cache_update() -> None: + for layer_index in range(config.num_layers): + impl.do_kv_cache_update( + layer, + k_list[layer_index], + v_list[layer_index], + cache_list[layer_index], + common_metadata.slot_mapping, + ) + + for _ in range(config.warmup_iters): + run_kv_cache_update() + torch.accelerator.synchronize() + kv_tracer = record_function_tracer(str(args.output.parent)) + with kv_tracer: + for _ in range(config.repeats): + with torch.profiler.record_function("vidur_attn_kv_cache_save"): + run_kv_cache_update() + kv_time_stats = kv_tracer.get_operation_time_stats() + if "attn_kv_cache_save" not in kv_time_stats: + raise RuntimeError("missing KERNEL_ONLY KV-update stats") + kv_stats = { + f"{name}_ms": float(value) / config.num_layers + for name, value in kv_time_stats["attn_kv_cache_save"].items() + } + + result = BenchmarkResult( + config=config, + mean_time=core["mean"] / 1000.0, + std_time=core["std"] / 1000.0, + min_time=core["min"] / 1000.0, + max_time=core["max"] / 1000.0, + throughput_tokens_per_sec=( + total_q / (core["mean"] / 1000.0) if core["mean"] > 0 else 0.0 + ), + ) + return result, kv_stats + rows: list[dict[str, object]] = [] for tp in args.tp: for batch_spec in args.batch_specs: @@ -237,12 +389,18 @@ def main() -> None: kv_cache_dtype="auto", use_cuda_graphs=False, ) - result = runner.run_attention_benchmark(config) + if args.profile_method == "record_function": + result, kv_stats = profile_kernel_only(config) + else: + result = runner.run_attention_benchmark(config) + kv_stats = ( + profile_kv_cache_update(config) if args.profile_kv_update else None + ) row = result.to_dict() row["tensor_parallel_size"] = tp row["attention_core_excludes_kv_cache_update"] = True - if args.profile_kv_update: - row["kv_cache_update_time"] = profile_kv_cache_update(config) + if kv_stats is not None: + row["kv_cache_update_time"] = kv_stats rows.append(row) print( json.dumps( @@ -273,10 +431,10 @@ def main() -> None: "attention_backend": "FLASH_ATTN", "block_size": 16, "profile_kv_update": args.profile_kv_update, + "profile_method": args.profile_method, }, "rows": rows, } - args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text( json.dumps(payload, indent=2, sort_keys=True, default=json_default) + "\n" ) diff --git a/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_moe.py b/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_moe.py index 74da2d1..2eecef2 100644 --- a/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_moe.py +++ b/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_moe.py @@ -9,7 +9,7 @@ import math import statistics import subprocess from pathlib import Path -from typing import Any +from typing import Any, Callable import torch import vllm @@ -40,6 +40,12 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--repeats", type=int, default=5) parser.add_argument("--device", default="cuda:0") parser.add_argument("--check-reference", action="store_true") + parser.add_argument( + "--profile-method", + choices=("cuda_event", "record_function"), + default="cuda_event", + ) + parser.add_argument("--frontier-source", type=Path) return parser.parse_args() @@ -59,6 +65,34 @@ def stats_ms(samples: list[float]) -> dict[str, float]: } +def measure_kernel_only_ms( + fn: Callable[[], torch.Tensor], + *, + warmup_iters: int, + repeats: int, + trace_root: Path, + operation_name: str, + record_function_tracer: type, +) -> tuple[torch.Tensor, dict[str, float]]: + """Use Frontier's KERNEL_ONLY contract, not a CUDA-event relabel.""" + result = None + for _ in range(warmup_iters): + result = fn() + torch.accelerator.synchronize() + + tracer = record_function_tracer(str(trace_root)) + with tracer: + for _ in range(repeats): + with torch.profiler.record_function(f"vidur_{operation_name}"): + result = fn() + stats = tracer.get_operation_time_stats() + if operation_name not in stats: + raise RuntimeError(f"missing RecordFunctionTracer stats for {operation_name}") + if result is None: + raise RuntimeError("kernel-only profiler executed no MoE step") + return result, {name: float(value) for name, value in stats[operation_name].items()} + + def routing_inputs( mode: str, num_tokens: int, device: torch.device ) -> tuple[torch.Tensor, torch.Tensor, dict[str, Any]]: @@ -145,6 +179,8 @@ def main() -> None: raise SystemExit( f"model contract mismatch: expected {expected_model}, got {observed_model}" ) + if args.profile_method == "record_function" and args.frontier_source is None: + raise SystemExit("--frontier-source is required for --profile-method record_function") from vllm.config import ParallelConfig, VllmConfig, set_current_vllm_config from vllm.model_executor.layers.fused_moe.activation import MoEActivation @@ -163,10 +199,22 @@ def main() -> None: from vllm.utils.math_utils import next_power_of_2 from vllm.v1.worker.workspace import init_workspace_manager + record_function_tracer = None + if args.profile_method == "record_function": + import sys + + sys.path.insert(0, str(args.frontier_source.resolve())) + from frontier.profiling.utils.record_function_tracer import RecordFunctionTracer + + record_function_tracer = RecordFunctionTracer + device = torch.device(args.device) torch.accelerator.set_device_index(device) torch.manual_seed(20260716) init_workspace_manager(args.device) + args.output.parent.mkdir(parents=True, exist_ok=True) + if args.profile_method == "record_function": + (args.output.parent / "profiler_traces").mkdir(exist_ok=True) max_num_tokens = next_power_of_2(max(args.num_tokens)) rows: list[dict[str, Any]] = [] @@ -257,8 +305,8 @@ def main() -> None: routing_mode, num_tokens, device ) - for _ in range(args.warmup_iters): - output = kernel.apply( + def run_kernel() -> torch.Tensor: + return kernel.apply( hidden_states=hidden, w1=w13_kernel, w2=w2_kernel, @@ -269,27 +317,30 @@ def main() -> None: expert_map=None, apply_router_weight_on_input=False, ) - torch.accelerator.synchronize() - samples: list[float] = [] - for _ in range(args.repeats): - start = torch.cuda.Event(enable_timing=True) - end = torch.cuda.Event(enable_timing=True) - start.record() - output = kernel.apply( - hidden_states=hidden, - w1=w13_kernel, - w2=w2_kernel, - topk_weights=topk_weights, - topk_ids=topk_ids, - activation=MoEActivation.SILU, - global_num_experts=NUM_EXPERTS, - expert_map=None, - apply_router_weight_on_input=False, + if args.profile_method == "record_function": + output, time_ms = measure_kernel_only_ms( + run_kernel, + warmup_iters=args.warmup_iters, + repeats=args.repeats, + trace_root=args.output.parent, + operation_name="moe_grouped_gemm", + record_function_tracer=record_function_tracer, ) - end.record() + else: + for _ in range(args.warmup_iters): + output = run_kernel() torch.accelerator.synchronize() - samples.append(float(start.elapsed_time(end))) + samples: list[float] = [] + for _ in range(args.repeats): + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + start.record() + output = run_kernel() + end.record() + torch.accelerator.synchronize() + samples.append(float(start.elapsed_time(end))) + time_ms = stats_ms(samples) if output.shape != hidden.shape or not torch.isfinite(output).all(): raise SystemExit( @@ -316,7 +367,7 @@ def main() -> None: "backend": backend.value, "intermediate_size_per_partition": INTERMEDIATE_DIM // tp, "output_is_reduced": kernel.output_is_reduced(), - "time_ms": stats_ms(samples), + "time_ms": time_ms, "routing_load": load, } rows.append(row) @@ -350,6 +401,7 @@ def main() -> None: "weight_quantization": "none", "top_k": TOP_K, "norm_topk_prob": True, + "profile_method": args.profile_method, }, "measurement_scope": ( "one TP-local weight shard: vLLM modular MoE prepare+FlashInfer " @@ -357,7 +409,6 @@ def main() -> None: ), "rows": rows, } - args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") diff --git a/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_router.py b/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_router.py index 8da2070..eac341f 100644 --- a/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_router.py +++ b/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_router.py @@ -31,6 +31,12 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--warmup-iters", type=int, default=5) parser.add_argument("--repeats", type=int, default=20) parser.add_argument("--device", default="cuda:0") + parser.add_argument( + "--profile-method", + choices=("cuda_event", "record_function"), + default="cuda_event", + ) + parser.add_argument("--frontier-source", type=Path) return parser.parse_args() @@ -51,12 +57,34 @@ def stats_ms(samples: list[float]) -> dict[str, float]: def measure_ms( - fn: Callable[[], Any], warmup_iters: int, repeats: int + fn: Callable[[], Any], + warmup_iters: int, + repeats: int, + *, + profile_method: str, + trace_root: Path | None = None, + operation_name: str | None = None, + record_function_tracer: type | None = None, ) -> tuple[Any, dict[str, float]]: result = None for _ in range(warmup_iters): result = fn() torch.accelerator.synchronize() + if profile_method == "record_function": + if trace_root is None or operation_name is None or record_function_tracer is None: + raise ValueError("record_function profiling requires tracer metadata") + tracer = record_function_tracer(str(trace_root)) + with tracer: + for _ in range(repeats): + with torch.profiler.record_function(f"vidur_{operation_name}"): + result = fn() + stats = tracer.get_operation_time_stats() + if operation_name not in stats: + raise RuntimeError(f"missing RecordFunctionTracer stats for {operation_name}") + return result, { + name: float(value) for name, value in stats[operation_name].items() + } + samples: list[float] = [] for _ in range(repeats): start = torch.cuda.Event(enable_timing=True) @@ -76,6 +104,8 @@ def main() -> None: source_head = git_head(args.vllm_source) if source_head != VLLM_COMMIT: raise SystemExit(f"expected vLLM source {VLLM_COMMIT}, got {source_head}") + if args.profile_method == "record_function" and args.frontier_source is None: + raise SystemExit("--frontier-source is required for --profile-method record_function") raw_model_config = json.loads(args.model.joinpath("config.json").read_text()) observed = { @@ -103,6 +133,15 @@ def main() -> None: from vllm.model_executor.layers.fused_moe import fused_topk from vllm.model_executor.layers.linear import ReplicatedLinear + record_function_tracer = None + if args.profile_method == "record_function": + import sys + + sys.path.insert(0, str(args.frontier_source.resolve())) + from frontier.profiling.utils.record_function_tracer import RecordFunctionTracer + + record_function_tracer = RecordFunctionTracer + device = torch.device(args.device) torch.accelerator.set_device_index(device) torch.manual_seed(20260716) @@ -113,6 +152,9 @@ def main() -> None: skip_tokenizer_init=True, generation_config="vllm", ) + args.output.parent.mkdir(parents=True, exist_ok=True) + if args.profile_method == "record_function": + (args.output.parent / "profiler_traces").mkdir(exist_ok=True) rows: list[dict[str, Any]] = [] with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener: @@ -141,12 +183,22 @@ def main() -> None: (num_tokens, HIDDEN_DIM), device=device, dtype=torch.bfloat16 ).uniform_(-0.1, 0.1) logits, gate_time = measure_ms( - lambda: gate(hidden)[0], args.warmup_iters, args.repeats + lambda: gate(hidden)[0], + args.warmup_iters, + args.repeats, + profile_method=args.profile_method, + trace_root=args.output.parent, + operation_name="moe_gating_linear", + record_function_tracer=record_function_tracer, ) topk_result, topk_time = measure_ms( lambda: fused_topk(hidden, logits, TOP_K, renormalize=True), args.warmup_iters, args.repeats, + profile_method=args.profile_method, + trace_root=args.output.parent, + operation_name="moe_gating_routing_topk", + record_function_tracer=record_function_tracer, ) def gate_and_topk() -> tuple[ @@ -158,7 +210,13 @@ def main() -> None: ) combined_result, combined_time = measure_ms( - gate_and_topk, args.warmup_iters, args.repeats + gate_and_topk, + args.warmup_iters, + args.repeats, + profile_method=args.profile_method, + trace_root=args.output.parent, + operation_name="moe_gating_linear_and_routing_topk", + record_function_tracer=record_function_tracer, ) topk_weights, topk_ids, _ = topk_result combined_weights, combined_ids, _ = combined_result @@ -207,6 +265,7 @@ def main() -> None: "gate_replication": "replicated_across_tp", "top_k": TOP_K, "norm_topk_prob": True, + "profile_method": args.profile_method, }, "measurement_scope": ( "vLLM ReplicatedLinear gate and fused_topk; measured separately and " @@ -214,7 +273,6 @@ def main() -> None: ), "rows": rows, } - args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")