Add vLLM 0.20 per-TP FlashAttention profile smoke

2026-07-16 21:00:27 +08:00
parent 13f0713bf2
commit e9a4be6153
7 changed files with 371 additions and 0 deletions
--- a/runs/frontier-qwen30-vllm020-profile-v1/experiment-card.md
+++ b/runs/frontier-qwen30-vllm020-profile-v1/experiment-card.md
@@ -0,0 +1,43 @@
+# 实验 EXP-SIMFID-Q30-P020：同栈 per-TP operator profile 能否恢复 Frontier ranking？
+
+> **状态：** 运行中（用户已批准 5-step campaign）
+
+## Claim 与决策
+
+- **Parent claim：** Frontier 在 Qwen3-30B-A3B 的 TP×MNS surface 上出现 25.63% selection regret，原因可能不是 simulator scheduler 本身，而是 execution profile 与真实 serving stack 不一致。
+- **目的：** 区分 profile provenance mismatch 与 simulator composition/schema mismatch。
+- **Competing hypotheses：** H1：换成与真实 serving 相同的 per-TP operator profiles 后，profile-only ranking 达到 regret ≤5% 且 Kendall τ-b ≥0.8；H2：即使 operator profile 同栈，Frontier 的 operator composition 或 mixed-state abstraction 仍无法保持 ranking。
+- **事前预测：** 若 H1 成立，新 profile-only top set 应从旧结果的 TP4 移到真实 top family TP2，且不需要 end-to-end scalar calibration；若 H2 成立，operator microbench 单项可对齐，但组成后的 config rank 仍错，残差应随 phase、batch state 或 TP action 系统变化。
+- **判定规则：** `P-020` profile-only regret ≤5% 且 τ-b ≥0.8 → 支持 H1；否则进入 stage/action-conditioned residual 分解，禁止用本 surface 的 E2E measurement 拟合 scale。
+
+## Setup
+
+- **自变量：** execution profile root：`P-old`（历史 profile）与 `P-020`（本实验冻结 profile）。`P-cal` 历史 per-TP scalar calibration 只作上界参照。
+- **控制变量：** Frontier simulator code/commit、trace fixtures、KV capacity、SLO、12-cell config surface、random seed 与 analysis code保持完全相同。
+- **System context：** Qwen3-30B-A3B，BF16 weights/activations/KV；community vLLM 0.20.0；dash0 8×NVIDIA H20；TP∈{1,2,4}，DP=PP=EP=1；MBT=8192；MNS∈{8,16,32,64}；prefix caching、chunked prefill 与 async scheduling均启用。
+- **真实 runtime contract：** engine log 已确认 TP1/2/4 均为 FlashAttention 3 attention + FlashInfer CUTLASS unquantized MoE；TP2/4 使用 FlashInfer TRT-LLM all-reduce。旧 profile log 明确为 vLLM 0.11.1，attention backend 为 FlashInfer。
+- **Workload 或 trace：** 历史 ground truth `chat_w20260311_1000`，source 600 s window、replay time scale 0.1、input 0–8192、fixed output 128、prefix cache on。当前本地 recovered artifact 无法重建 exact source requests；本次只替换 frozen simulator profile，不重跑或改写 ground truth。
+- **Baselines：** old profile-only；new `P-020` profile-only；historical frozen per-TP calibrated。
+- **Metrics：** real/sim capacity per GPU、selected top set、worst selected-config regret、Kendall τ-b、informative-pair direction、逐 TP operator-time ratio、profile coverage/extrapolation ratio。
+
+## 预期产物与 review
+
+- **预期数据：** TP1/2/4 attention、KV-cache update、linear、MoE、collective raw profiles；冻结 manifest/sha256；12-cell simulation outputs；三 baseline comparison table。
+- **Figure prototype：** `mock-profile-ablation.png`；x 为 12 configs，y 为 normalized capacity；series 为 real、old profile-only、new profile-only、calibrated。图中数值明确标为 mock，不进入结论。
+- **人工 review：** 已批准。用户明确要求推进 5 个步骤，并要求 smoke 通过后直接完整运行。
+- **Review 意见：** 不能把 TP–MNS 耦合拆成互不相关问题；profile 只冻结 execution counterfactual 的证据，最终仍以完整 config ranking 判断。
+
+## 复现信息
+
+- **Code：** AITuner 当前 branch `codex/fidelity-prefix-pilot-20260714`；Frontier canonical commit `d9cfeb6d8791fbf2f295dd9744c56a666171776e`；vLLM commit/tag `88d34c6409e9fb3c7b8ca0c04756f061d2099eb1` / `0.20.0`。
+- **Environment：** `/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1`；hardware/driver/package freeze 写入每个 profile artifact。
+- **产物路径：** remote `/home/admin/cpfs/wjh/frontier_qwen30_vllm020_profiles/`; local harvest `runs/frontier-qwen30-vllm020-profile-v1/fleet-artifacts/`。
+- **已知 deviation：** 历史 real ground truth 来自 dash1；本次 profile 按用户要求在 dash0 生成。driver 也已从历史 570.133.20 漂移到 dash0 当前 580.95.05。结果只能归因于“profile stack alignment 在当前 dash0 上的效果”，不能声称严格复现旧 dash1 execution latency。
+
+## 结果
+
+- **观察事实：** 待完整 profile 与 simulation。
+- **异常：** Frontier 原生 attention profiler 只实现 `FLASHINFER`/`NO_OP`，不能调用真实 serving 的 `FLASH_ATTN`；其 MoE wrapper声明并硬编码 vLLM 0.10.x API。vLLM 0.20.0 自带的 attention benchmark可调用实际 FlashAttention backend，但 mixed prefill+decode 在 FA3 中是一个 fused varlen call，而 Frontier schema将其拆成 `attn_prefill` 与 `attn_decode` 两项。
+- **Interpretation 与剩余 alternatives：** 这既是 profiler compatibility gap，也是潜在 representational gap。先用真实 kernel smoke 判断能否无损物化 Frontier profile；不能无损时同时保留 fused total 与 schema projection，避免误把 projection error 当 kernel error。
+- **Claim update：** unchanged。
+- **下一步：** 通过 vLLM 0.20 exact-kernel smoke；冻结 trace-derived shape support；完整 TP1/2/4 profiling；profile-only rerun。
--- a/runs/frontier-qwen30-vllm020-profile-v1/fleet.toml
+++ b/runs/frontier-qwen30-vllm020-profile-v1/fleet.toml
@@ -0,0 +1,24 @@
+version = 1
+
+[paths]
+state_dir = "runs/frontier-qwen30-vllm020-profile-v1/fleet-state"
+artifacts_dir = "runs/frontier-qwen30-vllm020-profile-v1/fleet-artifacts"
+
+[ssh]
+connect_timeout_sec = 10
+
+[scheduler]
+gpu_free_memory_mb = 1024
+gpu_free_utilization_pct = 10
+prefer_pack = true
+
+[sync]
+mode = "scp"
+local_path = "runs/frontier-qwen30-vllm020-profile-v1"
+
+[[hosts]]
+name = "dash0"
+ssh_alias = "dash0"
+enabled = true
+sync_remote_path = "/home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-fleet"
+fleet_root = "/home/admin/cpfs/wjh/aituner/gpu-fleet-qwen30-vllm020-profile-v1"
--- a/runs/frontier-qwen30-vllm020-profile-v1/jobs_smoke.toml
+++ b/runs/frontier-qwen30-vllm020-profile-v1/jobs_smoke.toml
@@ -0,0 +1,15 @@
+version = 1
+
+[[jobs]]
+name = "qwen30-vllm020-flashattn-smoke-20260716-v1"
+gpus = 1
+gpu_model = "H20"
+hosts = ["dash0"]
+command = "cd /home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-v1/runs/frontier-qwen30-vllm020-profile-v1 && timeout --signal=TERM --kill-after=30s 900 bash run_flashattn_smoke.sh"
+artifacts = ["artifacts/flashattn-smoke"]
+
+[jobs.env]
+OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-fleet/artifacts/flashattn-smoke"
+VENV_ROOT = "/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1"
+VLLM_SOURCE = "/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build"
+MODEL = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B"
--- a/runs/frontier-qwen30-vllm020-profile-v1/mock-profile-ablation.png
+++ b/runs/frontier-qwen30-vllm020-profile-v1/mock-profile-ablation.png
--- a/runs/frontier-qwen30-vllm020-profile-v1/plot_mock_ablation.py
+++ b/runs/frontier-qwen30-vllm020-profile-v1/plot_mock_ablation.py
@@ -0,0 +1,41 @@
+#!/usr/bin/env python3
+"""Render the preregistered profile-ablation figure with schematic data."""
+
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+
+
+CONFIGS = [f"TP{tp}\nMNS{mns}" for tp in (1, 2, 4) for mns in (8, 16, 32, 64)]
+
+# Schematic only. These values are deliberately not derived from experiment data.
+MOCK = {
+    "Real (mock)": [0.42, 0.50, 0.58, 0.57, 0.60, 0.76, 1.00, 0.99, 0.55, 0.70, 0.74, 0.73],
+    "Old profile-only (mock)": [0.38, 0.44, 0.51, 0.50, 0.47, 0.58, 0.70, 0.69, 0.66, 0.84, 0.95, 0.96],
+    "New P-020 profile-only (H1 mock)": [0.40, 0.49, 0.57, 0.56, 0.58, 0.74, 0.98, 0.97, 0.53, 0.69, 0.75, 0.74],
+    "Per-TP calibrated upper bound (mock)": [0.41, 0.50, 0.58, 0.57, 0.59, 0.75, 0.99, 0.98, 0.54, 0.69, 0.75, 0.74],
+}
+
+
+def main() -> None:
+    output = Path(__file__).with_name("mock-profile-ablation.png")
+    fig, ax = plt.subplots(figsize=(13.5, 5.8), constrained_layout=True)
+    x = list(range(len(CONFIGS)))
+    styles = ["o-", "s--", "^-", "D:"]
+    for (label, values), style in zip(MOCK.items(), styles, strict=True):
+        ax.plot(x, values, style, linewidth=2, markersize=5, label=label)
+    ax.axvline(3.5, color="0.75", linewidth=1)
+    ax.axvline(7.5, color="0.75", linewidth=1)
+    ax.set_xticks(x, CONFIGS)
+    ax.set_ylabel("Normalized SLO-feasible throughput (schematic)")
+    ax.set_xlabel("Configuration")
+    ax.set_ylim(0.3, 1.08)
+    ax.grid(axis="y", alpha=0.25)
+    ax.legend(ncol=2, frameon=False, loc="upper left")
+    ax.set_title("MOCK / SCHEMATIC — expected discriminative result, not experiment data")
+    fig.savefig(output, dpi=180)
+    print(output)
+
+
+if __name__ == "__main__":
+    main()
--- a/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_flashattn.py
+++ b/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_flashattn.py
@@ -0,0 +1,190 @@
+#!/usr/bin/env python3
+"""Profile the exact vLLM 0.20 FlashAttention backend at TP-local shapes.
+
+This deliberately uses vLLM's own v0.20.0 attention benchmark runner instead
+of Frontier's FlashInfer-only attention wrapper.  The output is raw evidence;
+projection into Frontier's split attention CSV schema is a separate step.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import sys
+import types
+from pathlib import Path
+
+import torch
+import vllm
+
+
+VLLM_VERSION = "0.20.0"
+VLLM_COMMIT = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1"
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--vllm-source", type=Path, required=True)
+    parser.add_argument("--model", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--tp", type=int, choices=(1, 2, 4), nargs="+", default=[1, 2, 4])
+    parser.add_argument(
+        "--batch-specs",
+        nargs="+",
+        default=["q128", "4q1s128", "q128_4q1s128"],
+    )
+    parser.add_argument("--warmup-iters", type=int, default=3)
+    parser.add_argument("--repeats", type=int, default=5)
+    parser.add_argument("--device", default="cuda:0")
+    return parser.parse_args()
+
+
+def git_head(repo: Path) -> str:
+    return subprocess.check_output(
+        ["git", "-C", str(repo), "rev-parse", "HEAD"], text=True
+    ).strip()
+
+
+def main() -> None:
+    args = parse_args()
+    if vllm.__version__ != VLLM_VERSION:
+        raise SystemExit(f"expected vLLM {VLLM_VERSION}, got {vllm.__version__}")
+    source_head = git_head(args.vllm_source)
+    if source_head != VLLM_COMMIT:
+        raise SystemExit(f"expected vLLM source {VLLM_COMMIT}, got {source_head}")
+    if not args.model.joinpath("config.json").is_file():
+        raise SystemExit(f"missing model config: {args.model / 'config.json'}")
+
+    bench_dir = args.vllm_source / "benchmarks" / "attention_benchmarks"
+    sys.path.insert(0, str(bench_dir))
+    import runner  # type: ignore[import-not-found]  # noqa: PLC0415
+    from common import BenchmarkConfig  # type: ignore[import-not-found]  # noqa: PLC0415
+    from vllm.config import (  # noqa: PLC0415
+        CacheConfig,
+        CompilationConfig,
+        DeviceConfig,
+        LoadConfig,
+        ModelConfig,
+        ParallelConfig,
+        SchedulerConfig,
+        VllmConfig,
+    )
+    from vllm.v1.worker.workspace import init_workspace_manager  # noqa: PLC0415
+
+    def create_vllm_config(config: BenchmarkConfig, max_num_blocks: int) -> VllmConfig:
+        model_config = ModelConfig(
+            model=str(args.model),
+            tokenizer=str(args.model),
+            trust_remote_code=False,
+            dtype="bfloat16",
+            seed=0,
+            max_model_len=40960,
+        )
+        cache_config = CacheConfig(block_size=config.block_size, cache_dtype="auto")
+        cache_config.num_gpu_blocks = max_num_blocks
+        cache_config.num_cpu_blocks = 0
+        parallel_config = ParallelConfig(tensor_parallel_size=1)
+        scheduler_config = SchedulerConfig(
+            max_num_seqs=256,
+            max_num_batched_tokens=8192,
+            max_model_len=40960,
+            is_encoder_decoder=False,
+            enable_chunked_prefill=True,
+        )
+        model_config.get_num_layers = types.MethodType(
+            lambda self: config.num_layers, model_config
+        )
+        model_config.get_sliding_window_for_layer = types.MethodType(
+            lambda self, i: None, model_config
+        )
+        model_config.get_logits_soft_cap_for_layer = types.MethodType(
+            lambda self, i: 0.0, model_config
+        )
+        model_config.get_sm_scale_for_layer = types.MethodType(
+            lambda self, i: 1.0 / config.head_dim**0.5, model_config
+        )
+        model_config.get_num_attention_heads = types.MethodType(
+            lambda self, parallel_config=None: config.num_q_heads, model_config
+        )
+        model_config.get_num_kv_heads = types.MethodType(
+            lambda self, parallel_config=None: config.num_kv_heads, model_config
+        )
+        model_config.get_head_size = types.MethodType(
+            lambda self: config.head_dim, model_config
+        )
+        model_config.get_sliding_window = types.MethodType(
+            lambda self: None, model_config
+        )
+        return VllmConfig(
+            model_config=model_config,
+            cache_config=cache_config,
+            parallel_config=parallel_config,
+            scheduler_config=scheduler_config,
+            device_config=DeviceConfig(),
+            load_config=LoadConfig(),
+            compilation_config=CompilationConfig(),
+        )
+
+    runner._create_vllm_config = create_vllm_config
+    init_workspace_manager(args.device)
+
+    rows: list[dict[str, object]] = []
+    for tp in args.tp:
+        for batch_spec in args.batch_specs:
+            config = BenchmarkConfig(
+                backend="FLASH_ATTN",
+                batch_spec=batch_spec,
+                num_layers=1,
+                head_dim=128,
+                num_q_heads=32 // tp,
+                num_kv_heads=4 // tp,
+                block_size=16,
+                device=args.device,
+                dtype=torch.bfloat16,
+                repeats=args.repeats,
+                warmup_iters=args.warmup_iters,
+                profile_memory=True,
+                kv_cache_dtype="auto",
+                use_cuda_graphs=False,
+            )
+            result = runner.run_attention_benchmark(config)
+            row = result.to_dict()
+            row["tensor_parallel_size"] = tp
+            rows.append(row)
+            print(
+                json.dumps(
+                    {
+                        "tp": tp,
+                        "batch_spec": batch_spec,
+                        "mean_time_s": result.mean_time,
+                        "error": result.error,
+                    },
+                    sort_keys=True,
+                ),
+                flush=True,
+            )
+            if not result.success:
+                raise SystemExit(f"attention profile failed: {row}")
+
+    payload = {
+        "schema_version": "qwen30_vllm020_flashattn_raw.v1",
+        "environment": {
+            "vllm_version": vllm.__version__,
+            "vllm_source_commit": source_head,
+            "torch_version": torch.__version__,
+            "torch_cuda": torch.version.cuda,
+            "gpu": torch.cuda.get_device_name(torch.device(args.device)),
+            "model": str(args.model),
+            "dtype": "bfloat16",
+            "attention_backend": "FLASH_ATTN",
+            "block_size": 16,
+        },
+        "rows": rows,
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+
+
+if __name__ == "__main__":
+    main()
--- a/runs/frontier-qwen30-vllm020-profile-v1/run_flashattn_smoke.sh
+++ b/runs/frontier-qwen30-vllm020-profile-v1/run_flashattn_smoke.sh
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+OUTPUT_ROOT="${OUTPUT_ROOT:-$(pwd)/artifacts/flashattn-smoke}"
+VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}"
+VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}"
+MODEL="${MODEL:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}"
+LOG_DIR="${OUTPUT_ROOT}/logs"
+PROVENANCE_DIR="${OUTPUT_ROOT}/provenance"
+
+mkdir -p "${LOG_DIR}" "${PROVENANCE_DIR}"
+exec > >(tee -a "${LOG_DIR}/smoke.log") 2>&1
+
+if [[ -z "${CUDA_VISIBLE_DEVICES:-}" ]]; then
+  echo "ERROR: CUDA_VISIBLE_DEVICES must contain the fleet-allocated GPU" >&2
+  exit 1
+fi
+IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES}"
+if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then
+  echo "ERROR: expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2
+  exit 1
+fi
+
+echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} model=${MODEL} runtime=vLLM-0.20.0+cu129 operator=FlashAttention3 tp_local_shapes=1,2,4 specs=q128,4q1s128,q128_4q1s128 dtype=BF16 block=16 output=${OUTPUT_ROOT} expected_wall=3-8m hard_wall=900s hard_gpu_cap=0.15_H20h"
+date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ"
+nvidia-smi --query-gpu=index,name,driver_version,memory.used,utilization.gpu --format=csv,noheader
+
+test -x "${VENV_ROOT}/bin/python"
+test -f "${VLLM_SOURCE}/benchmarks/attention_benchmarks/runner.py"
+test -f "${MODEL}/config.json"
+test -f profile_vllm020_flashattn.py
+
+git rev-parse HEAD > "${PROVENANCE_DIR}/aituner.commit"
+git -C "${VLLM_SOURCE}" rev-parse HEAD > "${PROVENANCE_DIR}/vllm-source.commit"
+sha256sum profile_vllm020_flashattn.py run_flashattn_smoke.sh \
+  > "${PROVENANCE_DIR}/source.sha256"
+uv pip freeze --python "${VENV_ROOT}/bin/python" \
+  > "${PROVENANCE_DIR}/pip-freeze.txt"
+nvidia-smi --query-gpu=index,uuid,name,driver_version,memory.total \
+  --format=csv,noheader > "${PROVENANCE_DIR}/gpus.csv"
+
+timeout --signal=TERM --kill-after=30s 780 \
+  "${VENV_ROOT}/bin/python" profile_vllm020_flashattn.py \
+  --vllm-source "${VLLM_SOURCE}" \
+  --model "${MODEL}" \
+  --output "${OUTPUT_ROOT}/raw/flashattn-smoke.json" \
+  --tp 1 2 4 \
+  --batch-specs q128 4q1s128 q128_4q1s128 \
+  --warmup-iters 3 \
+  --repeats 5
+
+test -s "${OUTPUT_ROOT}/raw/flashattn-smoke.json"
+sha256sum "${OUTPUT_ROOT}/raw/flashattn-smoke.json" \
+  "${PROVENANCE_DIR}"/* > "${OUTPUT_ROOT}/artifacts.sha256"
+nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader
+date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ"
+echo "FLASHATTN_SMOKE_COMPLETE"