diff --git a/runs/frontier-qwen30-vllm020-profile-v1/experiment-card.md b/runs/frontier-qwen30-vllm020-profile-v1/experiment-card.md new file mode 100644 index 0000000..a51365c --- /dev/null +++ b/runs/frontier-qwen30-vllm020-profile-v1/experiment-card.md @@ -0,0 +1,43 @@ +# 实验 EXP-SIMFID-Q30-P020:同栈 per-TP operator profile 能否恢复 Frontier ranking? + +> **状态:** 运行中(用户已批准 5-step campaign) + +## Claim 与决策 + +- **Parent claim:** Frontier 在 Qwen3-30B-A3B 的 TP×MNS surface 上出现 25.63% selection regret,原因可能不是 simulator scheduler 本身,而是 execution profile 与真实 serving stack 不一致。 +- **目的:** 区分 profile provenance mismatch 与 simulator composition/schema mismatch。 +- **Competing hypotheses:** H1:换成与真实 serving 相同的 per-TP operator profiles 后,profile-only ranking 达到 regret ≤5% 且 Kendall τ-b ≥0.8;H2:即使 operator profile 同栈,Frontier 的 operator composition 或 mixed-state abstraction 仍无法保持 ranking。 +- **事前预测:** 若 H1 成立,新 profile-only top set 应从旧结果的 TP4 移到真实 top family TP2,且不需要 end-to-end scalar calibration;若 H2 成立,operator microbench 单项可对齐,但组成后的 config rank 仍错,残差应随 phase、batch state 或 TP action 系统变化。 +- **判定规则:** `P-020` profile-only regret ≤5% 且 τ-b ≥0.8 → 支持 H1;否则进入 stage/action-conditioned residual 分解,禁止用本 surface 的 E2E measurement 拟合 scale。 + +## Setup + +- **自变量:** execution profile root:`P-old`(历史 profile)与 `P-020`(本实验冻结 profile)。`P-cal` 历史 per-TP scalar calibration 只作上界参照。 +- **控制变量:** Frontier simulator code/commit、trace fixtures、KV capacity、SLO、12-cell config surface、random seed 与 analysis code保持完全相同。 +- **System context:** Qwen3-30B-A3B,BF16 weights/activations/KV;community vLLM 0.20.0;dash0 8×NVIDIA H20;TP∈{1,2,4},DP=PP=EP=1;MBT=8192;MNS∈{8,16,32,64};prefix caching、chunked prefill 与 async scheduling均启用。 +- **真实 runtime contract:** engine log 已确认 TP1/2/4 均为 FlashAttention 3 attention + FlashInfer CUTLASS unquantized MoE;TP2/4 使用 FlashInfer TRT-LLM all-reduce。旧 profile log 明确为 vLLM 0.11.1,attention backend 为 FlashInfer。 +- **Workload 或 trace:** 历史 ground truth `chat_w20260311_1000`,source 600 s window、replay time scale 0.1、input 0–8192、fixed output 128、prefix cache on。当前本地 recovered artifact 无法重建 exact source requests;本次只替换 frozen simulator profile,不重跑或改写 ground truth。 +- **Baselines:** old profile-only;new `P-020` profile-only;historical frozen per-TP calibrated。 +- **Metrics:** real/sim capacity per GPU、selected top set、worst selected-config regret、Kendall τ-b、informative-pair direction、逐 TP operator-time ratio、profile coverage/extrapolation ratio。 + +## 预期产物与 review + +- **预期数据:** TP1/2/4 attention、KV-cache update、linear、MoE、collective raw profiles;冻结 manifest/sha256;12-cell simulation outputs;三 baseline comparison table。 +- **Figure prototype:** `mock-profile-ablation.png`;x 为 12 configs,y 为 normalized capacity;series 为 real、old profile-only、new profile-only、calibrated。图中数值明确标为 mock,不进入结论。 +- **人工 review:** 已批准。用户明确要求推进 5 个步骤,并要求 smoke 通过后直接完整运行。 +- **Review 意见:** 不能把 TP–MNS 耦合拆成互不相关问题;profile 只冻结 execution counterfactual 的证据,最终仍以完整 config ranking 判断。 + +## 复现信息 + +- **Code:** AITuner 当前 branch `codex/fidelity-prefix-pilot-20260714`;Frontier canonical commit `d9cfeb6d8791fbf2f295dd9744c56a666171776e`;vLLM commit/tag `88d34c6409e9fb3c7b8ca0c04756f061d2099eb1` / `0.20.0`。 +- **Environment:** `/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1`;hardware/driver/package freeze 写入每个 profile artifact。 +- **产物路径:** remote `/home/admin/cpfs/wjh/frontier_qwen30_vllm020_profiles/`; local harvest `runs/frontier-qwen30-vllm020-profile-v1/fleet-artifacts/`。 +- **已知 deviation:** 历史 real ground truth 来自 dash1;本次 profile 按用户要求在 dash0 生成。driver 也已从历史 570.133.20 漂移到 dash0 当前 580.95.05。结果只能归因于“profile stack alignment 在当前 dash0 上的效果”,不能声称严格复现旧 dash1 execution latency。 + +## 结果 + +- **观察事实:** 待完整 profile 与 simulation。 +- **异常:** Frontier 原生 attention profiler 只实现 `FLASHINFER`/`NO_OP`,不能调用真实 serving 的 `FLASH_ATTN`;其 MoE wrapper声明并硬编码 vLLM 0.10.x API。vLLM 0.20.0 自带的 attention benchmark可调用实际 FlashAttention backend,但 mixed prefill+decode 在 FA3 中是一个 fused varlen call,而 Frontier schema将其拆成 `attn_prefill` 与 `attn_decode` 两项。 +- **Interpretation 与剩余 alternatives:** 这既是 profiler compatibility gap,也是潜在 representational gap。先用真实 kernel smoke 判断能否无损物化 Frontier profile;不能无损时同时保留 fused total 与 schema projection,避免误把 projection error 当 kernel error。 +- **Claim update:** unchanged。 +- **下一步:** 通过 vLLM 0.20 exact-kernel smoke;冻结 trace-derived shape support;完整 TP1/2/4 profiling;profile-only rerun。 diff --git a/runs/frontier-qwen30-vllm020-profile-v1/fleet.toml b/runs/frontier-qwen30-vllm020-profile-v1/fleet.toml new file mode 100644 index 0000000..e560549 --- /dev/null +++ b/runs/frontier-qwen30-vllm020-profile-v1/fleet.toml @@ -0,0 +1,24 @@ +version = 1 + +[paths] +state_dir = "runs/frontier-qwen30-vllm020-profile-v1/fleet-state" +artifacts_dir = "runs/frontier-qwen30-vllm020-profile-v1/fleet-artifacts" + +[ssh] +connect_timeout_sec = 10 + +[scheduler] +gpu_free_memory_mb = 1024 +gpu_free_utilization_pct = 10 +prefer_pack = true + +[sync] +mode = "scp" +local_path = "runs/frontier-qwen30-vllm020-profile-v1" + +[[hosts]] +name = "dash0" +ssh_alias = "dash0" +enabled = true +sync_remote_path = "/home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-fleet" +fleet_root = "/home/admin/cpfs/wjh/aituner/gpu-fleet-qwen30-vllm020-profile-v1" diff --git a/runs/frontier-qwen30-vllm020-profile-v1/jobs_smoke.toml b/runs/frontier-qwen30-vllm020-profile-v1/jobs_smoke.toml new file mode 100644 index 0000000..eee4f5a --- /dev/null +++ b/runs/frontier-qwen30-vllm020-profile-v1/jobs_smoke.toml @@ -0,0 +1,15 @@ +version = 1 + +[[jobs]] +name = "qwen30-vllm020-flashattn-smoke-20260716-v1" +gpus = 1 +gpu_model = "H20" +hosts = ["dash0"] +command = "cd /home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-v1/runs/frontier-qwen30-vllm020-profile-v1 && timeout --signal=TERM --kill-after=30s 900 bash run_flashattn_smoke.sh" +artifacts = ["artifacts/flashattn-smoke"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-fleet/artifacts/flashattn-smoke" +VENV_ROOT = "/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1" +VLLM_SOURCE = "/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build" +MODEL = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B" diff --git a/runs/frontier-qwen30-vllm020-profile-v1/mock-profile-ablation.png b/runs/frontier-qwen30-vllm020-profile-v1/mock-profile-ablation.png new file mode 100644 index 0000000..0457cd2 Binary files /dev/null and b/runs/frontier-qwen30-vllm020-profile-v1/mock-profile-ablation.png differ diff --git a/runs/frontier-qwen30-vllm020-profile-v1/plot_mock_ablation.py b/runs/frontier-qwen30-vllm020-profile-v1/plot_mock_ablation.py new file mode 100644 index 0000000..c61deac --- /dev/null +++ b/runs/frontier-qwen30-vllm020-profile-v1/plot_mock_ablation.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +"""Render the preregistered profile-ablation figure with schematic data.""" + +from pathlib import Path + +import matplotlib.pyplot as plt + + +CONFIGS = [f"TP{tp}\nMNS{mns}" for tp in (1, 2, 4) for mns in (8, 16, 32, 64)] + +# Schematic only. These values are deliberately not derived from experiment data. +MOCK = { + "Real (mock)": [0.42, 0.50, 0.58, 0.57, 0.60, 0.76, 1.00, 0.99, 0.55, 0.70, 0.74, 0.73], + "Old profile-only (mock)": [0.38, 0.44, 0.51, 0.50, 0.47, 0.58, 0.70, 0.69, 0.66, 0.84, 0.95, 0.96], + "New P-020 profile-only (H1 mock)": [0.40, 0.49, 0.57, 0.56, 0.58, 0.74, 0.98, 0.97, 0.53, 0.69, 0.75, 0.74], + "Per-TP calibrated upper bound (mock)": [0.41, 0.50, 0.58, 0.57, 0.59, 0.75, 0.99, 0.98, 0.54, 0.69, 0.75, 0.74], +} + + +def main() -> None: + output = Path(__file__).with_name("mock-profile-ablation.png") + fig, ax = plt.subplots(figsize=(13.5, 5.8), constrained_layout=True) + x = list(range(len(CONFIGS))) + styles = ["o-", "s--", "^-", "D:"] + for (label, values), style in zip(MOCK.items(), styles, strict=True): + ax.plot(x, values, style, linewidth=2, markersize=5, label=label) + ax.axvline(3.5, color="0.75", linewidth=1) + ax.axvline(7.5, color="0.75", linewidth=1) + ax.set_xticks(x, CONFIGS) + ax.set_ylabel("Normalized SLO-feasible throughput (schematic)") + ax.set_xlabel("Configuration") + ax.set_ylim(0.3, 1.08) + ax.grid(axis="y", alpha=0.25) + ax.legend(ncol=2, frameon=False, loc="upper left") + ax.set_title("MOCK / SCHEMATIC — expected discriminative result, not experiment data") + fig.savefig(output, dpi=180) + print(output) + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_flashattn.py b/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_flashattn.py new file mode 100644 index 0000000..bfba530 --- /dev/null +++ b/runs/frontier-qwen30-vllm020-profile-v1/profile_vllm020_flashattn.py @@ -0,0 +1,190 @@ +#!/usr/bin/env python3 +"""Profile the exact vLLM 0.20 FlashAttention backend at TP-local shapes. + +This deliberately uses vLLM's own v0.20.0 attention benchmark runner instead +of Frontier's FlashInfer-only attention wrapper. The output is raw evidence; +projection into Frontier's split attention CSV schema is a separate step. +""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +import types +from pathlib import Path + +import torch +import vllm + + +VLLM_VERSION = "0.20.0" +VLLM_COMMIT = "88d34c6409e9fb3c7b8ca0c04756f061d2099eb1" + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--vllm-source", type=Path, required=True) + parser.add_argument("--model", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--tp", type=int, choices=(1, 2, 4), nargs="+", default=[1, 2, 4]) + parser.add_argument( + "--batch-specs", + nargs="+", + default=["q128", "4q1s128", "q128_4q1s128"], + ) + parser.add_argument("--warmup-iters", type=int, default=3) + parser.add_argument("--repeats", type=int, default=5) + parser.add_argument("--device", default="cuda:0") + return parser.parse_args() + + +def git_head(repo: Path) -> str: + return subprocess.check_output( + ["git", "-C", str(repo), "rev-parse", "HEAD"], text=True + ).strip() + + +def main() -> None: + args = parse_args() + if vllm.__version__ != VLLM_VERSION: + raise SystemExit(f"expected vLLM {VLLM_VERSION}, got {vllm.__version__}") + source_head = git_head(args.vllm_source) + if source_head != VLLM_COMMIT: + raise SystemExit(f"expected vLLM source {VLLM_COMMIT}, got {source_head}") + if not args.model.joinpath("config.json").is_file(): + raise SystemExit(f"missing model config: {args.model / 'config.json'}") + + bench_dir = args.vllm_source / "benchmarks" / "attention_benchmarks" + sys.path.insert(0, str(bench_dir)) + import runner # type: ignore[import-not-found] # noqa: PLC0415 + from common import BenchmarkConfig # type: ignore[import-not-found] # noqa: PLC0415 + from vllm.config import ( # noqa: PLC0415 + CacheConfig, + CompilationConfig, + DeviceConfig, + LoadConfig, + ModelConfig, + ParallelConfig, + SchedulerConfig, + VllmConfig, + ) + from vllm.v1.worker.workspace import init_workspace_manager # noqa: PLC0415 + + def create_vllm_config(config: BenchmarkConfig, max_num_blocks: int) -> VllmConfig: + model_config = ModelConfig( + model=str(args.model), + tokenizer=str(args.model), + trust_remote_code=False, + dtype="bfloat16", + seed=0, + max_model_len=40960, + ) + cache_config = CacheConfig(block_size=config.block_size, cache_dtype="auto") + cache_config.num_gpu_blocks = max_num_blocks + cache_config.num_cpu_blocks = 0 + parallel_config = ParallelConfig(tensor_parallel_size=1) + scheduler_config = SchedulerConfig( + max_num_seqs=256, + max_num_batched_tokens=8192, + max_model_len=40960, + is_encoder_decoder=False, + enable_chunked_prefill=True, + ) + model_config.get_num_layers = types.MethodType( + lambda self: config.num_layers, model_config + ) + model_config.get_sliding_window_for_layer = types.MethodType( + lambda self, i: None, model_config + ) + model_config.get_logits_soft_cap_for_layer = types.MethodType( + lambda self, i: 0.0, model_config + ) + model_config.get_sm_scale_for_layer = types.MethodType( + lambda self, i: 1.0 / config.head_dim**0.5, model_config + ) + model_config.get_num_attention_heads = types.MethodType( + lambda self, parallel_config=None: config.num_q_heads, model_config + ) + model_config.get_num_kv_heads = types.MethodType( + lambda self, parallel_config=None: config.num_kv_heads, model_config + ) + model_config.get_head_size = types.MethodType( + lambda self: config.head_dim, model_config + ) + model_config.get_sliding_window = types.MethodType( + lambda self: None, model_config + ) + return VllmConfig( + model_config=model_config, + cache_config=cache_config, + parallel_config=parallel_config, + scheduler_config=scheduler_config, + device_config=DeviceConfig(), + load_config=LoadConfig(), + compilation_config=CompilationConfig(), + ) + + runner._create_vllm_config = create_vllm_config + init_workspace_manager(args.device) + + rows: list[dict[str, object]] = [] + for tp in args.tp: + for batch_spec in args.batch_specs: + config = BenchmarkConfig( + backend="FLASH_ATTN", + batch_spec=batch_spec, + num_layers=1, + head_dim=128, + num_q_heads=32 // tp, + num_kv_heads=4 // tp, + block_size=16, + device=args.device, + dtype=torch.bfloat16, + repeats=args.repeats, + warmup_iters=args.warmup_iters, + profile_memory=True, + kv_cache_dtype="auto", + use_cuda_graphs=False, + ) + result = runner.run_attention_benchmark(config) + row = result.to_dict() + row["tensor_parallel_size"] = tp + rows.append(row) + print( + json.dumps( + { + "tp": tp, + "batch_spec": batch_spec, + "mean_time_s": result.mean_time, + "error": result.error, + }, + sort_keys=True, + ), + flush=True, + ) + if not result.success: + raise SystemExit(f"attention profile failed: {row}") + + payload = { + "schema_version": "qwen30_vllm020_flashattn_raw.v1", + "environment": { + "vllm_version": vllm.__version__, + "vllm_source_commit": source_head, + "torch_version": torch.__version__, + "torch_cuda": torch.version.cuda, + "gpu": torch.cuda.get_device_name(torch.device(args.device)), + "model": str(args.model), + "dtype": "bfloat16", + "attention_backend": "FLASH_ATTN", + "block_size": 16, + }, + "rows": rows, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-qwen30-vllm020-profile-v1/run_flashattn_smoke.sh b/runs/frontier-qwen30-vllm020-profile-v1/run_flashattn_smoke.sh new file mode 100644 index 0000000..04318bb --- /dev/null +++ b/runs/frontier-qwen30-vllm020-profile-v1/run_flashattn_smoke.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash + +set -euo pipefail + +OUTPUT_ROOT="${OUTPUT_ROOT:-$(pwd)/artifacts/flashattn-smoke}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}" +VLLM_SOURCE="${VLLM_SOURCE:-/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build}" +MODEL="${MODEL:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}" +LOG_DIR="${OUTPUT_ROOT}/logs" +PROVENANCE_DIR="${OUTPUT_ROOT}/provenance" + +mkdir -p "${LOG_DIR}" "${PROVENANCE_DIR}" +exec > >(tee -a "${LOG_DIR}/smoke.log") 2>&1 + +if [[ -z "${CUDA_VISIBLE_DEVICES:-}" ]]; then + echo "ERROR: CUDA_VISIBLE_DEVICES must contain the fleet-allocated GPU" >&2 + exit 1 +fi +IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES}" +if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then + echo "ERROR: expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2 + exit 1 +fi + +echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} model=${MODEL} runtime=vLLM-0.20.0+cu129 operator=FlashAttention3 tp_local_shapes=1,2,4 specs=q128,4q1s128,q128_4q1s128 dtype=BF16 block=16 output=${OUTPUT_ROOT} expected_wall=3-8m hard_wall=900s hard_gpu_cap=0.15_H20h" +date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ" +nvidia-smi --query-gpu=index,name,driver_version,memory.used,utilization.gpu --format=csv,noheader + +test -x "${VENV_ROOT}/bin/python" +test -f "${VLLM_SOURCE}/benchmarks/attention_benchmarks/runner.py" +test -f "${MODEL}/config.json" +test -f profile_vllm020_flashattn.py + +git rev-parse HEAD > "${PROVENANCE_DIR}/aituner.commit" +git -C "${VLLM_SOURCE}" rev-parse HEAD > "${PROVENANCE_DIR}/vllm-source.commit" +sha256sum profile_vllm020_flashattn.py run_flashattn_smoke.sh \ + > "${PROVENANCE_DIR}/source.sha256" +uv pip freeze --python "${VENV_ROOT}/bin/python" \ + > "${PROVENANCE_DIR}/pip-freeze.txt" +nvidia-smi --query-gpu=index,uuid,name,driver_version,memory.total \ + --format=csv,noheader > "${PROVENANCE_DIR}/gpus.csv" + +timeout --signal=TERM --kill-after=30s 780 \ + "${VENV_ROOT}/bin/python" profile_vllm020_flashattn.py \ + --vllm-source "${VLLM_SOURCE}" \ + --model "${MODEL}" \ + --output "${OUTPUT_ROOT}/raw/flashattn-smoke.json" \ + --tp 1 2 4 \ + --batch-specs q128 4q1s128 q128_4q1s128 \ + --warmup-iters 3 \ + --repeats 5 + +test -s "${OUTPUT_ROOT}/raw/flashattn-smoke.json" +sha256sum "${OUTPUT_ROOT}/raw/flashattn-smoke.json" \ + "${PROVENANCE_DIR}"/* > "${OUTPUT_ROOT}/artifacts.sha256" +nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader +date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ" +echo "FLASHATTN_SMOKE_COMPLETE"