aituner/runs/simulator-tuning-latency-matrix-v0/experiment-card.md

EXP-SIM-TUNING-LATENCY-MATRIX-V0：现有 simulator 能否选择真机低延迟 serving config？

状态： protocol 已冻结；Qwen3.6 release-runtime topology smoke 已通过（2026-07-17）。 没有 Frontier Qwen3.6 contract/profile、trace replay 或 latency-selection 结果；本 card 不授权 600-s matrix launch。

Claim 与决策

• 研究问题： 对冻结的 (H20 hardware, model, serving engine, request trace, config candidate surface)，现有 simulator 能否为一个明确的 latency objective 选出真机最优、或 统计上等价的 serving-engine config？
• Parent claim： “simulator 已经解决 config tuning”只有在完整 candidate surface 上产生 正确的真机 latency selection 时才成立；绝对 latency error 或单个吞吐/SLO 结果均不充分。
• Hypothesis： simulator 在 fixed-shape / prefill-only 条件下可能足够，但在 decode、真实 arrival 与 prefix/KV state 同时存在时会因 state-conditioned residual 或 coverage failure 而 失去 selection fidelity。
• Competing hypothesis： 经冻结、非 evaluation 的 operator/collective calibration 后， simulator 能跨 non-MoE hybrid/MoE 与四种 workload condition 稳定选出真机低延迟 config。
• 判定： 每个 model × workload × metric 是独立 tuning task，且 config 越低 latency 越好。 只有所有预注册且 simulator 声称支持的 task 均通过下面的 coverage 与 selection gates，才能 写“该 simulator 已解决其声明范围内的 tuning”；任一失败 cell 是明确 boundary，不被平均值 掩盖。

固定系统与模型覆盖

角色	状态	运行时契约	当前可复用 artifact / 含义
MoE	可开始准备	Qwen3-30B-A3B，BF16，community vLLM 0.20.0，dash0 H20；Frontier/Vidur-class fork	frontier-phase-factorial-v0 有 ISL=2048, OSL=1, r=4 的 request-level real/sim pilot；frontier-fidelity-envelope-v1 有 fixed-PD 与 exact Trace-PD materializers。历史 pilot 不是本 card 的最终 600-s evaluation。
Non-MoE hybrid (not all-attention dense)	TP1/2/4 smoke passed; profile pending	Qwen/Qwen3.6-27B@cea40373b9214dd387123e68841890af30dcd469，community release vLLM 0.20.2 / Torch 2.11.0 / Transformers 5.14.1，dash0 H20；Frontier profile 必须在此 exact stack 采集	Exact release serves text-only 2048→1 at TP1/2/4. vLLM resolves a 784-token attention page for its GDN/Mamba state, so requested block=16 is not the physical engine state. Qwen3.6 trace, Frontier hybrid model contract, and profile remain missing.

所有公开结果固定在单机 dash0（8×H20）；每个 model 使用相同 engine build 的 real server 与 simulator contract。Qwen3.6 bring-up 已确认真实 runtime/page legality；其余最低成本 gate 是冻结 trace/tokenizer 与完成 Frontier hybrid profile-schema audit。若不能做到，论文结论 必须限于 MoE，不得写跨模型 generalization。

Qwen3.6 prerequisite audit (runtime + smoke, 2026-07-17)

Prerequisite	Evidence	Verdict	Decision
Exact model snapshot	Immutable ModelScope snapshot Qwen/Qwen3.6-27B@cea40373b9214dd387123e68841890af30dcd469 is staged at /home/admin/cpfs/wjh/models/Qwen/Qwen3.6-27B; its sorted-file inventory digest is 031cea103726cbf27732b6971e898e210aee77f0c32e4bc703c947e0372349c4.	PASS	Bind every Qwen3.6 trace, real server, and profile manifest to this revision and inventory.
Community vLLM runtime	Clean environment /home/admin/cpfs/wjh/venvs/qwen36-vllm-0.20.2 contains official wheel vLLM 0.20.2, Torch 2.11.0, and Transformers 5.14.1; pip check passes and CPU-only get_config(..., trust_remote_code=false) resolves Qwen3_5ForConditionalGeneration.	PASS (CPU)	Use exactly this release environment for real serving and profile collection.
Topology compatibility	Fresh TP1/2/4 servers each completed a text-only 2048-prompt/1-completion request with exact usage; all GPUs were released after cleanup. Every topology resolved the attention page from requested 16 to 784 tokens for Mamba/GDN state.	PASS (GPU smoke)	This is an engine-configuration legality result, not a latency rank or simulator result. Profile/simulator input must expose the 784-token state/page contract.
Qwen3.6 Frontier model/profile	No local or remote Qwen3.6-named Frontier profile/model directory was found	MISSING	New dense profile root is required; Qwen30/MoE profile CSVs are forbidden inputs.
Old dense artifacts	Existing Qwen3.5-27B trace is internal-vLLM/hybrid/speculative	N/A as baseline	Keep only as historical context; never feed it to this matrix or profile calibration.

This audit is not a failed simulator result. The topology gate is closed, but the model contract, profile root, and frozen trace are still reproducibility blockers: the Qwen3.6 half is NOT RUN, not FAIL. For a paper claim about a conventional all-attention dense transformer, Qwen3.6 is not admissible evidence; it is only a non-MoE hybrid case.

Workload matrix

每个 model 都运行四个固定 trace。P 是 PD disaggregation 的 P-node workload，不是缺失 decode 的坏数据；它的 OSL=1 是设计变量。固定 cases 与 trace cases都使用 open-loop replay， 600 秒 observation window；Fixed 的 request count 为 ceil(QPS × 600)。

ID	trace contract	output contract	prefix/KV contract	可评 latency objectives
Fixed-P	固定 ISL=2048、uniform fixed QPS=4 req/s，token vector deterministic	OSL=1	no shared logical prefix；prefix cache off	mean/p90 TTFT；mean/p90 E2E；TPOT=N/A
Fixed-PD	固定 ISL=2048、uniform fixed QPS=4 req/s	OSL=128，min=max=128，ignore_eos=true	no shared logical prefix；prefix cache off	mean/p90 TTFT、TPOT、E2E
Trace-P	held-out production trace 的逐 request prompt-token vector、arrival time、request order、session root 与 prefix block relation 原样保留	只将 replay output 强制为 OSL=1	source prefix equivalence、warm/cold state、cache policy 原样保留	mean/p90 TTFT；mean/p90 E2E；TPOT=N/A
Trace-PD	held-out production trace 的逐 request input/output lengths、arrival time、order、session root 与 prefix block relation 原样保留	原始 min=max=source OSL，ignore_eos=true	source prefix equivalence、warm/cold state、cache policy 原样保留	mean/p90 TTFT、TPOT、E2E

Trace-P 应明确称为 trace-derived prefill-only：它忠实保留输入、到达、会话与 cache state，但不忠实保留原始 output-length distribution；不能被描述成完整 trace-faithful replay。 Trace-PD 才是完整 input/output/arrival/prefix-KV joint replay。

Trace、correctness 与 cache 的不可变契约

• 每个 trace manifest 列出：request ID、arrival timestamp、input/output token counts、session ID、runtime block identities、trace vector SHA-256、source trace SHA-256、requested and resolved physical page/block size、cache policy、initial cache state、warmup trace digest 与 model tokenizer/version。
• 当 simulator 要模拟 vLLM prefix caching 时，block_hash_ids 只能描述可复用的完整 runtime blocks：adapter 必须导出 floor(ISL / block_size) 个 prompt identities，不能将 ceil 得到的 partial final block 当作 cache hit。该 simulator-facing projection 有独立 digest；private real replay 仍保留完整 prompt 与原始 token vector。
• Trace-P 产生单独 manifest：source OSL 和 replay OSL 都写入 digest；只允许 source OSL → 1 这一个变换。Trace-PD、real client 与 simulator CSV 的 row-vector digest 必须完全 相同。
• 真实端逐 request 核验 prompt/completion usage 与 manifest 一致；simulator 核验 request ID、 input/output shapes、完成数与 trace digest。一项不符即该 cell 的 metric 无效。
• 每个 (config, workload, trial) fresh server；server warmup 用独立 manifest，warmup request 不进入指标，也不得改变 evaluation cache 初态。prefix-on trace 以同一明示 initial cache state重放，不能用随机前史替代。

Candidate-surface 与 baseline 规则

1. 对每个 model 冻结 C = TP∈{1,2,4} × MNS∈{8,16,32,64}（12 个 configs），并固定 DP=PP=EP=1、MBT=8192、chunked prefill on。每个 candidate 还必须冻结 真实 engine resolved page/block and state-cache contract；requested CLI --block-size 16 不能替代它。 Qwen3.6 smoke 在所有 TP 都将其改写为 784 tokens。任何不被某 model/engine 静态支持的 config 要在运行前从该 model 的 C 整体删除，并记录理由；不能看结果后删除。
2. C_real-valid 是 server launch 成功、所有三个 fresh trials 全部请求完成且 usage/hash 合法的 configs。真机 OOM、timeout 或 request error 保留为 real infeasible/safety outcome，而非给它 赋无限 latency。
3. 对每个 simulator，selection coverage 必须涵盖全部 C_real-valid。crash、scheduler stall、 trace mismatch、缺 request metrics 都是 simulator coverage failure，不可当成高 latency 或 infeasible 来产生排序。只有完整 coverage 才能形成 tuning verdict。
4. Frontier、官方 Vidur 与 APEX 分别独立报告版本、patch、profile 与 capability surface。APEX 若只支持 DP/PP/TP parallel plan，则它只评 parallel-plan tuning 交集；不得用该子问题结果 宣称解决 MNS/MBT/cache 等完整 engine tuning。

Metrics、统计与 selection score

每个完整 request 使用同一 client-boundary 定义：

• TTFT：客户端发出请求至收到第一个 generated token；包含 serving queue。
• E2E：客户端发出请求至完成 [DONE]。
• TPOT：(E2E - TTFT) / (completion_tokens - 1)；只对 completion_tokens > 1 的请求定义。
• 每 trial 的 mean 是 request-level arithmetic mean；p90 是 nearest-rank ceil(0.90 × N)。P-only 的 TPOT 是 JSON null / table N/A，绝不写 0。

同一 config 的 real 运行三次 fresh-server trial，随机化 config order；每个 trial 都对整个 trace 计算六项或四项指标。real ranking point estimate 是三个 trial metric 的 median，且报告 pooled request statistic、每-trial statistic 和 95% hierarchical bootstrap interval（先重采样 trial，再在 trial 内重采样 request）。simulator 的 seed、determinism 和 repeat policy同样记录。

对每一个 objective，输出：absolute real/sim surface、real bootstrap top set、simulator exact top set、top-set hit、sim-chosen config 的 best/worst tie-break regret、Kendall τ-b、non-tied pair direction accuracy、coverage 与 error/infeasible table。Latency regret 定义为 (real_metric(sim_choice) - real_metric(real_best)) / real_metric(real_best)。

预注册 decision gates

Gate	Pass	Failure handling
Trace/metric validity	100% planned requests have matching usage/shape and finite applicable metrics	cell invalid；诊断 harness，不作 performance conclusion
Simulator coverage	100% of C_real-valid output legal request metrics	FAIL: coverage；不把 missing cell 排成最差
Selection	simulator top set 与 bootstrap real top set 有交集	FAIL: selection
Risk	simulator top-set 的 worst real tie-break regret ≤5%	FAIL: regret，即使 best tie-break 碰巧为 0
Global rank diagnostic	τ-b ≥0.8；若没有非并列 pair 则写 N/A，不当通过	NEEDS EVIDENCE / FAIL 与 top-set gate 分开报告

这里不使用 SLO、capacity 或 throughput 作 selection objective；它们最多作为安全/诊断附录指标。 P-only 共 4 tasks，PD 共 6 tasks，所以完整 non-MoE-hybrid/MoE × 4 workloads matrix 共有 40 个 独立 latency-selection tasks。

Calibration 与 information boundary

• 允许 simulator 使用同硬件/同 model/同 runtime 的 offline operator、collective 或 memory profiles，但 profile command、shape grid、cost 和 hash 必须在 evaluation 前冻结。
• calibration 不得读取 evaluation 的 real request latency、top config 或 E2E scalar；禁止 per-TP 或 per-config serving E2E scale。若有 trace-driven calibration，必须只使用按 session root 切分 的 calibration fold；Trace-P/PD 使用完全不相交的 held-out root sessions。
• Fixed traces 与 trace evaluation 都不能同时作为 calibration workload。任何 required simulator patch 在查看该 cell real result 前冻结；patched baseline 与 original release 分开作图。

Minimal Qwen3.6-27B bring-up and profile plan

This is the only permitted path to add the dense case. It intentionally separates cheap provenance work from GPU work, and it does not authorize either stage by itself.

1. Preflight and topology smoke (complete): the snapshot/runtime are locked; fresh TP1/2/4 servers completed exact-usage text requests. The actual vLLM state/page setting is 784 tokens, not the requested 16. This establishes the profile input contract only.
2. Hybrid Frontier contract (not run): verify Frontier can represent Gated DeltaNet/linear-state, gated attention, FFN, true mixed execution, TP collectives, the 784-token page rule and capacity, without substituting Qwen3.5/Qwen30 fields.
3. Minimal profile set (GPU, separately approved): on the exact Qwen3.6/community-vLLM/dash0 stack, collect per-TP Gated-DeltaNet, gated-attention prefill/decode, state/KV and TP2/TP4 all-reduce rows. The shape grid must cover the pre-registered Fixed-P/PD and held-out Trace-P/PD ranges, but must not contain evaluation E2E latency or a fitted time scale. Freeze CSV/schema/command hashes before running any evaluation trace.
4. Profile acceptance (CPU): verify model/runtime identity in every profile manifest; require requested shape coverage, finite values, and no fallback to a MoE or Qwen3.5 profile. Only then may a parent-approved real/simulator latency surface be scheduled.

预期产物、复现与成本

• Figure prototype： latency-selection-matrix-schematic.svg （明确标注 SCHEMATIC — NOT MEASURED DATA）。最终图替换为每个 task 的 top-set hit / regret / coverage heatmap，并附 per-config latency surfaces。
• Raw layout： artifacts/<model>/<workload>/<real|simulator>/<config>/trial-*/；每个目录含 command、environment/machine fingerprint、server flags、trace/profile hash、raw request records、 stdout/stderr、duration 与 SHA-256 inventory。
• Existing reuse audit： MoE F0 可检验 P-only raw metric reader；现有 F2 可检验 fixed-PD trace materialization；T1 可检验 Trace-PD digest and simulator coverage reporting。它们并不构成 这个 600-s no-SLO matrix 的完整 real surface。Qwen3.6 requires a new same-stack trace, model contract and profile root; old Qwen3.5 internal-vLLM artifacts are excluded.
• Cost estimate, not authorization： 单 config×trial 假定 10 min replay + 5 min fresh-server startup/warmup = 15 min。完整 12-config sweep 的 GPU allocation 为 4 × (TP1 + TP2 + TP4) = 28 GPU; 因而每个 model×workload 的三 trial 约 28 × 0.25 × 3 = 21 H20-GPUh。8 cells 的 real serving core is 168 H20-GPUh. The Qwen3.6 bring-up/profile set adds at most 6 H20-GPUh (three topology smokes plus dense/attention/ collective rows), giving a 174 H20-GPUh core estimate and a 190–215 H20-GPUh planning envelope with retries. Every launch must recompute this from the staged trace duration and actual server preflight; this paragraph is not authorization.

Required remote launch echo (not executed)

NOT EXECUTED — awaiting explicit approval after Qwen3.6 model/runtime/profile gates
host=dash0; remote_root=/home/admin/cpfs/wjh/aituner/aituner
models=non-moe-hybrid:Qwen/Qwen3.6-27B@<FROZEN_SNAPSHOT>,moe:Qwen3-30B-A3B-BF16
engine=community-open-source-vLLM@<FROZEN_SOURCE_OR_WHEEL>; simulator=Frontier/Vidur-class,<official baselines if installed>
workloads=Fixed-P(ISL2048,OSL1,QPS4,600s),Fixed-PD(ISL2048,OSL128,QPS4,600s),
          Trace-P(heldout exact input/arrival/prefix,OSL1),Trace-PD(heldout exact joint trace)
surface=TP{1,2,4}×MNS{8,16,32,64}; trials=3; fresh_server=true
real_paths=runs/simulator-tuning-latency-matrix-v0/artifacts/<model>/<workload>/real/
expected_cost=174 H20-GPUh core; 190–215 H20-GPUh planning envelope; expected_wall=multiple queued hours
gate=Qwen3.6 snapshot + community-vLLM fingerprint + resolved-784 hybrid Frontier profile completeness + trace/profile hashes + user/parent approval

Expected outcomes and stop/pivot rules

• If a simulator passes every supported task, stop diagnosing that simulator and look for a new problem outside its evaluated scope (e.g., unsupported engine knobs or alignment cost).
• If it fails coverage, repair/liveness is a separate intervention; original capability result remains frozen. First rerun simulator only; do not spend GPU-hours until it can produce a complete surface.
• If it covers but selects incorrectly, retain the counterexample and run one minimal discriminative mechanism experiment (scheduler state, batch composition, graph/kernel family, or MoE/collective), changing one factor at a time.
• If Qwen3.6 cannot be profiled on its exact community-vLLM stack, stop the non-MoE-hybrid half rather than treating any internal-vLLM or Qwen3.5 result as a substitute. If the headline requires a pure all-attention dense-transformer case, add that model explicitly rather than relabelling Qwen3.6.

Benchmark-design audit

Risk	Pre-registered control
SLO/capacity result misrepresented as tuning	selection uses only per-request mean/p90 TTFT, TPOT, E2E on the same trace
P-only judged by nonexistent TPOT	OSL=1 and TPOT=N/A are explicit workload/metric contracts
Trace simplification hidden	Trace-P is labelled output-normalized; only Trace-PD claims full joint replay
calibration equals evaluation	microprofile/trace calibration folds are disjoint; no E2E correction on evaluation
simulator crash treated as a bad config	coverage gate separates missing metrics from latency ranking
cross-stack or wrong-architecture reuse	only Qwen3.6-27B hybrid profiles collected on its exact community-vLLM stack are admissible; old Qwen3.5/internal-vLLM or Qwen30/MoE artifacts are excluded
selected winning cells only	all 8 cells and all legal configs are frozen before real results
tail/noise omitted	p90, three fresh trials, per-trial values and bootstrap top sets are mandatory