229 lines
18 KiB
Markdown
229 lines
18 KiB
Markdown
# EXP-SIM-TUNING-LATENCY-MATRIX-V0：现有 simulator 能否选择真机低延迟 serving config？
|
||
|
||
> **状态：** protocol 已冻结；Qwen3.6 release-runtime topology smoke 已通过（2026-07-17）。
|
||
> 没有 Frontier Qwen3.6 contract/profile、trace replay 或 latency-selection 结果；本 card
|
||
> 不授权 600-s matrix launch。
|
||
|
||
## Claim 与决策
|
||
|
||
- **研究问题：** 对冻结的 `(H20 hardware, model, serving engine, request trace, config
|
||
candidate surface)`，现有 simulator 能否为一个明确的 latency objective 选出真机最优、或
|
||
统计上等价的 serving-engine config？
|
||
- **Parent claim：** “simulator 已经解决 config tuning”只有在完整 candidate surface 上产生
|
||
正确的真机 latency selection 时才成立；绝对 latency error 或单个吞吐/SLO 结果均不充分。
|
||
- **Hypothesis：** simulator 在 fixed-shape / prefill-only 条件下可能足够，但在 decode、真实
|
||
arrival 与 prefix/KV state 同时存在时会因 state-conditioned residual 或 coverage failure 而
|
||
失去 selection fidelity。
|
||
- **Competing hypothesis：** 经冻结、非 evaluation 的 operator/collective calibration 后，
|
||
simulator 能跨 non-MoE hybrid/MoE 与四种 workload condition 稳定选出真机低延迟 config。
|
||
- **判定：** 每个 `model × workload × metric` 是独立 tuning task，且 config 越低 latency 越好。
|
||
只有所有预注册且 simulator 声称支持的 task 均通过下面的 coverage 与 selection gates，才能
|
||
写“该 simulator 已解决其声明范围内的 tuning”；任一失败 cell 是明确 boundary，不被平均值
|
||
掩盖。
|
||
|
||
## 固定系统与模型覆盖
|
||
|
||
| 角色 | 状态 | 运行时契约 | 当前可复用 artifact / 含义 |
|
||
|---|---|---|---|
|
||
| MoE | **可开始准备** | Qwen3-30B-A3B，BF16，community vLLM 0.20.0，dash0 H20；Frontier/Vidur-class fork | `frontier-phase-factorial-v0` 有 `ISL=2048, OSL=1, r=4` 的 request-level real/sim pilot；`frontier-fidelity-envelope-v1` 有 fixed-PD 与 exact Trace-PD materializers。历史 pilot 不是本 card 的最终 600-s evaluation。 |
|
||
| Non-MoE hybrid (not all-attention dense) | **TP1/2/4 smoke passed; profile pending** | `Qwen/Qwen3.6-27B@cea40373b9214dd387123e68841890af30dcd469`，community release `vLLM 0.20.2` / Torch `2.11.0` / Transformers `5.14.1`，dash0 H20；Frontier profile 必须在此 exact stack 采集 | Exact release serves text-only 2048→1 at TP1/2/4. vLLM resolves a 784-token attention page for its GDN/Mamba state, so requested block=16 is not the physical engine state. Qwen3.6 trace, Frontier hybrid model contract, and profile remain missing. |
|
||
|
||
所有公开结果固定在单机 `dash0`（8×H20）；每个 model 使用相同 engine build 的 real
|
||
server 与 simulator contract。Qwen3.6 bring-up 已确认真实 runtime/page legality；其余最低成本
|
||
gate 是冻结 trace/tokenizer 与完成 Frontier hybrid profile-schema audit。若不能做到，论文结论
|
||
必须限于 MoE，不得写跨模型 generalization。
|
||
|
||
### Qwen3.6 prerequisite audit (runtime + smoke, 2026-07-17)
|
||
|
||
| Prerequisite | Evidence | Verdict | Decision |
|
||
|---|---|---|---|
|
||
| Exact model snapshot | Immutable ModelScope snapshot `Qwen/Qwen3.6-27B@cea40373b9214dd387123e68841890af30dcd469` is staged at `/home/admin/cpfs/wjh/models/Qwen/Qwen3.6-27B`; its sorted-file inventory digest is `031cea103726cbf27732b6971e898e210aee77f0c32e4bc703c947e0372349c4`. | **PASS** | Bind every Qwen3.6 trace, real server, and profile manifest to this revision and inventory. |
|
||
| Community vLLM runtime | Clean environment `/home/admin/cpfs/wjh/venvs/qwen36-vllm-0.20.2` contains official wheel `vLLM 0.20.2`, Torch `2.11.0`, and Transformers `5.14.1`; `pip check` passes and CPU-only `get_config(..., trust_remote_code=false)` resolves `Qwen3_5ForConditionalGeneration`. | **PASS (CPU)** | Use exactly this release environment for real serving and profile collection. |
|
||
| Topology compatibility | Fresh TP1/2/4 servers each completed a text-only 2048-prompt/1-completion request with exact usage; all GPUs were released after cleanup. Every topology resolved the attention page from requested 16 to 784 tokens for Mamba/GDN state. | **PASS (GPU smoke)** | This is an engine-configuration legality result, not a latency rank or simulator result. Profile/simulator input must expose the 784-token state/page contract. |
|
||
| Qwen3.6 Frontier model/profile | No local or remote Qwen3.6-named Frontier profile/model directory was found | **MISSING** | New dense profile root is required; Qwen30/MoE profile CSVs are forbidden inputs. |
|
||
| Old dense artifacts | Existing Qwen3.5-27B trace is internal-vLLM/hybrid/speculative | **N/A as baseline** | Keep only as historical context; never feed it to this matrix or profile calibration. |
|
||
|
||
This audit is not a failed simulator result. The topology gate is closed, but the model contract,
|
||
profile root, and frozen trace are still reproducibility blockers: the Qwen3.6 half is `NOT RUN`,
|
||
not `FAIL`. For a paper claim about a conventional all-attention dense transformer, Qwen3.6 is not
|
||
admissible evidence; it is only a non-MoE hybrid case.
|
||
|
||
## Workload matrix
|
||
|
||
每个 model 都运行四个固定 trace。`P` 是 PD disaggregation 的 P-node workload，不是缺失
|
||
decode 的坏数据；它的 `OSL=1` 是设计变量。固定 cases 与 trace cases都使用 open-loop replay，
|
||
600 秒 observation window；`Fixed` 的 request count 为 `ceil(QPS × 600)`。
|
||
|
||
| ID | trace contract | output contract | prefix/KV contract | 可评 latency objectives |
|
||
|---|---|---|---|---|
|
||
| Fixed-P | 固定 `ISL=2048`、uniform fixed QPS=`4 req/s`，token vector deterministic | `OSL=1` | no shared logical prefix；prefix cache off | mean/p90 TTFT；mean/p90 E2E；TPOT=`N/A` |
|
||
| Fixed-PD | 固定 `ISL=2048`、uniform fixed QPS=`4 req/s` | `OSL=128`，`min=max=128`，`ignore_eos=true` | no shared logical prefix；prefix cache off | mean/p90 TTFT、TPOT、E2E |
|
||
| Trace-P | held-out production trace 的逐 request prompt-token vector、arrival time、request order、session root 与 prefix block relation 原样保留 | **只将 replay output 强制为 `OSL=1`** | source prefix equivalence、warm/cold state、cache policy 原样保留 | mean/p90 TTFT；mean/p90 E2E；TPOT=`N/A` |
|
||
| Trace-PD | held-out production trace 的逐 request input/output lengths、arrival time、order、session root 与 prefix block relation 原样保留 | 原始 `min=max=source OSL`，`ignore_eos=true` | source prefix equivalence、warm/cold state、cache policy 原样保留 | mean/p90 TTFT、TPOT、E2E |
|
||
|
||
`Trace-P` 应明确称为 **trace-derived prefill-only**：它忠实保留输入、到达、会话与 cache
|
||
state，但不忠实保留原始 output-length distribution；不能被描述成完整 trace-faithful replay。
|
||
`Trace-PD` 才是完整 input/output/arrival/prefix-KV joint replay。
|
||
|
||
### Trace、correctness 与 cache 的不可变契约
|
||
|
||
- 每个 trace manifest 列出：request ID、arrival timestamp、input/output token counts、session
|
||
ID、runtime block identities、trace vector SHA-256、source trace SHA-256、requested and
|
||
resolved physical page/block size、cache policy、initial cache state、warmup trace digest 与
|
||
model tokenizer/version。
|
||
- 当 simulator 要模拟 vLLM prefix caching 时，`block_hash_ids` 只能描述可复用的**完整**
|
||
runtime blocks：adapter 必须导出 `floor(ISL / block_size)` 个 prompt identities，不能将
|
||
`ceil` 得到的 partial final block 当作 cache hit。该 simulator-facing projection 有独立
|
||
digest；private real replay 仍保留完整 prompt 与原始 token vector。
|
||
- `Trace-P` 产生单独 manifest：source OSL 和 replay OSL 都写入 digest；只允许 source
|
||
OSL → 1 这一个变换。`Trace-PD`、real client 与 simulator CSV 的 row-vector digest 必须完全
|
||
相同。
|
||
- 真实端逐 request 核验 prompt/completion usage 与 manifest 一致；simulator 核验 request ID、
|
||
input/output shapes、完成数与 trace digest。一项不符即该 cell 的 metric 无效。
|
||
- 每个 `(config, workload, trial)` fresh server；server warmup 用独立 manifest，warmup request
|
||
不进入指标，也不得改变 evaluation cache 初态。prefix-on trace 以同一明示 initial cache
|
||
state重放，不能用随机前史替代。
|
||
|
||
## Candidate-surface 与 baseline 规则
|
||
|
||
1. 对每个 model 冻结 `C = TP∈{1,2,4} × MNS∈{8,16,32,64}`（12 个 configs），并固定
|
||
`DP=PP=EP=1`、MBT=8192、chunked prefill on。每个 candidate 还必须冻结 **真实 engine
|
||
resolved** page/block and state-cache contract；requested CLI `--block-size 16` 不能替代它。
|
||
Qwen3.6 smoke 在所有 TP 都将其改写为 784 tokens。任何不被某 model/engine 静态支持的
|
||
config 要在运行前从该 model 的 `C` 整体删除，并记录理由；不能看结果后删除。
|
||
2. `C_real-valid` 是 server launch 成功、所有三个 fresh trials 全部请求完成且 usage/hash 合法的
|
||
configs。真机 OOM、timeout 或 request error 保留为 real infeasible/safety outcome，而非给它
|
||
赋无限 latency。
|
||
3. 对每个 simulator，selection coverage 必须涵盖全部 `C_real-valid`。crash、scheduler stall、
|
||
trace mismatch、缺 request metrics 都是 simulator **coverage failure**，不可当成高 latency 或
|
||
infeasible 来产生排序。只有完整 coverage 才能形成 tuning verdict。
|
||
4. Frontier、官方 Vidur 与 APEX 分别独立报告版本、patch、profile 与 capability surface。APEX
|
||
若只支持 DP/PP/TP parallel plan，则它只评 `parallel-plan tuning` 交集；不得用该子问题结果
|
||
宣称解决 MNS/MBT/cache 等完整 engine tuning。
|
||
|
||
## Metrics、统计与 selection score
|
||
|
||
每个完整 request 使用同一 client-boundary 定义：
|
||
|
||
- `TTFT`：客户端发出请求至收到第一个 generated token；包含 serving queue。
|
||
- `E2E`：客户端发出请求至完成 `[DONE]`。
|
||
- `TPOT`：`(E2E - TTFT) / (completion_tokens - 1)`；只对 `completion_tokens > 1` 的请求定义。
|
||
- 每 trial 的 `mean` 是 request-level arithmetic mean；`p90` 是 nearest-rank
|
||
`ceil(0.90 × N)`。P-only 的 TPOT 是 JSON `null` / table `N/A`，绝不写 0。
|
||
|
||
同一 config 的 real 运行三次 fresh-server trial，随机化 config order；每个 trial 都对整个
|
||
trace 计算六项或四项指标。real ranking point estimate 是三个 trial metric 的 median，且报告
|
||
pooled request statistic、每-trial statistic 和 95% hierarchical bootstrap interval（先重采样
|
||
trial，再在 trial 内重采样 request）。simulator 的 seed、determinism 和 repeat policy同样记录。
|
||
|
||
对每一个 objective，输出：absolute real/sim surface、real bootstrap top set、simulator exact
|
||
top set、top-set hit、sim-chosen config 的 best/worst tie-break regret、Kendall τ-b、non-tied
|
||
pair direction accuracy、coverage 与 error/infeasible table。Latency regret 定义为
|
||
`(real_metric(sim_choice) - real_metric(real_best)) / real_metric(real_best)`。
|
||
|
||
## 预注册 decision gates
|
||
|
||
| Gate | Pass | Failure handling |
|
||
|---|---|---|
|
||
| Trace/metric validity | 100% planned requests have matching usage/shape and finite applicable metrics | cell invalid；诊断 harness，不作 performance conclusion |
|
||
| Simulator coverage | 100% of `C_real-valid` output legal request metrics | `FAIL: coverage`；不把 missing cell 排成最差 |
|
||
| Selection | simulator top set 与 bootstrap real top set 有交集 | `FAIL: selection` |
|
||
| Risk | simulator top-set 的 **worst** real tie-break regret `≤5%` | `FAIL: regret`，即使 best tie-break 碰巧为 0 |
|
||
| Global rank diagnostic | τ-b `≥0.8`；若没有非并列 pair 则写 `N/A`，不当通过 | `NEEDS EVIDENCE` / `FAIL` 与 top-set gate 分开报告 |
|
||
|
||
这里不使用 SLO、capacity 或 throughput 作 selection objective；它们最多作为安全/诊断附录指标。
|
||
P-only 共 4 tasks，PD 共 6 tasks，所以完整 `non-MoE-hybrid/MoE × 4 workloads` matrix 共有 40 个
|
||
独立 latency-selection tasks。
|
||
|
||
## Calibration 与 information boundary
|
||
|
||
- 允许 simulator 使用同硬件/同 model/同 runtime 的 offline operator、collective 或 memory
|
||
profiles，但 profile command、shape grid、cost 和 hash 必须在 evaluation 前冻结。
|
||
- calibration 不得读取 evaluation 的 real request latency、top config 或 E2E scalar；禁止 per-TP
|
||
或 per-config serving E2E scale。若有 trace-driven calibration，必须只使用按 session root 切分
|
||
的 calibration fold；Trace-P/PD 使用完全不相交的 held-out root sessions。
|
||
- Fixed traces 与 trace evaluation 都不能同时作为 calibration workload。任何 required simulator
|
||
patch 在查看该 cell real result 前冻结；patched baseline 与 original release 分开作图。
|
||
|
||
## Minimal Qwen3.6-27B bring-up and profile plan
|
||
|
||
This is the only permitted path to add the dense case. It intentionally separates cheap provenance
|
||
work from GPU work, and it does not authorize either stage by itself.
|
||
|
||
1. **Preflight and topology smoke (complete):** the snapshot/runtime are locked; fresh TP1/2/4
|
||
servers completed exact-usage text requests. The actual vLLM state/page setting is 784 tokens,
|
||
not the requested 16. This establishes the profile input contract only.
|
||
2. **Hybrid Frontier contract (not run):** verify Frontier can represent Gated DeltaNet/linear-state,
|
||
gated attention, FFN, true mixed execution, TP collectives, the 784-token page rule and capacity,
|
||
without substituting Qwen3.5/Qwen30 fields.
|
||
3. **Minimal profile set (GPU, separately approved):** on the exact Qwen3.6/community-vLLM/dash0
|
||
stack, collect per-TP Gated-DeltaNet, gated-attention prefill/decode, state/KV and TP2/TP4 all-reduce
|
||
rows. The shape grid must cover the pre-registered Fixed-P/PD and held-out Trace-P/PD ranges, but
|
||
must not contain evaluation E2E latency or a fitted time scale. Freeze CSV/schema/command hashes
|
||
before running any evaluation trace.
|
||
4. **Profile acceptance (CPU):** verify model/runtime identity in every profile manifest; require
|
||
requested shape coverage, finite values, and no fallback to a MoE or Qwen3.5 profile. Only then may
|
||
a parent-approved real/simulator latency surface be scheduled.
|
||
|
||
## 预期产物、复现与成本
|
||
|
||
- **Figure prototype：** [latency-selection-matrix-schematic.svg](latency-selection-matrix-schematic.svg)
|
||
（明确标注 `SCHEMATIC — NOT MEASURED DATA`）。最终图替换为每个 task 的 top-set hit / regret /
|
||
coverage heatmap，并附 per-config latency surfaces。
|
||
- **Raw layout：** `artifacts/<model>/<workload>/<real|simulator>/<config>/trial-*/`；每个目录含
|
||
command、environment/machine fingerprint、server flags、trace/profile hash、raw request records、
|
||
stdout/stderr、duration 与 SHA-256 inventory。
|
||
- **Existing reuse audit：** MoE F0 可检验 P-only raw metric reader；现有 F2 可检验 fixed-PD
|
||
trace materialization；T1 可检验 Trace-PD digest and simulator coverage reporting。它们并不构成
|
||
这个 600-s no-SLO matrix 的完整 real surface。Qwen3.6 requires a new same-stack trace,
|
||
model contract and profile root; old Qwen3.5 internal-vLLM artifacts are excluded.
|
||
- **Cost estimate, not authorization：** 单 config×trial 假定 `10 min replay + 5 min fresh-server
|
||
startup/warmup = 15 min`。完整 12-config sweep 的 GPU allocation 为
|
||
`4 × (TP1 + TP2 + TP4) = 28 GPU`; 因而每个 `model×workload` 的三 trial 约
|
||
`28 × 0.25 × 3 = 21 H20-GPUh`。8 cells 的 real serving core is **168 H20-GPUh**. The Qwen3.6
|
||
bring-up/profile set adds at most **6 H20-GPUh** (three topology smokes plus dense/attention/
|
||
collective rows), giving a **174 H20-GPUh** core estimate and a **190–215 H20-GPUh** planning
|
||
envelope with retries. Every launch must recompute this from the staged trace duration and actual
|
||
server preflight; this paragraph is not authorization.
|
||
|
||
### Required remote launch echo (not executed)
|
||
|
||
```text
|
||
NOT EXECUTED — awaiting explicit approval after Qwen3.6 model/runtime/profile gates
|
||
host=dash0; remote_root=/home/admin/cpfs/wjh/aituner/aituner
|
||
models=non-moe-hybrid:Qwen/Qwen3.6-27B@<FROZEN_SNAPSHOT>,moe:Qwen3-30B-A3B-BF16
|
||
engine=community-open-source-vLLM@<FROZEN_SOURCE_OR_WHEEL>; simulator=Frontier/Vidur-class,<official baselines if installed>
|
||
workloads=Fixed-P(ISL2048,OSL1,QPS4,600s),Fixed-PD(ISL2048,OSL128,QPS4,600s),
|
||
Trace-P(heldout exact input/arrival/prefix,OSL1),Trace-PD(heldout exact joint trace)
|
||
surface=TP{1,2,4}×MNS{8,16,32,64}; trials=3; fresh_server=true
|
||
real_paths=runs/simulator-tuning-latency-matrix-v0/artifacts/<model>/<workload>/real/
|
||
expected_cost=174 H20-GPUh core; 190–215 H20-GPUh planning envelope; expected_wall=multiple queued hours
|
||
gate=Qwen3.6 snapshot + community-vLLM fingerprint + resolved-784 hybrid Frontier profile completeness + trace/profile hashes + user/parent approval
|
||
```
|
||
|
||
## Expected outcomes and stop/pivot rules
|
||
|
||
- If a simulator passes every supported task, stop diagnosing that simulator and look for a new problem
|
||
outside its evaluated scope (e.g., unsupported engine knobs or alignment cost).
|
||
- If it fails coverage, repair/liveness is a separate intervention; original capability result remains
|
||
frozen. First rerun simulator only; do not spend GPU-hours until it can produce a complete surface.
|
||
- If it covers but selects incorrectly, retain the counterexample and run one minimal discriminative
|
||
mechanism experiment (scheduler state, batch composition, graph/kernel family, or MoE/collective),
|
||
changing one factor at a time.
|
||
- If Qwen3.6 cannot be profiled on its exact community-vLLM stack, stop the non-MoE-hybrid half rather
|
||
than treating any internal-vLLM or Qwen3.5 result as a substitute. If the headline requires a pure
|
||
all-attention dense-transformer case, add that model explicitly rather than relabelling Qwen3.6.
|
||
|
||
## Benchmark-design audit
|
||
|
||
| Risk | Pre-registered control |
|
||
|---|---|
|
||
| SLO/capacity result misrepresented as tuning | selection uses only per-request mean/p90 TTFT, TPOT, E2E on the same trace |
|
||
| P-only judged by nonexistent TPOT | OSL=1 and TPOT=`N/A` are explicit workload/metric contracts |
|
||
| Trace simplification hidden | Trace-P is labelled output-normalized; only Trace-PD claims full joint replay |
|
||
| calibration equals evaluation | microprofile/trace calibration folds are disjoint; no E2E correction on evaluation |
|
||
| simulator crash treated as a bad config | coverage gate separates missing metrics from latency ranking |
|
||
| cross-stack or wrong-architecture reuse | only Qwen3.6-27B hybrid profiles collected on its exact community-vLLM stack are admissible; old Qwen3.5/internal-vLLM or Qwen30/MoE artifacts are excluded |
|
||
| selected winning cells only | all 8 cells and all legal configs are frozen before real results |
|
||
| tail/noise omitted | p90, three fresh trials, per-trial values and bootstrap top sets are mandatory |
|