85 lines
5.0 KiB
Markdown
85 lines
5.0 KiB
Markdown
# Frontier coverage audit for Qwen3.6-27B
|
|
|
|
> **Verdict (2026-07-17): `FAIL: original Frontier coverage`.** This is a
|
|
> static capability result, not a latency-fidelity result. It says that the
|
|
> examined existing Frontier cannot represent the real Qwen3.6 execution
|
|
> contract; it does not say that a future extended simulator will be wrong.
|
|
|
|
## Question
|
|
|
|
Before spending profile or replay GPU-hours, can the checked Frontier baseline
|
|
simulate the same model/engine state that the real server executes?
|
|
|
|
## Compared contracts
|
|
|
|
| Side | Observed contract |
|
|
|---|---|
|
|
| Real server | ModelScope `Qwen/Qwen3.6-27B@cea40373b9214dd387123e68841890af30dcd469`; vLLM 0.20.2, BF16; `Qwen3_5ForConditionalGeneration`; 64-layer non-MoE hybrid with three Gated DeltaNet/linear-state layers followed by one gated-attention layer per block. The release vLLM run used a FlashInfer GDN prefill kernel and changed requested `--block-size 16` to a 784-token attention/state page at TP1/2/4. |
|
|
| Frontier baseline | `/home/admin/cpfs/wjh/frontier-qwen30-vllm020-profile-v1/Frontier`, commit `d9cfeb6d8791fbf2f295dd9744c56a666171776e`. Its model-spec directory has Qwen2/Qwen3-MoE and all-attention dense examples, but no Qwen3.6 or Qwen3_5 spec. Source search across `*.py`, `*.json`, `*.md` returned no hit for `qwen3_5`, `qwen3.6`, `GatedDeltaNet`, `DeltaNet`, `mamba`, `linear attention`, `state cache`, or `gdn`. |
|
|
|
|
## Frontier source model-spec inventory
|
|
|
|
This is the complete `frontend/configs/model_configs/*.json` inventory at the
|
|
checked commit. It establishes that the simulator can instantiate these
|
|
*structural specifications*; it is **not** evidence that a matching vLLM
|
|
profile, runtime contract, or validated tuning result exists for each model.
|
|
|
|
| Class | Source specs |
|
|
|---|---|
|
|
| All-attention, non-MoE | `Llama-3.1-405B-Instruct-FP8`, `Llama-3.2-1B-Instruct`, `llama2_7b_dense_example`, `llama3.1-405b`, `llama3.1-8b`, `llama3.3-70b`, `qwen2_dense_test` |
|
|
| MoE | `Phi-tiny-MoE-instruct`, `Qwen3-235B-A22B`, `Qwen3-235B-A22B__layers96`, `Qwen3-30B-A3B-tiny`, `deepseek-v3`, `mixtral_8x7b_moe`, `qwen2_moe_example`, `qwen3-a3b-30b-moe`, `qwen3-next-80b-a3b-instruct-reduced-l2`, `qwen3-next-80b-a3b-instruct-reduced-l20`, `Step2Mini-tiny`, `step-moe-noquant-small`, `step-moe-noquant`, `step-moe` |
|
|
|
|
The exact JSON architecture labels are `llama`, `qwen2`, `phimoe`,
|
|
`qwen3_moe`, `deepseek_v3`, `mixtral`, `qwen2_moe`, `qwen3_next`,
|
|
`step2_mini`, and `step3_text`. The separately held experimental artifacts
|
|
in this project establish profile-and-surface evidence only for
|
|
`Qwen3-30B-A3B` and `Qwen3-235B-A22B`; the remaining names are not
|
|
automatically runnable or validated on H20/vLLM merely because a JSON exists.
|
|
|
|
## Why the missing model file is semantic, not administrative
|
|
|
|
The Frontier execution-time taxonomy separates `attention_prefill_execution_time`
|
|
and `attention_decode_execution_time`; its operator context and parameter
|
|
accounting are built from attention Q/KV heads, attention head dimension, FFN,
|
|
and optional MoE routing. No Gated-DeltaNet/linear-state operator, state-cache
|
|
capacity rule, or its mixed prefill/decode scheduling state was found.
|
|
|
|
Adding a JSON whose layer count merely matches Qwen3.6 would therefore map
|
|
three quarters of its text layers onto an all-attention approximation. Reusing
|
|
Qwen30/MoE or Qwen3.5 profiles has the same defect. Neither is a valid
|
|
execution of the existing baseline.
|
|
|
|
## Decision
|
|
|
|
1. Record Frontier as **coverage failure** for the Qwen3.6 non-MoE-hybrid
|
|
branch. Do not rank configs, fabricate high latency, or spend GPU-hours on
|
|
an all-attention profile substitute.
|
|
2. A new Gated-DeltaNet/state-cache model and profiling implementation would
|
|
be an **extended Frontier** intervention. It must be evaluated separately
|
|
from the capability of the original simulator and cannot support the claim
|
|
that existing Frontier already solves tuning.
|
|
3. The Qwen3.6 result cannot be presented as evidence for a conventional pure
|
|
dense-transformer class. It is a useful non-MoE/hybrid boundary case.
|
|
4. No runnable local checkout/version of Vidur or APEX was found; only a
|
|
historical Vidur patch exists. Their coverage remains **unassessed**, not
|
|
passed or failed, until their exact released artifacts are frozen and
|
|
checked against this model/runtime contract.
|
|
|
|
## Next critical experiment
|
|
|
|
Do not extend Frontier before establishing the requested no-SLO metric on a
|
|
model it already represents. The next direct test is the existing
|
|
Qwen3-30B-A3B MoE branch under one same-trace Fixed-P replay surface, scored
|
|
by mean/p90 TTFT and E2E (TPOT `N/A`), with simulator-only coverage checked
|
|
before any real 600-s sweep. This separates a known architectural coverage
|
|
failure from a fidelity failure on an in-scope model.
|
|
|
|
## Evidence locations
|
|
|
|
- Real topology smoke and resolved page state:
|
|
`artifacts/qwen36/smoke-v4-20260717/` in this run directory.
|
|
- Real model/runtime audit:
|
|
[dense-preflight.md](dense-preflight.md).
|
|
- No-SLO latency protocol and selection gates:
|
|
[experiment-card.md](experiment-card.md).
|