Related Work Matrix — Workload Pattern → Serving Engine Config

| Paper                                                                                                  | Venue / Year | 主要研究对象                            | Workload signals / patterns considered                 | Optimized knobs / decisions                                                               | Objective                                         | 方法类型                                 | 与“workload→config”关系    | 对 AITuner 的直接启发                                                     | 是否已提出关键 insight                              |
| ------------------------------------------------------------------------------------------------------ | ------------ | --------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------ | ----------------------- | ------------------------------------------------------------------- | -------------------------------------------- |
| AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving              | arXiv 2026   | 多框架 LLM serving config 优化         | production serving workloads；模型类型；硬件平台；SLA/TTFT/TPOT需求 | TP/PP/EP、CUDA graphs、KV-cache memory fraction、max token capacity、framework-specific flags | latency / throughput / goodput 风格目标               | 性能建模 + 快速配置搜索                        | 最直接命中                   | 证明“最优 engine config 明显 workload-dependent”，而且 config 空间不止 scheduler | 是，且最直接                                       |
| Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service (Kareto)             | arXiv 2026   | KV cache 分层存储配置                   | real-world traces；KV block access pattern；reuse；不同优化目标 | HBM/DRAM/disk tier config；eviction/group-specific cache mgmt                              | latency / throughput / cost Pareto frontier       | simulator + pruning + adaptive tuner | 直接命中，但只在 KV/storage 子空间 | 说明 memory-side knobs 必须 workload-aware，且天然是多目标优化                    | 是，但局限于 KV tiering                            |
| Serving Generative Large Language Models on Preemptible Instances (SpotServe)                          | ASPLOS 2024  | 抢占式云环境下的 LLM serving              | fluctuating workload；实例可用性变化；preemption trace          | distributed parallelization config；migration strategy                                     | latency / throughput / monetary cost              | 系统设计 + 动态重配置                         | 部分命中                    | 说明 config = f(workload, resource-state)，不是只由 workload 决定            | 是，但偏部署/并行配置                                  |
| Efficient and Workload-Aware LLM Serving via Runtime Layer Swapping and KV Cache Resizing (MorphServe) | arXiv 2025   | workload-aware runtime adaptation | bursty workload；real-time memory pressure；load surge   | layer quantization/swapping；KV cache resizing                                             | SLO violations、TTFT、quality-preserving efficiency | runtime adaptive serving             | 部分命中                    | 说明静态最优 config 不适合 bursty workload，online adaptive config 很重要        | 是，但偏 runtime morphology                      |
| Llumnix: Dynamic Scheduling for Large Language Model Serving                                           | OSDI 2024    | heterogeneous requests 的动态调度      | heterogeneous / unpredictable requests；不同优先级/SLO       | request placement、migration、instance-level scheduling                                     | tail latency、priority acceleration、cost saving    | 动态 rescheduling + live migration     | 邻近，不是 config tuning     | 说明高异质 workload 下，先做实例级隔离/迁移可能比局部 knob tuning 更重要                    | 提出了 workload→architecture/scheduling insight |

- 1/2

- 1
- 2

| Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage (semi-PD) | arXiv 2025 | phase-wise 资源划分 | request rate；phase contention；SLO pressure | prefill/decode partition ratio；resource controller；unified storage | latency、SLO attainment | 系统设计 + dynamic partitioning | 部分命中 | 说明 phase split ratio 本身就是重要 config knob，而且 workload-dependent | 是，但聚焦 phase partition |  
| Workload Characterization and Generation of Large Language Model Serving in Production (ServeGen) | arXiv 2025 | 真实 workload characterization | per-client composition；temporal structure；model mix；multimodal/reasoning；large-scale production traces | 不直接优化 config | benchmark realism；avoid under-provisioning | characterization + workload generator | 基础支撑，不直接命中 | 说明如果 workload 表征不真实，任何 workload→config 规律都可能失真 | 提出了 workload realism 的关键 insight |  
| BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems | arXiv / KDD | 真实 serving traces 数据集 | burstiness；conversation patterns；response lengths；system failures | 不直接优化 config | evaluation realism；stress realistic serving behavior | trace dataset / characterization | 基础支撑，不直接命中 | 说明 burstiness、会话结构、output-length joint stats 都是重要 workload features | 是，提出真实 workload 重要性 |  
| Comparative Analysis of Large Language Model Inference Serving Systems: vLLM and HuggingFace TGI | arXiv 2025 | serving framework 对比 | concurrency level；interactive vs batch use case；model size | framework choice（vLLM vs TGI） | throughput、tail latency、memory、scalability | empirical comparison | 粗粒度命中 | 说明 engine choice 本身就是 coarse-grained config decision | 部分提出 |  
| Understanding and Optimizing Multi-Stage AI Inference Pipelines (HERMES) | arXiv 2025 | multi-stage inference pipeline | reasoning, RAG, KV retrieval, routing, heterogeneous hardware, stage diversity | stage-specific batching / HW-SW design choices | end-to-end latency；pipeline optimization | simulator / design-space exploration | 扩展边界相关 | 提醒你：未来 workload→config 问题会扩展到 multi-stage，不再只看 prefill/decode | 部分提出，多阶段视角 |

我建议你在论文里把它们再压成 4 类

A. Workload characterization / benchmark realism

这些 paper 不直接调 config，但决定你的论证是否站得住。

|Paper|你该怎么引用它|
|---|---|
|ServeGen|用来证明：真实 workload 很复杂，synthetic trace 容易误导配置结论|
|BurstGPT|用来证明：burstiness、conversation pattern、response length 这些 workload feature 不能忽略|

B. Workload-aware scheduling / routing / partitioning

这些不是在调 engine knobs，但给了你 workload pattern 的核心结构。

|Paper|核心已知 insight|
|---|---|
|Llumnix|heterogeneous workload 需要跨实例 rescheduling / isolation|
|EWSJF|mixed workload 应先分组/分 regime，再优化|
|CascadeInfer|length heterogeneity 是核心 bottleneck|
|Sarathi-Serve|prefill/decode 冲突是第一性瓶颈|
|semi-PD|phase split ratio 应随 workload / SLO 改变|

C. Workload-aware configuration / adaptive resource shaping

这些和你最接近。

|Paper|命中子空间|
|---|---|
|AIConfigurator|通用 serving config search|
|Kareto|KV / storage configuration|
|SpotServe|distributed parallelization configuration|
|MorphServe|online adaptive resource / precision configuration|

D. Framework / pipeline-level coarse configuration

这些告诉你研究边界在哪里。

|Paper|边界意义|
|---|---|
|Comparative vLLM vs TGI|engine choice 本身就是 coarse-grained config|
|HERMES|multi-stage pipeline 会让 workload→config 问题更复杂|

如果你想在 paper 里更突出 gap，可以直接用这张“缺口矩阵”

|维度|现有工作覆盖情况|你的空间|
|---|---|---|
|真实 workload characterization|强|不必重复造数据集，但要吸收其 feature 定义|
|workload-aware scheduling|很强|不要和它们正面重合|
|单一子空间 config adaptation（KV / parallelism / runtime precision）|中等|可以统一这些子空间 insight|
|通用 workload→engine-knob mapping|弱|这是你的主要机会|
|在 length-dependent TTFT + TPOT + goodput 统一目标下的配置优化|很弱|这是你最强的位置|

- 1/2

- 1
- 2

我帮你提炼成一句话版 related work positioning

你后面写论文时，可以这么说：

Prior work has extensively studied realistic workload characterization and workload-aware scheduling for LLM serving, and several recent systems have explored workload-dependent adaptation in specific configuration subspaces such as parallelization, KV-cache tiering, and runtime precision scaling. However, a general and interpretable mapping from workload patterns to serving-engine configurations remains underexplored, especially under length-dependent TTFT and TPOT constraints with SLO-constrained goodput as the optimization target.

这句话基本能把你和上面这些 work 的关系讲清楚。

我建议你下一步补一张“feature × knob”矩阵

也就是：  

- 行：workload features
    

- arrival burstiness
    

- input length mean/variance/tail
    

- output length mean/variance/tail
    

- mixed-workload ratio
    

- prefix/KV locality
    

- SLO strictness
    

- 列：config knobs
    

- max_num_batched_tokens
    

- scheduler mode
    

- prefill/decode split
    

- KV memory fraction
    

- block size
    

- parallelism
    

- disaggregation mode
    

- cache/tier policy
    

然后标每篇 paper 实际覆盖了哪些 feature→knob 边。  
  
这个会比 paper matrix 更直接服务你的项目。  
  
如果你要，我下一条就直接给你整理 feature × knob insight matrix。



[ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production](https://arxiv.org/pdf/2505.09999)

[Glia: A Human-Inspired AI for Automated Systems Design and Optimization](https://arxiv.org/pdf/2510.27176)
[VIDUR: A LARGE-SCALE SIMULATION FRAMEWORK FOR LLM INFERENCE](https://arxiv.org/pdf/2405.05465)

[DynamoLLM](https://arxiv.org/pdf/2408.00741)：profile（不同的 TP、workload） + search（target：满足 SLO 下最小化能耗） + dynamic 调整 parallelism configuration
[NanoFlow](https://arxiv.org/pdf/2408.12757)：Theoretical model + automated pipeline search（split sequence / operation，target：最大化 compute/memory/network 利用率）


https://github.com/llm-d/llm-d

AIBrix 实验性的提出了异构 GPU 推理，workload 有不同的 size，不同的 GPU 适合不同 request size 的甜点区间，offline profile，online scheduling，异构硬件 serving 时最小化 cost
这存在老生常谈的问题：offline profile 与实际 serving 的 gap
https://github.com/vllm-project/aibrix/blob/main/docs/source/features/heterogeneous-gpu.rst


agent related:
- StorageXTuner: An LLM Agent-Driven Automatic Tuning Framework for Heterogeneous Storage Systems https://arxiv.org/pdf/2510.25017
- ASAP: an Agentic Solution to Auto-optimize Performance of Large-Scale LLM Training https://arxiv.org/pdf/2511.03844
- STELLAR: Storage Tuning Engine Leveraging LLM Autonomous Reasoning for High Performance Parallel File Systems https://dl.acm.org/doi/epdf/10.1145/3712285.3759887

AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving https://arxiv.org/pdf/2601.06288

SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines https://arxiv.org/pdf/2408.04323
https://github.com/antgroup/SCOOT-SLO-Oriented-Performance-Tuning

> Request traces are collected from four LLM inference services at Ant Group, including applications of text-to-SQL (SQL), chatbot (BOT), classification (CLS), and recommendation (REC).
> 
> SCOOT uses requests with the longest 50% of output lengths for stress testing. The input and output lengths of these requests are presented in Fig.

![[projects/auto-tuner/related-works.figs/260410-105227.png]]

SLO-Aware SchedulingforLarge Language Model Inferences
https://arxiv.org/pdf/2504.14966


https://ut-sysml.ece.utexas.edu/publications/prints/sc2025_liakopoulos.pdf


FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
https://arxiv.org/abs/2602.22593






你想要的“统一范式”其实在多篇顶会论文里被非常直白地描述过：**定义搜索空间/动作 → 运行试验（trial）→ 测量目标 → 用算法选择下一步动作**。

- **ATC’18（Cao et al.）**在解释黑箱 auto-tuning 时，把机制写得非常标准化：_“迭代尝试不同配置、测量目标函数、基于已学信息选择下一批配置”_。
https://www.usenix.org/system/files/conference/atc18/atc18-cao.pdf

- **OSDI’23（Hydro）**对超参/配置调优同样给出“工作流定义”：用户指定搜索空间；算法生成 trials；系统协调执行直到找出最优配置。
https://www.usenix.org/system/files/osdi23-hu.pdf
	
- **ATC’18（Metis）**把自己定位为“black-box optimization service”，在真实生产系统上调参并以 tail latency 为主要评估指标。
https://www.usenix.org/system/files/conference/atc18/atc18-li-zhao.pdf
    
- **OSDI’18（µTune）**属于“在线自适应”流派，但它仍然是同一闭环：监控/估计负载 → 用模型预测每个配置下的 tail latency → 切换到预测最优的配置。
https://www.usenix.org/system/files/osdi18-sriraman.pdf
    
- **SOSP’21（POP）**是“数学优化/求解器”流派，它甚至把系统资源分配问题明确定义为“可写成数学优化问题”的形式，并讨论求解速度与 SLA 的权衡。
   https://people.eecs.berkeley.edu/~matei/papers/2021/sosp_pop.pdf 

**结论**：无论是黑箱搜索、在线控制、强化学习、还是数学规划，顶会系统优化基本都落在同一个元结构上：

> **在预算与约束下，围绕某个“可执行试验”的闭环，迭代地产生动作并更新策略。**

---
# related works

[DynamoLLM](https://arxiv.org/pdf/2408.00741)：profile（不同的 TP、workload） + search（target：满足 SLO 下最小化能耗） + dynamic 调整 parallelism configuration
[NanoFlow](https://arxiv.org/pdf/2408.12757)：Theoretical model + automated pipeline search（split sequence / operation，target：最大化 compute/memory/network 利用率）
[MorphServe](https://arxiv.org/pdf/2506.02006v1)
[Autocomp](https://arxiv.org/pdf/2505.18574v3)



# Questions

1. [Mooncake 访谈](https://www.infoq.cn/video/vUNUp9tkjBHxGqaO33WK) 10:00 处，业务场景变得极为丰富，需要不同的配置，PD 分离时几P几D，P 和 D 节点内分别跑什么样的并行模式都需要调优，Qwen 线上是否有明确的需要不同配置的需求？或者目前 Qwen 的现状还是根据人工测试调整，找到一个整体相对比较优的配置固定作为一个模型的配置，并不涉及线上的负载感知与动态调整？我认为阿里这里有「异构硬件 x 多模型 x 多样的负载」线上是否需要对每个维度都人工 finetune 找一个好的推理配置？还是有一些更好的工作流？
2. Qwen 系列模型是否已经上线 EP？如果上线了 EP，为什么不使用 DeepSeek 的 DBO (dual batch overlap) 方案？我理解 DBO 在 EP 的场景总是能做到一定程度的 overlap 计算和通信从而提高性能？
3. Qwen-Next 可以看到针对 linear-attention 的优化 (GDNAttention)、针对超大稀疏 experts 的 EP 优化，可以认为这是一个普遍的趋势吗？


---
# Backup





> 说明：这里的“自动优化”聚焦**自动化调度/并行/批处理/能耗-成本/KV-Cache 管理**等系统层机制，而非单纯模型改造或手工参数调优。每条给出一句话核心观点与公开报告的性能主张（以作者给出的 headline 为准）。

### 速览对照表

| 工作（年份/来源）                                                                     | 一句话核心                                                  | 关键自动化机制                                                                | 公开性能主张（作者给出）                                | 与相近工作的差异点                                      | 在 vLLM / SGLang 落地                                         |
| ----------------------------------------------------------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------- | ------------------------------------------- | ---------------------------------------------- | ---------------------------------------------------------- |
| **DynamoLLM** (2024, HPCA’25) ([arXiv][1])                                    | 面向**集群级**的能耗/成本最优化：在满足 SLO 的前提下**自动重构**推理集群配置          | 负载-功耗感知的**动态集群重构**、分层控制                                                | 节能 ~52–53%，运营碳减 38%，成本降 61%，SLO 保持          | 目标是**服务级/集群级**运行点最优化，而非 GPU 内核/批处理细节           | 与 vLLM/SGLang **并行部署**（外部编排层），非内嵌；可用作上层资源控制器               |
| **NanoFlow** (2024→2025) ([arXiv][2])                                         | 在**单 GPU**上把一次请求切成多个**nano-batches**并自动搜寻**并行-重叠流水线**  | **自动搜索**nano-batch 的数量/大小/顺序/资源配额；算子共调度以重叠算力/显存/通信                     | 对多基线（vLLM、FastGen、TRT-LLM）最高 **1.91×** 吞吐提升 | 强调**设备内**并行与流水重叠；对 decode 轻/ prefll 重的结构做细粒度流水 | 目前为**独立运行时**（开源实现），未并入 vLLM/SGLang 主干；可作替代后端 ([GitHub][3]) |
| **Sarathi & Sarathi-Serve** (2023–2024) ([arXiv][4])                          | 把长**prefill**切块，并与**decode**混合成连续混批，减少流水“气泡”           | **Chunked prefill** + **continuous hybrid batching**（decode piggyback） | 端到端最高 **1.91×**；decode 吞吐最高 **10×**         | 最早系统化提出“**prefill-decode 混合**一批”的通用策略          | vLLM/SGLang 均已支持/吸收该思路（见下文“落地”） ([VLLM Documentation][5])  |
| **DeepSpeed-FastGen** (2024) ([arXiv][6])                                     | **Blocked KV** + **Dynamic SplitFuse 连续批处理**，兼顾低延迟与高吞吐 | 动态拆分/融合批次，KV 分块复用                                                      | 多模型/硬件上优于 vLLM（作者报告）                        | 方案与 vLLM 的 PagedAttention 路线相近但实现不同            | 作为**独立后端**；与 vLLM/SGLang 互为替代                              |
| **POD-Attention** (ASPLOS’25) ([Microsoft][7])                                | 追求**prefill 与 decode 的全重叠**，降低 Token Break Time（TBT）   | Prefill/Decode **完全并行化调度**与相容内核                                        | 在长上下文/高 TBT 场景显著降停顿（定性+量化）                  | 把 Sarathi 的“混合批”更推进到**近全重叠**                   | 学术原型；思想可迁移到 vLLM/SGLang 的调度/内核层                            |
| **Fluid-Guided Online Scheduling（WAIT/Nested WAIT）** (2025, SSRN) ([SSRN][8]) | 把 LLM 推理抽象为**带 KV 内存约束**的多阶段**在线调度**，给出近似最优策略          | 基于流体模型的**在线批量与内存配给**决策                                                 | 实验优于 vLLM/Sarathi 的吞吐/时延（作者报告）              | 强理论导向，**内存-批量-时延**三者联动的在线算法                    | 研究性，尚未并入主流实现                                               |
| **Memory-aware 动态批处理** (2025) ([arXiv][9])                                    | 运行时监控**显存与 SLA**，**自适应**批大小与解码过程                       | 显存感知的批调度 + 延迟反馈回路                                                      | 在 Llama-7B+A100 上优于固定超参的吞吐/时延               | 更工程化的**在线批超参**调节方法                             | 思路与 vLLM/SGLang 的自调度兼容；尚无主干合并记录                            |
| **HyGen** (2025) ([arXiv][10])                                                | **线上/离线**融合：两阶段调度在不破坏线上 SLO 的前提下挤出“离线”算力               | 两阶段 SLO-aware 调度与隔离                                                    | 在离线共置下维持在线延迟并提升整体利用率                        | 关注**业务混部**而非单一推理吞吐                             | 可与任一后端并行部署；非 vLLM/SGLang 内核改动                              |
| **PrefillOnly** (2025) ([arXiv][11])                                          | 针对**只需 1 token 输出**的“prefill-only”工作负载做**极简 KV**与路径优化  | 仅保留**最后一层** KV、轻量运行路径                                                  | 对检索/分类式工作负载显著提速降时延                          | 面向特定负载类型的**路径裁剪**                              | 可作为特型后端/路径对接主流引擎                                           |
| **vLLM / PagedAttention** (2023→) ([arXiv][12])                               | **分页化 KV-Cache** + 预占式调度，近零碎片、易做连续批处理                  | 块级内存管理、连续批、请求抢占                                                        | 对 HF baseline 最高 **24×** 吞吐（早期报告）           | 率先把**内存分页**与**连续批**标准化                         | 已成为事实上的开源主流服务引擎之一                                          |
| **Throughput-Optimal Scheduling for LLM Serving** (2025) ([arXiv][13])        | 从理论上给**连续批处理**的吞吐上界/最优策略                               | 令牌级排队/匹配策略                                                             | 给出与实践接轨的理论最优性结果                             | 偏理论基线，指导工程实现                                   | 待工程化吸收                                                     |
| **Learning-to-Rank Scheduler** (2024)            | 预测请求**相对长度排序**，更逼近 SJF 以降延迟/提吞吐                              | LTR 训练的调度器，按预测顺序排队/并批                      | 相比现有基线更优的延迟/完成时间            | 与两者同属**自动排队/批形成**                   | 可接入 vLLM/SGLang 的 admission/排队阶段。 ([arXiv][5])                       |
| **Online Scheduling with KV Constraints** (2025) | 把 LLM 推理抽象为**带 KV 内存约束**的在线调度，给出近优策略                         | 流体/队列模型 + 在线批/KV 决策                        | 优于 vLLM/Sarathi 基线（作者报告）    | 与两者同属**自动在线调度**理论化                  | 可做 vLLM/SGLang 的**策略插件**（需低开销遥测）。 ([arXiv][8])                       |
| **Fairness-Aware Batch Formation** (2025.10)     | 在**连续/混合批**下自动平衡新/旧请求的“计算公平性”与吞吐                             | 批内配额/重排策略                                  | 在保持吞吐下显著改善不公平               | 同属批形成自动化（与 Sarathi/Chunked 互补）      | 可改造 vLLM/SGLang 的 batcher。 ([arXiv][10])                             |
| **Drift（PD-Multiplexing）** (2025)                | **相位解耦复用**：预填与解码相位分离并“就地”复用，缓解吞吐-时延拉扯                        | 相位解耦 + in-place compute 复用                 | 在多负载下同时提吞吐与守 SLO            | 与 NanoFlow/POD 同属**相位级重叠**路线        | 需要内核/调度联动，适合作为引擎深改方向。 ([Han Zhao 赵涵][11])                            |

[1]: https://arxiv.org/html/2408.00741v1?utm_source=chatgpt.com "DynamoLLM: Designing LLM Inference Clusters for ..."
[2]: https://arxiv.org/abs/2408.12757?utm_source=chatgpt.com "NanoFlow: Towards Optimal Large Language Model ..."
[3]: https://github.com/efeslab/Nanoflow?utm_source=chatgpt.com "efeslab/Nanoflow: A throughput-oriented high-performance ..."
[4]: https://arxiv.org/abs/2308.16369?utm_source=chatgpt.com "SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills"
[5]: https://docs.vllm.ai/en/v0.4.2/models/performance.html?utm_source=chatgpt.com "Performance and Tuning - vLLM"
[6]: https://arxiv.org/pdf/2401.08671?utm_source=chatgpt.com "DeepSpeed-FastGen: High-throughput Text Generation for ..."
[7]: https://www.microsoft.com/en-us/research/wp-content/uploads/2025/03/POD-Attention-ASPLOS25.pdf?utm_source=chatgpt.com "POD-Attention: Unlocking Full Prefill-Decode Overlap for ..."
[8]: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5195463&utm_source=chatgpt.com "Optimizing LLM Inference: Fluid-Guided Online Scheduling ..."
[9]: https://arxiv.org/pdf/2503.05248?utm_source=chatgpt.com "Optimizing LLM Inference Throughput via Memory-aware ..."
[10]: https://arxiv.org/html/2501.14808v2?utm_source=chatgpt.com "1 Introduction"
[11]: https://arxiv.org/html/2505.07203v1?utm_source=chatgpt.com "PrefillOnly: An Inference Engine for Prefill-only Workloads ..."
[12]: https://arxiv.org/pdf/2309.06180?utm_source=chatgpt.com "Efficient Memory Management for Large Language Model ..."
[13]: https://arxiv.org/html/2504.07347v1?utm_source=chatgpt.com "Throughput-Optimal Scheduling Algorithms for LLM ..."
[14]: https://homes.cs.washington.edu/~arvind/papers/nanoflow.pdf?utm_source=chatgpt.com "NanoFlow: Towards Optimal Large Language Model ..."
[15]: https://discuss.vllm.ai/t/does-the-vllm-v1-support-speculative-decoding-now/191?utm_source=chatgpt.com "Does the vLLM v1 support Speculative Decoding now?"
[16]: https://github.com/sgl-project/sglang/issues/2273?utm_source=chatgpt.com "[Kernel] Launch two kernels for mixed chunked prefill #2273"
[17]: https://github.com/sgl-project/sglang/issues/6553?utm_source=chatgpt.com "[PD] Support Multi-Process for TokenizerManager #6553"
[18]: https://arxiv.org/html/2312.07104v1?utm_source=chatgpt.com "Efficiently Programming Large Language Models using ..."
[19]: https://iacoma.cs.uiuc.edu/iacoma-papers/hpca25_2.pdf?utm_source=chatgpt.com "DynamoLLM: Designing LLM Inference Clusters for ..."




## Qwen optimization


Catalogued how Alibaba’s Qwen-family models have been tuned across the codebase.

**Optimization Matrix**  

| Model scope                                | Optimization & commits                                                                                                                                                                                                                                        | Why it was added                                                                                                                | Feature / hardware fit                                                                                 |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| Qwen3-VL                                   | Vectorized fast_pos_embed_interpolate (30d08911f, af7dfb0d1) → vllm/model_executor/models/qwen3_vl.py:443                                                                                                                                                     | Removes nested Python loops and reuses tensor meshgrids to cut interpolation overhead on large vision grids                     | Handles arbitrary grid_thw shapes for image/video batches, improving deepstack throughput              |
| Qwen3-VL                                   | Non-blocking data flow & CPU-built cu_seqlens (b2155ed31, 2c1c7dfb3, 0426e3c5e) → vllm/model_executor/models/qwen3_vl.py:492, 529-549, 1296                                                                                                                   | Keeps host/device copies async, avoids redundant concatenate, flattens 3D MM tensors without reallocation                       | Sustains high-frame video pipelines and DP sharding without CUDA synchronizations                      |
| Qwen3-VL                                   | Triton interleaved MRoPE + ViT data-parallel/back-end gating (cea91a32f, 3127274d0, c242c9803) → vllm/model_executor/layers/rotary_embedding/mrope.py:19, vllm/model_executor/models/qwen3_vl.py:316, 1420                                                    | Custom kernel accelerates interleaved 3D RoPE; DP path removes TP-only constraint while forcing SDPA on Blackwell for stability | Matches Qwen3-VL vision tower on GB200/B200 with MoE+DP and interleaved rotary requirements            |
| Qwen2.5-VL                                 | Flash/xFormers friendly attention staging (02ed8a1fb, 70b808fe1, 47c712621, c242c9803) → vllm/model_executor/models/qwen2_5_vl.py:274, 627, 814                                                                                                               | Precomputes seqlens once, routes through backend-specific fast paths, and supports upstream FlashAttn fallback                  | Smooths attention across FlashAttn, TORCH_SDPA, xFormers on long video contexts and new Blackwell GPUs |
| Qwen2.5-VL                                 | O(n) inverse permutation & pinned staging (67da5720d4, e283976f3, b0d1213ac) → vllm/model_executor/models/qwen2_5_vl.py:769-897, 829-831, 892-897                                                                                                             | Replaces argsort with in-place inverse, pins buffers, and streams seqlens to GPU asynchronously                                 | Large-window adapters and deepstack vision merges stay latency-neutral under heavy video loads         |
| Qwen2-VL                                   | Vision data-parallel mode & memoized seqlens (c98be0a23, 70b808fe1) → vllm/model_executor/models/qwen2_vl.py:244-716, 767-805                                                                                                                                 | Allows disabling TP in favor of DP for the encoder and caches attention metadata                                                | Optimizes multi-GPU deployments handling many images per request                                       |
| Qwen2/2.5-VL startup                       | Max-token heuristics replace dummy image probing (2c5302fad) → vllm/model_executor/models/qwen2_vl.py:931-940, vllm/multimodal/profiling.py:353-360                                                                                                           | Avoids generating huge fake inputs to find token limits, slashing initialization time                                           | Keeps autoregressive limits aligned with true max resolution / frame counts before serving             |
| Qwen series (CUDA & ROCm)                  | Rotary dispatch abstraction (5e4a8223c) → vllm/model_executor/layers/rotary_embedding/common.py:74, vllm/model_executor/models/qwen2_vl.py:274                                                                                                                | Dynamically selects FlashAttn kernels (CUDA/ROCm) instead of Python fallback                                                    | Maintains high-throughput rotary embedding on both CUDA and ROCm stacks                                |
| Qwen2 text & MoE                           | FP8 KV-scale remap + tuned A100 configs (da971ec7a, bd4397352) → vllm/model_executor/models/qwen2.py:436-451, vllm/model_executor/layers/fused_moe/configs/E=64,N=1280,device_name=NVIDIA_A100-SXM4-80GB.json#L1                                              | Loads FP8 cache scales correctly and ships empirically tuned fused-MoE kernels                                                  | Targets FP8-capable GPUs and boosts Qwen2-57B-A14B throughput on A100 TP=2/4                           |
| Qwen3 MoE family (Next / Coder / Thinking) | Blackwell / H100 / H200 FP8 fused configs (238c4c170, 482e52f56, 75334956c, 9f04d9d55, 12a8414d8, f82f7a899, 7a70a7189, 569bf1c9c, c733bd5e8) → e.g. vllm/model_executor/layers/fused_moe/configs/E=512,N=128,device_name=NVIDIA_GB200,dtype=fp8_w8a8.json#L1 | Provides block-size / warp tuning for each TP×EP layout and precision mix                                                       | Lets Qwen3 MoE variants saturate GB200/H200/H100 SKUs at FP8 W8A8 without manual retuning              |

Key observations

- Multiple commits converge on eliminating host/device syncs via pinned buffers and asynchronous transfers (vllm/model_executor/models/qwen2_5_vl.py:831, 892-897), indicating a broad push to keep multimodal pipelines GPU-bound.
- Data-parallel encoder paths (vllm/model_executor/models/qwen3_vl.py:316, vllm/model_executor/models/qwen2_vl.py:664-716) mirror each other, suggesting a reusable pattern for future Qwen multimodal towers.
- Fused-MoE JSON configs proliferate across hardware tiers; keeping them in sync with upstream kernel tuning is now central to sustaining Qwen3-* throughput.

Next steps

1. Spot-check lm-eval / throughput on GB200 and B200 with the new MoE configs to confirm expected gains on production workloads.
2. Document the DP vs TP trade-offs for operators now that encoder data-parallel is available in both Qwen2-VL and Qwen3-VL, so deployment teams pick the right topology.




| Model                  | Optimization                                                                                                                                                                            | Why it was added                                                                                                                                 | Feature / hardware fit                                                          |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- |
| Qwen3‑Next             | SharedFusedMoE wraps shared experts and only overlaps them when it pays off (vllm/model_executor/models/qwen3_next.py:161, vllm/model_executor/layers/fused_moe/shared_fused_moe.py:31) | Avoids double-computing shared experts when Expert Parallel (EP) or FlashInfer overlap would be wasted, keeping the block torch.compile friendly | Hybrid MoE layers with shared experts on DP/EP deployments                      |
| Qwen3‑Next             | Sequence-parallel MoE routing (vllm/model_executor/models/qwen3_next.py:183, vllm/model_executor/models/utils.py:784, vllm/config/parallel.py:342)                                      | Guarantees tokens are sharded across TP ranks before expert dispatch so TP × EP + DeepEP doesn’t repeat work                                     | Large-TG throughput setups using TP+EP or DeepEP all-to-all                     |
| Qwen3‑Next             | Passes GDN causal-conv metadata through the attention path (vllm/model_executor/models/qwen3_next.py:528, vllm/v1/attention/backends/gdn_attn.py:59)                                    | Supplies Triton conv kernels with precomputed offsets so gated delta nets can reuse cached state cheaply during prefills                         | Keeps the hybrid Mamba/GDN stack fast on long prefills and speculative decoding |
| Qwen3‑Next             | Splits the in-projection so FP8 checkpoints load cleanly (vllm/model_executor/models/qwen3_next.py:284, 471)                                                                            | Separates QKVZ vs BA projections, matching blockwise FP8 quant requirements and avoiding unsupported merges                                      | Enables FP8 W8A8 checkpoints across FlashInfer / Triton paths                   |
| Qwen3‑Next             | Removes CUDA-only device assumptions (vllm/model_executor/models/qwen3_next.py:306)                                                                                                     | Lets the gated delta net initialize via the current platform hook instead of torch.cuda.current_device()                                         | Unlocks ROCm and future device backends without code forks                      |
| Qwen3‑Next MoE configs | Adds FP8 tuning files for GB200 / H100 / H200 (e.g. vllm/model_executor/layers/fused_moe/configs/E=512,N=128,device_name=NVIDIA_GB200,dtype=fp8_w8a8.json:1)                            | Ships per-topology block sizes and warp counts that match Alibaba’s published sweeps                                                             | Drop-in configs for Qwen3-Next MoE on Blackwell, Hopper, and Grace Hopper       |
| Qwen3 dense            | Dual chunk attention plumbing (vllm/model_executor/models/qwen3.py:118, 199)                                                                                                            | Exposes transformers’ dual-chunk cache so Qwen3 can serve ultra-long prompts without KV blow-up                                                  | Long-context serving (>128 K) on dense Qwen3 checkpoints                        |
| Qwen3‑MoE              | Built-in sequence-parallel dispatch (vllm/model_executor/models/qwen3_moe.py:139, 192)                                                                                                  | Uses the same chunk op to keep EP workloads strictly partitioned, avoiding duplicate expert calls                                                | TP × EP launches with DeepEP / allgather-reducescatter all-to-all               |
| Qwen3‑MoE              | Lets FusedMoE return reduced tensors directly (vllm/model_executor/models/qwen3_moe.py:167)                                                                                             | Eliminates a second tensor-parallel all-reduce at block exit, shaving latency on every layer                                                     | Any MoE run with TP>1, especially bandwidth-bound clusters                      |
| Qwen3‑MoE              | FP8 batched expert kernels (vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py:937, 991; vllm/model_executor/layers/quantization/fp8.py:1021)         | Chooses BatchedTriton experts and Cutlass FP8 paths automatically when FP8 W8A8 checkpoints are loaded                                           | Delivers FP8 throughput on Triton / Cutlass backends for 64–128 experts         |
| Qwen3‑MoE              | Enables 4-bit bitsandbytes pre-quant loading (vllm/model_executor/model_loader/bitsandbytes_loader.py:467)                                                                              | Lifts the restriction on pre-quantized BNB weights for MoE blocks, keeping only the 8-bit ban                                                    | Lets Qwen3-MoE serve 4-bit BNB “prequant” checkpoints alongside dense models    |

Highlights & context

- The TP×EP fix (sequence parallel chunk) now applies uniformly to Qwen3 Next and Qwen3 MoE, so hybrid attention and experts both avoid redundant dispatch when DeepEP or naive A2A is active.
- SharedFusedMoE carved the shared expert overlap logic out of the MoE block; if your deployment disables FlashInfer or EP, the shared MLP simply executes once, which also makes torch.compile tracing cleaner.
- Dual-chunk attention landed in the base Qwen3 dense model, aligning it with Alibaba’s long-context releases; no config change is needed beyond picking a checkpoint with dual_chunk_attention_config set.

Next steps

1. Benchmark FP8 and 4-bit BNB variants on the intended GPU SKU (GB200/H100/H200) to confirm the shipped configs hit expected throughput.
2. If you rely on custom parallel settings, double-check parallel_config.use_sequence_parallel_moe so the sequence-parallel path activates where you expect.



Evidence: when the team first tuned Qwen2’s 57 B MoE on A100 they landed on the JSON E=64,N=1280,device_name=NVIDIA_A100-SXM4-80GB.json, and the commit that introduced it measured a move from 10.53 req/s (11 058 tok/s) to 12.47 req/s (13 089 tok/s) at TP=2 (see commit bd43973522ea17be50e10fbb222a22f673c8067e). That improvement came solely from swapping the default heuristics with hand‑picked block and warp sizes for that GPU. The newer GB200 FP8 tables you’re looking at (E=512,N=128,...) were produced with the same profiling workflow, just targeting the Blackwell SM layout and FP8 balance – that’s why they use more aggressive BLOCK_SIZE_N and higher num_stages.




---

**Runtime Optimizations**

| Feature / Area                          | Commits (Date)                                                      | Optimization Highlights                                                                                                                                                      | Perf Notes                                                      |
| --------------------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| Qwen2/2.5-VL vision attention rearrange | 5c2acb27 (2025-10-18)                                               | Dropped redundant .contiguous() after rearrange so Q/K/V stay view-backed before rotary embeddings (qwen2_vl.py, qwen2_5_vl.py).                                             | Not reported; removes extra tensor copies.                      |
| Qwen3-next gated RMSNorm                | 82e64c7 (2025-10-12)                                                | Reworked Triton LayerNorm to tile multiple rows, cache SM count, and keep ops in fp32 while avoiding host transfers; added coverage tests (layernorm_guard.py).              | No metric; fewer kernel launches and better occupancy.          |
| Qwen3-next MTP bool-mask handling       | 785d8b6 (2025-10-16)                                                | Replaced boolean tensor indexing with index_select/index_copy in qwen3_next.py and GDN attention utils to eliminate device→host copies during multi-token prediction.        | Not stated; expected higher MTP throughput.                     |
| Qwen3-VL fast_pos_embed_interpolate     | 30d0891 (2025-09-21); af7dfb0 (2025-09-22); a6049be7 (2025-10-12)   | Successive vectorizations of the bilinear weight/index build, switching to meshgrids, tensor stacking, and summed reductions instead of Python loops in qwen3_vl.py.         | No metric; reduces Python overhead for large image/video grids. |
| Qwen3-VL multimodal tensor prep         | 0426e3c5 & 2c1c7dfb (2025-10-09)                                    | _validate_and_reshape_mm_tensor now reshapes instead of concatenating lists; cu_seqlens is built via torch.cat to avoid an extra padding kernel.                             | Not reported; fewer allocations/kernels.                        |
| Qwen3-VL deepstack splitting            | 1dfea5f4 (2025-09-19)                                               | Caches vision dims (visual_dim, multiscale_dim) and removes repeated .contiguous() when splitting deepstack embeddings, trimming bookkeeping in qwen3_vl.py/qwen3_vl_moe.py. | No metric; lighter host-side work.                              |
| Qwen3-VL interleaved MRoPE              | cea91a32 (2025-09-19)                                               | Added Triton kernel that handles interleaved temporal/spatial rotary sections, keeping rotations GPU-bound (mrope.py).                                                       | Not reported; avoids slow native fallback.                      |
| Qwen3Moe fused output                   | 4f510bc2 (2025-08-20)                                               | Lets experts perform the tensor-parallel all-reduce internally (reduce_results=True), removing a redundant explicit all-reduce in qwen3_moe.py.                              | Not stated; less collective traffic.                            |
| Qwen3-next FP8 checkpoints              | ef7eefe1 (2025-09-18)                                               | Split merged linear projection into FP8-compatible pieces (in_proj_qkvz/in_proj_ba) so blockwise FP8 weights load correctly in qwen3_next.py.                                | No metric; enables FP8 inference path.                          |
| Qwen3 FP8 accuracy guard                | a258ad8b (2025-08-17)                                               | Adjusted FP8 quantization helpers to address accuracy drift for Qwen3 MoE (quantization/fp8.py).                                                                             | Not reported; quality fix to keep FP8 speed viable.             |
| Qwen3 fused RMSNorm                     | f80ae5bd (2025-05-07)                                               | Swapped attention Q/K norm calls to the fused RMSNorm custom op across Qwen3 dense and MoE models.                                                                           | No metric; fewer RMSNorm kernels.                               |
| Qwen3 reasoning parser                  | 015069b0 (2025-05-01)                                               | Streamlined reasoning text extraction to avoid repeated string splits in qwen3_reasoning_parser.py.                                                                          | Not measured; faster host parsing.                              |
| Qwen2.5-VL rotary/window pipeline       | 67da5720 (2025-05-16); e283976f (2025-09-09)                        | Recomputed rotary embeddings per (t,h,w) on device, pre-built window indices, and replaced torch.argsort with an O(n) inverse permutation in qwen2_5_vl.py.                  | No metrics; eliminates repeated cudaMemcpy and sorting.         |
| Qwen2/2.5-VL CUDA sync avoidance        | 6772bb0f (2025-08-13); 60f0843e (2025-09-08)                        | Convert grid products to Python lists before splitting embeddings to avoid implicit GPU syncs in qwen2_vl.py/qwen2_5_vl.py.                                                  | Not reported; prevents blocking host waits.                     |
| Qwen2.5-VL normalization & SDPA         | 02ed8a1f (2025-02-13); cbc8457b (2025-08-07)                        | Adopted shared RMSNorm, rewrote SDPA path to operate per-window to cut VRAM, and routed to fused RMSNorm in qwen2_5_vl.py.                                                   | No metric; more efficient norm/attention.                       |
| Qwen2/2.5-VL init & DP                  | 1298c677 (2025-08-19); 2c5302fa (2025-06-21); d49adea1 (2025-06-18) | Enabled vision-tower data parallel mode, cached profiling inputs, and switched to fast HF processors to shrink startup latency.                                              | Not reported; faster init and lower memory.                     |
| Qwen2/2.5-VL attention masks            | 47c7126 (2025-03-21)                                                | Restored attention mask precomputation for mixed window/full attention paths.                                                                                                | No metrics; reduces per-forward recompute.                      |
| Qwen2-VL cudaMemcpy reduction           | 70b808fe (2025-03-11)                                               | Builds seqlens/window metadata on CPU and reuses them to limit GPU copies during vision attention.                                                                           | Not reported; lowers cudaMemcpyAsync volume.                    |
| Qwen2 FP8 KV cache                      | da971ec7 (2024-06-19)                                               | Added FP8 KV-cache storage option in qwen2.py to shrink cache footprint/bandwidth.                                                                                           | No metrics; expected memory savings.                            |
| Qwen2 pipeline parallel                 | 1d2e7fb7 (2024-08-01)                                               | Implemented stage splits and related plumbing so Qwen2/Qwen2Moe can pipeline across GPUs.                                                                                    | No metric; enables scale-out throughput.                        |
| Qwen LoRA punica specialization         | 1f567421 (2024-06-21); 8435b207 (2024-05-17)                        | Extended Punica BGMV configs for Qwen2 and Qwen1.5 LoRA ranks to avoid generic kernels.                                                                                      | No metrics; better LoRA throughput.                             |

**Kernel Config Tuning (Fused MoE / FP8)**

|Commits (Date)|Model / HW Target|Optimization|Perf Metrics|
|---|---|---|---|
|4d0f2661 (2025-10-20)|Qwen3-30B A3/A3B on H100 (FP8 & BF16)|Added tuned Triton fused_moe configs (block shape [128,128]) for both dtypes.|Not reported.|
|f96bc364 (2025-10-15)|Qwen3-Next FP8 on H100 TP=2|Introduced TP2-specific FP8 fused_moe config.|Not reported.|
|238c4c17 (2025-09-15); 7533495 (2025-09-06); 482e52f (2025-09-04); 62f66be (2025-09-07)|Qwen3 Next / Thinking / Coder across NVIDIA GB200, B200, H200, H100|New and corrected fused_moe tuning tables for multiple expert counts; fixed Qwen3-coder block shapes and benchmarking coverage.|Not reported.|
|8ed01e32 (2025-07-25); 04ff4be3 (2025-07-28)|Qwen3-Coder-480B-A35B on NVIDIA H20-3e|Added FP8 fused_moe configs for large coder variant.|Not reported.|
|2d40665 (2025-06-11)|Qwen3-30B A3B on NVIDIA B200|Introduced B200-specific FP8 fused_moe config.|Not reported.|
|22c3c0aa (2025-06-11); 989dcee (2025-06-08); 597051e (2025-05-08)|Qwen3-235B A22B on NVIDIA H20-3e & A100|Added FP8/BF16 fused_moe configs for new hardware (H20-3e, A100).|Not reported.|
|8fc88d63 (2025-04-28)|Qwen3 MoE (H100/H200/H20 targets)|Bulk addition of tuned fused_moe configs for E=128 across multiple expert counts; updated benchmarks README.|Not reported.|
|dcbac4cb (2025-04-28)|Qwen3 dense FP8|Adjusted linear layers so FP8 compatibility works with fused kernels.|Not reported.|
|2007d4d5 & f5a3c655 (2025-05-01)|Qwen3-30B A3B / Qwen3-235B A22B FP8 on AMD MI300X|Added ROCm-specific fused_moe configs for MI300X (block shape [128,128]).|Not reported.|
|bd439735 (2024-06-14)|Qwen2-57B-A14B (TP2/TP4)|Tuned fused_moe configs for A100/H100; benchmarks show +18–20% requests/s (10.53→12.47 @TP2, 17.77→20.20 @TP4) and +18–14% tokens/s.|Throughput gains published in commit message.|

Next steps (optional):

1. Run targeted throughput tests on the most recent Qwen3-next and Qwen3-VL changes to quantify wins that weren’t reported.
2. For hardware configs, validate them on the intended devices (H100/B200/H20-3e/MI300X) to confirm stability before baking into release builds.



Model optimization methods:
- 减少 host/device 之间的 copy
- 优化 fused_moe configs，让 Triton 算子在特定 E/N/device/quantization 下更高效
- 提高 CPU/GPU 之间的 overlap



---
## notes

Mingxing Zhang:

一个 agent 的不同阶段可能业务需求就不一样，业务需求不同，需要不同 parallelism

value = function / cost
GPU 利用率已经 70~80%，单一场景空间继续提升很困难，提升多场景下，不同 function 使用不同的 cost，value 最大化


mooncake 适配 vllm 和 sglang 时，工程方法不一样，需要不同的做法，需要分别优化适配推理框架


---
## Model Optimization Summary

### Kernel

fused_moe/configs/xxx 针对不同的 expert 数量、N、device、quant 给定配置文件指导 Triton 生成算子，保证 SM 的利用率、且 share memory、registers 不爆掉

GDNAttention for Qwen3-Next

### Data Movement



---
Hybrid Qwen3-Next support in vLLM centers on a Gated DeltaNet (GDN) linear-attention stack plus shared MoE and multi-token prediction so the runtime matches the checkpoint’s architecture while staying compatible with speculative decoding and parallelism features.

|Optimization|Match-to-model feature|Performance effect|Evidence/data|
|---|---|---|---|
|Gated DeltaNet linear-attention backend with fused gating kernels|Custom Qwen3NextGatedDeltaNet layer drives causal-conv + recurrent updates, registered as a native op and backed by a dedicated metadata builder to interop with speculative decoding and CUDA graphs|Converts most layers to linear-time recurrence while keeping full-attention layers where needed; fused gating reduces extra projections; speculative metadata avoids recompute during draft acceptance|Implementation in vllm/model_executor/models/qwen3_next.py:206, custom op + kernel at vllm/model_executor/models/qwen3_next.py:1266 and 1292, backend/metadata in vllm/v1/attention/backends/gdn_attn.py:22 and 61, runtime wiring for spec decode in vllm/v1/worker/gpu_model_runner.py:1374; no benchmark numbers checked into repo|
|Shared Fused MoE with expert parallel load balancing|Qwen3NextSparseMoeBlock wires shared experts, redundant experts, and EPLB-aware routing via SharedFusedMoE and runtime hooks for updating expert maps|Maintains Qwen3 Next’s shared + routed expert mix while overlapping communication and balancing per-expert load across EP/SP, reducing straggler cost|Implementation in vllm/model_executor/models/qwen3_next.py:99; EPLB state management at vllm/model_executor/models/qwen3_next.py:1168; no MoE benchmark data present|
|NextN multi-token predictor (MTP) path|Qwen3NextMultiTokenPredictor reuses decoder layers for draft tokens and config overrides remap the HF config into the MTP runner|Enables draft-token speculation without loading a separate model, improving decode throughput when speculative decoding is enabled|MTP module in vllm/model_executor/models/qwen3_next_mtp.py:43; config remapping in vllm/config/speculative.py:202; no throughput figures published|
|Mamba-style state management for speculative decode|Inherits from MambaBase, provides dtype/shape calculators, and relaxes the default “no speculative” guard specifically for Qwen3 Next|Keeps convolution + SSM state caches in an efficient layout and allows speculative decoding—otherwise disallowed for generic Mamba models|State helpers in vllm/model_executor/models/qwen3_next.py:1218; speculative allowance in vllm/model_executor/layers/mamba/abstract.py:50; no direct measurements supplied|

No repository benchmarks quantify the above: benchmarks/ scripts cover generic MoE kernels but do not include recorded Qwen3-Next runs.

Next steps (optional): 1) run benchmarks/kernels/benchmark_moe.py or a decode throughput script with your Qwen3-Next checkpoint to obtain empirical numbers; 2) capture profiling traces to confirm the GDN layers hit the fused kernels.


引入 PD 分离后，EPxDP 时 调度器 影响更大，在线的并行模式调整，可能不如调度器的调整，让每个 rank 收到的 pattern 相对固定

