# 现有工作总结

## Insights

- 层内 expert 具有 load imbalance 的特性
- 层间的 expert 激活具有可预测性
- 单个请求在 decoding 过程中的 expert 激活具有 skewness，少部分 expert 会在 decoding 过程中被高频选择

## expert prefetch 与 cache

![[250624-194444.png]]
- iteration-wise (token)：比较 input tokens 的相似性，根据历史记录进行预测（原理：相似的 input tokens 具有相似的 embedding，经过 gate 也应该会得到相似的 expert 选择）
- layer-wise (skip)：相邻层的 embedding 变化不大，用第 i 层的 expert 预测第 i+1 层的 expert
![[250624-195519.png]]

- 单个请求具有 skewness，多个请求的 statistics 会相对比较均衡
- 单个 sequence 的 decoding 过程具有 skewness，< 5% 的 experts 会高频 activate
![[250627-111237.png]]

## expert placement 优化

$x_{l, e, g} \in \{0, 1\}$，表述第 $l$ 层的 expert $e$ 是否放在 GPU $g$ 上，目标：
1. load balance：对任意的 $l_0$，让 $L(g) = \sum_{e} x_{l, e, g} \cdot N_{e}$ 的标准差最小
2. communication：$\sum x_{l, e_1, g_1} \cdot x_{l+1, e_2, g_2} \cdot N_{e_1, e_2}$ 最小


## 集合通信优化








---
# Paper list

#### [fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving](https://arxiv.org/abs/2502.05370)

#### [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444)

- 重算 10%-20% HKVD tokens 就足够把与 $A_{full}$ 的 deviation 降到很低
- 不同 layer 之间的 HKVD tokens 相似 -> 第一层完全重算，计算 deviation 得到 HKVD tokens，在第二层对这些 tokens 重算，第二层又可计算 deviation 得到 HKVD tokens 用于下一层，以此类推
![[250611-091821.jpeg]]
- MoE router 时计算 experts 的概率的 input 为 attention 的 embedding，那么是否相邻层的 embedding 相同，会使得相邻层的 activation pattern 具有相似性？

#### [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://arxiv.org/abs/2401.06066)

#### [ProMoE: Fast MoE-based LLM Serving using Proactive Caching](https://arxiv.org/abs/2410.22134)

- ploys a small neural network to learn correlations between layer inputs and expert selections

#### [Accelerating Distributed MoE Training and Inference with Lina](https://arxiv.org/abs/2210.17223)

#### [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://arxiv.org/abs/2201.05596)

#### [DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency](https://arxiv.org/abs/2408.00741)

#### [HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference](https://arxiv.org/abs/2411.01433)

#### [Optimizing Distributed Deployment of Mixture-of-Experts Model Inference in Serverless Computing](https://arxiv.org/abs/2501.05313)

Assume $f$ is the token feature vector of a token, in which $f_1$ is the token ID, $f_2$ is the position ID and $f_3$ is the attention ID.
![[250701-102443.png]]

#### [SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models](# SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models)

#### [MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache](https://arxiv.org/abs/2401.14361v3)

#### [MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing](https://arxiv.org/abs/2502.06643v1)

- token paths across layers are not random but instead follow structured and predictable patterns
- The primary goal of MOETUNER is to develop an expert placement strategy that minimizes two critical factors: the imbalance of token processing load across GPUs and the inter-GPU communication overhead.
- 统计前一层激活的 expert $E_{l, i}$ 和后一层激活的 expert $E_{l+1, j}$ 的 pair $\langle E_{l, i}, E_{l+1, j} \rangle$
- 该工作没有考虑动态负载的变化

#### [Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models with Adaptive Expert Placement](https://arxiv.org/abs/2407.04656)

- Adaptive Expert Allocation
- Expert Placement Algorithm
- Flexible Token Dispatcher

#### [Accelerating Mixture-of-Experts Training with Adaptive Expert Replication](https://arxiv.org/abs/2504.19925)


#### [ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling]

- allows the communication and computation tasks in training MoE models to be scheduled in an optimal way
- all-to-all collective which better utilizes intra- and inter-connect bandwidths
- supports easy extensions of customized all-to-all collectives and data compression approaches

#### [Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing](https://arxiv.org/abs/2404.16914)

- defined the transient state with “obvious load fluctuation” and the stable state with “temporal locality” (the loads of each expert are similar in adjacent iterations)



- [ ] 总结百炼的架构，分析整体系统有什么可能可以做的
- [ ] expert scaling
	前提：流量 spike，模型切换（数量？）需要启动，模型为 MoE 模型
- [ ] train 和 inference 的 A2A 的区别？和 deepep 的区别？
- [ ] 容错问题？本质 GPU 数量增多，更容易挂，EP 是不是最容易容错的？kvcache 的容错
	- [ ] failure 的概率？
	- [ ] replication 除了容错，能服务什么



https://github.com/withinmiaov/A-Survey-on-Mixture-of-Experts-in-LLMs