--- title: "大模型时代的混合专家系统优化综述 — 史宏志 2025" description: "Mixture of experts (MoE) has become a leading architecture for large models." authors: ["史宏志", "赵健", "赵雅倩", "李茹杨", "魏辉", "胡克坤", "温东超", "金良"] year: 2025 venue: "计算机研究与发展 (Journal of Computer Research and Development)" source: "https://crad.ict.ac.cn/article/doi/10.7544/issn1000-1239.202440016" doi: "10.7544/issn1000-1239.202440016" licence: "Free access (crad.ict.ac.cn), no open licence stated — readonly; © 计算机研究与发展" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/shi-2025-system-optimization-mixture/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/shi-2025-system-optimization-mixture/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2025](https://md.research.elamur.ai/free/papers/2025/page/3/) / 大模型时代的混合专家系统优化综述 # 大模型时代的混合专家系统优化综述 **Authors:** 史宏志, 赵健, 赵雅倩, 李茹杨, 魏辉, 胡克坤, 温东超, 金良 · **Year:** 2025 · **Venue:** 计算机研究与发展 (Journal of Computer Research and Development) · **Licence:** Free access (crad.ict.ac.cn), no open licence stated — readonly; © 计算机研究与发展 **Links:** [Publisher's page](https://doi.org/10.7544/issn1000-1239.202440016) · [PDF](https://crad.ict.ac.cn/cn/article/pdf/preview/10.7544/issn1000-1239.202440016.pdf) · [DOI 10.7544/issn1000-1239.202440016](https://doi.org/10.7544/issn1000-1239.202440016) ## Summary Mixture of experts (MoE) has become a leading architecture for large models. Because only a few experts are activated per input, parameter counts can grow while compute rises only sublinearly. At scale, however, MoE systems face four bottlenecks: - heavy memory demand; - costly All-to-All exchanges between devices; - wasted computation from zero-padding and uneven expert load; - difficulty picking a parallel strategy that suits constantly shifting workloads. This Chinese-language survey, by authors at IEIT SYSTEMS/Inspur, collects recent work on these system-side problems. It first explains core MoE ideas: Top-k gating, expert capacity, expert parallelism, and how MoE differs from classic ensemble learning. It then traces model development from the sparsely gated LSTM-based MoE through GShard, Switch Transformers, Mixtral-8x7B and DeepSeekMoE. Next, it sorts optimization techniques into four groups: memory, communication, computation and parallel scaling. For each, it compares the main ideas, suitable scenarios and remaining weaknesses, and it ends with future research directions. No new dataset or model is introduced. Its value is that, by the authors' account, it is the first review focused on MoE systems engineering rather than algorithms. ## Key points - **Taxonomy of techniques:** - *Memory:* offloading to CPU memory/SSD; parameter compression. - *Communication:* hierarchical communication; redundancy elimination; topology-aware routing. - *Computation:* dynamic capacity; expert/device load balancing; statistics- and ML-based expert activation prediction; static vs. adaptive pipelining; kernel optimization. - *Parallel scaling:* static vs. adaptive parallelism. - **Memory gap:** top-end GPU memory stayed at 80 GB during 2021–2023. Switch Transformers already had 1.6 trillion parameters in 2021. - **GLaM:** 1.2T parameters, about 7× GPT-3, yet needs roughly a third of its compute and outperforms GPT-3 on 29 NLP benchmarks. - **GShard:** introduced expert capacity and expert parallelism, and built a 600B-parameter MoE Transformer for translation. - **Source:** Journal of Computer Research and Development 62(5):1164–1189, 2025; DOI 10.7544/issn1000-1239.202440016. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{2025, title = {{大模型时代的混合专家系统优化综述}}, author = {史宏志 and 赵健 and 赵雅倩 and 李茹杨 and 魏辉 and 胡克坤 and 温东超 and 金良}, journal = {计算机研究与发展 (Journal of Computer Research and Development)}, year = {2025}, doi = {10.7544/issn1000-1239.202440016}, url = {https://doi.org/10.7544/issn1000-1239.202440016} } ``` - [← previous 基于词性对齐与依存关系的中文排比句生成方法](https://md.research.elamur.ai/free/papers/zhong-2025-chinese-parallelism-generation/) - [next → 大语言模型和知识图谱协同的跨域异质数据查询框架](https://md.research.elamur.ai/free/papers/wu-2025-synergetic-llm-kg/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/papers/2025/page/3/) ## References 6 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. experts[J]. arXiv preprint, arXiv: 2407.06204, 2024 — [arXiv:2407.06204](https://arxiv.org/abs/2407.06204) [find](https://scholar.google.com/scholar?q=experts%5BJ%5D%2E+arXiv+preprint%2C+arXiv%3A+2407%2E06204%2C+2024+%E2%80%94+%5BarXiv%3A2407%2E06204%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2407%2E06204%29) 2. ［36］ Lepikhin D, Lee H, Xu Yuandong, et al. GShard: Scaling giant models with conditional computation and automatic sharding [J]. arXiv preprint, arXiv: 2006.16668, 2020 [find](https://scholar.google.com/scholar?q=%EF%BC%BB36%EF%BC%BD+Lepikhin+D%2C+Lee+H%2C+Xu+Yuandong%2C+et+al%2E+GShard%3A+Scaling+giant+models+with+conditional+computation+and+automatic+sharding+%5BJ%5D%2E+arXiv+preprint%2C+arXiv%3A+2006%2E16668%2C+2020) 3. ［37］ Zhang Zhengyan, Lin Yankai, Liu Zhiyuan, et al. MoEfication: Transformer feed-forward layers are mixtures of experts[J]. arXiv preprint, arXiv: 2110.01786, 2021 [find](https://scholar.google.com/scholar?q=%EF%BC%BB37%EF%BC%BD+Zhang+Zhengyan%2C+Lin+Yankai%2C+Liu+Zhiyuan%2C+et+al%2E+MoEfication%3A+Transformer+feed%2Dforward+layers+are+mixtures+of+experts%5BJ%5D%2E+arXiv+preprint%2C+arXiv%3A+2110%2E01786%2C+2021) 4. ［38］ Zuo Simiao, Zhang Qingru, Liang Chen, et al. MoEBERT: From BERT to mixture-of-experts via importance-guided adaptation[J]. arXiv preprint, arXiv: 2204.07675, 2022 [find](https://scholar.google.com/scholar?q=%EF%BC%BB38%EF%BC%BD+Zuo+Simiao%2C+Zhang+Qingru%2C+Liang+Chen%2C+et+al%2E+MoEBERT%3A+From+BERT+to+mixture%2Dof%2Dexperts+via+importance%2Dguided+adaptation%5BJ%5D%2E+arXiv+preprint%2C+arXiv%3A+2204%2E07675%2C+2022) 5. ［39］ Zhu Tong, Qu Xiaoye, Dong Daize, et al. LLaMA-MoE: Building mixture-of-experts from LLaMA with continual pre-training[J]. arXiv preprint, arXiv: 2406.16554, 2024 [find](https://scholar.google.com/scholar?q=%EF%BC%BB39%EF%BC%BD+Zhu+Tong%2C+Qu+Xiaoye%2C+Dong+Daize%2C+et+al%2E+LLaMA%2DMoE%3A+Building+mixture%2Dof%2Dexperts+from+LLaMA+with+continual+pre%2Dtraining%5BJ%5D%2E+arXiv+preprint%2C+arXiv%3A+2406%2E16554%2C+2024) 6. ［40］ Dai Damai, Deng Chenqi, Zhao Chenggang, et al. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models[J]. arXiv preprint, arXiv: 2401.06066, 2024 [find](https://scholar.google.com/scholar?q=%EF%BC%BB40%EF%BC%BD+Dai+Damai%2C+Deng+Chenqi%2C+Zhao+Chenggang%2C+et+al%2E+DeepSeekMoE%3A+Towards+ultimate+expert+specialization+in+mixture%2Dof%2Dexperts+language+models%5BJ%5D%2E+arXiv+preprint%2C+arXiv%3A+2401%2E06066%2C+2024)