---
title: "Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs"
description: "Training MoE language models with nearly a trillion parameters is hard: the hardware is underused… 63 references, cited by 8 papers here."
authors: ["Yehui Tang", "Yichun Yin", "Yaoyuan Wang", "Hang Zhou", "Yu Pan", "Wei Guo", "Ziyang Zhang", "Miao Rang", "Fangcheng Liu", "Naifu Zhang", "Binghan Li", "Yonghan Dong", "Xiaojun Meng", "Yasheng Wang", "Dong Li", "Yin Li", "Dandan Tu", "Can Chen", "Youliang Yan", "Fisher Yu", "Ruiming Tang", "Yunhe Wang", "Botian Huang", "Bo Wang", "Boxiao Liu", "Changzheng Zhang", "Da Kuang", "Fei Liu", "Gang Huang", "Jiansheng Wei", "Jiarui Qin", "Jie Ran", "Jinpeng Li", "Jun Zhao", "Liang Dai", "Lin Li", "Liqun Deng", "Peifeng Qin", "Pengyuan Zeng", "Qiang Gu", "Shaohua Tang", "Shengjun Cheng", "Tao Gao", "Tao Yu", "Tianshu Li", "Tianyu Bi", "Wei He", "Weikai Mao", "Wenyong Huang", "Wulong Liu", "Xiabing Li", "Xianzhi Yu", "Xueyu Wu", "Xu He", "Yangkai Du", "Yan Xu", "Ye Tian", "Yimeng Wu", "Yongbing Huang", "Yong Tian", "Yong Zhu", "Yue Li", "Yufei Wang", "Yuhang Gai", "Yujun Li", "Yu Luo", "Yunsheng Ni", "Yusen Sun", "Zelin Chen", "Zhe Liu", "Zhicheng Liu", "Zhipeng Tu", "Zilin Ding", "Zongyuan Zhan"]
year: 2025
venue: "arXiv:2505.04519"
source: "https://arxiv.org/abs/2505.04519"
doi: "10.48550/arxiv.2505.04519"
arxiv: "2505.04519"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tang-2025-pangu-ultra-moe-train/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tang-2025-pangu-ultra-moe-train/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) / Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs

# Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs

**Authors:** [Yehui Tang](https://md.research.elamur.ai/free/en/authors/yehui-tang/), [Yichun Yin](https://md.research.elamur.ai/free/en/authors/yichun-yin/), [Yaoyuan Wang](https://md.research.elamur.ai/free/en/authors/yaoyuan-wang/), [Hang Zhou](https://md.research.elamur.ai/free/en/authors/hang-zhou/), [Yu Pan](https://md.research.elamur.ai/free/en/authors/yu-pan/), [Wei Guo](https://md.research.elamur.ai/free/en/authors/wei-guo/), Ziyang Zhang, Miao Rang, [Fangcheng Liu](https://md.research.elamur.ai/free/en/authors/fangcheng-liu/), Naifu Zhang, Binghan Li, Yonghan Dong, [Xiaojun Meng](https://md.research.elamur.ai/free/en/authors/xiaojun-meng/), [Yasheng Wang](https://md.research.elamur.ai/free/en/authors/yasheng-wang/), [Dong Li](https://md.research.elamur.ai/free/en/authors/dong-li/), Yin Li, [Dandan Tu](https://md.research.elamur.ai/free/en/authors/dandan-tu/), [Can Chen](https://md.research.elamur.ai/free/en/authors/can-chen/), Youliang Yan, Fisher Yu, [Ruiming Tang](https://md.research.elamur.ai/free/en/authors/ruiming-tang/), [Yunhe Wang](https://md.research.elamur.ai/free/en/authors/yunhe-wang/), Botian Huang, [Bo Wang](https://md.research.elamur.ai/free/en/authors/bo-wang/), Boxiao Liu, Changzheng Zhang, Da Kuang, [Fei Liu](https://md.research.elamur.ai/free/en/authors/fei-liu/), Gang Huang, [Jiansheng Wei](https://md.research.elamur.ai/free/en/authors/jiansheng-wei/), Jiarui Qin, Jie Ran, [Jinpeng Li](https://md.research.elamur.ai/free/en/authors/jinpeng-li/), [Jun Zhao](https://md.research.elamur.ai/free/en/authors/jun-zhao/), Liang Dai, [Lin Li](https://md.research.elamur.ai/free/en/authors/lin-li/), Liqun Deng, Peifeng Qin, Pengyuan Zeng, Qiang Gu, Shaohua Tang, Shengjun Cheng, Tao Gao, [Tao Yu](https://md.research.elamur.ai/free/en/authors/tao-yu/), Tianshu Li, Tianyu Bi, [Wei He](https://md.research.elamur.ai/free/en/authors/wei-he/), Weikai Mao, [Wenyong Huang](https://md.research.elamur.ai/free/en/authors/wenyong-huang/), Wulong Liu, Xiabing Li, [Xianzhi Yu](https://md.research.elamur.ai/free/en/authors/xianzhi-yu/), Xueyu Wu, Xu He, Yangkai Du, [Yan Xu](https://md.research.elamur.ai/free/en/authors/yan-xu/), [Ye Tian](https://md.research.elamur.ai/free/en/authors/ye-tian/), [Yimeng Wu](https://md.research.elamur.ai/free/en/authors/yimeng-wu/), Yongbing Huang, Yong Tian, Yong Zhu, Yue Li, [Yufei Wang](https://md.research.elamur.ai/free/en/authors/yufei-wang/), Yuhang Gai, Yujun Li, Yu Luo, Yunsheng Ni, Yusen Sun, Zelin Chen, Zhe Liu, [Zhicheng Liu](https://md.research.elamur.ai/free/en/authors/zhicheng-liu/), Zhipeng Tu, Zilin Ding and Zongyuan Zhan · **Year:** 2025 · **Venue:** arXiv:2505.04519 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2505.04519) · [PDF](https://arxiv.org/pdf/2505.04519) · [DOI 10.48550/arxiv.2505.04519](https://doi.org/10.48550/arxiv.2505.04519) · [arXiv 2505.04519](https://arxiv.org/abs/2505.04519) · [OpenAlex](https://openalex.org/W4416014127)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/tang-2025-pangu-ultra-moe-train/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training MoE language models with nearly a trillion parameters is hard: the hardware is underused because the set of experts each token activates changes at runtime, and the scale strains communication and memory. This report from Huawei's Pangu Team presents a recipe for such training on Ascend NPUs. On the modelling side, small pilot runs settled the expert count and the use of a shared expert. A simulator, calibrated bottom-up from operator measurements on Ascend hardware, then scored candidate architectures without costly full trials. This process produced Pangu Ultra MoE, a sparse model with 718B parameters. On the systems side, the authors restructured Expert Parallelism communication, overlapped it with computation, cut memory pressure, and balanced expert load across devices. Training on 6K Ascend NPUs reached 30.0% MFU, and the resulting model performs on par with DeepSeek R1, with particular strength on medical benchmarks. The work matters because it shows that frontier-scale sparse LLM training can run end to end on non-NVIDIA hardware.

## Key points
- **Architecture choices:** Pilot ablations used a ~20B-parameter MoE with 1.6B parameters active per token.
  - 256 experts beat 64 or 128; going to 512 added little.
  - A shared expert gave lower loss than an equal-compute variant without one.
  - The final model routes each token to 8 of 256 experts per layer.
- **Simulation-based search:** Analytical pruning, including a depth-to-width scaling fit, reduced the design space to about 10,000 configurations. These were then ranked by simulated throughput on Ascend NPUs.
- **System techniques:**
  - Hierarchical EP All-to-All Communication handles inter-node and intra-node traffic separately.
  - Adaptive Pipe Overlap hides communication behind computation.
  - Fine-grained per-operator recomputation and tensor swapping to host memory reduce NPU memory use.
  - Dynamic device-level load balancing predicts expert load and places experts adaptively.
- **Efficiency gains:** Throughput rose from a baseline of 18.9% MFU and 0.61M TPS (on 4K NPUs) to 30.0% MFU and 1.46M TPS (on 6K NPUs).
- **Model behaviour analysis:** Expert routing was examined on C4, MMLU, C-Eval, Math and LiveCodeBench.
  - Experts specialize more in deeper layers (layers 3, 30 and 60 compared).
  - Routed and shared experts contribute output of similar magnitude.
  - Experts rarely co-activate, which suggests low redundancy among them.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 81 works

## Cite

Cite as: Yehui Tang, Yichun Yin, Yaoyuan Wang, Hang Zhou, Yu Pan, Wei Guo, Ziyang Zhang, Miao Rang, Fangcheng Liu, Naifu Zhang, Binghan Li, Yonghan Dong, Xiaojun Meng, Yasheng Wang, Dong Li, Yin Li, Dandan Tu, Can Chen, Youliang Yan, Fisher Yu, Ruiming Tang, Yunhe Wang, Botian Huang, Bo Wang, Boxiao Liu, Changzheng Zhang, Da Kuang, Fei Liu, Gang Huang, Jiansheng Wei, Jiarui Qin, Jie Ran, Jinpeng Li, Jun Zhao, Liang Dai, Lin Li, Liqun Deng, Peifeng Qin, Pengyuan Zeng, Qiang Gu, Shaohua Tang, Shengjun Cheng, Tao Gao, Tao Yu, Tianshu Li, Tianyu Bi, Wei He, Weikai Mao, Wenyong Huang, Wulong Liu, Xiabing Li, Xianzhi Yu, Xueyu Wu, Xu He, Yangkai Du, Yan Xu, Ye Tian, Yimeng Wu, Yongbing Huang, Yong Tian, Yong Zhu, Yue Li, Yufei Wang, Yuhang Gai, Yujun Li, Yu Luo, Yunsheng Ni, Yusen Sun, Zelin Chen, Zhe Liu, Zhicheng Liu, Zhipeng Tu, Zilin Ding and Zongyuan Zhan. 2025. Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs. arXiv:2505.04519.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tang2025pangu,
  title = {{Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs}},
  author = {Yehui Tang, Yichun Yin, Yaoyuan Wang, Hang Zhou, Yu Pan, Wei Guo, Ziyang Zhang, Miao Rang, Fangcheng Liu, Naifu Zhang, Binghan Li, Yonghan Dong, Xiaojun Meng, Yasheng Wang, Dong Li, Yin Li, Dandan Tu, Can Chen, Youliang Yan, Fisher Yu, Ruiming Tang, Yunhe Wang, Botian Huang, Bo Wang, Boxiao Liu, Changzheng Zhang, Da Kuang, Fei Liu, Gang Huang, Jiansheng Wei, Jiarui Qin, Jie Ran, Jinpeng Li, Jun Zhao, Liang Dai, Lin Li, Liqun Deng, Peifeng Qin, Pengyuan Zeng, Qiang Gu, Shaohua Tang, Shengjun Cheng, Tao Gao, Tao Yu, Tianshu Li, Tianyu Bi, Wei He, Weikai Mao, Wenyong Huang, Wulong Liu, Xiabing Li, Xianzhi Yu, Xueyu Wu, Xu He, Yangkai Du, Yan Xu, Ye Tian, Yimeng Wu, Yongbing Huang, Yong Tian, Yong Zhu, Yue Li, Yufei Wang, Yuhang Gai, Yujun Li, Yu Luo, Yunsheng Ni, Yusen Sun, Zelin Chen, Zhe Liu, Zhicheng Liu, Zhipeng Tu, Zilin Ding and Zongyuan Zhan},
  year = {2025},
  doi = {10.48550/arxiv.2505.04519},
  eprint = {2505.04519},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.04519}
}
```

- [← previous PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier](https://md.research.elamur.ai/free/en/papers/jiang-2025-pag-multi-turn-reinforced-llm/)
- [next → Piecing It All Together: Verifying Multi-Hop Multimodal Claims](https://md.research.elamur.ai/free/en/papers/wang-2025-piecing-it-all-together-verifying/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/)

## References

63 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. CoC (Communication Over Computation). [https://gitee.com/ascend/MindSpeed-LLM/blob/master/docs/features/communication-over-computation.md](https://gitee.com/ascend/MindSpeed-LLM/blob/master/docs/features/communication-over-computation.md). [↗](https://gitee.com/ascend/MindSpeed-LLM/blob/master/docs/features/communication-over-computation.md)
2. Megatron-LM. [https://github.com/NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM). [↗](https://github.com/NVIDIA/Megatron-LM)
3. MindSpeed. [https://gitee.com/ascend/MindSpeed](https://gitee.com/ascend/MindSpeed). [↗](https://gitee.com/ascend/MindSpeed)
4. Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025. [find](https://scholar.google.com/scholar?q=Meta+AI%2E+The+llama+4+herd%3A+The+beginning+of+a+new+era+of+natively+multimodal+ai+innovation%2C+2025%2E)
5. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/), 2023.
6. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
7. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, 2016. arXiv preprint arXiv:1604.06174, 2016. [find](https://scholar.google.com/scholar?q=Tianqi+Chen%2C+Bing+Xu%2C+Chiyuan+Zhang%2C+and+Carlos+Guestrin%2E+Training+deep+nets+with+sublinear+memory+cost%2C+2016%2E+arXiv+preprint+arXiv%3A1604%2E06174%2C+2016%2E)
8. Peizhuang Cong, Aomufei Yuan, Shimao Chen, Yuxuan Tian, Bowen Ye, and Tong Yang. Prediction is all moe needs: Expert load distribution goes from fluctuating to stabilizing. arXiv preprint arXiv:2404.16914, 2024. — [arXiv:2404.16914](https://arxiv.org/abs/2404.16914) [find](https://scholar.google.com/scholar?q=Peizhuang+Cong%2C+Aomufei+Yuan%2C+Shimao+Chen%2C+Yuxuan+Tian%2C+Bowen+Ye%2C+and+Tong+Yang%2E+Prediction+is+all+moe+needs%3A+Expert+load+distribution+goes+from+fluctuating+to+stabilizing%2E+arXiv+preprint+arXiv%3A2404%2E1)
9. Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.… [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Chengqi+Deng%2C+Chenggang+Zhao%2C+R%2E+X%2E+Xu%2C+Huazuo+Gao%2C+Deli+Chen%2C+Jiashi+Li%2C+Wangding+Zeng%2C+Xingkai+Yu%2C+Y%2E+Wu%2C+Zhenda+Xie%2C+Y%2E+K%2E+Li%2C+Panpan+Huang%2C+Fuli+Luo%2C+Chong+Ruan%2C+Zhifang+Sui%2C+and+Wenfen)
10. DeepSeek-AI. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/), 2025.
11. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Journal+of+Machine+Learning+Research%2C+23%28120%29%3A1%E2%80%9339%2C+2022%2E)
12. Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems, 5:288–304, 2023. [find](https://scholar.google.com/scholar?q=Trevor+Gale%2C+Deepak+Narayanan%2C+Cliff+Young%2C+and+Matei+Zaharia%2E+Megablocks%3A+Efficient+sparse+training+with+mixture%2Dof%2Dexperts%2E+Proceedings+of+Machine+Learning+and+Systems%2C+5%3A288%E2%80%93304%2C+2023%2E)
13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
14. Chien-Chin Huang, Gu Jin, and Jinyang Li. Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 1341–1355, 2020. [find](https://scholar.google.com/scholar?q=Chien%2DChin+Huang%2C+Gu+Jin%2C+and+Jinyang+Li%2E+Swapadvisor%3A+Pushing+deep+learning+beyond+the+gpu+memory+limit+via+smart+swapping%2E+In+Proceedings+of+the+Twenty%2DFifth+International+Conference+on+Architectura)
15. Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder tasks need more experts: Dynamic routing in moe models. arXiv preprint arXiv:2403.07652, 2024. — [arXiv:2403.07652](https://arxiv.org/abs/2403.07652) [find](https://scholar.google.com/scholar?q=Quzhe+Huang%2C+Zhenwei+An%2C+Nan+Zhuang%2C+Mingxu+Tao%2C+Chen+Zhang%2C+Yang+Jin%2C+Kun+Xu%2C+Liwei+Chen%2C+Songfang+Huang%2C+and+Yansong+Feng%2E+Harder+tasks+need+more+experts%3A+Dynamic+routing+in+moe+models%2E+arXiv+prepri)
16. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism, 2019. [find](https://scholar.google.com/scholar?q=Yanping+Huang%2C+Youlong+Cheng%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Mia+Xu+Chen%2C+Dehao+Chen%2C+HyoukJoong+Lee%2C+Jiquan+Ngiam%2C+Quoc+V%2E+Le%2C+Yonghui+Wu%2C+and+Zhifeng+Chen%2E+Gpipe%3A+Efficient+training+of+giant+neural+netwo)
17. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. [C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). Advances in Neural Information Processing Systems, 36:62991–63010, 2023.
18. Mikhail Isaev, Nic Mcdonald, Larry Dennison, and Richard Vuduc. Calculon: a methodology and tool for high-level co-design of systems and large language models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23, New York, NY, USA,… [find](https://scholar.google.com/scholar?q=Mikhail+Isaev%2C+Nic+Mcdonald%2C+Larry+Dennison%2C+and+Richard+Vuduc%2E+Calculon%3A+a+methodology+and+tool+for+high%2Dlevel+co%2Ddesign+of+systems+and+large+language+models%2E+In+Proceedings+of+the+International+Conf)
19. Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. [Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models](https://md.research.elamur.ai/free/en/papers/jacobs-2023-deepspeed-ulysses-system-optimizations-enabling/), 2023.
20. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv preprint arXiv:2403.07974, 2024.
21. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+Mixtral+of+expe)
22. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421, 2021.
23. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
24. Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. [Reducing activation recomputation in large transformer models](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/), 2022.
25. Tung D. Le, Haruki Imai, Yasushi Negishi, and Kiyokuni Kawachiya. Tflms: Large model support in tensorflow by graph rewriting, 2019. [find](https://scholar.google.com/scholar?q=Tung+D%2E+Le%2C+Haruki+Imai%2C+Yasushi+Negishi%2C+and+Kiyokuni+Kawachiya%2E+Tflms%3A+Large+model+support+in+tensorflow+by+graph+rewriting%2C+2019%2E)
26. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020. [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and+autom)
27. Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. The depth-to-width interplay in self-attention. arXiv preprint arXiv:2006.12467, 2020. — [arXiv:2006.12467](https://arxiv.org/abs/2006.12467) [find](https://scholar.google.com/scholar?q=Yoav+Levine%2C+Noam+Wies%2C+Or+Sharir%2C+Hofit+Bata%2C+and+Amnon+Shashua%2E+The+depth%2Dto%2Dwidth+interplay+in+self%2Dattention%2E+arXiv+preprint+arXiv%3A2006%2E12467%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E12467%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006)
28. Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. [Minimax-01: Scaling foundation models with lightning attention](https://md.research.elamur.ai/free/en/papers/minimax-2025-minimax-01-scaling-foundation/). arXiv preprint arXiv:2501.08313, 2025.
29. Jing Li, Zhijie Sun, Xuan He, Li Zeng, Yi Lin, Entong Li, Binfan Zheng, Rongqian Zhao, and Xin Chen. Locmoe: A low-overhead moe for large language model training. arXiv preprint arXiv:2401.13920, 2024. — [arXiv:2401.13920](https://arxiv.org/abs/2401.13920) [find](https://scholar.google.com/scholar?q=Jing+Li%2C+Zhijie+Sun%2C+Xuan+He%2C+Li+Zeng%2C+Yi+Lin%2C+Entong+Li%2C+Binfan+Zheng%2C+Rongqian+Zhao%2C+and+Xin+Chen%2E+Locmoe%3A+A+low%2Doverhead+moe+for+large+language+model+training%2E+arXiv+preprint+arXiv%3A2401%2E13920%2C+2024)
30. Heng Liao, Jiajin Tu, Jing Xia, and Xiping Zhou. Davinci: A scalable architecture for neural network computing. In 2019 IEEE Hot Chips 31 Symposium (HCS), pages 1–44, 2019. [find](https://scholar.google.com/scholar?q=Heng+Liao%2C+Jiajin+Tu%2C+Jing+Xia%2C+and+Xiping+Zhou%2E+Davinci%3A+A+scalable+architecture+for+neural+network+computing%2E+In+2019+IEEE+Hot+Chips+31+Symposium+%28HCS%29%2C+pages+1%E2%80%9344%2C+2019%2E)
31. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023. — [Let's Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
32. Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bin+Wang%2C+Bingxuan+Wang%2C+Bo+Liu%2C+Chenggang+Zhao%2C+Chengqi+Dengr%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+et+al%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+languag)
33. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+et+al%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437%2C+2024%2E)
34. Liyuan Liu, Young Jin Kim, Shuohang Wang, Chen Liang, Yelong Shen, Hao Cheng, Xiaodong Liu, Masahiro Tanaka, Xiaoxia Wu, Wenxiang Hu, et al. [Grin: Gradient-informed moe](https://md.research.elamur.ai/free/en/papers/liu-2024-grin-gradient-informed-moe/). arXiv preprint arXiv:2409.12136, 2024.
35. MAA. Codeforces. American Invitational Mathematics Examination - AIME 2024, 2024. [https://maa.org/math-competitions/american-invitational-mathematics-examination-aime](https://maa.org/math-competitions/american-invitational-mathematics-examination-aime). [↗](https://maa.org/math-competitions/american-invitational-mathematics-examination-aime)
36. MAA. Codeforces. American Invitational Mathematics Examination - AIME 2025, 2025. [https://maa.org/math-competitions/american-invitational-mathematics-examination-aime](https://maa.org/math-competitions/american-invitational-mathematics-examination-aime). [↗](https://maa.org/math-competitions/american-invitational-mathematics-examination-aime)
37. Chen Meng, Minmin Sun, Jun Yang, Minghui Qiu, and Yang Gu. Training deeper models by gpu memory optimization on tensorflow. In Proc. of ML Systems Workshop in NIPS, volume 7, page 26, 2017. [find](https://scholar.google.com/scholar?q=Chen+Meng%2C+Minmin+Sun%2C+Jun+Yang%2C+Minghui+Qiu%2C+and+Yang+Gu%2E+Training+deeper+models+by+gpu+memory+optimization+on+tensorflow%2E+In+Proc%2E+of+ML+Systems+Workshop+in+NIPS%2C+volume+7%2C+page+26%2C+2017%2E)
38. Saiful A Mojumder, Yifan Sun, Leila Delshadtehrani, Yenai Ma, Trinayan Baruah, José L Abellán, John Kim, David Kaeli, and Ajay Joshi. Mgpu-tsm: A multi-gpu system with truly shared memory. arXiv preprint arXiv:2008.02300, 2020. [find](https://scholar.google.com/scholar?q=Saiful+A+Mojumder%2C+Yifan+Sun%2C+Leila+Delshadtehrani%2C+Yenai+Ma%2C+Trinayan+Baruah%2C+Jos%C3%A9+L+Abell%C3%A1n%2C+John+Kim%2C+David+Kaeli%2C+and+Ajay+Joshi%2E+Mgpu%2Dtsm%3A+A+multi%2Dgpu+system+with+truly+shared+memory%2E+arXiv+prepr)
39. Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Luca+Soldaini%2C+Dirk+Groeneveld%2C+Kyle+Lo%2C+Jacob+Morrison%2C+Sewon+Min%2C+Weijia+Shi%2C+Pete+Walsh%2C+Oyvind+Tafjord%2C+Nathan+Lambert%2C+et+al%2E+Olmoe%3A+Open+mixture%2Dof%2Dexperts+language+models%2E+a)
40. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the international… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+et+al%2E+Efficient+large%2D)
41. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm,… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Anand+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+Amar+Phanishayee%2C)
42. Xiaonan Nie, Xupeng Miao, Zilong Wang, Zichao Yang, Jilong Xue, Lingxiao Ma, Gang Cao, and Bin Cui. Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement. Proceedings of the ACM on Management of Data, 1(1):1–19, 2023. [find](https://scholar.google.com/scholar?q=Xiaonan+Nie%2C+Xupeng+Miao%2C+Zilong+Wang%2C+Zichao+Yang%2C+Jilong+Xue%2C+Lingxiao+Ma%2C+Gang+Cao%2C+and+Bin+Cui%2E+Flexmoe%3A+Scaling+large%2Dscale+sparse+pre%2Dtrained+model+training+via+dynamic+device+placement%2E+Proceed)
43. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR, 2022. — [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/)
44. Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. Capuchin: Tensor-based gpu memory management for deep learning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages… [find](https://scholar.google.com/scholar?q=Xuan+Peng%2C+Xuanhua+Shi%2C+Hulin+Dai%2C+Hai+Jin%2C+Weiliang+Ma%2C+Qian+Xiong%2C+Fan+Yang%2C+and+Xuehai+Qian%2E+Capuchin%3A+Tensor%2Dbased+gpu+memory+management+for+deep+learning%2E+In+Proceedings+of+the+Twenty%2DFifth+Inter)
45. Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin. [Demons in the detail: On implementing load balancing loss for training specialized mixture-of-expert models](https://md.research.elamur.ai/free/en/papers/qiu-2025-demons-detail-implementing-load-balancing/). arXiv preprint arXiv:2501.11873, 2025.
46. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv e-prints, 2019.
47. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2C+2020%2E)
48. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling, 2024.
49. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+arXiv+preprint)
50. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprin)
51. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2C+2020%2E)
52. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2C+2020%2E)
53. Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, et al. [Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent](https://md.research.elamur.ai/free/en/papers/sun-2024-hunyuan-large-open-source/). arXiv preprint arXiv:2411.02265, 2024.
54. Qwen Team. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+Qwen1%2E5%2Dmoe%3A+Matching+7b+model+performance+with+1%2F3+activated+parameters%22%2C+February+2024%2E)
55. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). In The Thirty-eight Conference on Neural Information Processing Systems Datase…
56. Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lü, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, et al. [Skywork-moe: A deep dive into training techniques for mixture-of-experts language models](https://md.research.elamur.ai/free/en/papers/wei-2024-skywork-moe-deep-dive/). arXiv preprint arXiv:2406.06563, 2024.
57. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. [Clue: A chinese language understanding evaluation benchmark](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/). arXiv preprint arXiv:2004.05986, 2020.
58. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+et+al%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2C+2024%2E)
59. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+et+al%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2C+2024%2E)
60. Yichun Yin, Wenyong Huang, Kaikai Song, Yehui Tang, Xueyu Wu, Wei Guo, Peng Guo, Yaoyuan Wang, Xiaojun Meng, Yasheng Wang, et al. [Pangu ultra: Pushing the limits of dense large language models on ascend npus](https://md.research.elamur.ai/free/en/papers/yin-2025-pangu-ultra-pushing-limits/). arXiv preprint arXiv:2504.07866, 2025.
61. Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, and Jidong Zhai. SmartMoE: Efficiently training sparsely-activated models through combining offline and online oarallelization. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 961–975, Boston, MA, July 2023. USENIX Associa… [find](https://scholar.google.com/scholar?q=Mingshu+Zhai%2C+Jiaao+He%2C+Zixuan+Ma%2C+Zan+Zong%2C+Runqing+Zhang%2C+and+Jidong+Zhai%2E+SmartMoE%3A+Efficiently+training+sparsely%2Dactivated+models+through+combining+offline+and+online+oarallelization%2E+In+2023+USEN)
62. Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, and Jidong Zhai. {SmartMoE}: Efficiently training {Sparsely-Activated} models through combining offline and online parallelization. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 961–975, 2023. [find](https://scholar.google.com/scholar?q=Mingshu+Zhai%2C+Jiaao+He%2C+Zixuan+Ma%2C+Zan+Zong%2C+Runqing+Zhang%2C+and+Jidong+Zhai%2E+%7BSmartMoE%7D%3A+Efficiently+training+%7BSparsely%2DActivated%7D+models+through+combining+offline+and+online+parallelization%2E+In+2023+)
63. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). arXiv preprint arXiv:2311.07911, 2023.

*Updated 1 November 2026*