---
title: "Skywork-MoE — paper by Wei et al. (2024) — summary and ideas"
description: "The authors also propose two techniques: Gating logit normalization , which makes routing decisions… 37 references, cited by 17 papers here."
authors: ["Tianwen Wei", "Bo Zhu", "Liang Zhao", "Cheng Cheng", "Biye Li", "Weiwei L\\\"u", "Peng Cheng", "Jianhao Zhang", "Xiaoyu Zhang", "Liang Zeng", "Xiaokun Wang", "Yutuan Ma", "Rui Hu", "Shuicheng Yan", "Han Fang", "Yahui Zhou"]
year: 2024
venue: "arXiv:2406.06563"
source: "https://arxiv.org/abs/2406.06563"
doi: "10.48550/arxiv.2406.06563"
arxiv: "2406.06563"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wei-2024-skywork-moe-deep-dive/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wei-2024-skywork-moe-deep-dive/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models

# Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models

**Authors:** [Tianwen Wei](https://md.research.elamur.ai/free/en/authors/tianwen-wei/), [Bo Zhu](https://md.research.elamur.ai/free/en/authors/bo-zhu/), [Liang Zhao](https://md.research.elamur.ai/free/en/authors/liang-zhao/), [Cheng Cheng](https://md.research.elamur.ai/free/en/authors/cheng-cheng/), [Biye Li](https://md.research.elamur.ai/free/en/authors/biye-li/), [Weiwei L\\"u](https://md.research.elamur.ai/free/en/authors/weiwei-l-u/), [Peng Cheng](https://md.research.elamur.ai/free/en/authors/peng-cheng/), [Jianhao Zhang](https://md.research.elamur.ai/free/en/authors/jianhao-zhang/), Xiaoyu Zhang, [Liang Zeng](https://md.research.elamur.ai/free/en/authors/liang-zeng/), [Xiaokun Wang](https://md.research.elamur.ai/free/en/authors/xiaokun-wang/), Yutuan Ma, [Rui Hu](https://md.research.elamur.ai/free/en/authors/rui-hu/), [Shuicheng Yan](https://md.research.elamur.ai/free/en/authors/shuicheng-yan/), [Han Fang](https://md.research.elamur.ai/free/en/authors/han-fang/) and [Yahui Zhou](https://md.research.elamur.ai/free/en/authors/yahui-zhou/) · **Year:** 2024 · **Venue:** arXiv:2406.06563 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.06563) · [PDF](https://arxiv.org/pdf/2406.06563) · [DOI 10.48550/arxiv.2406.06563](https://doi.org/10.48550/arxiv.2406.06563) · [arXiv 2406.06563](https://arxiv.org/abs/2406.06563) · [OpenAlex](https://openalex.org/W4399597325)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [17](https://md.research.elamur.ai/free/en/papers/wei-2024-skywork-moe-deep-dive/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training sparse Mixture-of-Experts (MoE) LLMs involves several practical choices whose effects are poorly understood. Two examples are whether to start from a dense checkpoint or from random weights, and how to keep experts both distinct and evenly loaded. This technical report from Kunlun's Skywork Team addresses these choices while building Skywork-MoE. The model has 146B total parameters, 22B of which are active per token. It uses 16 experts with top-2 routing and is initialized from Skywork-13B.

Small-scale experiments with a 0.3B dense model and 8-expert MoEs show that upcycling is not always the better start. With a larger training budget, a randomly initialized MoE overtakes the upcycled variants. The authors also propose two techniques:
- **Gating logit normalization**, which makes routing decisions sharper.
- **Adaptive auxiliary loss coefficients**, which tune the load-balancing weight separately for each layer.

The final model was trained mainly on a filtered part of the SkyPile corpus plus synthetic data. It beats Deepseek-67B and Llama2-70B on most benchmarks but stays behind Llama3-70B and larger MoEs. The work offers practical guidance for planning MoE training budgets.

## Key points
- **Upcycling vs. scratch:**
  - With a 100B-token budget, the from-scratch MoE roughly matched the one upcycled from the 100B-token checkpoint.
  - With 300B tokens, the from-scratch MoE outperformed every upcycled version.
  - Upcycled runs with a too-low learning rate did worst.
- **Expert similarity:** average similarity between experts falls during upcycled training, so the authors suggest it as a useful monitoring signal. The worst run kept the highest similarity, while from-scratch experts stayed near zero similarity throughout.
- **Gating logit normalization:** this yielded a moderately lower loss and fewer dropped tokens. Without it, the gap between the top gate probabilities shrinks toward a ratio of one, which the authors consider harmful.
- **Adaptive auxiliary loss coefficients:** the per-layer coefficient follows each layer's token drop rate, rising when imbalance grows.
- **Data and results:**
  - The training mix is about 7:2:1 English:Chinese:code.
  - Benchmark scores are CEVAL 82.2, CMMLU 79.5, MMLU 77.4, GSM8K 76.1, MATH 31.9 and HumanEval 43.9.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei L\\"u, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, Xiaokun Wang, Yutuan Ma, Rui Hu, Shuicheng Yan, Han Fang and Yahui Zhou. 2024. Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models. arXiv:2406.06563.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wei2024skywork,
  title = {{Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models}},
  author = {Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei L"u, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, Xiaokun Wang, Yutuan Ma, Rui Hu, Shuicheng Yan, Han Fang and Yahui Zhou},
  year = {2024},
  doi = {10.48550/arxiv.2406.06563},
  eprint = {2406.06563},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.06563}
}
```

- [← previous Personality Alignment of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2024-personality-alignment-large-language-models/)
- [next → Source-Aware Training Enables Knowledge Attribution in Language Models](https://md.research.elamur.ai/free/en/papers/khalifa-2024-source-aware-training-enables-knowledge/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

37 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language mo… [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Chengqi+Deng%2C+Chenggang+Zhao%2C+R%2E+X%2E+Xu%2C+Huazuo+Gao%2C+Deli+Chen%2C+Jiashi+Li%2C+Wangding+Zeng%2C+Xingkai+Yu%2C+Y%2E+Wu%2C+Zhenda+Xie%2C+Y%2E+K%2E+Li%2C+Panpan+Huang%2C+Fuli+Luo%2C+Chong+Ruan%2C+Zhifang+Sui%2C+and+Wenfen)
2. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E)
3. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeff Wang, Luke Zettle… [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Shruti+Bhosale%2C+Naman+Goyal%2C+Todor+Mihaylov%2C+Myle+Ott%2C+Sam+Shleifer%2C+Xi+Victoria+Lin%2C+Jingfei+Du%2C+Srinivasan+Iyer%2C+Ramakanth+Pasunuru%2C+Giri+Anantharaman%2C+Xian+Li%2C+Shuohui+Chen%2C+Halil+Ak)
4. Databricks. 2024. Dbrx-instruct. [find](https://scholar.google.com/scholar?q=Databricks%2E+2024%2E+Dbrx%2Dinstruct%2E)
5. DeepSeek-AI. 2024a. Deepseek llm: Scaling open-source language models with longtermism. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+2024a%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E)
6. DeepSeek-AI. 2024b. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+2024b%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+language+model%2E)
7. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/).
8. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas… — [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
9. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
10. David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. 2014. Learning factored representations in a deep mixture of experts. [find](https://scholar.google.com/scholar?q=David+Eigen%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Ilya+Sutskever%2E+2014%2E+Learning+factored+representations+in+a+deep+mixture+of+experts%2E)
11. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2022%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E)
12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/).
13. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
14. Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, Dav… [find](https://scholar.google.com/scholar?q=Aidan+Clark%2C+Diego+de+las+Casas%2C+Aurelia+Guy%2C+Arthur+Mensch%2C+Michela+Paganini%2C+Jordan+Hoffmann%2C+Bogdan+Damoc%2C+Blake+Hechtman%2C+Trevor+Cai%2C+Sebastian+Borgeaud%2C+George+van+den+Driessche%2C+Eliza+Rutherford)
15. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. [Ceval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). arXiv preprint arXiv:2305.08322.
16. Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. 1991. Adaptive mixture of local expert. Neural Computation, 3:78–88. [find](https://scholar.google.com/scholar?q=Robert+Jacobs%2C+Michael+Jordan%2C+Steven+Nowlan%2C+and+Geoffrey+Hinton%2E+1991%2E+Adaptive+mixture+of+local+expert%2E+Neural+Computation%2C+3%3A78%E2%80%9388%2E)
17. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/).
18. Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2023. Sparse upcycling: Training mixture-of-experts from dense checkpoints. [find](https://scholar.google.com/scholar?q=Aran+Komatsuzaki%2C+Joan+Puigcerver%2C+James+Lee%2DThorp%2C+Carlos+Riquelme+Ruiz%2C+Basil+Mustafa%2C+Joshua+Ainslie%2C+Yi+Tay%2C+Mostafa+Dehghani%2C+and+Neil+Houlsby%2E+2023%2E+Sparse+upcycling%3A+Training+mixture%2Dof%2Dexperts)
19. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+2020%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and)
20. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2022. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/).
21. Gemini Team. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/).
22. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. [Cmmlu: Measuring massive multitask language understanding in chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/).
23. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/).
24. Meta-AI. 2024. Llama 3 model card. [find](https://scholar.google.com/scholar?q=Meta%2DAI%2E+2024%2E+Llama+3+model+card%2E)
25. Mistral-AI. 2023. Mixtral-8x7B-v0.1. [find](https://scholar.google.com/scholar?q=Mistral%2DAI%2E+2023%2E+Mixtral%2D8x7B%2Dv0%2E1%2E)
26. Mistral-AI. 2024. Mixtral-8x22B-v0.1. [find](https://scholar.google.com/scholar?q=Mistral%2DAI%2E+2024%2E+Mixtral%2D8x22B%2Dv0%2E1%2E)
27. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on gpu clusters using megatr… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Anand+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+Amar+Phanishayee%2C)
28. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
29. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Cur…
31. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
32. Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yongy… — [Skywork: A More Open Bilingual Foundation Model](https://md.research.elamur.ai/free/en/papers/wei-2023-skywork-more-open-bilingual/)
33. Qwen Team. 2023. QWEN technical report. [https://github.com/QwenLM/Qwen](https://github.com/QwenLM/Qwen) . [↗](https://github.com/QwenLM/Qwen)
34. Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. [Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale](https://md.research.elamur.ai/free/en/papers/rajbhandari-2022-deepspeed-moe-advancing-mixture-experts/).
35. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E)
36. Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Biao+Zhang+and+Rico+Sennrich%2E+2019%2E+Root+Mean+Square+Layer+Normalization%2E+In+Advances+in+Neural+Information+Processing+Systems+32%2C+Vancouver%2C+Canada%2E)
37. Noam Shazeer. 2020. Glu variants improve transformer. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2020%2E+Glu+variants+improve+transformer%2E)

*Updated 31 October 2026*