---
title: "OpenMoE — paper by Xue et al. (2024) — summary and ideas"
description: "The authors openly describe their own mistakes and suggest remedies, making this a practical guide… 40 references, cited by 52 papers here."
authors: ["Fuzhao Xue", "Zian Zheng", "Yao Fu", "Jinjie Ni", "Zangwei Zheng", "Wangchunshu Zhou", "Yang You"]
year: 2024
venue: "arXiv:2402.01739"
source: "https://arxiv.org/abs/2402.01739"
doi: "10.48550/arxiv.2402.01739"
arxiv: "2402.01739"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xue-2024-openmoe-early-effort-open-mixture/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xue-2024-openmoe-early-effort-open-mixture/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

# OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

**Authors:** [Fuzhao Xue](https://md.research.elamur.ai/free/en/authors/fuzhao-xue/), Zian Zheng, [Yao Fu](https://md.research.elamur.ai/free/en/authors/yao-fu/), Jinjie Ni, Zangwei Zheng, [Wangchunshu Zhou](https://md.research.elamur.ai/free/en/authors/wangchunshu-zhou/) and [Yang You](https://md.research.elamur.ai/free/en/authors/yang-you/) · **Year:** 2024 · **Venue:** arXiv:2402.01739 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.01739) · [PDF](https://arxiv.org/pdf/2402.01739) · [DOI 10.48550/arxiv.2402.01739](https://doi.org/10.48550/arxiv.2402.01739) · [arXiv 2402.01739](https://arxiv.org/abs/2402.01739) · [OpenAlex](https://openalex.org/W4391622923)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/xue-2024-openmoe-early-effort-open-mixture/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Before this work, few open Mixture-of-Experts (MoE) language models had been trained on trillion-scale, diverse data, and little was known about how their routers actually behave. The authors built and publicly released OpenMoE, a family of decoder-only MoE models with 650M to 34B parameters. The architecture follows ST-MoE and uses top-2 routing, a residual FFN alongside the experts, and MoE blocks placed only every few layers. The models were pre-trained on a code-heavy mix of RedPajama and The Stack Dedup, using the UL2 objective and the umT5 tokenizer. OpenMoE-8B/32E performs on par with dense OpenLLaMA-3B and TinyLLaMA-1.1B, which cost more to train, supporting the cost-efficiency argument for MoE. The analysis of routing found three patterns. Experts specialize by token ID rather than context. Routing becomes fixed early in training. Tokens late in a sequence are dropped more often, which hurts multi-turn chat. The authors openly describe their own mistakes and suggest remedies, making this a practical guide for future open MoE work.

## Key points
- **Released models:** OpenMoE-Base/16E (0.65B), OpenMoE-8B/32E (about 2B active parameters, trained on >1T tokens), OpenMoE-8B/32E-Chat, and OpenMoE-34B/32E (6B active, 200B tokens).
- **Fine-tuning and checkpoints:** the Chat model was tuned on a 100K subset of WildChat and clearly beats the dense baselines on single-turn MT-Bench. Five intermediate checkpoints are also released, spaced 200B tokens apart.
- **Code share:** code made up as much as 52.25% of early pre-training data, which the authors now consider excessive. They recommend roughly 30%, the share used for OpenMoE-34B/32E.
- **Tokenizer:** the 256K-vocabulary umT5 tokenizer is far more compact than the LLaMA tokenizer on multilingual TED text (Arabic needs only ~0.34× as many tokens). However, it gives no savings on code.
- **Proposed fixes:**
  - Drop the trainable router after warmup.
  - Use parallel Transformer layers.
  - Overlap attention computation with all-to-all communication.
  - Mix instruction data into pre-training warmup so that load balancing also holds for chat-style inputs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 255 works · 68,771 citations
- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 84 works · 32,204 citations
- [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) — University, Zurich, Switzerland · 36 works · 4,534 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou and Yang You. 2024. OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models. arXiv:2402.01739.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xue2024openmoe,
  title = {{OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models}},
  author = {Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou and Yang You},
  year = {2024},
  doi = {10.48550/arxiv.2402.01739},
  eprint = {2402.01739},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.01739}
}
```

- [← previous One-Shot Learning as Instruction Data Prospector for Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-one-shot-learning-as-instruction/)
- [next → ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-allava-harnessing-gpt4v-synthesized-data/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

40 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Anil et al., “[Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/),” arXiv preprint arXiv:2305.10403, 2023. Anonymous, “(inthe)wildchat: 570k chatGPT interaction logs in the wild,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: [https://openreview.net/forum?id=Bl8u7ZRlbM](https://openreview.net/forum?id=Bl8u7ZRlbM). M. A… [↗](https://openreview.net/forum?id=Bl8u7ZRlbM)
2. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer, “[TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/),” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M.-Y. Kan, Eds., Vancouver,…
3. J. D. M.-W. C. Kenton and L. K. Toutanova, “[Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/),” in Proceedings of naacL-HLT, vol. 1, 2019, p. 2.
4. D. Kocetkov et al., “The stack: 3 tb of permissively licensed source code,” Preprint, 2022. [find](https://scholar.google.com/scholar?q=D%2E+Kocetkov+et+al%2E%2C+%E2%80%9CThe+stack%3A+3+tb+of+permissively+licensed+source+code%2C%E2%80%9D+Preprint%2C+2022%2E)
5. A. Komatsuzaki et al., “Sparse upcycling: Training mixture-of-experts from dense checkpoints,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: [https://openreview.net/forum?id=T5nUQDrM4u](https://openreview.net/forum?id=T5nUQDrM4u). [↗](https://openreview.net/forum?id=T5nUQDrM4u)
6. D. Lepikhin et al., “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668, 2020. [find](https://scholar.google.com/scholar?q=D%2E+Lepikhin+et+al%2E%2C+%E2%80%9CGshard%3A+Scaling+giant+models+with+conditional+computation+and+automatic+sharding%2C%E2%80%9D+arXiv+preprint+arXiv%3A2006%2E16668%2C+2020%2E)
7. M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer, “Base layers: Simplifying training of large, sparse models,” in International Conference on Machine Learning, PMLR, 2021, pp. 6265–6274. [find](https://scholar.google.com/scholar?q=M%2E+Lewis%2C+S%2E+Bhosale%2C+T%2E+Dettmers%2C+N%2E+Goyal%2C+and+L%2E+Zettlemoyer%2C+%E2%80%9CBase+layers%3A+Simplifying+training+of+large%2C+sparse+models%2C%E2%80%9D+in+International+Conference+on+Machine+Learning%2C+PMLR%2C+2021%2C+pp%2E+6265%E2%80%936274)
8. J. Li, Z. Zhang, and H. Zhao, “Self-prompting large language models for open-domain qa,” arXiv preprint arXiv:2212.08635, 2022. — [arXiv:2212.08635](https://arxiv.org/abs/2212.08635) [find](https://scholar.google.com/scholar?q=J%2E+Li%2C+Z%2E+Zhang%2C+and+H%2E+Zhao%2C+%E2%80%9CSelf%2Dprompting+large+language+models+for+open%2Ddomain+qa%2C%E2%80%9D+arXiv+preprint+arXiv%3A2212%2E08635%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E08635%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2212%2E08635%29)
9. R. Li et al., “Starcoder: May the source be with you!” arXiv preprint arXiv:2305.06161, 2023. [find](https://scholar.google.com/scholar?q=R%2E+Li+et+al%2E%2C+%E2%80%9CStarcoder%3A+May+the+source+be+with+you%21%E2%80%9D+arXiv+preprint+arXiv%3A2305%2E06161%2C+2023%2E)
10. Y. Liu et al., “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Y%2E+Liu+et+al%2E%2C+%E2%80%9CRoberta%3A+A+robustly+optimized+bert+pretraining+approach%2C%E2%80%9D+arXiv+preprint+arXiv%3A1907%2E11692%2C+2019%2E)
11. Y. Lou, F. Xue, Z. Zheng, and Y. You, “Cross-token modeling with conditional computation,” arXiv preprint arXiv:2109.02008, 2021. — [arXiv:2109.02008](https://arxiv.org/abs/2109.02008) [find](https://scholar.google.com/scholar?q=Y%2E+Lou%2C+F%2E+Xue%2C+Z%2E+Zheng%2C+and+Y%2E+You%2C+%E2%80%9CCross%2Dtoken+modeling+with+conditional+computation%2C%E2%80%9D+arXiv+preprint+arXiv%3A2109%2E02008%2C+2021%2E+%E2%80%94+%5BarXiv%3A2109%2E02008%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2109%2E02008%29)
12. B. Mustafa, C. Riquelme, J. Puigcerver, R. Jenatton, and N. Houlsby, “Multimodal contrastive learning with limoe: The language-image mixture of experts,” Advances in Neural Information Processing Systems, vol. 35, pp. 9564–9576, 2022. [find](https://scholar.google.com/scholar?q=B%2E+Mustafa%2C+C%2E+Riquelme%2C+J%2E+Puigcerver%2C+R%2E+Jenatton%2C+and+N%2E+Houlsby%2C+%E2%80%9CMultimodal+contrastive+learning+with+limoe%3A+The+language%2Dimage+mixture+of+experts%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Syst)
13. E. Nijkamp et al., “Xgen-7b technical report,” arXiv preprint arXiv:2309.03450, 2023. — [arXiv:2309.03450](https://arxiv.org/abs/2309.03450) [find](https://scholar.google.com/scholar?q=E%2E+Nijkamp+et+al%2E%2C+%E2%80%9CXgen%2D7b+technical+report%2C%E2%80%9D+arXiv+preprint+arXiv%3A2309%2E03450%2C+2023%2E+%E2%80%94+%5BarXiv%3A2309%2E03450%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E03450%29)
14. J. Puigcerver, C. Riquelme, B. Mustafa, and N. Houlsby, “From sparse to soft mixtures of experts,” arXiv preprint arXiv:2308.00951, 2023. — [arXiv:2308.00951](https://arxiv.org/abs/2308.00951) [find](https://scholar.google.com/scholar?q=J%2E+Puigcerver%2C+C%2E+Riquelme%2C+B%2E+Mustafa%2C+and+N%2E+Houlsby%2C+%E2%80%9CFrom+sparse+to+soft+mixtures+of+experts%2C%E2%80%9D+arXiv+preprint+arXiv%3A2308%2E00951%2C+2023%2E+%E2%80%94+%5BarXiv%3A2308%2E00951%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2308%2E00951%29)
15. J. W. Rae et al., “Scaling language models: Methods, analysis & insights from training gopher,” arXiv preprint arXiv:2112.11446, 2021. [find](https://scholar.google.com/scholar?q=J%2E+W%2E+Rae+et+al%2E%2C+%E2%80%9CScaling+language+models%3A+Methods%2C+analysis+%26+insights+from+training+gopher%2C%E2%80%9D+arXiv+preprint+arXiv%3A2112%2E11446%2C+2021%2E)
16. C. Raffel et al., “[Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/),” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020. [Online]. Available: [http://jmlr.org/papers/v21/20-074.html](http://jmlr.org/papers/v21/20-074.html). [↗](http://jmlr.org/papers/v21/20-074.html)
17. C. Riquelme et al., “Scaling vision with sparse mixture of experts,” Advances in Neural Information Processing Systems, vol. 34, pp. 8583–8595, 2021. [find](https://scholar.google.com/scholar?q=C%2E+Riquelme+et+al%2E%2C+%E2%80%9CScaling+vision+with+sparse+mixture+of+experts%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+34%2C+pp%2E+8583%E2%80%938595%2C+2021%2E)
18. S. Roller, S. Sukhbaatar, J. Weston, et al., “Hash layers for large sparse models,” Advances in Neural Information Processing Systems, vol. 34, pp. 17 555–17 566, 2021. [find](https://scholar.google.com/scholar?q=S%2E+Roller%2C+S%2E+Sukhbaatar%2C+J%2E+Weston%2C+et+al%2E%2C+%E2%80%9CHash+layers+for+large+sparse+models%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+34%2C+pp%2E+17+555%E2%80%9317+566%2C+2021%2E)
19. B. Roziere et al., “[Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/),” arXiv preprint arXiv:2308.12950, 2023.
20. N. Shazeer, “Glu variants improve transformer,” arXiv preprint arXiv:2002.05202, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2C+%E2%80%9CGlu+variants+improve+transformer%2C%E2%80%9D+arXiv+preprint+arXiv%3A2002%2E05202%2C+2020%2E)
21. N. Shazeer et al., “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538, 2017. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer+et+al%2E%2C+%E2%80%9COutrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2C%E2%80%9D+arXiv+preprint+arXiv%3A1701%2E06538%2C+2017%2E)
22. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=M%2E+Shoeybi%2C+M%2E+Patwary%2C+R%2E+Puri%2C+P%2E+LeGresley%2C+J%2E+Casper%2C+and+B%2E+Catanzaro%2C+%E2%80%9CMegatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2C%E2%80%9D+arXiv+preprint+arXiv%3A1909%2E08053%2C+20)
23. L. Soldaini et al., “Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research,” arXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Soldaini+et+al%2E%2C+%E2%80%9CDolma%3A+An+Open+Corpus+of+Three+Trillion+Tokens+for+Language+Model+Pretraining+Research%2C%E2%80%9D+arXiv+preprint%2C+2023%2E)
24. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “[Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/),” Neurocomputing, vol. 568, p. 127 063, 2024.
25. Y. Tay et al., “[Ul2: Unifying language learning paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/),” in The Eleventh International Conference on Learning Representations, 2022.
26. Y. Tay et al., “Unifying language learning paradigms,” arXiv preprint arXiv:2205.[05131, 2022](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/).
27. L.-M. Team, [Llama-moe: Building mixture-of-experts from llama with continual pre-training](https://md.research.elamur.ai/free/en/papers/zhu-2024-llama-moe-building-mixture-experts/), Dec. 2023. [Online]. Available: [https://github.com/pjlab-sys4nlp/llama-moe](https://github.com/pjlab-sys4nlp/llama-moe). [↗](https://github.com/pjlab-sys4nlp/llama-moe)
28. H. Touvron et al., “[Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/),” arXiv preprint arXiv:2302.13971, 2023.
29. B. Wang and A. Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), May 2021. [↗](https://github.com/kingoflolz/mesh-transformer-jax)
30. G. Wenzek et al., “[CCNet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/),” English, in Proceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari et al., Eds., Marseille, France: European Language Resources Association, May 2020, pp. 4003–4012, ISBN : 979…
31. Y. Xu et al., “Gspmd: General and scalable parallelization for ml computation graphs,” arXiv preprint arXiv:2105.04663, 2021. — [arXiv:2105.04663](https://arxiv.org/abs/2105.04663) [find](https://scholar.google.com/scholar?q=Y%2E+Xu+et+al%2E%2C+%E2%80%9CGspmd%3A+General+and+scalable+parallelization+for+ml+computation+graphs%2C%E2%80%9D+arXiv+preprint+arXiv%3A2105%2E04663%2C+2021%2E+%E2%80%94+%5BarXiv%3A2105%2E04663%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2105%2E04663%29)
32. F. Xue, X. He, X. Ren, Y. Lou, and Y. You, “One student knows all experts know: From sparse to dense,” arXiv preprint arXiv:2201.10890, 2022. — [arXiv:2201.10890](https://arxiv.org/abs/2201.10890) [find](https://scholar.google.com/scholar?q=F%2E+Xue%2C+X%2E+He%2C+X%2E+Ren%2C+Y%2E+Lou%2C+and+Y%2E+You%2C+%E2%80%9COne+student+knows+all+experts+know%3A+From+sparse+to+dense%2C%E2%80%9D+arXiv+preprint+arXiv%3A2201%2E10890%2C+2022%2E+%E2%80%94+%5BarXiv%3A2201%2E10890%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2201%2E10890%29)
33. F. Xue, Z. Shi, F. Wei, Y. Lou, Y. Liu, and Y. You, “Go wider instead of deeper,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, 2022, pp. 8779–8787. [find](https://scholar.google.com/scholar?q=F%2E+Xue%2C+Z%2E+Shi%2C+F%2E+Wei%2C+Y%2E+Lou%2C+Y%2E+Liu%2C+and+Y%2E+You%2C+%E2%80%9CGo+wider+instead+of+deeper%2C%E2%80%9D+in+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+vol%2E+36%2C+2022%2C+pp%2E+8779%E2%80%938787%2E)
34. P. Yu et al., “Efficient language modeling with sparse all-mlp,” arXiv preprint arXiv:2203.06850, 2022. — [arXiv:2203.06850](https://arxiv.org/abs/2203.06850) [find](https://scholar.google.com/scholar?q=P%2E+Yu+et+al%2E%2C+%E2%80%9CEfficient+language+modeling+with+sparse+all%2Dmlp%2C%E2%80%9D+arXiv+preprint+arXiv%3A2203%2E06850%2C+2022%2E+%E2%80%94+%5BarXiv%3A2203%2E06850%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2203%2E06850%29)
35. P. Zhang, G. Zeng, T. Wang, and W. Lu, [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/), 2024. arXiv: 2401.02385 [cs.CL].
36. Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng, “Deep long-tailed learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Zhang%2C+B%2E+Kang%2C+B%2E+Hooi%2C+S%2E+Yan%2C+and+J%2E+Feng%2C+%E2%80%9CDeep+long%2Dtailed+learning%3A+A+survey%2C%E2%80%9D+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+2023%2E)
37. L. Zheng et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,” arXiv preprint arXiv:2306.05685, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Zheng+et+al%2E%2C+%E2%80%9CJudging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E05685%2C+2023%2E)
38. Y. Zhou et al., “Brainformers: Trading simplicity for efficiency,” in International Conference on Machine Learning, PMLR, 2023, pp. 42 531–42 542. [find](https://scholar.google.com/scholar?q=Y%2E+Zhou+et+al%2E%2C+%E2%80%9CBrainformers%3A+Trading+simplicity+for+efficiency%2C%E2%80%9D+in+International+Conference+on+Machine+Learning%2C+PMLR%2C+2023%2C+pp%2E+42+531%E2%80%9342+542%2E)
39. Y. Zhou et al., “Mixture-of-experts with expert choice routing,” Advances in Neural Information Processing Systems, vol. 35, pp. 7103–7114, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Zhou+et+al%2E%2C+%E2%80%9CMixture%2Dof%2Dexperts+with+expert+choice+routing%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+35%2C+pp%2E+7103%E2%80%937114%2C+2022%2E)
40. B. Zoph et al., “St-moe: Designing stable and transferable sparse expert models,” URL https://arxiv.org/abs/2202.08906, 2022. [↗](https://arxiv.org/abs/2202.08906)

*Updated 24 October 2026*