---
title: "JetMoE: Reaching Llama2 Performance with 0.1M Dollars (2024)"
description: "The architecture builds on ModuleFormer and uses top-k routing to sparsely activate both the… 80 references, cited by 24 papers here."
authors: ["Yikang Shen", "Zhen Guo", "Tianle Cai", "Zengyi Qin"]
year: 2024
venue: "arXiv:2404.07413"
source: "https://arxiv.org/abs/2404.07413"
doi: "10.48550/arxiv.2404.07413"
arxiv: "2404.07413"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shen-2024-jetmoe-reaching-llama2-performance-0/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shen-2024-jetmoe-reaching-llama2-performance-0/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / JetMoE: Reaching Llama2 Performance with 0.1M Dollars

# JetMoE: Reaching Llama2 Performance with 0.1M Dollars

**Authors:** [Yikang Shen](https://md.research.elamur.ai/free/en/authors/yikang-shen/), [Zhen Guo](https://md.research.elamur.ai/free/en/authors/zhen-guo/), [Tianle Cai](https://md.research.elamur.ai/free/en/authors/tianle-cai/) and Zengyi Qin · **Year:** 2024 · **Venue:** arXiv:2404.07413 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.07413) · [PDF](https://arxiv.org/pdf/2404.07413) · [DOI 10.48550/arxiv.2404.07413](https://doi.org/10.48550/arxiv.2404.07413) · [arXiv 2404.07413](https://arxiv.org/abs/2404.07413) · [OpenAlex](https://openalex.org/W4394780940)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [24](https://md.research.elamur.ai/free/en/papers/shen-2024-jetmoe-reaching-llama2-performance-0/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training a competitive large language model is usually thought to need budgets that only large companies have. Strong MoE models also rarely publish their recipes and often depend on private data. JetMoE-8B shows that a capable model can be built for under $100k, using about 30,000 H100 GPU hours and 1.25T tokens. All of that data comes from public corpora, including RefinedWeb, StarCoder data, Dolma and The Pile, plus math, code and synthetic instruction sets, arranged in a two-phase pretraining schedule. The architecture builds on ModuleFormer and uses top-k routing to sparsely activate both the attention layer and the feed-forward layer. The attention layer is a Mixture of Attention heads variant with shared key/value projections and RoPE. Only about 2B of the 8B parameters are used per token, which cuts inference compute by roughly 70% compared with Llama2-7B. The base model beats Llama2-7B on the OpenLLM Leaderboard average, and the aligned chat model scores above Llama-2-13b-chat on MT-Bench. Because the data mixtures, hyperparameters and code are all disclosed, the work gives academic groups a reproducible template for efficient open foundation models.

## Key points
- **Scale and parameters:** 8B total parameters with about 2.2B active per token (per Table 3). Trained on 1.25T tokens, versus 2T for Llama2-7B, DeepseekMoE and Gemma.
- **Router training:** combines the standard LM loss with the Fedus et al. frequency-based load-balancing loss and the router z-loss from Zoph et al.
- **OpenLLM Leaderboard:** JetMoE-8B averages 53.0, ahead of Llama2-7B (51.0), DeepseekMoE (51.1) and Gemma (46.4).
  - It leads on most tasks, e.g. GSM8k 27.8 vs 14.5 for Llama2-7B and MMLU 49.2.
  - It trails on ARC-challenge and WinoGrande.
  - It has the best MBPP score (34.2) but is weaker on HumanEval (14.6).
- **Alignment:** distilled SFT on UltraChat 200k, Airoboros-3.2, Code-Feedback, Orca-math, SystemChat and Capybara, followed by dDPO on UltraFeedback. The whole process takes 60 H100 GPU hours.
  - The resulting JetMoE-8B-Chat scores 6.681 on MT-Bench, versus 6.650 for Llama-2-13b-chat.
  - It remains weak on coding and extraction compared with GPT-3.5-turbo.
- **Limitation:** the budget allowed no architecture ablations, so hyperparameters and data mixtures were chosen based on prior work. Models and code are released at github.com/myshell-ai/JetMoE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 211 works · 69,000 citations
- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 139 works · 70,691 citations
- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 90 works · 39,109 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Yikang Shen, Zhen Guo, Tianle Cai and Zengyi Qin. 2024. JetMoE: Reaching Llama2 Performance with 0.1M Dollars. arXiv:2404.07413.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shen2024jetmoe,
  title = {{JetMoE: Reaching Llama2 Performance with 0.1M Dollars}},
  author = {Yikang Shen, Zhen Guo, Tianle Cai and Zengyi Qin},
  year = {2024},
  doi = {10.48550/arxiv.2404.07413},
  eprint = {2404.07413},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.07413}
}
```

- [← previous Can Language Models Solve Olympiad Programming?](https://md.research.elamur.ai/free/en/papers/shi-2024-language-models-solve-olympiad-programming/)
- [next → Multi-modal Stance Detection: New Datasets and Model](https://md.research.elamur.ai/free/en/papers/liang-2024-multi-modal-stance-detection-new/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

80 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. abacusai. Systemchat, 2024. SystemChat. [find](https://scholar.google.com/scholar?q=abacusai%2E+Systemchat%2C+2024%2E+SystemChat%2E)
2. URL [https://huggingface.co/datasets/abacusai/ajibawa](https://huggingface.co/datasets/abacusai/ajibawa) 2023. Code-290k-sharegpt, 2024. ajibawa-2023/Code-290k-ShareGPT. [↗](https://huggingface.co/datasets/abacusai/ajibawa)
3. URL — [PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation](https://md.research.elamur.ai/free/en/papers/ansel-2024-pytorch-2-faster-machine-learning/) [https://huggingface.co/datasets/Jason](https://huggingface.co/datasets/Jason) Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation, 2024. [↗](https://huggingface.co/datasets/Jason)
4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
5. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. [Llemma: An open language model for mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/), 2023.
6. Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models. [https://github.com/bigcode-project/bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness), 2022. [↗](https://github.com/bigcode-project/bigcode-evaluation-harness)
7. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373, 2023. [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+Pythia%3A+A+suite+for+a)
8. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432–7439, 2020.
9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
11. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna/…](https://lmsys.org/blog/2023-03-30-vicuna/…) [↗](https://lmsys.org/blog/2023-03-30-vicuna/…)
12. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. [find](https://scholar.google.com/scholar?q=Paul+Christiano%2C+Jan+Leike%2C+Tom+B%2E+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2C+2023%2E)
13. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044, 2019.
14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
15. CogStack. OpenGPT: A framework for creating grounded instruction based datasets and training conversational domain expert Large Language Models (LLMs). [https://github.com/CogStack/OpenGPT](https://github.com/CogStack/OpenGPT), 2023. [↗](https://github.com/CogStack/OpenGPT)
16. CollectiveCognition. Collective cognition chatgpt conversations, 2023. URL [https://huggingface.co/datasets/CollectiveCognition/chats-data-2023-09-22](https://huggingface.co/datasets/CollectiveCognition/chats-data-2023-09-22). [↗](https://huggingface.co/datasets/CollectiveCognition/chats-data-2023-09-22)
17. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2C+2023%2E)
18. Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024. [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Chengqi+Deng%2C+Chenggang+Zhao%2C+RX+Xu%2C+Huazuo+Gao%2C+Deli+Chen%2C+Jiashi+Li%2C+Wangding+Zeng%2C+Xingkai+Yu%2C+Y+Wu%2C+et+al%2E+Deepseekmoe%3A+Towards+ultimate+expert+specialization+in+mixture%2Dof%2Dexperts+lang)
19. Luigi Daniele and Suphavadeeprasit. Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training. arXiv preprint arXiv:(coming soon), 2023. URL [https://huggingface.co/datasets/LDJnr/Capybara](https://huggingface.co/datasets/LDJnr/Capybara). [↗](https://huggingface.co/datasets/LDJnr/Capybara)
20. Databricks. Dbrx: Resources and code examples. [https://github.com/databricks/dbrx](https://github.com/databricks/dbrx), 2024. [↗](https://github.com/databricks/dbrx)
21. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/), 2023.
22. Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. [Glam: Efficient scaling of language models with mixture-of-experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/). In International Conference on Machine Learning, pp. 5547–5569. PMLR, 2022.
23. Jon Durbin. airoboros: Customizable implementation of the self-instruct paper. [https://github.com/jondurbin/airoboros](https://github.com/jondurbin/airoboros), 2023. [↗](https://github.com/jondurbin/airoboros)
24. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2C+2021%2E)
25. Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems, 5, 2023. [find](https://scholar.google.com/scholar?q=Trevor+Gale%2C+Deepak+Narayanan%2C+Cliff+Young%2C+and+Matei+Zaharia%2E+Megablocks%3A+Efficient+sparse+training+with+mixture%2Dof%2Dexperts%2E+Proceedings+of+Machine+Learning+and+Systems%2C+5%2C+2023%2E)
26. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+language+)
27. Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL [https://github.com/openlm-research/open_llama](https://github.com/openlm-research/open_llama). glaiveai. [↗](https://github.com/openlm-research/open_llama)
28. Glaive-code-assistant, 2023. URL [https://huggingface.co/datasets/glaiveai/](https://huggingface.co/datasets/glaiveai/) glaive-code-assistant. [↗](https://huggingface.co/datasets/glaiveai/)
29. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024a. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Y%2E+Wu%2C+Y%2E+K%2E+Li%2C+Fuli+Luo%2C+Yingfei+Xiong%2C+and+Wenfeng+Liang%2E+Deepseek%2Dcoder%3A+When+the+large+language+model)
30. Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online ai feedback, 2024b. [find](https://scholar.google.com/scholar?q=Shangmin+Guo%2C+Biao+Zhang%2C+Tianlin+Liu%2C+Tianqi+Liu%2C+Misha+Khalman%2C+Felipe+Llinares%2C+Alexandre+Rame%2C+Thomas+Mesnard%2C+Yao+Zhao%2C+Bilal+Piot%2C+Johan+Ferret%2C+and+Mathieu+Blondel%2E+Direct+language+model+alignm)
31. Horace He and Shangdi Yu. Transcending runtime-memory tradeoffs in checkpointing by being fusion aware. Proceedings of Machine Learning and Systems, 5, 2023. [find](https://scholar.google.com/scholar?q=Horace+He+and+Shangdi+Yu%2E+Transcending+runtime%2Dmemory+tradeoffs+in+checkpointing+by+being+fusion+aware%2E+Proceedings+of+Machine+Learning+and+Systems%2C+5%2C+2023%2E)
32. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Mao… — [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/)
33. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
34. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
35. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). arXiv preprint arXiv:1705.03551, 2017.
36. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7:4…
37. LAION-AI. Open-Assistant: A chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically. [https://github.com/](https://github.com/) LAION-AI/Open-Assistant, 2023. [↗](https://github.com/)
38. Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms, 2024. [find](https://scholar.google.com/scholar?q=Ariel+N%2E+Lee%2C+Cole+J%2E+Hunter%2C+and+Nataniel+Ruiz%2E+Platypus%3A+Quick%2C+cheap%2C+and+powerful+refinement+of+llms%2C+2024%2E)
39. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for ”mind” exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. — [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/)
40. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicola… [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+Qian+Liu%2C+Evgenii+Zheltonozhskii%2C+Terry+Yue+Zhuo%2C+Thoma)
41. Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ”Teknium”. Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023. URL https://https://huggingface.co/Open-Orca/SlimOrca. [find](https://scholar.google.com/scholar?q=Wing+Lian%2C+Guan+Wang%2C+Bleys+Goodson%2C+Eugene+Pentland%2C+Austin+Cook%2C+Chanvichet+Vong%2C+and+%E2%80%9DTeknium%E2%80%9D%2E+Slimorca%3A+An+open+dataset+of+gpt%2D4+augmented+flan+reasoning+traces%2C+with+verification%2C+2023%2E+URL+http)
42. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). lm sys. FastChat: An open platform for training, serving, and evaluating large language model bas…
43. Locutusque. Ultratextbooks, 2024. URL [https://huggingface.co/datasets/Locutusque/](https://huggingface.co/datasets/Locutusque/) UltraTextbooks. [↗](https://huggingface.co/datasets/Locutusque/)
44. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/), 2023.
45. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2C+2017%2E)
46. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024. [find](https://scholar.google.com/scholar?q=Anton+Lozhkov%2C+Raymond+Li%2C+Loubna+Ben+Allal%2C+Federico+Cassano%2C+Joel+Lamy%2DPoirier%2C+Nouamane+Tazi%2C+Ao+Tang%2C+Dmytro+Pykhtar%2C+Jiawei+Liu%2C+Yuxiang+Wei%2C+et+al%2E+Starcoder+2+and+the+stack+v2%3A+The+next+generat)
47. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2C+2023%2E)
48. Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. [Orca-math: Unlocking the potential of slms in grade school math](https://md.research.elamur.ai/free/en/papers/mitra-2024-orca-math-unlocking-potential-slms/), 2024.
49. Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124, 2023a. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Qian+Liu%2C+Armel+Zebaze%2C+Qinkai+Zheng%2C+Binyuan+Hui%2C+Terry+Yue+Zhuo%2C+Swayam+Singh%2C+Xiangru+Tang%2C+Leandro+von+Werra%2C+and+Shayne+Longpre%2E+Octopack%3A+Instruction+tuning+code+large+langua)
50. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… — [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
51. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/), 2023.
52. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+et+al%2E+Efficient+large%2D)
53. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
54. Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, and Rameswar Panda. Dense training, sparse inference: Rethinking training of mixture-of-experts language models, 2024. [find](https://scholar.google.com/scholar?q=Bowen+Pan%2C+Yikang+Shen%2C+Haokun+Liu%2C+Mayank+Mishra%2C+Gaoyuan+Zhang%2C+Aude+Oliva%2C+Colin+Raffel%2C+and+Rameswar+Panda%2E+Dense+training%2C+sparse+inference%3A+Rethinking+training+of+mixture%2Dof%2Dexperts+language+mod)
55. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. [The lambada dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). arXiv preprint arXiv:1606.06031, 2016.
56. Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. [Openwebmath: An open dataset of high-quality mathematical web text](https://md.research.elamur.ai/free/en/papers/paster-2023-openwebmath-open-dataset-high-quality/), 2023.
57. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+refinedweb+dataset+for+falcon+)
58. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). arXiv preprint arXiv:2304.03277, 2023.
59. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/), 2023.
60. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67, 2020. URL [http://jmlr.org/papers/v21/20…](http://jmlr.org/papers/v21/20…) [↗](http://jmlr.org/papers/v21/20…)
61. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2C+2020%2E)
62. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosse… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
63. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2C+2021%2E)
64. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). arXiv preprint arXiv:1904.09728, 2019.
65. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+arXiv+preprint)
66. Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, and Chuang Gan. Moduleformer: Learning modular large language models from uncurated data. arXiv preprint arXiv:2306.[04640, 2023](https://md.research.elamur.ai/free/en/papers/shen-2023-moduleformer-modularity-emerges-mixture-experts/).
67. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprin)
68. Qingyi Si, Tong Wang, Zheng Lin, Xu Zhang, Yanan Cao, and Weiping Wang. An empirical study of instruction-tuning large language models in chinese, 2023. [find](https://scholar.google.com/scholar?q=Qingyi+Si%2C+Tong+Wang%2C+Zheng+Lin%2C+Xu+Zhang%2C+Yanan+Cao%2C+and+Weiping+Wang%2E+An+empirical+study+of+instruction%2Dtuning+large+language+models+in+chinese%2C+2023%2E)
69. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Na… [find](https://scholar.google.com/scholar?q=Luca+Soldaini%2C+Rodney+Kinney%2C+Akshita+Bhagia%2C+Dustin+Schwenk%2C+David+Atkinson%2C+Russell+Authur%2C+Ben+Bogin%2C+Khyathi+Chandu%2C+Jennifer+Dumas%2C+Yanai+Elazar%2C+Valentin+Hofmann%2C+Ananya+Harsh+Jha%2C+Sachin+Kumar%2C)
70. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063, 2024.
71. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
72. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivie%CC%80re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+et+al%2E+Gemma%3A+Open+models+based+on+gemi)
73. Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL [https://huggingface.co/datasets/teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5). [↗](https://huggingface.co/datasets/teknium/OpenHermes-2.5)
74. Teknium1. GPTeacher: A collection of modular datasets generated by GPT-4. [https://github.com/teknium1/GPTeacher](https://github.com/teknium1/GPTeacher), 2023. [↗](https://github.com/teknium1/GPTeacher)
75. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
76. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. The alignment handbook. [https://github.com/huggingface/alignment-handbook](https://github.com/huggingface/alignment-handbook), 2023a. [↗](https://github.com/huggingface/alignment-handbook)
77. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment, 2023b. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cle%CC%81mentine+Fourrier%2C+Nathan+Habib%2C+Nathan+Sarrazin%2C+Omar+Sanseviero%2C+A)
78. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
79. Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints, 2023a. [find](https://scholar.google.com/scholar?q=Chaoqi+Wang%2C+Yibo+Jiang%2C+Chenghao+Yang%2C+Han+Liu%2C+and+Yuxin+Chen%2E+Beyond+reverse+kl%3A+Generalizing+direct+preference+optimization+with+diverse+divergence+constraints%2C+2023a%2E)
80. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. [Scibench: Evaluating college-level scientific problem-solving abilities of large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/), 2023b.

*Updated 30 October 2026*