---
title: "SeaLLMs — Large Language Models for Southeast Asia (2023)"
description: "Mainstream LLMs are trained mostly on English. 41 references, cited by 85 papers here. Among them: 7."
authors: ["Xuan-Phi Nguyen", "Wenxuan Zhang", "Xin Li", "Mahani Aljunied", "Zhiqiang Hu", "Chenhui Shen", "Yew Ken Chia", "Xingxuan Li", "Jianyu Wang", "Qingyu Tan", "Liying Cheng", "Guanzheng Chen", "Yue Deng", "Sen Yang", "Chaoqun Liu", "Hang Zhang", "Lidong Bing"]
year: 2023
venue: "arXiv:2312.00738"
source: "https://arxiv.org/abs/2312.00738"
doi: "10.48550/arxiv.2312.00738"
arxiv: "2312.00738"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nguyen-2023-seallms-large-language-models-southeast/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nguyen-2023-seallms-large-language-models-southeast/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / SeaLLMs — Large Language Models for Southeast Asia

# SeaLLMs — Large Language Models for Southeast Asia

**Authors:** [Xuan-Phi Nguyen](https://md.research.elamur.ai/free/en/authors/xuan-phi-nguyen/), [Wenxuan Zhang](https://md.research.elamur.ai/free/en/authors/wenxuan-zhang/), [Xin Li](https://md.research.elamur.ai/free/en/authors/xin-li/), Mahani Aljunied, [Zhiqiang Hu](https://md.research.elamur.ai/free/en/authors/zhiqiang-hu/), [Chenhui Shen](https://md.research.elamur.ai/free/en/authors/chenhui-shen/), [Yew Ken Chia](https://md.research.elamur.ai/free/en/authors/yew-ken-chia/), [Xingxuan Li](https://md.research.elamur.ai/free/en/authors/xingxuan-li/), [Jianyu Wang](https://md.research.elamur.ai/free/en/authors/jianyu-wang/), [Qingyu Tan](https://md.research.elamur.ai/free/en/authors/qingyu-tan/), [Liying Cheng](https://md.research.elamur.ai/free/en/authors/liying-cheng/), [Guanzheng Chen](https://md.research.elamur.ai/free/en/authors/guanzheng-chen/), [Yue Deng](https://md.research.elamur.ai/free/en/authors/yue-deng/), [Sen Yang](https://md.research.elamur.ai/free/en/authors/sen-yang/), Chaoqun Liu, [Hang Zhang](https://md.research.elamur.ai/free/en/authors/hang-zhang/) and [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/) · **Year:** 2023 · **Venue:** arXiv:2312.00738 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2312.00738) · [PDF](https://arxiv.org/pdf/2312.00738) · [DOI 10.48550/arxiv.2312.00738](https://doi.org/10.48550/arxiv.2312.00738) · [arXiv 2312.00738](https://arxiv.org/abs/2312.00738) · [OpenAlex](https://openalex.org/W4389326213)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [85](https://md.research.elamur.ai/free/en/papers/nguyen-2023-seallms-large-language-models-southeast/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Mainstream LLMs are trained mostly on English. As a result, they handle Southeast Asian languages poorly, especially those written in non-Latin scripts, which standard BPE tokenizers break into very long token sequences. SeaLLMs are open models for Southeast Asian languages, built by adapting English-centric backbones: Llama-2-13B for v1, Mistral-7B for v2, and Gemma-7B for v2.5. Training has four stages:
1. Continued pre-training on filtered regional web, news and Wikipedia text mixed with high-quality English RedPajama data. For v1 and v2, this uses a vocabulary expanded with pieces taken from the NLLB tokenizer.
2. A hybrid stage that combines pre-training data with English instruction data.
3. Supervised fine-tuning on balanced multilingual data, partly synthetic and checked by native speakers, including country-specific safety examples.
4. DPO alignment on preference pairs that the model labels itself.

The models were evaluated on the new Sea-Bench (GPT-4 as judge), M3Exam, MMLU, GSM8K, MATH and Flores-200. They beat comparable open models and often surpass ChatGPT-3.5, with the largest gains in Thai, Khmer, Lao and Burmese. This makes useful assistants for under-served languages possible at low cost.

## Key points
- The models cover Vietnamese, Chinese, Thai, Indonesian, Khmer, Lao, Malay, Burmese and Tagalog, plus English. Three releases exist: SeaLLM-13B-v1, SeaLLM-7B-v2 and SeaLLM-7B-v2.5.
- Tokenization cost for Khmer, measured against English with the ChatGPT tokenizer, falls from about 15.6× to about 2.7× with the SeaLLM tokenizer. Gemma's 250K vocabulary needed no expansion.
- SeaLLM-7B-v2.5 scores 78.5 on GSM8K and 34.9 on MATH in English. On Thai GSM8K it reaches 62.0, against 35.8 for ChatGPT-3.5.
- SeaLLM-7B-v2 scores 7.54 on English MT-Bench, which is strong for a 7B multilingual model.
- On Flores-200 (chrF++), SeaLLM-13B-v1 clearly outperforms ChatGPT-3.5 for Lao and Khmer, including direct translation between SEA language pairs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 64 works · 14,113 citations
- [Alibaba DAMO Academy](https://md.research.elamur.ai/free/en/companies/alibaba-damo-academy/) — Company, Hangzhou, China · 28 works · 6,115 citations
- [Alibaba Group (Cayman Islands)](https://md.research.elamur.ai/free/en/companies/alibaba-group-cayman-islands/) — Company, George Town, Cayman Islands · 15 works · 3,209 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang and Lidong Bing. 2023. SeaLLMs — Large Language Models for Southeast Asia. arXiv:2312.00738.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nguyen2023seallms,
  title = {{SeaLLMs — Large Language Models for Southeast Asia}},
  author = {Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang and Lidong Bing},
  year = {2023},
  doi = {10.48550/arxiv.2312.00738},
  eprint = {2312.00738},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.00738}
}
```

- [← previous INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models](https://md.research.elamur.ai/free/en/papers/chia-2023-instructeval-towards-holistic-evaluation-instruction/)
- [next → ChatHaruhi: Reviving Anime Character in Reality via Large Language Model](https://md.research.elamur.ai/free/en/papers/li-2023-chatharuhi-reviving-anime-character-reality/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

41 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. [MEGA: multilingual evaluation of generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/). CoRR, abs/2303.12528.
2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the Association for Computational Linguistics, 5:135–146.
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
6. Together Computer. 2023. Redpajama: An open source recipe to reproduce llama training dataset. [find](https://scholar.google.com/scholar?q=Together+Computer%2E+2023%2E+Redpajama%3A+An+open+source+recipe+to+reproduce+llama+training+dataset%2E)
7. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+2022%2E+No+language+left+behind%3A+Scaling+h)
8. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023. [Multilingual jailbreak challenges in large language models](https://md.research.elamur.ai/free/en/papers/deng-2023-multilingual-jailbreak-challenges-large-language/). arXiv preprint arXiv:2310.06474.
9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations (ICLR).
10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS.
11. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
12. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+2024%2E+Mixtral+o)
13. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. [Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). CoRR, abs/2304.05613.
14. Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. 2023. Platypus: Quick, cheap, and powerful refinement of llms. [find](https://scholar.google.com/scholar?q=Ariel+N%2E+Lee%2C+Cole+J%2E+Hunter%2C+and+Nataniel+Ruiz%2E+2023%2E+Platypus%3A+Quick%2C+cheap%2C+and+powerful+refinement+of+llms%2E)
15. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca. [find](https://scholar.google.com/scholar?q=Wing+Lian%2C+Bleys+Goodson%2C+Eugene+Pentland%2C+Austin+Cook%2C+Chanvichet+Vong%2C+and+%22Teknium%22%2E+2023%2E+Openorca%3A+An+open+dataset+of+gpt+augmented+flan+reasoning+traces%2E+https%3A%2F%2Fhttps%3A%2F%2Fhuggingface%2Eco%2FOpen%2DOrca)
16. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/).
17. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+2023%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Df)
18. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. [Crosslingual generalization through multitask finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/). arXiv preprint arXiv:2211.01786.
19. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/).
20. Xuan-Phi Nguyen, Sharifah Mahani Aljunied, Shafiq Joty, and Lidong Bing. 2023. Democratizing llms for low-resource languages by leveraging their english dominant abilities with linguistically-diverse prompts. arXiv preprint arXiv:2306.11372. [find](https://scholar.google.com/scholar?q=Xuan%2DPhi+Nguyen%2C+Sharifah+Mahani+Aljunied%2C+Shafiq+Joty%2C+and+Lidong+Bing%2E+2023%2E+Democratizing+llms+for+low%2Dresource+languages+by+leveraging+their+english+dominant+abilities+with+linguistically%2Ddiverse+)
21. OpenAI. 2023a. Chatgpt (june 2023 version. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023a%2E+Chatgpt+%28june+2023+version%2E)
22. OpenAI. 2023b. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint.
23. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
24. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290.
25. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv e-prints.
26. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725. Association for Computational Linguistics.
28. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008– 3021. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+2020%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural)
29. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Beno%C3%AEt+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+pipeline+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+In+7th+Workshop+on+the+Challenges+in+the)
30. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+et+al%2E+2024%2E+Gemma%3A+Open+models+based+on)
31. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239.
32. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv p…
33. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023b%2E+Llama+2%3A+Open+foundation+and+f)
34. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. Zephyr: Direct distillation of lm alignment. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+Nathan+Sarrazin%2C+Omar+Sanseviero%2C+Al)
35. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+pre)
36. Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023. Polylm: An open source polyglot large language model. arXiv preprint arXiv:2307.06018. [find](https://scholar.google.com/scholar?q=Xiangpeng+Wei%2C+Haoran+Wei%2C+Huan+Lin%2C+Tianhao+Li%2C+Pei+Zhang%2C+Xingzhang+Ren%2C+Mei+Li%2C+Yu+Wan%2C+Zhiwei+Cao%2C+Binbin+Xie%2C+et+al%2E+2023%2E+Polylm%3A+An+open+source+polyglot+large+language+model%2E+arXiv+preprint+arX)
37. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. [CCNet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/). In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003–4012, Marsei…
38. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284.
39. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+Weston%2E+2024%2E+Self%2Drewarding+language+models%2E+arXiv+preprint+arXiv%3A2401%2E10020%2E)
40. Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. CoRR, abs/2306.05179. [find](https://scholar.google.com/scholar?q=Wenxuan+Zhang%2C+Sharifah+Mahani+Aljunied%2C+Chang+Gao%2C+Yew+Ken+Chia%2C+and+Lidong+Bing%2E+2023%2E+M3exam%3A+A+multilingual%2C+multimodal%2C+multilevel+benchmark+for+examining+large+language+models%2E+CoRR%2C+abs%2F2306%2E05)
41. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+2023%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+a)

*Updated 24 October 2026*