---
title: "MaLA-500 — paper by Lin et al. (2024) — summary and ideas"
description: "Most large language models are trained mainly on English or a few dozen languages, so they perform… 55 references, cited by 55 papers here."
authors: ["Peiqin Lin", "Shaoxiong Ji", "Jorg Tiedemann", "Andre F. T. Martins", "Hinrich Schutze"]
year: 2024
venue: "arXiv:2401.13303"
source: "https://arxiv.org/abs/2401.13303"
doi: "10.48550/arxiv.2401.13303"
arxiv: "2401.13303"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lin-2024-mala-500-massive-language-adaptation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-2024-mala-500-massive-language-adaptation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / MaLA-500: Massive Language Adaptation of Large Language Models

# MaLA-500: Massive Language Adaptation of Large Language Models

**Authors:** [Peiqin Lin](https://md.research.elamur.ai/free/en/authors/peiqin-lin/), [Shaoxiong Ji](https://md.research.elamur.ai/free/en/authors/shaoxiong-ji/), [Jorg Tiedemann](https://md.research.elamur.ai/free/en/authors/jorg-tiedemann/), [Andre F. T. Martins](https://md.research.elamur.ai/free/en/authors/andre-f-t-martins/) and [Hinrich Schutze](https://md.research.elamur.ai/free/en/authors/hinrich-schutze/) · **Year:** 2024 · **Venue:** arXiv:2401.13303 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.13303) · [PDF](https://arxiv.org/pdf/2401.13303) · [DOI 10.48550/arxiv.2401.13303](https://doi.org/10.48550/arxiv.2401.13303) · [arXiv 2401.13303](https://arxiv.org/abs/2401.13303) · [OpenAlex](https://openalex.org/W4391244633)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/lin-2024-mala-500-massive-language-adaptation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most large language models are trained mainly on English or a few dozen languages, so they perform poorly on low-resource languages. Earlier massively multilingual work, such as Glot500-m, reached hundreds of languages but only with small encoder models. MaLA-500 adapts a generative decoder model, LLaMA 2 7B, to 534 languages using the Glot500-c corpus.

The authors train a SentencePiece tokenizer on sampled Glot500-c data and merge it with the original LLaMA 2 vocabulary. They then continue pretraining with LoRA. The original transformer weights stay frozen, while the embeddings and the language modeling head are updated in full.

Evaluation has two parts. Intrinsic tests measure negative log-likelihood on the Glot500-c test set and the Parallel Bible Corpus. Extrinsic tests use few-shot in-context topic classification on SIB200 and Taxi1500. MaLA-500 models low-resource text better than comparable open multilingual LLMs. Its macro-average accuracy beats prior models by 11.68% on SIB200 and 4.82% on Taxi1500.

The weights and code are public, which gives an open generative baseline for many underserved languages.

## Key points
- **Data:** Glot500-c covers 534 languages, identified by ISO 639-3 code plus script. Languages were resampled with a multinomial distribution (α = 0.3), giving about 1 billion training sentences (600K–8M per language). About 50M sentences were used to build the vocabulary.
- **Tokenizer:** a 250K SentencePiece vocabulary was merged with LLaMA 2's 32,000 tokens, giving 260,164 tokens. Segmentation length shrank by 8% for English and by up to 88% for Oriya. Some Latin-script languages, such as Kabiyè and Vietnamese, gained around 50%.
- **Training setup:**
  - Model size: 8.6B after embedding resizing, about 10B total, of which roughly 2B are trainable.
  - LoRA settings: applied to the query and value matrices, with rank 8, alpha 32 and dropout 0.05.
  - Run: 4k context, global batch size 384, three epochs, on 24 A100 GPUs using DeepSpeed/ZeRO.
- **Evaluation coverage:**
  - Model-comparable NLL on all 534 Glot500-c languages.
  - Language-comparable NLL on PBC (370 languages).
  - Accuracy on SIB200 (177 languages, 7 topic classes) and Taxi1500 (351 languages).
- **Release and limits:** weights are at huggingface.co/MaLA-LM and code at github.com/MaLA-LM/mala-500. The model is not instruction-tuned, and machine translation is left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Ludwig-Maximilians-Universität München](https://md.research.elamur.ai/free/en/universities/ludwig-maximilians-universitat-munchen/) — University, Munich, Germany · 107 works · 56,478 citations
- [University of Helsinki](https://md.research.elamur.ai/free/en/universities/university-of-helsinki/) — University, Helsinki, Finland · 50 works · 5,318 citations
- [Munich Center for Machine Learning](https://md.research.elamur.ai/free/en/universities/munich-center-for-machine-learning/) — University, Munich, Germany · 57 works · 1,094 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Peiqin Lin, Shaoxiong Ji, Jorg Tiedemann, Andre F. T. Martins and Hinrich Schutze. 2024. MaLA-500: Massive Language Adaptation of Large Language Models. arXiv:2401.13303.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lin2024mala,
  title = {{MaLA-500: Massive Language Adaptation of Large Language Models}},
  author = {Peiqin Lin, Shaoxiong Ji, Jorg Tiedemann, Andre F. T. Martins and Hinrich Schutze},
  year = {2024},
  doi = {10.48550/arxiv.2401.13303},
  eprint = {2401.13303},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.13303}
}
```

- [← previous ChatQA: Surpassing GPT-4 on Conversational QA and RAG](https://md.research.elamur.ai/free/en/papers/liu-2024-chatqa-surpassing-gpt-4-conversational/)
- [next → MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance](https://md.research.elamur.ai/free/en/papers/pi-2024-mllm-protector-ensuring-mllm-s/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

55 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte. SERENGETI: Massively multilingual language models for africa. arXiv preprint arXiv:2212.10785, 2022. [find](https://scholar.google.com/scholar?q=Ife+Adebara%2C+AbdelRahim+Elmadany%2C+Muhammad+Abdul%2DMageed%2C+and+Alcides+Alcoba+Inciarte%2E+SERENGETI%3A+Massively+multilingual+language+models+for+africa%2E+arXiv+preprint+arXiv%3A2212%2E10785%2C+2022%2E)
2. David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. [SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects](https://md.research.elamur.ai/free/en/papers/adelani-2023-sib-200-simple-inclusive-big/). CoRR, abs/2309.07445, 2023. doi: 10.48550/arXiv.2309.…
3. Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. [MEGA: multilingual evaluation of generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/). CoRR, abs/2303.12528, 2023a. doi: 10.48550/ arXiv.2303.12528. URL https://doi.org/10.4…
4. Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. [MEGAVERSE: benchmarking large language models across languages, modalities, models and tasks](https://md.research.elamur.ai/free/en/papers/ahuja-2024-megaverse-benchmarking-large-language-models/). CoRR, abs/2311.07463, 2023b. doi:…
5. Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. [Adapting pre-trained language models to african languages via multilingual adaptive fine-tuning](https://md.research.elamur.ai/free/en/papers/alabi-2022-adapting-pre-trained-language-models/). In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen,…
6. Duarte Alves, Nuno Guerreiro, João Alves, José Pombal, Ricardo Rei, José de Souza, Pierre Colombo, and André FT Martins. Steering large language models for machine translation with finetuning and in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1… [find](https://scholar.google.com/scholar?q=Duarte+Alves%2C+Nuno+Guerreiro%2C+Joa%CC%83o+Alves%2C+Jose%CC%81+Pombal%2C+Ricardo+Rei%2C+Jose%CC%81+de+Souza%2C+Pierre+Colombo%2C+and+Andre%CC%81+FT+Martins%2E+Steering+large+language+models+for+machine+translation+with+finetuning+and+)
7. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. [A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). CoRR, abs/2302.04023, 20…
8. Samuel Cahyawijaya, Holy Lovenia, Tiezheng Yu, Willy Chung, and Pascale Fung. Instruct-align: Teaching novel languages with to LLMs through alignment-based cross-lingual instruction. CoRR, abs/2305.[13627, 2023](https://md.research.elamur.ai/free/en/papers/cahyawijaya-2023-instructalign-high-low-resource-language/). doi: 10.48550/arXiv.2305.13627. URL https://doi.org/10.48550/arXiv.2305.13627. [↗](https://doi.org/10.48550/arXiv.2305.13627)
9. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. Parsing with multilingual bert, a small treebank, and a small corpus. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, [Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings…](https://md.research.elamur.ai/free/en/papers/chau-2020-parsing-multilingual-bert-small-corpus/)
10. Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Barry Haddow, and Kenneth Heafield. Monolingual or multilingual instruction tuning: Which makes a better Alpaca. In Findings of the Association for Computational Linguistics: EACL, 2024. URL https://doi.org/10. 48550/arXiv.2309.08958. [find](https://scholar.google.com/scholar?q=Pinzhen+Chen%2C+Shaoxiong+Ji%2C+Nikolay+Bogoychev%2C+Barry+Haddow%2C+and+Kenneth+Heafield%2E+Monolingual+or+multilingual+instruction+tuning%3A+Which+makes+a+better+Alpaca%2E+In+Findings+of+the+Association+for+Compu)
11. Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifeoluwa Adelani, Pontus Stenetorp, Sebastian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. CoRR, abs/2307.01163, 2023. doi: 10.48550/arXiv.2307.01163. URL https://doi.org/10.48550/arXiv.2307.011… — [doi:10.48550/arxiv.2307.01163](https://doi.org/10.48550/arxiv.2307.01163) [↗](https://doi.org/10.48550/arXiv.2307.011…)
12. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=r1xMH1BtvB](https://openreview.net/forum?id=r1xMH1BtvB). [↗](https://openreview.net/forum?id=r1xMH1BtvB)
13. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Co… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+E%CC%81douard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Unsupervised+cross%2Dlingual+r)
14. Yiming Cui, Ziqing Yang, and Xin Yao. [Efficient and effective text encoding for Chinese LLaMA and Alpaca](https://md.research.elamur.ai/free/en/papers/cui-2023-efficient-effective-text-encoding-chinese/). CoRR, abs/2304.08177, 2023. doi: 10.48550/ARXIV.2304.08177. URL https://doi.org/10.48550/arXiv.2304.08177. [↗](https://doi.org/10.48550/arXiv.2304.08177)
15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
16. C. M. Downey, Terra Blevins, Nora Goldfine, and Shane Steinert-Threlkeld. Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages. CoRR, abs/2309.04679, 2023. doi: 10.48550/arXiv.2309.04679. URL https://doi.org/10.48550/arXiv.2309.04679. [↗](https://doi.org/10.48550/arXiv.2309.04679)
17. Abteen Ebrahimi and Katharina Kann. [How to adapt your pretrained multilingual model to 1600 languages](https://md.research.elamur.ai/free/en/papers/ebrahimi-2021-how-adapt-your-pretrained-multilingual/). In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc…
18. Fahim Faisal and Antonios Anastasopoulos. Phylogeny-inspired adaptation of multilingual models to new languages. CoRR, abs/2205.09634, 2022. doi: 10.48550/arXiv.2205.09634. URL https://doi.org/10.48550/arXiv.2205.09634. [↗](https://doi.org/10.48550/arXiv.2205.09634)
19. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2022. URL [https://openreview.net/](https://openreview.net/) forum?id=nZeVKeeFYf9. [↗](https://openreview.net/)
20. Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze. Glot500: Scaling multilingual corpora and language models to 500 languages. In Anna Rogers, Jordan Boyd-Graber… [find](https://scholar.google.com/scholar?q=Ayyoob+ImaniGooghari%2C+Peiqin+Lin%2C+Amir+Hossein+Kargaran%2C+Silvia+Severini%2C+Masoud+Jalili+Sabet%2C+Nora+Kassner%2C+Chunlan+Ma%2C+Helmut+Schmid%2C+Andre%CC%81+Martins%2C+Franc%CC%A7ois+Yvon%2C+and+Hinrich+Schu%CC%88tze%2E+Glot500%3A+S)
21. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroi… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+Le%CC%81lio+Renard+La)
22. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+Mixtral+of+expe)
23. Taku Kudo and John Richardson. [SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations… — [doi:10.18653/v1/](https://doi.org/10.18653/v1/)
24. Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, et al. [MADLAD-400: A multilingual and document-level large audited dataset](https://md.research.elamur.ai/free/en/papers/kudugunta-2023-madlad-400-multilingual-document-level/). arXiv preprint arXiv:2309.04662, 2023.
25. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. [ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). CoRR, abs/2304.05613, 2023. doi: 10.48550/arXiv.2304.05613. URL https://…
26. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoy… — [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
27. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. [What makes good in-context examples for gpt-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) In Eneko Agirre, Marianna Apidianaki, and Ivan Vulic (eds.), Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep…
28. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742, 2020. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactions+of+)
29. Chunlan Ma, Ayyoob ImaniGooghari, Haotian Ye, Ehsaneddin Asgari, and Hinrich Schütze. Taxi1500: A multilingual dataset for text classification in 1500 languages, 2023. [find](https://scholar.google.com/scholar?q=Chunlan+Ma%2C+Ayyoob+ImaniGooghari%2C+Haotian+Ye%2C+Ehsaneddin+Asgari%2C+and+Hinrich+Schu%CC%88tze%2E+Taxi1500%3A+A+multilingual+dataset+for+text+classification+in+1500+languages%2C+2023%2E)
30. Thomas Mayer and Michael Cysouw. [Creating a massively parallel bible corpus](https://md.research.elamur.ai/free/en/papers/mayer-2014-creating-massively-parallel-bible-corpus/). In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asunción Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Ninth International Conference on Langu…
31. Benjamin Müller, Benoı̂t Sagot, and Djamé Seddah. [Can multilingual language models transfer to an unseen dialect? A case study on north african arabizi](https://md.research.elamur.ai/free/en/papers/muller-2020-multilingual-language-models-transfer-unseen/). CoRR, abs/2005.00318, 2020. URL https://arxiv.org/abs/2005.00318. [↗](https://arxiv.org/abs/2005.00318)
32. Benjamin Müller, Antonios Anastasopoulos, Benoı̂t Sagot, and Djamé Seddah. [When being unseen from mbert is just the beginning: Handling new languages with multilingual language models](https://md.research.elamur.ai/free/en/papers/muller-2021-when-being-unseen-mbert-is/). In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cot…
33. Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. [Trankit: A light-weight transformer-based toolkit for multilingual natural language processing](https://md.research.elamur.ai/free/en/papers/nguyen-2021-trankit-light-weight-transformer-based/). In Dimitra Gkatzia and Djamé Seddah (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for…
34. Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. [MAD-X: an adapter-based framework for multi-task cross-lingual transfer](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/). In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020,…
35. Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. [Unks everywhere: Adapting multilingual language models to new scripts](https://md.research.elamur.ai/free/en/papers/pfeiffer-2021-unks-everywhere-adapting-multilingual-language/). In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro…
36. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–16. IEEE, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+ZeRO%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing)
37. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505–3506, 2020. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+DeepSpeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26th+ACM+)
38. Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natur… [find](https://scholar.google.com/scholar?q=Masoud+Jalili+Sabet%2C+Philipp+Dufter%2C+Franc%CC%A7ois+Yvon%2C+and+Hinrich+Schu%CC%88tze%2E+Simalign%3A+High+quality+word+alignments+without+parallel+training+data+using+static+and+contextualized+embeddings%2E+In+Trevor+C)
39. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagne%CC%81%2C+Alexandra+Sasha+Luccioni%2C+Franc%CC%A7ois+Yvon%2C+Matthias+Galle%CC%81%2C+et+al%2E+BLOOM%3A+A+176b%2Dparameter+op)
40. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. [mGPT: Few-shot learners go multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/). CoRR, abs/2204.07580, 2022. doi: 10.48550/arXiv.2204.07580. URL https://doi.org/10.48550/ arXiv.2204.07580.
41. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. [UL2: Unifying language learning paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/). In The Eleventh International Conference on Learning Representations, 2022.
42. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, abs/2…
43. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
44. Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. Udapter: Language adaptation for truly universal dependency parsing. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, O… [find](https://scholar.google.com/scholar?q=Ahmet+U%CC%88stu%CC%88n%2C+Arianna+Bisazza%2C+Gosse+Bouma%2C+and+Gertjan+van+Noord%2E+Udapter%3A+Language+adaptation+for+truly+universal+dependency+parsing%2E+In+Bonnie+Webber%2C+Trevor+Cohn%2C+Yulan+He%2C+and+Yang+Liu+%28eds%2E%29%2C+P)
45. Xinyi Wang, Sebastian Ruder, and Graham Neubig. [Expanding pretrained models to thousands more languages via lexicon-based adaptation](https://md.research.elamur.ai/free/en/papers/wang-2022-expanding-pretrained-models-thousands-more/). In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:…
46. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. Extending multilingual BERT to low-resource languages. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of AC… [find](https://scholar.google.com/scholar?q=Zihan+Wang%2C+Karthikeyan+K%2C+Stephen+Mayhew%2C+and+Dan+Roth%2E+Extending+multilingual+BERT+to+low%2Dresource+languages%2E+In+Trevor+Cohn%2C+Yulan+He%2C+and+Yang+Liu+%28eds%2E%29%2C+Findings+of+the+Association+for+Computati)
47. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 conference on empirical methods in natural language process…
48. Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine translation: Boosting translation performance of large language models. CoRR, abs/2309.11674, 2023. doi: 10.48550/ARXIV.2309.11674. URL https://doi.org/10. 48550/arXiv.2309.11674. [find](https://scholar.google.com/scholar?q=Haoran+Xu%2C+Young+Jin+Kim%2C+Amr+Sharaf%2C+and+Hany+Hassan+Awadalla%2E+A+paradigm+shift+in+machine+translation%3A+Boosting+translation+performance+of+large+language+models%2E+CoRR%2C+abs%2F2309%2E11674%2C+2023%2E+doi%3A+10%2E)
49. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistic…
50. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Trans. Assoc. Comput. Linguistics, 10:291–306, 2022. doi: 10.1162/ tacl \\ a \\ 00461. URL https://doi.org/10.1162/… [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Aditya+Barua%2C+Noah+Constant%2C+Rami+Al%2DRfou%2C+Sharan+Narang%2C+Mihir+Kale%2C+Adam+Roberts%2C+and+Colin+Raffel%2E+Byt5%3A+Towards+a+token%2Dfree+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E+Trans%2E+Assoc%2E)
51. Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. Bigtrans: Augmenting large language models with multilingual translation capability over 100 languages. CoRR, abs/2305.18098, 2023. doi: 10.48550/arXiv.2305.18098. URL https://doi.org/10.48550/ arXiv.2305.18098. — [doi:10.48550/arxiv.2305.18098](https://doi.org/10.48550/arxiv.2305.18098) [find](https://scholar.google.com/scholar?q=Wen+Yang%2C+Chong+Li%2C+Jiajun+Zhang%2C+and+Chengqing+Zong%2E+Bigtrans%3A+Augmenting+large+language+models+with+multilingual+translation+capability+over+100+languages%2E+CoRR%2C+abs%2F2305%2E18098%2C+2023%2E+doi%3A+10%2E48550%2F)
52. Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M. Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Indra Winata, Stella Biderman, Dragomir Radev, and Vassilina Nikoulina. BLOOM+1: adding language support to BLOOM for z… — [BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting](https://md.research.elamur.ai/free/en/papers/yong-2023-bloom-1-adding-language-support/)
53. Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. MEGABYTE: predicting million-byte sequences with multiscale transformers. CoRR, abs/2305.07185, 2023. doi: 10.48550/arXiv.2305.07185. URL https://doi.org/10. 48550/arXiv.2305.07185. [find](https://scholar.google.com/scholar?q=Lili+Yu%2C+Daniel+Simig%2C+Colin+Flaherty%2C+Armen+Aghajanyan%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+MEGABYTE%3A+predicting+million%2Dbyte+sequences+with+multiscale+transformers%2E+CoRR%2C+abs%2F2305%2E07185%2C+2023%2E+doi%3A+10)
54. Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. LLaMA beyond English: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055, 2024. [find](https://scholar.google.com/scholar?q=Jun+Zhao%2C+Zhihao+Zhang%2C+Qi+Zhang%2C+Tao+Gui%2C+and+Xuanjing+Huang%2E+LLaMA+beyond+English%3A+An+empirical+study+on+language+capability+transfer%2E+arXiv+preprint+arXiv%3A2401%2E01055%2C+2024%2E)
55. Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Extrapolating large language models to non-english by aligning languages. CoRR, abs/2308.04948, 2023. doi: 10.48550/arXiv.2308.04948. URL https://doi.org/10.48550/arXiv.2308.04948. [↗](https://doi.org/10.48550/arXiv.2308.04948)

*Updated 31 October 2026*