---
title: "Expanding Pretrained Models to Thousands More Languages via"
description: "Multilingual pretrained models such as mBERT work poorly for languages that had little or no text… 38 references, cited by 47 papers here."
authors: ["Xinyi Wang", "Sebastian Ruder", "Graham Neubig"]
year: 2022
venue: "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2022.acl-long.61/"
doi: "10.18653/v1/2022.acl-long.61"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2022-expanding-pretrained-models-thousands-more/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2022-expanding-pretrained-models-thousands-more/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation

# Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation

**Authors:** [Xinyi Wang](https://md.research.elamur.ai/free/en/authors/xinyi-wang/), [Sebastian Ruder](https://md.research.elamur.ai/free/en/authors/sebastian-ruder/) and [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) · **Year:** 2022 · **Venue:** [Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2022](https://md.research.elamur.ai/free/en/venues/acl/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.acl-long.61/) · [PDF](https://aclanthology.org/2022.acl-long.61.pdf) · [DOI 10.18653/v1/2022.acl-long.61](https://doi.org/10.18653/v1/2022.acl-long.61) · [OpenAlex](https://openalex.org/W4226395792)

**Citations:** 25 citations (OpenAlex, 2026-10-14) · [47](https://md.research.elamur.ai/free/en/papers/wang-2022-expanding-pretrained-models-thousands-more/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Multilingual pretrained models such as mBERT work poorly for languages that had little or no text during pretraining. Standard fixes, like continued MLM training or translate-train, need monolingual or parallel corpora, and most of the world's languages lack them. The authors turn to bilingual lexicons, which exist for far more languages. Using PanLex entries, they translate English sentences word by word to create synthetic target-language data in two ways:
- **Pseudo MLM:** pseudo-monolingual text for continued pretraining.
- **Pseudo Trans-train:** pseudo-labeled task data for fine-tuning.

Words missing from the lexicon are left in English. The authors also test ways to clean this noisy data, such as label distillation and using a little parallel text. They evaluate two scenarios: No-Text, with essentially no target text, and Few-Text, with a small amount of monolingual data. Across 19 under-represented languages and 3 tasks, the synthetic data helps consistently. The best strategy depends on how much real text is available. The work suggests a cheap way to extend NLP to thousands of underserved languages.

## Key points
- Estimated share of the world's ~7,000 languages covered by each resource: mBERT about 1%, Wikipedia/CommonCrawl 4%, the Bible 23%, and lexicons roughly 70%.
- Reported improvements reach up to 15 points without extra monolingual text and up to 5 points when such text is added.
- The No-Text and Few-Text settings correspond to Joshi et al.'s "Left-Behinds" and "Scraping-Bys" groups. Together these make up about 85% of languages.
- Experiments use English as the only source language and are limited to Latin-script targets. Morphology and word-order differences are ignored because analyzers are lacking for these languages.
- Code and data are released on GitHub (cindyxinyiwang/expand-via-lexicon-based-adaptation). PanLex lists over 5,000 languages, though some have only a few words.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Xinyi Wang, Sebastian Ruder and Graham Neubig. 2022. Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2022expanding,
  title = {{Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation}},
  author = {Xinyi Wang, Sebastian Ruder and Graham Neubig},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.61},
  url = {https://doi.org/10.18653/v1/2022.acl-long.61}
}
```

- [← previous Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT](https://md.research.elamur.ai/free/en/papers/jarrar-2022-wojood-nested-arabic-named-entity/)
- [next → Infusing Knowledge from Wikipedia to Enhance Stance Detection](https://md.research.elamur.ai/free/en/papers/he-2022-infusing-knowledge-wikipedia-enhance-stance/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

38 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [natius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa…](https://md.research.elamur.ai/free/en/papers/adelani-2021-masakhaner-named-entity-recognition-african/)
2. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. [Parsing with multilingual BERT, a small corpus, and a small treebank](https://md.research.elamur.ai/free/en/papers/chau-2020-parsing-multilingual-bert-small-corpus/). In Findings of EMNLP 2020.
3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
4. Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In NeurIPS. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Crosslingual+language+model+pretraining%2E+In+NeurIPS%2E)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL.
6. Dan Kondratyuk and Milan Straka. 2019. [75 languages, 1 model: Parsing universal dependencies universally](https://md.research.elamur.ai/free/en/papers/kondratyuk-2019-75-languages-1-model-parsing/). In EMNLP, Hong Kong, China.
7. Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2016. Learning crosslingual word embeddings without bilingual corpora. arXiv preprint arXiv:1606.09403. [find](https://scholar.google.com/scholar?q=Long+Duong%2C+Hiroshi+Kanayama%2C+Tengfei+Ma%2C+Steven+Bird%2C+and+Trevor+Cohn%2E+2016%2E+Learning+crosslingual+word+embeddings+without+bilingual+corpora%2E+arXiv+preprint+arXiv%3A1606%2E09403%2E)
8. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. [From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers](https://md.research.elamur.ai/free/en/papers/lauscher-2020-zero-hero-limitations-zero-shot/). In EMNLP, Online. Association for Computational Linguistics.
9. Abteen Ebrahimi and Katharina Kann. 2021. [How to adapt your pretrained multilingual model to 1600 languages](https://md.research.elamur.ai/free/en/papers/ebrahimi-2021-how-adapt-your-pretrained-multilingual/). In ACL, Online. Association for Computational Linguistics.
10. Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. [Cheap translation for cross-lingual named entity recognition](https://md.research.elamur.ai/free/en/papers/mayhew-2017-cheap-translation-cross-lingual-named/). In EMNLP, Copenhagen, Denmark. Association for Computational Linguistics.
11. Jost Gippert, Nikolaus Himmelmann, Ulrike Mosel, et al. 2006. Essentials of language documentation. Mouton de Gruyter Berlín. [find](https://scholar.google.com/scholar?q=Jost+Gippert%2C+Nikolaus+Himmelmann%2C+Ulrike+Mosel%2C+et+al%2E+2006%2E+Essentials+of+language+documentation%2E+Mouton+de+Gruyter+Berl%C3%ADn%2E)
12. Stephan Gouws and Anders Søgaard. 2015. Simple task-specific bilingual word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1386–1390, Denver, Colorado. Association for Computational… [find](https://scholar.google.com/scholar?q=Stephan+Gouws+and+Anders+S%C3%B8gaard%2E+2015%2E+Simple+task%2Dspecific+bilingual+word+embeddings%2E+In+Proceedings+of+the+2015+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguis)
13. Emily Kennedy Gref. 2016. Publishing in North American Indigenous Languages. Ph.D. thesis, University of London. [find](https://scholar.google.com/scholar?q=Emily+Kennedy+Gref%2E+2016%2E+Publishing+in+North+American+Indigenous+Languages%2E+Ph%2ED%2E+thesis%2C+University+of+London%2E)
14. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In ACL, Online. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+In+ACL%2C+Online%2E)
15. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Quoc+V+Le%2C+and+Ilya+Sutskever%2E+2013%2E+Exploiting+similarities+among+languages+for+machine+translation%2E+arXiv+preprint+arXiv%3A1309%2E4168%2E)
16. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. [When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models](https://md.research.elamur.ai/free/en/papers/muller-2021-when-being-unseen-mbert-is/). In NAACL, Online.
17. Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. 2018. Universal dependencies 2.2. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Mitchell+Abrams%2C+%C5%BDeljko+Agic%CC%81%2C+Lars+Ahrenberg%2C+Lene+Antonsen%2C+Maria+Jesus+Aranzabe%2C+Gashaw+Arutie%2C+Masayuki+Asahara%2C+Luma+Ateyah%2C+Mohammed+Attia%2C+et+al%2E+2018%2E+Universal+dependencies+2%2E2%2E)
18. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL 2018. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+Language+Model+Fine%2Dtuning+for+Text+Classification%2E+In+Proceedings+of+ACL+2018%2E)
19. Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, and Graham Neubig. 2021. Explicit Alignment Objectives for Multilingual Bidirectional Encoders. In Proceedings of NAACL 2021. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Melvin+Johnson%2C+Orhan+Firat%2C+Aditya+Siddhant%2C+and+Graham+Neubig%2E+2021%2E+Explicit+Alignment+Objectives+for+Multilingual+Bidirectional+Encoders%2E+In+Proceedings+of+NAACL+2021%2E)
20. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. [Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/). In ICML.
21. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. [The state and fate of linguistic diversity and inclusion in the NLP world](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/). In ACL, Online. Association for Computational Linguistics.
22. Yash Khemchandani, Sarvesh Mehtani, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, and Sunita Sarawagi. 2021. Exploiting language relatedness for low web-resource language model adaptation: An Indic languages study. In ACL, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yash+Khemchandani%2C+Sarvesh+Mehtani%2C+Vaidehi+Patil%2C+Abhijeet+Awasthi%2C+Partha+Talukdar%2C+and+Sunita+Sarawagi%2E+2021%2E+Exploiting+language+relatedness+for+low+web%2Dresource+language+model+adaptation%3A+An+Indi)
23. Arya D. McCarthy, Rachel Wicks, Dylan Lewis, Aaron Mueller, Winston Wu, Oliver Adams, Garrett Nicolai, Matt Post, and David Yarowsky. 2020. [The Johns Hopkins University Bible corpus: 1600+ tongues for typological exploration](https://md.research.elamur.ai/free/en/papers/mccarthy-2020-johns-hopkins-university-bible-corpus/). In LREC, pages 2884–2892, Marseille, France. European Language Resources A…
24. Robert Östling and Jörg Tiedemann. 2016. Efficient word alignment with Markov Chain Monte Carlo. Prague Bulletin of Mathematical Linguistics, 106:125–146. [find](https://scholar.google.com/scholar?q=Robert+%C3%96stling+and+J%C3%B6rg+Tiedemann%2E+2016%2E+Efficient+word+alignment+with+Markov+Chain+Monte+Carlo%2E+Prague+Bulletin+of+Mathematical+Linguistics%2C+106%3A125%E2%80%93146%2E)
25. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In Proceedings of NAACL-HLT 2019: Demonstrations.
26. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. [Cross-lingual name tagging and linking for 282 languages](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/). In ACL, pages 1946–1958, Vancouver, Canada. ACL.
27. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/). In EMNLP, Online. Association for Computational Linguistics.
28. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021. [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://md.research.elamur.ai/free/en/papers/pfeiffer-2021-unks-everywhere-adapting-multilingual-language/). In Proceedings of EMNLP 2021.
29. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. [How multilingual is multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) In ACL, Florence, Italy.
30. Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulić, and Siva Reddy. 2021. Modelling latent translations for cross-lingual transfer. In Arxiv. [find](https://scholar.google.com/scholar?q=Edoardo+Maria+Ponti%2C+Julia+Kreutzer%2C+Ivan+Vulic%CC%81%2C+and+Siva+Reddy%2E+2021%2E+Modelling+latent+translations+for+cross%2Dlingual+transfer%2E+In+Arxiv%2E)
31. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. [Stanza: A Python natural language processing toolkit for many human languages](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/). In ACL.
32. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. [Massively multilingual transfer for NER](https://md.research.elamur.ai/free/en/papers/rahimi-2019-massively-multilingual-transfer-ner/). In ACL, pages 151–164, Florence, Italy. Association for Computational Linguistics.
33. Machel Reid and Mikel Artetxe. 2021. PAR-ADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining. arXiv preprint arXiv:2108.01887. — [arXiv:2108.01887](https://arxiv.org/abs/2108.01887) [find](https://scholar.google.com/scholar?q=Machel+Reid+and+Mikel+Artetxe%2E+2021%2E+PAR%2DADISE%3A+Exploiting+Parallel+Data+for+Multilingual+Sequence%2Dto%2DSequence+Pretraining%2E+arXiv+preprint+arXiv%3A2108%2E01887%2E+%E2%80%94+%5BarXiv%3A2108%2E01887%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
34. Shruti Rijhwani, Antonios Anastasopoulos, and Graham Neubig. 2020. OCR Post Correction for Endangered Language Texts. In EMNLP, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shruti+Rijhwani%2C+Antonios+Anastasopoulos%2C+and+Graham+Neubig%2E+2020%2E+OCR+Post+Correction+for+Endangered+Language+Texts%2E+In+EMNLP%2C+Online%2E+Association+for+Computational+Linguistics%2E)
35. Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson. 2021. [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://md.research.elamur.ai/free/en/papers/ruder-2021-xtreme-r-more-challenging-nuanced/). In Proceedings of EMNLP 2021.
36. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569–631. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Ivan+Vulic%CC%81%2C+and+Anders+S%C3%B8gaard%2E+2019%2E+A+survey+of+cross%2Dlingual+word+embedding+models%2E+Journal+of+Artificial+Intelligence+Research%2C+65%3A569%E2%80%93631%2E)
37. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In EMNLP-Findings, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zihan+Wang%2C+Karthikeyan+K%2C+Stephen+Mayhew%2C+and+Dan+Roth%2E+2020%2E+Extending+multilingual+BERT+to+low%2Dresource+languages%2E+In+EMNLP%2DFindings%2C+Online%2E+Association+for+Computational+Linguistics%2E)
38. Shijie Wu and Mark Dredze. 2019. [Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/). In EMNLP. *Figure 6: Improvements of using combined lexicons compared to PanLex lexicons for Pseudo MLM. Languages with fewer PanLex lexicons tend to benefit more from the combined lexicons.* A *…

*Updated 24 October 2026*