---
title: "XLM-V — paper by Liang et al. (2023) — summary and ideas"
description: "The authors build a one-million-token vocabulary. 3.5 points. 33 references, cited by 75 papers here."
authors: ["Davis Liang", "Hila Gonen", "Yuning Mao", "Rui Hou", "Naman Goyal", "Marjan Ghazvininejad", "Luke Zettlemoyer", "Madian Khabsa"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.813/"
doi: "10.18653/v1/2023.emnlp-main.813"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liang-2023-xlm-v-overcoming-vocabulary-bottleneck/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liang-2023-xlm-v-overcoming-vocabulary-bottleneck/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models

# XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models

**Authors:** [Davis Liang](https://md.research.elamur.ai/free/en/authors/davis-liang/), [Hila Gonen](https://md.research.elamur.ai/free/en/authors/hila-gonen/), [Yuning Mao](https://md.research.elamur.ai/free/en/authors/yuning-mao/), [Rui Hou](https://md.research.elamur.ai/free/en/authors/rui-hou/), [Naman Goyal](https://md.research.elamur.ai/free/en/authors/naman-goyal/), [Marjan Ghazvininejad](https://md.research.elamur.ai/free/en/authors/marjan-ghazvininejad/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) and [Madian Khabsa](https://md.research.elamur.ai/free/en/authors/madian-khabsa/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.813/) · [PDF](https://aclanthology.org/2023.emnlp-main.813.pdf) · [DOI 10.18653/v1/2023.emnlp-main.813](https://doi.org/10.18653/v1/2023.emnlp-main.813) · [OpenAlex](https://openalex.org/W4389524246)

**Citations:** 42 citations (OpenAlex, 2026-10-14) · [75](https://md.research.elamur.ai/free/en/papers/liang-2023-xlm-v-overcoming-vocabulary-bottleneck/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Multilingual encoders such as XLM-R have grown in depth and parameter count, yet their vocabularies have stayed near 250K tokens shared by more than 100 languages. That leaves only a few thousand distinctive tokens per language and hurts token-level tasks. The authors build a one-million-token vocabulary. They train monolingual sentencepiece models on text sampled from CC100 and describe each language by the unigram log probabilities of its tokens rather than by binary presence vectors. They then group the languages with K-Means (k = 8), split vocabulary budget across clusters using the ALP-based allocations of Zheng et al. (2021), and merge the per-cluster vocabularies. With this vocabulary they pretrain XLM-V on CC100 using the XLM-R recipe. XLM-V beats XLM-R on every benchmark tested, and the largest gains come in low-resource languages. Its segmentations are also shorter and more meaningful. The work shows that vocabulary size is an underused scaling dimension, which matters for under-represented languages, Ukrainian included.

## Key points
- **Vocabulary construction:**
  - Each language gets a 30,000-token sentencepiece model, trained on 1B lines sampled from CC100 with temperature 2.
  - Lexically unrelated languages are kept in separate clusters, so tokens shared between dissimilar languages are discouraged.
  - About 90% of each cluster's tokens are unique.
- **Capacity allocation:**
  - The Chinese/Japanese cluster receives 102,722 tokens, compared with 28,593 under the allocation method of Chung et al. (2020).
  - Uncovered tail languages get a fixed budget of 2,000 tokens.
- **Downstream results:**
  - Averaged over XQuAD, MLQA, TyDiQA, XNLI and WikiAnn, XLM-V scores 3.5 points higher than XLM-R.
  - On low-resource benchmarks the gains are larger: +11.2 F1 on MasakhaNER and +5.8 accuracy on AmericasNLI.
- **Tokenization efficiency:** On FLoRes-200, sequences are 11.5% shorter than with the XLM-R tokenizer.
- **Cost and limits:**
  - Embeddings make up 93% of parameters, versus 77% in XLM-R.
  - Moving from a 250K to a 1M vocabulary roughly triples trainable parameters (3.3x) while adding only about 25% to training time.
  - Expanding the vocabulary beyond 1M tokens degraded downstream results.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works

## Cite

Cite as: Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer and Madian Khabsa. 2023. XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liang2023xlm,
  title = {{XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models}},
  author = {Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer and Madian Khabsa},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.813},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.813}
}
```

- [← previous A Length-Extrapolatable Transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/)
- [next → How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN](https://md.research.elamur.ai/free/en/papers/mccoy-2023-how-much-do-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. 2021. [Masakhaner: named entity recognition for african languages](https://md.research.elamur.ai/free/en/papers/adelani-2021-masakhaner-named-entity-recognition-african/). Transactions of the Association for Computational Li…
2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). arXiv preprint arXiv:1910.11856.
3. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. [find](https://scholar.google.com/scholar?q=Alexei+Baevski+and+Michael+Auli%2E+2018%2E+Adaptive+input+representations+for+neural+language+modeling%2E+arXiv+preprint+arXiv%3A1809%2E10853%2E)
4. Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. 2015. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297. — [arXiv:1511.06297](https://arxiv.org/abs/1511.06297) [find](https://scholar.google.com/scholar?q=Emmanuel+Bengio%2C+Pierre%2DLuc+Bacon%2C+Joelle+Pineau%2C+and+Doina+Precup%2E+2015%2E+Conditional+computation+in+neural+networks+for+faster+models%2E+arXiv+preprint+arXiv%3A1511%2E06297%2E+%E2%80%94+%5BarXiv%3A1511%2E06297%5D%28https%3A%2F%2Far)
5. Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. EMNLP. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Dan+Garrette%2C+Kiat+Chuan+Tan%2C+and+Jason+Riesa%2E+2020%2E+Improving+multilingual+models+with+language%2Dclustered+vocabularies%2E+EMNLP%2E)
6. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. [Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics, 8:454–470.
7. Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. [Canine: Pre-training an efficient tokenization-free encoder for language representation](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/). Transactions of the Association for Computational Linguistics, 10:73–91.
8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Unsupervised+cross%2Dlingu)
9. Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Guillaume+Lample%2C+Ruty+Rinott%2C+Adina+Williams%2C+Samuel+R+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+arXiv+preprint+arXi)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
11. Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir, et al. 2021. [Americasnli: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages](https://md.research.elamur.ai/free/en/papers/ebrahimi-2022-americasnli-evaluating-zero-shot-natural/). arXiv…
12. Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021. [Larger-scale transformers for multilingual masked language modeling](https://md.research.elamur.ai/free/en/papers/goyal-2021-larger-scale-transformers-multilingual-masked/). arXiv preprint arXiv:2105.00572.
13. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computat… [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzman%2C+and+Angela+Fan%2E+2022%2E+The+flores%2D101+evaluation+benchmark)
14. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+Xtreme%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
15. Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou, et al. 2017. [Efficient softmax approximation for gpus](https://md.research.elamur.ai/free/en/papers/grave-2016-efficient-softmax-approximation-gpus/). In International conference on machine learning, pages 1302–1310. PMLR.
16. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
17. Taku Kudo and John Richardson. 2018. [Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). arXiv preprint arXiv:1808.06226.
18. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+arXiv+preprint+arXiv%3A1901%2E07291%2E)
19. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. [Mlqa: Evaluating cross-lingual extractive question answering](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/). arXiv preprint arXiv:1910.07475.
20. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. [Few-shot learning with multilingual language models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/). arXiv preprint arXiv:2112.10668.
21. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactio)
22. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2020. [How good is your tokenizer? on the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). arXiv preprint arXiv:2012.15613.
23. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
24. Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory, 13(2):260–269. [find](https://scholar.google.com/scholar?q=Andrew+Viterbi%2E+1967%2E+Error+bounds+for+convolutional+codes+and+an+asymptotically+optimum+decoding+algorithm%2E+IEEE+transactions+on+Information+Theory%2C+13%282%29%3A260%E2%80%93269%2E)
25. Hai Wang, Dian Yu, Kai Sun, Janshu Chen, and Dong Yu. 2019. Improving pre-trained multilingual models with vocabulary expansion. arXiv preprint arXiv:1909.12440. — [arXiv:1909.12440](https://arxiv.org/abs/1909.12440) [find](https://scholar.google.com/scholar?q=Hai+Wang%2C+Dian+Yu%2C+Kai+Sun%2C+Janshu+Chen%2C+and+Dong+Yu%2E+2019%2E+Improving+pre%2Dtrained+multilingual+models+with+vocabulary+expansion%2E+arXiv+preprint+arXiv%3A1909%2E12440%2E+%E2%80%94+%5BarXiv%3A1909%2E12440%5D%28https%3A%2F%2Farxiv%2Eorg)
26. Wikipedia. 2023. Table of general standard chinese characters — wikipedia, the free encyclopedia. [http://en.wikipedia.org/w/index.php?](http://en.wikipedia.org/w/index.php?) title=Table%20of%20General%20Standard% 20Chinese%20Characters&oldid=1123968033. [Online; accessed 05-January-2023]. [↗](http://en.wikipedia.org/w/index.php?)
27. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. The multi-genre nli corpus. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R+Bowman%2E+2018%2E+The+multi%2Dgenre+nli+corpus%2E)
28. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Aditya+Barua%2C+Noah+Constant%2C+Rami+Al%2DRfou%2C+Sharan+Narang%2C+Mihir+Kale%2C+Adam+Roberts%2C+and+Colin+Raffel%2E+2022%2E+Byt5%3A+Towards+a+token%2Dfree+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E+Transac)
29. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2E)
30. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). arXiv preprint arXiv:2010.11934.
31. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. [Fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). arXiv preprint arXiv:1904.01038.
32. Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021. Allocating large vocabulary capacity for cross-lingual language model pre-training. EMNLP. [find](https://scholar.google.com/scholar?q=Bo+Zheng%2C+Li+Dong%2C+Shaohan+Huang%2C+Saksham+Singhal%2C+Wanxiang+Che%2C+Ting+Liu%2C+Xia+Song%2C+and+Furu+Wei%2E+2021%2E+Allocating+large+vocabulary+capacity+for+cross%2Dlingual+language+model+pre%2Dtraining%2E+EMNLP%2E)
33. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. [Cross-lingual name tagging and linking for 282 languages](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958.

*Updated 24 October 2026*