---
title: "mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer — Xue 2021"
description: "The authors also describe \"accidental translation\": in the zero-shot setting, the model… 85.0% accuracy on XNLI. 49 references, cited by 1,574 papers here."
authors: ["Linting Xue", "Noah Constant", "Adam Roberts", "Mihir Kale", "Rami Al-Rfou", "Aditya Siddhant", "Aditya Barua", "Colin Raffel"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.41/"
doi: "10.18653/v1/2021.naacl-main.41"
arxiv: "2010.11934"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/) / mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

# mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

**Authors:** Linting Xue, [Noah Constant](https://md.research.elamur.ai/free/en/authors/noah-constant/), [Adam Roberts](https://md.research.elamur.ai/free/en/authors/adam-roberts/), Mihir Kale, [Rami Al-Rfou](https://md.research.elamur.ai/free/en/authors/rami-al-rfou/), [Aditya Siddhant](https://md.research.elamur.ai/free/en/authors/aditya-siddhant/), Aditya Barua and [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/) · **Year:** 2021 · **Venue:** Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.41/) · [PDF](https://aclanthology.org/2021.naacl-main.41.pdf) · [DOI 10.18653/v1/2021.naacl-main.41](https://doi.org/10.18653/v1/2021.naacl-main.41) · [arXiv 2010.11934](https://arxiv.org/abs/2010.11934)

**Citations:** [1574](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Most popular pre-trained language models, including T5, are trained only on English text, so they are of limited use for other languages. The authors from Google Research created mT5, a multilingual version of T5. It stays as close as possible to the original approach: a unified "text-to-text" format, an encoder-decoder architecture, the span-corruption objective, and the improved T5.1.1 variant. For training, they assembled a new corpus, mC4, covering 101 languages. Its source was all 71 monthly Common Crawl snapshots. The English-centric C4 filters were replaced with a line-length filter, and the language of pages was identified with the cld3 tool. To keep low-resource languages from being drowned out by high-resource ones, examples were sampled with an exponent α=0.3. Models ranging from 300M to 13B parameters achieved state-of-the-art or near state-of-the-art results on multilingual benchmarks. The authors also describe "accidental translation": in the zero-shot setting, the model partially answers in English instead of the language of the query. They propose a simple technique that noticeably mitigates this problem. The code, checkpoints and data are publicly released, which makes mT5 a convenient foundation for NLP tasks in low-resource languages, including Ukrainian.

## Key points
- **mC4**: a web page was kept if it contained at least three lines of 200 or more characters and cld3 identified its language with a confidence of at least 70%. Languages with at least 10,000 pages were included in the corpus. This yielded 107 "languages" according to the cld3 classification, 6 of which are Latin-script versions of other languages (such as ru-Latn), so the actual number of languages is 101. Ukrainian (uk) is included in the corpus.
- **Model**: five sizes (Small, Base, Large, XL, XXL), a SentencePiece vocabulary of 250,000 wordpieces with byte-fallback enabled. The authors compared three values of α: 0.2, 0.3 and 0.7. The best balance between high-resource and low-resource languages was achieved with α=0.3.
- **Results**: in zero-shot transfer from English, the largest model, mT5-XXL, reaches 85.0% accuracy on XNLI. On TyDi QA-GoldP it obtained 82.0 F1 and 67.3 EM, outperforming XLM-R, InfoXLM, VECO and RemBERT. Quality improves markedly as model size increases.
- **Accidental translation**: it occurs most often in Small and Base, especially for Greek and Russian. The model may even switch to English in the middle of a word, for example "chlorопласт".
- **Solution (DPT)**: during fine-tuning on XQuAD, a small amount of unlabeled mC4 data is mixed in at a 1:100 ratio, with α=0.1 and without sentinel tokens. This reduces the rate of illegal predictions by more than 70% (relative).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/)

## Cite

Cite as: Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua and Colin Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xue2021mt5,
  title = {{mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer}},
  author = {Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua and Colin Raffel},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.41},
  eprint = {2010.11934},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.41}
}
```

- [← previous Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
- [next → Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/)

## References

49 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019. [our card](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
3. Diedre Carmo, Marcos Piau, Israel Campiotti, Rodrigo Nogueira, and Roberto Lotufo. 2020. PTT5: Pre-training and validating the t5 model on brazilian portuguese data. arXiv preprint arXiv:2008.09144. to the spirit of multilingual models and zero-shot evaluation. 11 [https://goo.gle/mt5-code](https://goo.gle/mt5-code) — [arXiv:2008.09144](https://arxiv.org/abs/2008.09144) [↗](https://goo.gle/mt5-code)
4. Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2020. Rethinking embedding coupling in pre-trained language models. arXiv preprint arXiv:2010.12821. [our card](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/)
5. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454– 47… [our card](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)
6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
7. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32, pages 7059–7069. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+32%2C+pages+7059%E2%80%937069%2E)
8. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
9. David Crystal. 2008. Two thousand million? English today, 24(1):3–6. [find](https://scholar.google.com/scholar?q=David+Crystal%2E+2008%2E+Two+thousand+million%3F+English+today%2C+24%281%29%3A3%E2%80%936%2E)
10. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT model. arXiv preprint arXiv:1912.09582. [our card](https://md.research.elamur.ai/free/en/papers/de-vries-2019-bertje-dutch-bert-model/)
11. Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. RobBERT: a dutch RoBERTa-based language model. arXiv preprint arXiv:2001.06286. [our card](https://md.research.elamur.ai/free/en/papers/delobelle-2020-robbert-dutch-roberta-based-language/)
12. Jacob Devlin. 2018. Multilingual BERT README. [https://github.com/](https://github.com/) google-research/bert/blob/master/ multilingual.md. [↗](https://github.com/)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
14. Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. 2020. FILTER: An enhanced fusion method for cross-lingual language understanding. arXiv preprint arXiv:2009.05166. — [arXiv:2009.05166](https://arxiv.org/abs/2009.05166) [find](https://scholar.google.com/scholar?q=Yuwei+Fang%2C+Shuohang+Wang%2C+Zhe+Gan%2C+Siqi+Sun%2C+and+Jingjing+Liu%2E+2020%2E+FILTER%3A+An+enhanced+fusion+method+for+cross%2Dlingual+language+understanding%2E+arXiv+preprint+arXiv%3A2009%2E05166%2E+%E2%80%94+%5BarXiv%3A2009%2E05166%5D%28)
15. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. A… [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+In+Proceedings+of+)
16. Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Demonstrations%2C+pages+66%E2%80%9371%2C+Brussels%2C+Belgium%2E+Association+for+Computational+Linguistics%2E)
17. Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised language model pre-training for French. In Proceedings of the 12th Language Resources and Evaluation Conference,… [our card](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/)
18. Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020a. Pre-training via paraphrasing. arXiv preprint arXiv:2006.15020. [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-pre-training-paraphrasing/)
19. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
20. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080. [our card](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/)
21. Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282. [our card](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/)
22. Mihir Kale. 2020. Text-to-text pre-training for data-to-text tasks. arXiv preprint arXiv:2005.10433. [find](https://scholar.google.com/scholar?q=Mihir+Kale%2E+2020%2E+Text%2Dto%2Dtext+pre%2Dtraining+for+data%2Dto%2Dtext+tasks%2E+arXiv+preprint+arXiv%3A2005%2E10433%2E)
23. Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Unifying question answering and text classification via span extraction. arXiv preprint arXiv:1904.09286. — [arXiv:1904.09286](https://arxiv.org/abs/1904.09286) [find](https://scholar.google.com/scholar?q=Nitish+Shirish+Keskar%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2019%2E+Unifying+question+answering+and+text+classification+via+span+extraction%2E+arXiv+preprint+arXiv%3A1904%2E09286%2E+%E2%80%94+%5BarXiv%3A1904%2E09)
24. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UnifiedQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, Online. Association for Com… [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/)
25. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)
26. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvinine… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
27. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. MLQA: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475. [our card](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/)
28. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020a. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020a%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+arXiv+pre)
29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
30. Zihan Liu, Genta Indra Winata, Andrea Madotto, and Pascale Fung. 2020b. Exploring fine-tuning techniques for pre-trained cross-lingual models via continual learning. arXiv preprint arXiv:2004.14218. — [arXiv:2004.14218](https://arxiv.org/abs/2004.14218) [find](https://scholar.google.com/scholar?q=Zihan+Liu%2C+Genta+Indra+Winata%2C+Andrea+Madotto%2C+and+Pascale+Fung%2E+2020b%2E+Exploring+fine%2Dtuning+techniques+for+pre%2Dtrained+cross%2Dlingual+models+via+continual+learning%2E+arXiv+preprint+arXiv%3A2004%2E14218%2E+%E2%80%94)
31. Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. 2020. Veco: Variable encoder-decoder pre-training for cross-lingual understanding and generation. arXiv preprint arXiv:2010.16046. — [arXiv:2010.16046](https://arxiv.org/abs/2010.16046) [find](https://scholar.google.com/scholar?q=Fuli+Luo%2C+Wei+Wang%2C+Jiahao+Liu%2C+Yijia+Liu%2C+Bin+Bi%2C+Songfang+Huang%2C+Fei+Huang%2C+and+Luo+Si%2E+2020%2E+Veco%3A+Variable+encoder%2Ddecoder+pre%2Dtraining+for+cross%2Dlingual+understanding+and+generation%2E+arXiv+prepri)
32. Martin Malmsten, Love Börjeson, and Chris Haffenden. 2020. Playing with words at the national library of sweden–making a swedish BERT. arXiv preprint arXiv:2007.01658. [our card](https://md.research.elamur.ai/free/en/papers/malmsten-2020-playing-words-national-library-sweden/)
33. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Language Processing (EMNLP), pages 5418–5426, Online. Associa… [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
34. Association for Computational Linguistics, pages 7203–7219, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Association+for+Computational+Linguistics%2C+pages+7203%E2%80%937219%2C+Online%2E+Association+for+Computational+Linguistics%2E)
35. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. [our card](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/)
36. Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. WT5?! Training text-to-text models to explain their predictions. arXiv preprint arXiv:2004.14546. [find](https://scholar.google.com/scholar?q=Sharan+Narang%2C+Colin+Raffel%2C+Katherine+Lee%2C+Adam+Roberts%2C+Noah+Fiedel%2C+and+Karishma+Malkan%2E+2020%2E+WT5%3F%21+Training+text%2Dto%2Dtext+models+to+explain+their+predictions%2E+arXiv+preprint+arXiv%3A2004%2E14546%2E)
37. Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1037–1042, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/nguyen-2020-phobert-pre-trained-language-models/)
38. Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-trained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira%2C+Zhiying+Jiang%2C+Ronak+Pradeep%2C+and+Jimmy+Lin%2E+2020%2E+Document+ranking+with+a+pre%2Dtrained+sequence%2Dto%2Dsequence+model%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP)
39. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. A… [our card](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/)
40. Jason Phang, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Iacer Calixto, and Samuel R Bowman. 2020. English intermediate-task training improves zero-shot cross-lingual transfer too. arXiv preprint arXiv:2005.13013. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Phu+Mon+Htut%2C+Yada+Pruksachatkun%2C+Haokun+Liu%2C+Clara+Vania%2C+Katharina+Kann%2C+Iacer+Calixto%2C+and+Samuel+R+Bowman%2E+2020%2E+English+intermediate%2Dtask+training+improves+zero%2Dshot+cross%2Dlingual+tr)
41. Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pages 15–18, Minneapolis, Minnesota. Assoc… [our card](https://md.research.elamur.ai/free/en/papers/ruder-2019-transfer-learning-natural-language-processing/)
42. Noam Shazeer. 2020. GLU variants improve transformer. arXiv preprint arXiv:2002.05202. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2020%2E+GLU+variants+improve+transformer%2E+arXiv+preprint+arXiv%3A2002%2E05202%2E)
43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
44. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003–4012, Marsei… [our card](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/)
45. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin… [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/)
46. Marco Polignano, Pierpaolo Basile, Marco de Gemmis, Giovanni Semeraro, and Valerio Basile. 2019. AlBERTo: Italian BERT language understanding model for NLP challenging tasks based on tweets. In CLiC-it. [our card](https://md.research.elamur.ai/free/en/papers/polignano-2019-alberto-italian-bert-language-understanding/)
47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
48. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
49. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural)