---
title: "How Good is Your Tokenizer? On the Monolingual — Rust 2021"
description: "New monolingual BERT models keep being released for languages that multilingual models already cover."
authors: ["Phillip Rust", "Jonas Pfeiffer", "Ivan Vulić", "Sebastian Ruder", "Iryna Gurevych"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.243/"
doi: "10.18653/v1/2021.acl-long.243"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

# How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

**Authors:** Phillip Rust, [Jonas Pfeiffer](https://md.research.elamur.ai/free/en/authors/jonas-pfeiffer/), [Ivan Vulić](https://md.research.elamur.ai/free/en/authors/ivan-vulic/), [Sebastian Ruder](https://md.research.elamur.ai/free/en/authors/sebastian-ruder/), [Iryna Gurevych](https://md.research.elamur.ai/free/en/authors/iryna-gurevych/) · **Year:** 2021 · **Venue:** Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.243/) · [PDF](https://aclanthology.org/2021.acl-long.243.pdf) · [DOI 10.18653/v1/2021.acl-long.243](https://doi.org/10.18653/v1/2021.acl-long.243) · [OpenAlex](https://openalex.org/W3120253119)

**Citations:** 189 citations (openalex-snapshot · 2026-10-14) · [254](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary

New monolingual BERT models keep being released for languages that multilingual models already cover. The usual reason given is that multilingual models underperform on single-language tasks, but this claim has rarely been tested rigorously. Rust et al. test it with a controlled comparison of mBERT and existing monolingual BERT models across nine typologically diverse languages from eight families. They use five tasks: NER, sentiment analysis, QA, UD parsing and POS tagging.

To separate the effect of pretraining data volume from the effect of tokenization, they pretrain new monolingual models on identical data. One version uses the language's own tokenizer and the other uses the mBERT tokenizer. They also swap a monolingual tokenizer into mBERT.

The authors find that monolingual models usually win, but by smaller margins than commonly assumed. The gaps come largely from two sources: more pretraining data and a tokenizer fitted to the language. For languages that mBERT's vocabulary already represents well, the difference is negligible. The practical lesson is that improving multilingual tokenizers, or adapting them to a specific language, can close much of the gap without training a model from scratch.

## Key points

- **Languages and tasks:** The study covers languages including AR, EN, FI, ID, JA, KO, TR and ZH, evaluated on NER, SA, QA, UDP and POS.
- **Tokenizer quality matters:** Monolingual tokenizers with lower subword fertility and fewer continued (split) words gave consistent gains for AR, FI, ID, KO and TR. This held for both the MonoModel-* and mBERTModel-* variants.
- **Small gaps for well-covered languages:** For EN, JA and ZH, gaps were minimal, and mBERT even came out ahead on QA.
- **Correlation analysis (Spearman's ρ, ID excluded):** Reductions in fertility and in the continued-word proportion relate to downstream gains about as strongly as larger pretraining corpora do. Overall ρ is roughly 0.36 for fertility versus 0.38 for corpus size. The links are strongest for UDP and QA and weakest for NER and POS.
- **Resources:** mBERT's shared WordPiece vocabulary has 119,547 tokens. Code, pretrained models and adapters are released at github.com/Adapter-Hub/hgiyt.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/)

## Cited by

Cited by

254 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 242 not yet.

- [Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM Benchmarks](https://md.research.elamur.ai/free/en/papers/aclbulk-2026-inoue-do-diacritics-matter-evaluating-impact/)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/)
- [MEGA: Multilingual Evaluation of Generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/)
- [SantaCoder: don't reach for the stars!](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/)
- [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://md.research.elamur.ai/free/en/papers/barbieri-2022-xlm-t-multilingual-language-models/)
- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/)
- [Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models](https://md.research.elamur.ai/free/en/papers/langstream-2024-alrefaie-exploring-tokenization-strategies-vocabulary-sizes/)
- [Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya](https://md.research.elamur.ai/free/en/papers/langstream-2026-teklehaymanot-expanding-lexicon-ge-ez-based/)
- [Morphemes without Borders: Evaluating Root–Pattern Morphology in Arabic Tokenizers and LLMs](https://md.research.elamur.ai/free/en/papers/lingbulk-2026-alakeel-morphemes-without-borders-evaluating-root/)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2022-lifting-curse-multilinguality-pre-training/)
- [mGPT: Few-Shot Learners Go Multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rust2021how,
  title = {{How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models}},
  author = {Rust, Phillip and Pfeiffer, Jonas and Vulić, Ivan and Ruder, Sebastian and Gurevych, Iryna},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.243},
  url = {https://doi.org/10.18653/v1/2021.acl-long.243}
}
```

- [← previous How focus particles and accents affect attachment](https://md.research.elamur.ai/free/en/papers/carlson-2021-how-focus-particles-accents-affect/)
- [next → How many data points is a prompt worth?](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

70 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics. Judit Ács. 2019. Exploring BERT’s Vocabulary. Blog Post. [find](https://scholar.google.com/scholar?q=in+Natural+Language+Processing%2C+pages+2475%E2%80%932485%2C+Brussels%2C+Belgium%2E+Association+for+Computational+Linguistics%2E+Judit+A%CC%81cs%2E+2019%2E+Exploring+BERT%E2%80%99s+Vocabulary%2E+Blog+Post%2E)
2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Langu… [find](https://scholar.google.com/scholar?q=Wissam+Antoun%2C+Fady+Baly%2C+and+Hazem+Hajj%2E+2020%2E+AraBERT%3A+Transformer%2Dbased+model+for+Arabic+language+understanding%2E+In+Proceedings+of+the+4th+Workshop+on+Open%2DSource+Arabic+Corpora+and+Processing+Tool) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
3. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Sebastian+Ruder%2C+and+Dani+Yogatama%2E+2020%2E+On+the+cross%2Dlingual+transferability+of+monolingual+representations%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computati) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
4. Giusepppe Attardi. 2015. Repository. [find](https://scholar.google.com/scholar?q=Giusepppe+Attardi%2E+2015%2E+Repository%2E)
5. Wikiextractor. GitHub Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wikiextractor%2E+GitHub+Ethan+C%2E+Chau%2C+Lucy+H%2E+Lin%2C+and+Noah+A%2E+Smith%2E+2020%2E+Parsing+with+multilingual+BERT%2C+a+small+corpus%2C+and+a+small+treebank%2E+In+Findings+of+the+Association+for+Computational+Lingui)
6. Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536–4546, Online. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Dan+Garrette%2C+Kiat+Chuan+Tan%2C+and+Jason+Riesa%2E+2020%2E+Improving+multilingual+models+with+language%2Dclustered+vocabularies%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+)
7. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454– 47… [find](https://scholar.google.com/scholar?q=Jonathan+H%2E+Clark%2C+Eunsol+Choi%2C+Michael+Collins%2C+Dan+Garrette%2C+Tom+Kwiatkowski%2C+Vitaly+Nikolaev%2C+and+Jennimaria+Palomaki%2E+2020%2E+TyDi+QA%3A+A+benchmark+for+information%2Dseeking+question+answering+in+typol) [our card](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)
8. Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2021. CANINE: Pre-training an efficient tokenization-free encoder for language representation. arXiv preprint. [find](https://scholar.google.com/scholar?q=Jonathan+H%2E+Clark%2C+Dan+Garrette%2C+Iulia+Turc%2C+and+John+Wieting%2E+2021%2E+CANINE%3A+Pre%2Dtraining+an+efficient+tokenization%2Dfree+encoder+for+language+representation%2E+arXiv+preprint%2E)
9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
10. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods Erkin Demirtas and Mykola Pechenizkiy. 2013. Cross-lingual pol… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
12. Miguel Domingo, Mercedes Garcıa-Martınez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. 2019. How much does tokenization affect neural machine translation? arXiv preprint. [find](https://scholar.google.com/scholar?q=Miguel+Domingo%2C+Mercedes+Garc%C4%B1a%2DMart%C4%B1nez%2C+Alexandre+Helle%2C+Francisco+Casacuberta%2C+and+Manuel+Herranz%2E+2019%2E+How+much+does+tokenization+affect+neural+machine+translation%3F+arXiv+preprint%2E)
13. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France. OpenReview.net. [find](https://scholar.google.com/scholar?q=Timothy+Dozat+and+Christopher+D%2E+Manning%2E+2017%2E+Deep+biaffine+attention+for+neural+dependency+parsing%2E+In+Proceedings+of+the+5th+International+Conference+on+Learning+Representations+%28ICLR%29%2C+Toulon%2C+Fr) [our card](https://md.research.elamur.ai/free/en/papers/missing-2016-dozat-deep-biaffine-attention-neural-dependency/)
14. Pavel Efimov, Andrey Chertok, Leonid Boytsov, and Pavel Braslavski. 2020. SberQuAD – Russian Reading Comprehension Dataset: Description and analysis. In CLEF 2020: Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 3–15. Springer, Cham, Switzerland. [find](https://scholar.google.com/scholar?q=Pavel+Efimov%2C+Andrey+Chertok%2C+Leonid+Boytsov%2C+and+Pavel+Braslavski%2E+2020%2E+SberQuAD+%E2%80%93+Russian+Reading+Comprehension+Dataset%3A+Description+and+analysis%2E+In+CLEF+2020%3A+Experimental+IR+Meets+Multilingualit)
15. Ashraf Elnagar, Yasmin S. Khalifa, and Anas Einea. 2018. Hotel Arabic-Reviews Dataset Construction for Sentiment Analysis Applications. In Intelligent Natural Language Processing: Trends and Applications, pages 35–52. Springer, Cham, Switzerland. [find](https://scholar.google.com/scholar?q=Ashraf+Elnagar%2C+Yasmin+S%2E+Khalifa%2C+and+Anas+Einea%2E+2018%2E+Hotel+Arabic%2DReviews+Dataset+Construction+for+Sentiment+Analysis+Applications%2E+In+Intelligent+Natural+Language+Processing%3A+Trends+and+Applicati)
16. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 316–327, Brussels,… [find](https://scholar.google.com/scholar?q=Daniela+Gerz%2C+Ivan+Vulic%CC%81%2C+Edoardo+Maria+Ponti%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2018%2E+On+the+relation+between+linguistic+typology+and+%28limitations+of%29+multilingual+language+modeling%2E+In+Proceedings+o)
17. Goran Glavaš and Ivan Vulić. 2021. Is supervised syntactic parsing beneficial for language understanding tasks? an empirical investigation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3090–3104, Online. Associat… [find](https://scholar.google.com/scholar?q=Goran+Glavas%CC%8C+and+Ivan+Vulic%CC%81%2E+2021%2E+Is+supervised+syntactic+parsing+beneficial+for+language+understanding+tasks%3F+an+empirical+investigation%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chap)
18. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International 2011. Learning word vectors for sentiment analysis. In Procee… [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+XTREME%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
19. Conference on Machine Learning, pages 4411–4421, Virtual. PMLR. [find](https://scholar.google.com/scholar?q=Conference+on+Machine+Learning%2C+pages+4411%E2%80%934421%2C+Virtual%2E+PMLR%2E)
20. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp… [find](https://scholar.google.com/scholar?q=Pratik+Joshi%2C+Sebastin+Santy%2C+Amar+Budhiraja%2C+Kalika+Bali%2C+and+Monojit+Choudhury%2E+2020%2E+The+state+and+fate+of+linguistic+diversity+and+inclusion+in+the+NLP+world%2E+In+Proceedings+of+the+58th+Annual+Mee) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/)
21. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: an empirical study. In Proceedings of the 8th International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia. Open-Review.net. [find](https://scholar.google.com/scholar?q=Karthikeyan+K%2C+Zihan+Wang%2C+Stephen+Mayhew%2C+and+Dan+Roth%2E+2020%2E+Cross%2Dlingual+ability+of+multilingual+BERT%3A+an+empirical+study%2E+In+Proceedings+of+the+8th+International+Conference+on+Learning+Representa)
22. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+Proceedings+of+the+3rd+International+Conference+on+Learning+Representations+%28ICLR%29%2C+San+Diego%2C+CA%2C+USA%2E)
23. Yuri Kuratov and Mikhail Arkhipov. 2019. Adaptation of deep bidirectional multilingual transformers for russian language. arXiv preprint. [find](https://scholar.google.com/scholar?q=Yuri+Kuratov+and+Mikhail+Arkhipov%2E+2019%2E+Adaptation+of+deep+bidirectional+multilingual+transformers+for+russian+language%2E+arXiv+preprint%2E)
24. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483–4499, Online. Ass… [find](https://scholar.google.com/scholar?q=Anne+Lauscher%2C+Vinit+Ravishankar%2C+Ivan+Vulic%CC%81%2C+and+Goran+Glavas%CC%8C%2E+2020%2E+From+zero+to+hero%3A+On+the+limitations+of+zero%2Dshot+language+transfer+with+multilingual+Transformers%2E+In+Proceedings+of+the+2020+) [our card](https://md.research.elamur.ai/free/en/papers/lauscher-2020-zero-hero-limitations-zero-shot/)
25. Sangah Lee, Hansol Jang, Yunmee Baik, Suzi Park, and Hyopil Shin. 2020. KR-BERT: A small-scale Korean-specific language model. arXiv preprint. [find](https://scholar.google.com/scholar?q=Sangah+Lee%2C+Hansol+Jang%2C+Yunmee+Baik%2C+Suzi+Park%2C+and+Hyopil+Shin%2E+2020%2E+KR%2DBERT%3A+A+small%2Dscale+Korean%2Dspecific+language+model%2E+arXiv+preprint%2E)
26. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315– 7330, Online. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Barlas+Oguz%2C+Ruty+Rinott%2C+Sebastian+Riedel%2C+and+Holger+Schwenk%2E+2020%2E+MLQA%3A+Evaluating+cross%2Dlingual+extractive+question+answering%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Asso) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/)
27. Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019. KorQuAD1.0: Korean QA dataset for machine reading comprehension. arXiv preprint. [find](https://scholar.google.com/scholar?q=Seungyoung+Lim%2C+Myungji+Kim%2C+and+Jooyoul+Lee%2E+2019%2E+KorQuAD1%2E0%3A+Korean+QA+dataset+for+machine+reading+comprehension%2E+arXiv+preprint%2E)
28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
29. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations (ICLR), New Orleans, LA, USA. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+Proceedings+of+the+7th+International+Conference+on+Learning+Representations+%28ICLR%29%2C+New+Orleans%2C+LA%2C+USA%2E+OpenReview%2En)
30. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2020. CamemBERT: A tasty French language model. In Proceed… [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Raymond+E%2E+Daly%2C+Peter+T%2E+Pham%2C+Dan+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+Louis+Martin%2C+Benjamin+Muller%2C+Pedro+Javier+Ortiz+Sua%CC%81rez%2C+Yoann+Dupont%2C+Laurent+Romary%2C+E%CC%81ric+de+la+Cle) [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
31. Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019. Polyglot contextual representations improve crosslingual transfer. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers… [find](https://scholar.google.com/scholar?q=Phoebe+Mulcaire%2C+Jungo+Kasai%2C+and+Noah+A%2E+Smith%2E+2019%2E+Polyglot+contextual+representations+improve+crosslingual+transfer%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Ass)
32. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth I… [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+de+Marneffe%2C+Filip+Ginter%2C+Yoav+Goldberg%2C+Jan+Hajic%CC%8C%2C+Christopher+D%2E+Manning%2C+Ryan+McDonald%2C+Slav+Petrov%2C+Sampo+Pyysalo%2C+Natalia+Silveira%2C+Reut+Tsarfaty%2C+and+Daniel+Zeman) [our card](https://md.research.elamur.ai/free/en/papers/ud-2016-nivre-universal-dependencies-v1-multilingual-treebank/)
33. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluat… [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+de+Marneffe%2C+Filip+Ginter%2C+Jan+Hajic%CC%8C%2C+Christopher+D%2E+Manning%2C+Sampo+Pyysalo%2C+Sebastian+Schuster%2C+Francis+Tyers%2C+and+Daniel+Zeman%2E+2020%2E+Universal+Dependencies+v2%3A+An+eve)
34. Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [MASK]? Making sense of language-specific BERT models. arXiv preprint. [find](https://scholar.google.com/scholar?q=Debora+Nozza%2C+Federico+Bianchi%2C+and+Dirk+Hovy%2E+2020%2E+What+the+%5BMASK%5D%3F+Making+sense+of+language%2Dspecific+BERT+models%2E+arXiv+preprint%2E)
35. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. A… [find](https://scholar.google.com/scholar?q=Xiaoman+Pan%2C+Boliang+Zhang%2C+Jonathan+May%2C+Joel+Nothman%2C+Kevin+Knight%2C+and+Heng+Ji%2E+2017%2E+Cross%2Dlingual+name+tagging+and+linking+for+282+languages%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Asso)
36. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
37. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487–5… [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Aishwarya+Kamath%2C+Andreas+Ru%CC%88ckle%CC%81%2C+Kyunghyun+Cho%2C+and+Iryna+Gurevych%2E+2021%2E+AdapterFusion%3A+Non%2Ddestructive+task+composition+for+transfer+learning%2E+In+Proceedings+of+the+16th+Conferenc) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
38. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstration… [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Andreas+Ru%CC%88ckle%CC%81%2C+Clifton+Poth%2C+Aishwarya+Kamath%2C+Ivan+Vulic%CC%81%2C+Sebastian+Ruder%2C+Kyunghyun+Cho%2C+and+Iryna+Gurevych%2E+2020a%2E+AdapterHub%3A+A+framework+for+adapting+transformers%2E+In+Proceedi)
39. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020b. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Iryna+Gurevych%2C+and+Sebastian+Ruder%2E+2020b%2E+MAD%2DX%3A+An+Adapter%2DBased+Framework+for+Multi%2DTask+Cross%2DLingual+Transfer%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Met) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/)
40. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020c. UNKs Everywhere: Adapting Multilingual Language Models to New Scripts. arXiv preprint. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Iryna+Gurevych%2C+and+Sebastian+Ruder%2E+2020c%2E+UNKs+Everywhere%3A+Adapting+Multilingual+Language+Models+to+New+Scripts%2E+arXiv+preprint%2E)
41. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Telmo+Pires%2C+Eva+Schlinger%2C+and+Dan+Garrette%2E+2019%2E+How+multilingual+is+multilingual+BERT%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4996%E2%80%93+5001%2C) [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
42. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Associatio… [find](https://scholar.google.com/scholar?q=Edoardo+Maria+Ponti%2C+Goran+Glavas%CC%8C%2C+Olga+Majewska%2C+Qianchu+Liu%2C+Ivan+Vulic%CC%81%2C+and+Anna+Korhonen%2E+2020%2E+XCOPA%3A+A+multilingual+dataset+for+causal+commonsense+reasoning%2E+In+Proceedings+of+the+2020+Confere) [our card](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/)
43. Advanced Informatics: Concepts, Theory and Applications (ICAICTA), pages 1–5, Yogyakarta, Indonesia. IEEE. [find](https://scholar.google.com/scholar?q=Advanced+Informatics%3A+Concepts%2C+Theory+and+Applications+%28ICAICTA%29%2C+pages+1%E2%80%935%2C+Yogyakarta%2C+Indonesia%2E+IEEE%2E)
44. Sampo Pyysalo, Jenna Kanerva, Antti Virtanen, and Filip Ginter. 2020. WikiBERT models: Deep transfer learning for many languages. arXiv preprint. [find](https://scholar.google.com/scholar?q=Sampo+Pyysalo%2C+Jenna+Kanerva%2C+Antti+Virtanen%2C+and+Filip+Ginter%2E+2020%2E+WikiBERT+models%3A+Deep+transfer+learning+for+many+languages%2E+arXiv+preprint%2E)
45. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
46. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Afshin+Rahimi%2C+Yuan+Li%2C+and+Trevor+Cohn%2E+2019%2E+Massively+multilingual+transfer+for+NER%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+151%E2%80%93164%2C+Flore) [our card](https://md.research.elamur.ai/free/en/papers/rahimi-2019-massively-multilingual-transfer-ner/)
47. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
48. Teemu Ruokolainen, Pekka Kauppinen, Miikka Silfverberg, and Krister Lindén. 2020. A Finnish news corpus for named entity recognition. Language Resources and Evaluation, 54(1):247–272. [find](https://scholar.google.com/scholar?q=Teemu+Ruokolainen%2C+Pekka+Kauppinen%2C+Miikka+Silfverberg%2C+and+Krister+Linde%CC%81n%2E+2020%2E+A+Finnish+news+corpus+for+named+entity+recognition%2E+Language+Resources+and+Evaluation%2C+54%281%29%3A247%E2%80%93272%2E)
49. Samuel Rönnqvist, Jenna Kanerva, Tapio Salakoski, and Filip Ginter. 2019. Is multilingual BERT fluent in language generation? In Proceedings of the First NLPL Workshop on Deep Learning for Natural Language Processing, pages 29–36, Turku, Finland. Linköping University Electronic Press. [find](https://scholar.google.com/scholar?q=Samuel+Ro%CC%88nnqvist%2C+Jenna+Kanerva%2C+Tapio+Salakoski%2C+and+Filip+Ginter%2E+2019%2E+Is+multilingual+BERT+fluent+in+language+generation%3F+In+Proceedings+of+the+First+NLPL+Workshop+on+Deep+Learning+for+Natural+La)
50. Stefan Schweter. 2020. BERTurk - BERT models for Turkish. Zenodo. [find](https://scholar.google.com/scholar?q=Stefan+Schweter%2E+2020%2E+BERTurk+%2D+BERT+models+for+Turkish%2E+Zenodo%2E)
51. Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. 2019. DRCD: a Chinese machine reading comprehension dataset. arXiv preprint. [find](https://scholar.google.com/scholar?q=Chih+Chieh+Shao%2C+Trois+Liu%2C+Yuting+Lai%2C+Yiying+Tseng%2C+and+Sam+Tsai%2E+2019%2E+DRCD%3A+a+Chinese+machine+reading+comprehension+dataset%2E+arXiv+preprint%2E)
52. Edoardo Maria Ponti, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on typological linguistics for natural language processing. Computational Linguistics, 45(3):559–601. [find](https://scholar.google.com/scholar?q=Edoardo+Maria+Ponti%2C+Helen+O%E2%80%99Horan%2C+Yevgeni+Berzak%2C+Ivan+Vulic%CC%81%2C+Roi+Reichart%2C+Thierry+Poibeau%2C+Ekaterina+Shutova%2C+and+Anna+Korhonen%2E+2019%2E+Modeling+language+variation+and+universals%3A+A+survey+on+typo)
53. Sergey Smetanin and Michail Komarov. 2019. Sentiment analysis of product reviews in Russian using convolutional neural networks. In Proceedings of the 2019 IEEE 21st Conference on Business Informatics (CBI), pages 482–486, Moscow, Russia. IEEE. [find](https://scholar.google.com/scholar?q=Sergey+Smetanin+and+Michail+Komarov%2E+2019%2E+Sentiment+analysis+of+product+reviews+in+Russian+using+convolutional+neural+networks%2E+In+Proceedings+of+the+2019+IEEE+21st+Conference+on+Business+Informatics)
54. Lutz Prechelt. 1998. Early stopping-but when? In Neural Networks: Tricks of the Trade, pages 55–69. Springer, Berlin, Germany. [find](https://scholar.google.com/scholar?q=Lutz+Prechelt%2E+1998%2E+Early+stopping%2Dbut+when%3F+In+Neural+Networks%3A+Tricks+of+the+Trade%2C+pages+55%E2%80%9369%2E+Springer%2C+Berlin%2C+Germany%2E)
55. Milan Straka, Jan Hajič, and Jana Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4290– 4297… [find](https://scholar.google.com/scholar?q=Milan+Straka%2C+Jan+Hajic%CC%8C%2C+and+Jana+Strakova%CC%81%2E+2016%2E+UD%2DPipe%3A+Trainable+pipeline+for+processing+CoNLL%2DU+files+performing+tokenization%2C+morphological+analysis%2C+POS+tagging+and+parsing%2E+In+Proceedings+of)
56. Ayu Purwarianti and Ida Ayu Putu Ari Crisdayanti. 2019. Improving Bi-LSTM performance for Indonesian sentiment analysis using paragraph vector. In Proceedings of the 2019 International Conference of Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Languag… [find](https://scholar.google.com/scholar?q=Ayu+Purwarianti+and+Ida+Ayu+Putu+Ari+Crisdayanti%2E+2019%2E+Improving+Bi%2DLSTM+performance+for+Indonesian+sentiment+analysis+using+paragraph+vector%2E+In+Proceedings+of+the+2019+International+Conference+of+E)
57. Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020. UDapter: Language adaptation for truly Universal Dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2302–2315, Online. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Ahmet+U%CC%88stu%CC%88n%2C+Arianna+Bisazza%2C+Gosse+Bouma%2C+and+Gertjan+van+Noord%2E+2020%2E+UDapter%3A+Language+adaptation+for+truly+Universal+Dependency+parsing%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Method)
58. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844,… [find](https://scholar.google.com/scholar?q=Shijie+Wu+and+Mark+Dredze%2E+2019%2E+Beto%2C+bentz%2C+becas%3A+The+surprising+cross%2Dlingual+effectiveness+of+BERT%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+t) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/)
59. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shijie+Wu+and+Mark+Dredze%2E+2020%2E+Are+all+languages+created+equal+in+multilingual+BERT%3F+In+Proceedings+of+the+5th+Workshop+on+Representation+Learning+for+NLP%2C+pages+120%E2%80%93130%2C+Online%2E+Association+for+Com) [our card](https://md.research.elamur.ai/free/en/papers/wu-2020-are-all-languages-created-equal/)
60. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint. [find](https://scholar.google.com/scholar?q=Antti+Virtanen%2C+Jenna+Kanerva%2C+Rami+Ilo%2C+Jouni+Luoma%2C+Juhani+Luotolahti%2C+Tapio+Salakoski%2C+Filip+Ginter%2C+and+Sampo+Pyysalo%2E+2019%2E+Multilingual+is+not+enough%3A+BERT+for+Finnish%2E+arXiv+preprint%2E) [our card](https://md.research.elamur.ai/free/en/papers/virtanen-2019-multilingual-enough-bert-finnish/)
61. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+Luk)
62. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT Model. arXiv preprint. [find](https://scholar.google.com/scholar?q=Wietse+de+Vries%2C+Andreas+van+Cranenburgh%2C+Arianna+Bisazza%2C+Tommaso+Caselli%2C+Gertjan+van+Noord%2C+and+Malvina+Nissim%2E+2019%2E+BERTje%3A+A+Dutch+BERT+Model%2E+arXiv+preprint%2E) [our card](https://md.research.elamur.ai/free/en/papers/de-2019-vries-bertje-dutch-bert-model/)
63. Jingjing Xu, Ji Wen, Xu Sun, and Qi Su. 2017. A discourse-level named entity recognition and relation extraction dataset for Chinese literature text. arXiv preprint. [find](https://scholar.google.com/scholar?q=Jingjing+Xu%2C+Ji+Wen%2C+Xu+Sun%2C+and+Qi+Su%2E+2017%2E+A+discourse%2Dlevel+named+entity+recognition+and+relation+extraction+dataset+for+Chinese+literature+text%2E+arXiv+preprint%2E)
64. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computation… [find](https://scholar.google.com/scholar?q=Ivan+Vulic%CC%81%2C+Edoardo+Maria+Ponti%2C+Robert+Litschko%2C+Goran+Glavas%CC%8C%2C+and+Anna+Korhonen%2E+2020%2E+Probing+pretrained+language+models+for+lexical+semantics%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+) [our card](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/)
65. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Noah+Constant%2C+Adam+Roberts%2C+Mihir+Kale%2C+Rami+Al%2DRfou%2C+Aditya+Siddhant%2C+Aditya+Barua%2C+and+Colin+Raffel%2E+2021%2E+mT5%3A+A+massively+multilingual+pre%2Dtrained+text%2Dto%2Dtext+transformer%2E+In+Procee)
66. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008, Long Beach, CA, USA. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi)
67. Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In Proceedings of the 1… [find](https://scholar.google.com/scholar?q=Bryan+Wilie%2C+Karissa+Vincentio%2C+Genta+Indra+Winata%2C+Samuel+Cahyawijaya%2C+Xiaohong+Li%2C+Zhi+Yuan+Lim%2C+Sidik+Soleman%2C+Rahmad+Mahendra%2C+Pascale+Fung%2C+Syafri+Bahar%2C+and+Ayu+Purwarianti%2E+2020%2E+IndoNLU%3A+Bench)
68. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+Remi+Louf%2C+Morgan+Funtow%2Dicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
69. Daniel Zeman, Joakim Nivre, Mitchell Abrams, Elia Ackermann, Noëmi Aepli, Željko Agić, Lars Ahrenberg, Chika Kennedy Ajede, Gabrielė Aleksan-dravičiūtė, Lene Antonsen, Katya Aplonova, Angelina Aquino, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Furkan Atmaca, Mohammed… [find](https://scholar.google.com/scholar?q=Daniel+Zeman%2C+Joakim+Nivre%2C+Mitchell+Abrams%2C+Elia+Ackermann%2C+Noe%CC%88mi+Aepli%2C+Z%CC%8Celjko+Agic%CC%81%2C+Lars+Ahrenberg%2C+Chika+Kennedy+Ajede%2C+Gabriele%CC%87+Aleksan%2Ddravic%CC%8Ciu%CC%84te%CC%87%2C+Lene+Antonsen%2C+Katya+Aplonova%2C+Angelina+)
70. Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R. Bowman. 2020. When do you need billions of words of pretraining data? arXiv preprint. [find](https://scholar.google.com/scholar?q=Yian+Zhang%2C+Alex+Warstadt%2C+Haau%2DSing+Li%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+When+do+you+need+billions+of+words+of+pretraining+data%3F+arXiv+preprint%2E)