---
title: "How to Adapt Your Pretrained Multilingual Model to 1600"
description: "Ebrahimi and Kann test whether the New Testament can serve instead. 17.69% accuracy on POS. 43 references, cited by 57 papers here."
authors: ["Abteen Ebrahimi", "Katharina Kann"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.351/"
doi: "10.18653/v1/2021.acl-long.351"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ebrahimi-2021-how-adapt-your-pretrained-multilingual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ebrahimi-2021-how-adapt-your-pretrained-multilingual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / How to Adapt Your Pretrained Multilingual Model to 1600 Languages

# How to Adapt Your Pretrained Multilingual Model to 1600 Languages

**Authors:** [Abteen Ebrahimi](https://md.research.elamur.ai/free/en/authors/abteen-ebrahimi/) and [Katharina Kann](https://md.research.elamur.ai/free/en/authors/katharina-kann/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.351/) · [PDF](https://aclanthology.org/2021.acl-long.351.pdf) · [DOI 10.18653/v1/2021.acl-long.351](https://doi.org/10.18653/v1/2021.acl-long.351) · [OpenAlex](https://openalex.org/W3168767730)

**Citations:** 29 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/ebrahimi-2021-how-adapt-your-pretrained-multilingual/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Techniques for extending pretrained multilingual models (PMMs) to languages they never saw in pretraining are usually tested with large Wikipedia or Common Crawl corpora. Those sources cover only a few hundred languages, so the languages that most need adaptation are left out. Ebrahimi and Kann test whether the New Testament can serve instead. Translations exist for roughly 1,600 languages, but the text is small and limited to one domain.

The authors start from XLM-R and compare three adaptation strategies:
- continued MLM pretraining,
- vocabulary extension,
- MAD-X-style adapters.

The only target-language text is the New Testament, and there is no labeled target data. Models are evaluated zero-shot on POS tagging and NER for 30 typologically diverse languages that XLM-R never saw.

Even with so little data, most methods beat the unadapted baseline. The simplest one, continued pretraining, does best on both tasks. A Cyrillic-script case study separates the effects of domain and source language.

The work shows that scripture is a viable last-resort adaptation corpus for truly low-resource languages.

## Key points
- **Setup:** XLM-R is the base model. Adaptation uses New Testament text only, for 30 unseen languages, including Bashkir, Coptic, Maltese, Wolof and Yoruba. Tasks are POS tagging and NER, with no target-language labels.
- **Main result:** Continued pretraining (+MLM) gives the best results. Average gains over XLM-R are up to 17.69% accuracy on POS and 6.29 F1 on NER.
- **Case study design:** The case study covers Bashkir, Chechen and Chuvash. It swaps English for Russian as the finetuning source, and swaps Bible text for an equal amount of Wikipedia text.
- **Case study results:**
  - Using Russian as the source raises the average baseline by 18.96 F1.
  - Combining Wikipedia data with Russian labels gives the best results overall. In that setting, +MLM scores 30.81 F1 higher than the English-source baseline and 11.85 F1 higher than the Russian-source baseline.
- **Limitations noted by the authors:**
  - Only low-level tasks are evaluated.
  - The NER labels come from WikiANN and are machine-generated.
  - Bible quality varies across the 1611 available languages.
  - Only one model was studied.
  - Religious text may introduce extra biases.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Colorado Boulder](https://md.research.elamur.ai/free/en/universities/university-of-colorado-boulder/) — University, Boulder, United States · 52 works · 49,647 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works

## Cite

Cite as: Abteen Ebrahimi and Katharina Kann. 2021. How to Adapt Your Pretrained Multilingual Model to 1600 Languages. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ebrahimi2021how,
  title = {{How to Adapt Your Pretrained Multilingual Model to 1600 Languages}},
  author = {Abteen Ebrahimi and Katharina Kann},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.351},
  url = {https://doi.org/10.18653/v1/2021.acl-long.351}
}
```

- [← previous Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI](https://md.research.elamur.ai/free/en/papers/tian-2021-diagnosing-first-order-logical-reasoning/)
- [next → Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2021-are-references-really-needed-unbabel/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

43 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Guillaume+Lample%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Ludovic+Denoyer%2C+and+Herve%CC%81+Je%CC%81gou%2E+2017%2E+Word+translation+without+parallel+data%2E+arXiv+preprint+arXiv%3A1710%2E04087%2E)
2. Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith. 2016. [Massively multilingual word embeddings](https://md.research.elamur.ai/free/en/papers/ammar-2016-massively-multilingual-word-embeddings/). ArXiv, abs/1602.01925.
3. M. Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In ACL.
4. M. Artetxe and Holger Schwenk. 2019. [Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/). Transactions of the Association for Computational Linguistics, 7:597–610.
5. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. [Learning principled bilingual mappings of word embeddings while preserving monolingual invariance](https://md.research.elamur.ai/free/en/papers/artetxe-2016-learning-principled-bilingual-mappings-word/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289–2294, Austin, Texas. Association for Computati…
6. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. [Learning bilingual word embeddings with (almost) no bilingual data](https://md.research.elamur.ai/free/en/papers/artetxe-2017-learning-bilingual-word-embeddings-almost/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Vancouver, Canada. Association for Computational…
7. Giusepppe Attardi. 2015. Wikiextractor. [https://github.com/attardi/wikiextractor](https://github.com/attardi/wikiextractor) . [↗](https://github.com/attardi/wikiextractor)
8. M Saiful Bari, Shafiq R. Joty, and Prathyusha Jwalapuram. 2020. Zero-resource cross-lingual named entity recognition. In AAAI. [find](https://scholar.google.com/scholar?q=M+Saiful+Bari%2C+Shafiq+R%2E+Joty%2C+and+Prathyusha+Jwalapuram%2E+2020%2E+Zero%2Dresource+cross%2Dlingual+named+entity+recognition%2E+In+AAAI%2E)
9. Ramy Eskander, Smaranda Muresan, and Michael Collins. 2020. Unsupervised cross-lingual part-of-speech tagging for truly low-resource scenarios. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4820–4831, Online. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Ramy+Eskander%2C+Smaranda+Muresan%2C+and+Michael+Collins%2E+2020%2E+Unsupervised+cross%2Dlingual+part%2Dof%2Dspeech+tagging+for+truly+low%2Dresource+scenarios%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Metho)
10. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. [Learning word vectors for 157 languages](https://md.research.elamur.ai/free/en/papers/grave-2018-learning-word-vectors-157-languages/). In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
11. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. ArXiv, abs/2004.10964. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+ArXiv%2C+abs%2F2004%2E10)
12. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma… [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
13. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. [Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/). CoRR, abs/2003.11080.
14. Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. [Loss in translation: Learning bilingual word mapping with a retrieval criterion](https://md.research.elamur.ai/free/en/papers/joulin-2018-loss-translation-learning-bilingual-word/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in Neural Information Processing Systems, volume 26, pages 3111–3119. Curran Associates, Inc.
16. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondr%CC%8Cej+Bojar%2C+Alexandra+Const)
17. B. Muller, Antonis Anastasopoulos, Benoı̂t Sagot, and Djamé Seddah. 2020. [When being unseen from mbert is just the beginning: Handling new languages with multilingual language models](https://md.research.elamur.ai/free/en/papers/muller-2021-when-being-unseen-mbert-is/). ArXiv, abs/2010.12858.
18. Taku Kudo and John Richardson. 2018. [SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
19. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. [Measuring bias in contextualized word representations](https://md.research.elamur.ai/free/en/papers/kurita-2019-measuring-bias-contextualized-word-representations/).
20. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. [Quantifying the carbon emissions of machine learning](https://md.research.elamur.ai/free/en/papers/lacoste-2019-quantifying-carbon-emissions-machine-learning/). arXiv preprint arXiv:1910.09700.
21. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In NeurIPS. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+NeurIPS%2E)
22. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. [Unsupervised machine translation using monolingual corpora only](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/). arXiv preprint arXiv:1711.00043.
23. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. [From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers](https://md.research.elamur.ai/free/en/papers/lauscher-2020-zero-hero-limitations-zero-shot/). pages 4483–4499.
24. Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. [Cheap translation for cross-lingual named entity recognition](https://md.research.elamur.ai/free/en/papers/mayhew-2017-cheap-translation-cross-lingual-named/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2536–2545, Copenhagen, Denmark. Association for Computational Linguistics.
25. Arya D. McCarthy, Rachel Wicks, Dylan Lewis, Aaron Mueller, Winston Wu, Oliver Adams, Garrett Nicolai, Matt Post, and David Yarowsky. 2020. [The Johns Hopkins University Bible corpus: 1600+ tongues for typological exploration](https://md.research.elamur.ai/free/en/papers/mccarthy-2020-johns-hopkins-university-bible-corpus/). In Proceedings of the 12th Language Resources and Evaluation Conference, p…
26. Joakim Nivre, Marie-Catherine de Marneffe, F. Ginter, Jan Hajivc, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and D. Zeman. 2020. [Universal dependencies v2: An evergrowing multilingual treebank collection](https://md.research.elamur.ai/free/en/papers/nivre-2020-universal-dependencies-v2/). ArXiv, abs/2004.10643.
27. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. [Cross-lingual name tagging and linking for 282 languages](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. A…
28. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. [Adapterhub: A framework for adapting transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstration…
29. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020b. [Mad-x: An adapter-based framework for multi-task cross-lingual transfer](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/). In EMNLP.
30. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. [How multilingual is multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics.
31. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. [Stanza: A python natural language processing toolkit for many human languages](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101– 108, Online.…
32. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. [Massively multilingual transfer for NER](https://md.research.elamur.ai/free/en/papers/rahimi-2019-massively-multilingual-transfer-ner/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164, Florence, Italy. Association for Computational Linguistics.
33. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. [Learning multiple visual domains with residual adapters](https://md.research.elamur.ai/free/en/papers/rebuffi-2017-learning-multiple-visual-domains-residual/). In Advances in Neural Information Processing Systems, volume 30, pages 506– 516. Curran Associates, Inc.
34. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. [Inducing multilingual text analysis tools via robust projection across aligned corpora](https://md.research.elamur.ai/free/en/papers/yarowsky-2001-inducing-multilingual-text-analysis-tools/). In Proceedings of the First International Conference on Human Language Technology Research.
35. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569–631. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Ivan+Vulic%CC%81%2C+and+Anders+S%C3%B8gaard%2E+2019%2E+A+survey+of+cross%2Dlingual+word+embedding+models%2E+Journal+of+Artificial+Intelligence+Research%2C+65%3A569%E2%80%93631%2E)
36. Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, and Todd Ward. 2020. Multi-stage pre-training for low-resource domain adaptation. In Proceedings of the 2020 Conference on Empirical Methods in Natural La… [find](https://scholar.google.com/scholar?q=Rong+Zhang%2C+Revanth+Gangi+Reddy%2C+Md+Arafat+Sultan%2C+Vittorio+Castelli%2C+Anthony+Ferritto%2C+Radu+Florian%2C+Efsun+Sarioglu+Kayi%2C+Salim+Roukos%2C+Avi+Sil%2C+and+Todd+Ward%2E+2020%2E+Multi%2Dstage+pre%2Dtraining+for+low%2D)
37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
38. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. [Energy and policy considerations for deep learning in nlp](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/).
39. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649–2656, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zihan+Wang%2C+Karthikeyan+K%2C+Stephen+Mayhew%2C+and+Dan+Roth%2E+2020%2E+Extending+multilingual+BERT+to+low%2Dresource+languages%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+26)
40. [Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
41. Shijie Wu and Mark Dredze. 2019. [Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844,…
42. Shijie Wu and Mark Dredze. 2020. [Are all languages created equal in multilingual bert?](https://md.research.elamur.ai/free/en/papers/wu-2020-are-all-languages-created-equal/) In RepL4NLP@ACL.
43. Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018. [Neural cross-lingual named entity recognition with minimal resources](https://md.research.elamur.ai/free/en/papers/xie-2018-neural-cross-lingual-named-entity/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 369–379, Brussels, Belgium. Association for Comput…

*Updated 24 October 2026*