---
title: "WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer — Minixhofer 2022"
description: "The authors propose WECHSEL, a cheap way to move an English subword-based model into a new language. 54 references, cited by 87 papers here."
authors: ["Benjamin Minixhofer", "Fabian Paischer", "Navid Rekabsaz"]
year: 2022
venue: "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2022.naacl-main.293/"
doi: "10.18653/v1/2022.naacl-main.293"
arxiv: "2112.06598"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/minixhofer-2022-wechsel-effective-initialization-subword-embeddings/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/minixhofer-2022-wechsel-effective-initialization-subword-embeddings/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

# WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

**Authors:** Benjamin Minixhofer, Fabian Paischer and Navid Rekabsaz · **Year:** 2022 · **Venue:** Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.naacl-main.293/) · [PDF](https://aclanthology.org/2022.naacl-main.293.pdf) · [DOI 10.18653/v1/2022.naacl-main.293](https://doi.org/10.18653/v1/2022.naacl-main.293) · [arXiv 2112.06598](https://arxiv.org/abs/2112.06598) · [OpenAlex](https://openalex.org/W4200634521)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [87](https://md.research.elamur.ai/free/en/papers/minixhofer-2022-wechsel-effective-initialization-subword-embeddings/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Pretraining large language models is expensive, and most strong models exist only for English. Massively multilingual models lose quality as more languages are added, and training a monolingual model from scratch in another language needs roughly as much compute as the original English model. The authors propose WECHSEL, a cheap way to move an English subword-based model into a new language. The English model's inner (non-embedding) weights are copied, and its tokenizer is replaced with one built for the target language. Each new subword embedding is then initialized from the embeddings of semantically similar English subwords. These similarities come from multilingual static word embeddings that place English and the target language in a shared space.

The method is applied to RoBERTa and GPT-2 for French, German, Chinese and Swahili, and to GPT-2 for four very low-resource languages. Models are evaluated on NER, NLI and perplexity. WECHSEL consistently beats random initialization and the TransInner baseline. It also often matches or exceeds monolingual models trained far longer, which makes building new-language LMs cheaper and less environmentally costly.

## Key points
- **Baselines and compute:** WECHSEL is compared with FullRand (random initialization) and TransInner (de Vries and Nissim, 2021), which transfers only the inner parameters. All three are trained under equal budgets of about 4 days on a TPUv3-8.
- **Why embeddings matter:** Token embeddings make up about 31% of RoBERTa's parameters and about 33% of GPT-2's. Initializing them semantically rather than randomly therefore affects roughly a third of the model.
- **Comparison with models trained from scratch:**
  - French: WECHSEL-RoBERTa surpasses CamemBERT after 10% of its training steps, about 64x less effort.
  - German: it surpasses GBERT Base, about 39x less effort.
  - Chinese: it beats BERT Base-Chinese on NLI but not on NER.
- **Low-resource languages:** For Sundanese, Scottish Gaelic, Uyghur and Malagasy, WECHSEL-GPT2 yields lower perplexity across the board. The advantage tends to grow as training data shrinks, and these models need careful early model selection because they overfit quickly.
- **Freezing and release:** Unlike TransInner, WECHSEL does not require freezing the non-embedding parameters at the start of training. Code and models are released at github.com/cpjku/wechsel.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/)

## Institutions

- [Johannes Kepler University of Linz](https://md.research.elamur.ai/free/en/universities/johannes-kepler-university-of-linz/) — University, Linz, Austria · 2 works · 2,364 citations

## Cite

Cite as: Benjamin Minixhofer, Fabian Paischer and Navid Rekabsaz. 2022. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{minixhofer2022wechsel,
  title = {{WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models}},
  author = {Benjamin Minixhofer, Fabian Paischer and Navid Rekabsaz},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.293},
  eprint = {2112.06598},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.293}
}
```

- [← previous JGLUE: Japanese General Language Understanding Evaluation](https://md.research.elamur.ai/free/en/papers/kurihara-2022-jglue-japanese-general-language-understanding/)
- [next → The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs](https://md.research.elamur.ai/free/en/papers/ruis-2022-goldilocks-pragmatic-understanding-fine-tuning/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

54 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Langu… [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289–2294, Austin, Texas. Association for Computati… [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2016-learning-principled-bilingual-mappings-word/)
3. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Vancouver, Canada. Association for Computational… [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2017-learning-bilingual-word-embeddings-almost/)
4. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798, Melbourne, Australia… [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-vecmap-unsupervised/)
5. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
6. Piotr Bański and Beata Wójtowicz. 2009. Freedict: an open source repository of tei-encoded bilingual dictionaries. [find](https://scholar.google.com/scholar?q=Piotr+Ban%CC%81ski+and+Beata+W%C3%B3jtowicz%2E+2009%2E+Freedict%3A+an+open+source+repository+of+tei%2Dencoded+bilingual+dictionaries%2E)
7. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. [find](https://scholar.google.com/scholar?q=Emily+M+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+2021%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference+)
8. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [our card](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/)
9. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proc. of NeurIPS. [our card](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/)
10. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
11. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science. [find](https://scholar.google.com/scholar?q=Aylin+Caliskan%2C+Joanna+J+Bryson%2C+and+Arvind+Narayanan%2E+2017%2E+Semantics+derived+automatically+from+language+corpora+contain+human%2Dlike+biases%2E+Science%2E)
12. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/chan-2020-german-s-next-language-model/)
13. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+arXiv+preprint+arXiv%3A2003%2E10555%2E)
14. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
15. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Guillaume+Lample%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Ludovic+Denoyer%2C+and+Herv%C3%A9+J%C3%A9gou%2E+2017%2E+Word+translation+without+parallel+data%2E+arXiv+preprint+arXiv%3A1710%2E04087%2E)
16. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computation… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+R%2E+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+)
17. Wietse de Vries and Malvina Nissim. 2021. As good as new. how to successfully recycle English GPT-2 to make models for other languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836–846, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wietse+de+Vries+and+Malvina+Nissim%2E+2021%2E+As+good+as+new%2E+how+to+successfully+recycle+English+GPT%2D2+to+make+models+for+other+languages%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+AC)
18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
19. Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2016. Learning crosslingual word embeddings without bilingual corpora. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1285– 1295, Austin, Texas. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Long+Duong%2C+Hiroshi+Kanayama%2C+Tengfei+Ma%2C+Steven+Bird%2C+and+Trevor+Cohn%2E+2016%2E+Learning+crosslingual+word+embeddings+without+bilingual+corpora%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Method)
20. Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. [our card](https://md.research.elamur.ai/free/en/papers/elazar-2018-adversarial-removal-demographic-attributes-text/)
21. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2021%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+arXiv+preprint+arXiv%3A2101%2E03961%2E)
22. Christian Ganhör, David Penz, Navid Rekabsaz, Oleg Lesota, and Markus Schedl. 2022. Mitigating consumer biases in recommendations with adversarial training. In Proceedings of the 45th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2022. ACM. [find](https://scholar.google.com/scholar?q=Christian+Ganh%C3%B6r%2C+David+Penz%2C+Navid+Rekabsaz%2C+Oleg+Lesota%2C+and+Markus+Schedl%2E+2022%2E+Mitigating+consumer+biases+in+recommendations+with+adversarial+training%2E+In+Proceedings+of+the+45th+International+AC)
23. Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2979–2984, Brussels, Belgium. Associ… [our card](https://md.research.elamur.ai/free/en/papers/joulin-2018-loss-translation-learning-bilingual-word/)
24. Klara Krieg, Emilia Parada-Cabaleiro, Markus Schedl, and Navid Rekabsaz. 2022. Do perceived gender biases in retrieval results affect relevance judgements? In Proceedings of the Workshop on Algorithmic Bias in Search and Recommendation at the European Conference on Information Retrieval (ECIR-BIAS 2… [find](https://scholar.google.com/scholar?q=Klara+Krieg%2C+Emilia+Parada%2DCabaleiro%2C+Markus+Schedl%2C+and+Navid+Rekabsaz%2E+2022%2E+Do+perceived+gender+biases+in+retrieval+results+affect+relevance+judgements%3F+In+Proceedings+of+the+Workshop+on+Algorithmi)
25. Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Alexis+Conneau%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Ludovic+Denoyer%2C+and+Herv%C3%A9+J%C3%A9gou%2E+2018%2E+Word+translation+without+parallel+data%2E+In+International+Conference+on+Learning+Representations%2E)
26. Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020. Pre-training via paraphrasing. In Advances in Neural Information Processing Systems, volume 33, pages 18470–18481. Curran Associates, Inc. [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-pre-training-paraphrasing/)
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
28. Antoine Louis. 2020. BelGPT-2: a GPT-2 model pre-trained on French corpora. [https://github.com/antoiloui/belgpt2](https://github.com/antoiloui/belgpt2). [↗](https://github.com/antoiloui/belgpt2)
29. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151–159, Denver, Colorado. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-bilingual-word-representations-monolingual-quality/)
30. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, O… [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
31. Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [mask]? making sense of language-specific bert models. arXiv preprint arXiv:2003.02912. — [arXiv:2003.02912](https://arxiv.org/abs/2003.02912) [find](https://scholar.google.com/scholar?q=Debora+Nozza%2C+Federico+Bianchi%2C+and+Dirk+Hovy%2E+2020%2E+What+the+%5Bmask%5D%3F+making+sense+of+language%2Dspecific+bert+models%2E+arXiv+preprint+arXiv%3A2003%2E02912%2E+%E2%80%94+%5BarXiv%3A2003%2E02912%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2003%2E02)
32. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. A… [our card](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/)
33. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
34. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+with+unsupervised+learning%2E)
35. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
36. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164, Florence, Italy. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/rahimi-2019-massively-multilingual-transfer-ner/)
37. Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021. Few-shot question answering by pretraining span selection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Process… [find](https://scholar.google.com/scholar?q=Ori+Ram%2C+Yuval+Kirstain%2C+Jonathan+Berant%2C+Amir+Globerson%2C+and+Omer+Levy%2E+2021%2E+Few%2Dshot+question+answering+by+pretraining+span+selection%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+f)
38. Navid Rekabsaz, Simone Kopeinik, and Markus Schedl. 2021a. Societal biases in retrieved contents: Measurement framework and adversarial mitigation of bert rankers. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 306–316. [find](https://scholar.google.com/scholar?q=Navid+Rekabsaz%2C+Simone+Kopeinik%2C+and+Markus+Schedl%2E+2021a%2E+Societal+biases+in+retrieved+contents%3A+Measurement+framework+and+adversarial+mitigation+of+bert+rankers%2E+In+Proceedings+of+the+44th+Internati)
39. Alessandro B. Melchiorre, Navid Rekabsaz, Emilia Parada-Cabaleiro, Stefan Brandl, Oleg Lesota, and Markus Schedl. 2021. Investigating gender fairness of recommendation algorithms in the music domain. Information Processing and Management, 58(5):102666. [find](https://scholar.google.com/scholar?q=Alessandro+B%2E+Melchiorre%2C+Navid+Rekabsaz%2C+Emilia+Parada%2DCabaleiro%2C+Stefan+Brandl%2C+Oleg+Lesota%2C+and+Markus+Schedl%2E+2021%2E+Investigating+gender+fairness+of+recommendation+algorithms+in+the+music+domain%2E+)
40. Navid Rekabsaz, Nikolaos Pappas, James Henderson, Banriskhem K Khonglah, and Srikanth Madikeri. 2019. Regularization advantages of multilingual neural language models for low resource domains. arXiv preprint arXiv:1906.01496. — [arXiv:1906.01496](https://arxiv.org/abs/1906.01496) [find](https://scholar.google.com/scholar?q=Navid+Rekabsaz%2C+Nikolaos+Pappas%2C+James+Henderson%2C+Banriskhem+K+Khonglah%2C+and+Srikanth+Madikeri%2E+2019%2E+Regularization+advantages+of+multilingual+neural+language+models+for+low+resource+domains%2E+arXiv+p)
41. Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Navid Rekabsaz and Markus Schedl. 2020. Do neural ranki… [our card](https://md.research.elamur.ai/free/en/papers/nguyen-2017-transfer-learning-across-low-resource/)
42. Navid Rekabsaz, Robert West, James Henderson, and Allan Hanbury. 2021b. Measuring societal biases from text corpora with smoothed first-order co-occurrence. In Proceedings of the Fifteenth International AAAI Conference on Web and Social Media, ICWSM 2021, held virtually, June 7-10, 2021, pages 549–5… [find](https://scholar.google.com/scholar?q=Navid+Rekabsaz%2C+Robert+West%2C+James+Henderson%2C+and+Allan+Hanbury%2E+2021b%2E+Measuring+societal+biases+from+text+corpora+with+smoothed+first%2Dorder+co%2Doccurrence%2E+In+Proceedings+of+the+Fifteenth+Internation)
43. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join… [our card](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)
44. Peter H Schönemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10. [find](https://scholar.google.com/scholar?q=Peter+H+Sch%C3%B6nemann%2E+1966%2E+A+generalized+solution+of+the+orthogonal+procrustes+problem%2E+Psychometrika%2C+31%281%29%3A1%E2%80%9310%2E)
45. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/)
46. Ke Tran. 2020. From english to foreign languages: Transferring pre-trained language models. arXiv preprint arXiv:2002.07306. [find](https://scholar.google.com/scholar?q=Ke+Tran%2E+2020%2E+From+english+to+foreign+languages%3A+Transferring+pre%2Dtrained+language+models%2E+arXiv+preprint+arXiv%3A2002%2E07306%2E)
47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
48. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
49. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/wu-2020-are-all-languages-created-equal/)
50. Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006–1011, D… [our card](https://md.research.elamur.ai/free/en/papers/xing-2015-normalized-word-embedding-orthogonal-transform/)
51. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… [our card](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/)
52. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
53. George Zerveas, Navid Rekabsaz, Daniel Cohen, and Carsten Eickhoff. 2022. Mitigating bias in search results through set-based document reranking and neutrality regularization. In Proceedings of the 45th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 20… [find](https://scholar.google.com/scholar?q=George+Zerveas%2C+Navid+Rekabsaz%2C+Daniel+Cohen%2C+and+Carsten+Eickhoff%2E+2022%2E+Mitigating+bias+in+search+results+through+set%2Dbased+document+reranking+and+neutrality+regularization%2E+In+Proceedings+of+the+45)
54. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Barret+Zoph%2C+Deniz+Yuret%2C+Jonathan+May%2C+and+Kevin+Knight%2E+2016%2E+Transfer+learning+for+low%2Dresource+neural+machine+translation%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Lan)