---
title: "As Good as New. How to Successfully Recycle English GPT-2 to"
description: "First, they replace the embedding layer of GPT-2 small with a randomly initialised one for a new BPE… 39 references, cited by 47 papers here."
authors: ["Wietse de Vries", "Malvina Nissim"]
year: 2021
venue: "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021"
source: "https://aclanthology.org/2021.findings-acl.74/"
doi: "10.18653/v1/2021.findings-acl.74"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vries-2021-as-good-as-new-how/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vries-2021-as-good-as-new-how/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages

# As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages

**Authors:** [Wietse de Vries](https://md.research.elamur.ai/free/en/authors/wietse-de-vries/) and [Malvina Nissim](https://md.research.elamur.ai/free/en/authors/malvina-nissim/) · **Year:** 2021 · **Venue:** [Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2021](https://md.research.elamur.ai/free/en/venues/findings/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.findings-acl.74/) · [PDF](https://aclanthology.org/2021.findings-acl.74.pdf) · [DOI 10.18653/v1/2021.findings-acl.74](https://doi.org/10.18653/v1/2021.findings-acl.74) · [OpenAlex](https://openalex.org/W3112784227)

**Citations:** 44 citations (OpenAlex, 2026-10-14) · [47](https://md.research.elamur.ai/free/en/papers/vries-2021-as-good-as-new-how/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether an English generative model can be reused for other languages instead of training a new one from scratch, which costs a lot of data and compute. The authors adapt English GPT-2 to Italian and Dutch in stages. First, they replace the embedding layer of GPT-2 small with a randomly initialised one for a new BPE vocabulary and train only that layer, keeping all Transformer weights frozen. This yields target-language embeddings that line up with the original English ones. Next, they move to GPT-2 medium by mapping the small-model embeddings into the medium embedding space. Least-squares regression works best for this mapping, and the result is then refined. Finally, they unfreeze the whole network for a short finetuning run. In human judgements, sentences from the embedding-only models are still recognisable as machine-made. After full finetuning, the models are rated at least as well as GePpeTto, an Italian GPT-2 trained from scratch. The work offers a cheaper route to generative models for languages that lack large resources.

## Key points
- Base models are English GPT-2 small (12 layers) and medium (24 layers). Target languages are Italian and Dutch. Released models are on the Hugging Face hub under GroNLP, and the code is in the gpt2-recycle repository.
- Italian training data reuses the GePpeTto corpus: Wikipedia (2.8GB) plus ItWaC (11GB). Dutch data totals 13GB, drawn from Wikipedia, newspapers, books and news sites. It was filtered with a fastText language identifier and deduplicated on exact sentence matches. Dutch perplexity is evaluated on SoNaR, with SoNaR-1 used for development.
- To scale embeddings from small to medium, they compare mapping methods, and least-squares regression comes out best. It beats alternatives such as the orthogonal Procrustes approach common in embedding alignment.
- Human ratings were analysed with linear mixed-effects models.
  - Real text scored higher than all models, except that for Italian it was not significantly above the finetuned small model.
  - For Dutch, the finetuned small model was significantly better than the embedding-only models.
  - Longer Dutch sentences received lower ratings.
- Several tricks kept training cheap: 128-token windows during embedding-only training, mixed precision, bucketed sampling, and gradient accumulation to about 2000 examples per update. Training ran on eight V100 GPUs. The final full finetuning uses a learning rate around 1e-5.

## Interesting ideas
- Freezing the deep layers and relearning only the input/output lexicon forces the new vocabulary into the same space as the old one, so prior knowledge is kept rather than overwritten.
- A regression fitted on shared items between two embedding spaces (small and large model) can carry newly learned vectors from one space to the other, giving a warm start instead of random initialisation.
- Vocabulary size is chosen empirically: grow it step by step and stop when the average number of tokens per sentence stops dropping much.
- Human ratings are evaluated with mixed-effects models that treat raters and items as random effects and control for sentence length, with real text and a from-scratch model as reference points.
- The pipeline is staged: cheap partial training first, full finetuning last with a small learning rate, so that remaining structural differences such as word order get learned.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could be moved to a new tokenizer or embedding vocabulary by retraining only its lexical layer first, so the parsing layers learned from treebanks stay intact.
- **I2** (learn): when upgrading gparse from a small to a larger encoder, map existing learned embeddings with least squares and then briefly refine, cutting retraining cost.
- **I3** (tokens): the same stopping rule could size any subword lexicon used beside the en crate's tokenizer, avoiding rare, poorly covered entries.
- **I4** (learn): generation or answer evaluations could use this design so that differences between systems are not artefacts of particular judges, items or length.
- **I5** (back): a generator in the graph → text path could be adapted to a new domain lexicon first and only then lightly finetuned end-to-end.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Groningen](https://md.research.elamur.ai/free/en/universities/university-of-groningen/) — University, Groningen, Netherlands · 49 works · 4,843 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Wietse de Vries and Malvina Nissim. 2021. As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{vries2021as,
  title = {{As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages}},
  author = {Wietse {de Vries} and Malvina Nissim},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.74},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.74}
}
```

- [← previous SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning](https://md.research.elamur.ai/free/en/papers/mirzaee-2021-spartqa-textual-question-answering-benchmark/)
- [next → Frequency Effects on Syntactic Rule Learning in Transformers](https://md.research.elamur.ai/free/en/papers/wei-2021-frequency-effects-syntactic-rule-learning/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

39 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. [Learning principled bilingual mappings of word embeddings while preserving monolingual invariance](https://md.research.elamur.ai/free/en/papers/artetxe-2016-learning-principled-bilingual-mappings-word/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289–2294, Austin, Texas. Association for Computational Linguistics.
2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. [Unsupervised Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/). arXiv:1710.11041 [cs]. ArXiv: 1710.11041. [↗](https://arxiv.org/abs/1710.11041)
3. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. [On the Cross-lingual Transferability of Monolingual Representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
4. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209–226. [↗](https://doi.org/10.1007/s10579-009-9081-4)
5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.747)
6. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word Translation Without Parallel Data. arXiv:1710.04087 [cs]. ArXiv: 1710.04087. [↗](https://arxiv.org/abs/1710.04087)
7. Lorenzo De Mattei, Michele Cafagna, Felice Dell’Orletta, Malvina Nissim, and Marco Guerini. 2020. GePpeTto Carves Italian into a Language Model. arXiv:2004.14253 [cs]. [↗](https://arxiv.org/abs/2004.14253)
8. Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. [RobBERT: a Dutch RoBERTa-based Language Model](https://md.research.elamur.ai/free/en/papers/delobelle-2020-robbert-dutch-roberta-based-language/). arXiv:2001.06286 [cs]. [↗](https://arxiv.org/abs/2001.06286)
9. David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2020. Ethnologue: Languages of the World. Twenty-third edition. SIL International. [find](https://scholar.google.com/scholar?q=David+M%2E+Eberhard%2C+Gary+F%2E+Simons%2C+and+Charles+D%2E+Fennig%2E+2020%2E+Ethnologue%3A+Languages+of+the+World%2E+Twenty%2Dthird+edition%2E+SIL+International%2E)
10. W.A. Falcon. 2019. PyTorch Lightning. [find](https://scholar.google.com/scholar?q=W%2EA%2E+Falcon%2E+2019%2E+PyTorch+Lightning%2E)
11. Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 710– 721, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Goran+Glavas%CC%8C%2C+Robert+Litschko%2C+Sebastian+Ruder%2C+and+Ivan+Vulic%CC%81%2E+2019%2E+How+to+%28Properly%29+Evaluate+Cross%2DLingual+Word+Embeddings%3A+On+Strong+Baselines%2C+Comparative+Analyses%2C+and+Some+Misconceptions%2E+In)
12. Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. [Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion](https://md.research.elamur.ai/free/en/papers/joulin-2018-loss-translation-learning-bilingual-word/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2979–2984, Brussels, Belgium. Association for Computational Linguistics.
13. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2017%2E+Bag+of+Tricks+for+Efficient+Text+Classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associati)
14. Yunsu Kim, Yingbo Gao, and Hermann Ney. 2019. Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1246– 1257, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yunsu+Kim%2C+Yingbo+Gao%2C+and+Hermann+Ney%2E+2019%2E+Effective+Cross%2Dlingual+Transfer+of+Neural+Machine+Translation+Models+without+Shared+Vocabularies%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associ)
15. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. In arXiv:1412.6980 [cs]. [↗](https://arxiv.org/abs/1412.6980)
16. Tom Kocmi and Ondřej Bojar. 2018. [Trivial Transfer Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/kocmi-2018-trivial-transfer-learning-low-resource/). In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 244– 252, Brussels, Belgium. Association for Computational Linguistics.
17. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. [Improving Distributional Similarity with Lessons Learned from Word Embeddings](https://md.research.elamur.ai/free/en/papers/levy-2015-improving-distributional-similarity-lessons-learned/). Transactions of the Association for Computational Linguistics, 3:211–225. [↗](https://doi.org/10.1162/tacl_a_00134)
18. Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. [Choosing Transfer Languages for Cross-Lingual Learning](https://md.research.elamur.ai/free/en/papers/lin-2019-choosing-transfer-languages-cross-lingual/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125–3135, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1301)
19. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed Precision Training. arXiv:1710.03740 [cs, stat]. ArXiv: 1710.03740. [↗](https://arxiv.org/abs/1710.03740)
20. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting Similarities among Languages for Machine Translation. arXiv:1309.4168 [cs]. [↗](https://arxiv.org/abs/1309.4168)
21. Tasnim Mohiuddin, M Saiful Bari, and Shafiq Joty. 2020. LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2712–2723, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tasnim+Mohiuddin%2C+M+Saiful+Bari%2C+and+Shafiq+Joty%2E+2020%2E+LNMap%3A+Departures+from+Isomorphic+Assumption+in+Bilingual+Lexicon+Induction+Through+Non%2DLinear+Mapping+in+Latent+Space%2E+In+Proceedings+of+the+20)
22. Toan Q. Nguyen and David Chiang. 2017. [Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/nguyen-2017-transfer-learning-across-low-resource/). In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 296–301, Taipei, Taiwan. Asian Federation of Natural Language Processing. [↗](https://doi.org/10.48550/arxiv.1708.09803)
23. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2018. [RankME: Reliable Human Ratings for Natural Language Generation](https://md.research.elamur.ai/free/en/papers/novikova-2018-rankme-reliable-human-ratings-natural/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 72–78, New Orleans, Louisiana. Association for Computational Linguistics.
24. Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [MASK]? Making Sense of Language-Specific BERT Models. arXiv:2003.02912 [cs]. [↗](https://arxiv.org/abs/2003.02912)
25. Nelleke Oostdijk, Martin Reynaert, Véronique Hoste, and Ineke Schuurman. 2013. The Construction of a 500-Million-Word Reference Corpus of Contemporary Written Dutch. In Peter Spyns and Jan Odijk, editors, Essential Speech and Language Technology for Dutch: Results by the STEVIN programme, Theory and Applications of Natural Language Processing, pages 219–247. Springer, Berlin, Heidelberg. [↗](https://doi.org/10.1007/978-3-642-30910-6_13)
26. Ordelman, Roeland J.F., de Jong, Franciska M.G., van Hessen, Adrianus J., and Hondorp, G.H.W. 2007. TwNC: a Multifaceted Dutch News Corpus. ELRA Newsletter, 12(3-4). [find](https://scholar.google.com/scholar?q=Ordelman%2C+Roeland+J%2EF%2E%2C+de+Jong%2C+Franciska+M%2EG%2E%2C+van+Hessen%2C+Adrianus+J%2E%2C+and+Hondorp%2C+G%2EH%2EW%2E+2007%2E+TwNC%3A+a+Multifaceted+Dutch+News+Corpus%2E+ELRA+Newsletter%2C+12%283%2D4%29%2E)
27. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). In H. Wallach, H. Larochelle, A. Beygelzimer, F. d\textquotesingle Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8026– 8037. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.1912.01703)
28. Jonathan Peirce, Jeremy R. Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Erik Kastman, and Jonas Kristoffer Lindeløv. 2019. PsychoPy2: Experiments in behavior made easy. Behavior Research Methods, 51(1):195–203. [↗](https://doi.org/10.3758/s13428-018-01193-y)
29. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+Models+are+Unsupervised+Multitask+Learners%2E)
30. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A Survey of Cross-lingual Word Embedding Models. Journal of Artificial Intelligence Research, 65:569–631. [↗](https://doi.org/10.1613/jair.1.11640)
31. Leslie N. Smith. 2017. Cyclical Learning Rates for Training Neural Networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 464–472. [find](https://scholar.google.com/scholar?q=Leslie+N%2E+Smith%2E+2017%2E+Cyclical+Learning+Rates+for+Training+Neural+Networks%2E+In+2017+IEEE+Winter+Conference+on+Applications+of+Computer+Vision+%28WACV%29%2C+pages+464%E2%80%93472%2E)
32. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. [Energy and Policy Considerations for Deep Learning in NLP](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
33. Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018. [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://md.research.elamur.ai/free/en/papers/sogaard-2018-limitations-unsupervised-bilingual-dictionary-induction/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 778– 788, Melbourne, Australia. Association for Computational Linguistics.
34. Wietse de Vries, Andreas van Cranenburgh, and Malvina Nissim. 2020. What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4339–4350, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wietse+de+Vries%2C+Andreas+van+Cranenburgh%2C+and+Malvina+Nissim%2E+2020%2E+What%E2%80%99s+so+special+about+BERT%E2%80%99s+layers%3F+A+closer+look+at+the+NLP+pipeline+in+monolingual+and+multilingual+models%2E+In+Findings+of+the+)
35. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. [BERTje: A Dutch BERT Model](https://md.research.elamur.ai/free/en/papers/de-vries-2019-bertje-dutch-bert-model/). arXiv:1912.09582 [cs]. [↗](https://arxiv.org/abs/1912.09582)
36. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2020. HuggingFace’s [Transformers: State-of-the-art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv:1910.03771 [cs]. [↗](https://arxiv.org/abs/1910.03771)
37. Shijie Wu and Mark Dredze. 2020. [Are All Languages Created Equal in Multilingual BERT?](https://md.research.elamur.ai/free/en/papers/wu-2020-are-all-languages-created-equal/) arXiv:2005.09093 [cs]. ArXiv: 2005.09093. [↗](https://arxiv.org/abs/2005.09093)
38. Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. [Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation](https://md.research.elamur.ai/free/en/papers/xing-2015-normalized-word-embedding-orthogonal-transform/). In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006–1011, Denver, Colorado. Association for Computational Linguistics.
39. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer Learning for Low-Resource Neural Machine Translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Barret+Zoph%2C+Deniz+Yuret%2C+Jonathan+May%2C+and+Kevin+Knight%2E+2016%2E+Transfer+Learning+for+Low%2DResource+Neural+Machine+Translation%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Lan)

*Updated 24 October 2026*