---
title: "Efficient Language Model Training through Cross-Lingual and"
description: "Ostendorff and Rehm propose CLP-Transfer, which transfers both across languages and across… 49 references, cited by 23 papers here."
authors: ["Malte Ostendorff", "Georg Rehm"]
year: 2023
venue: "arXiv:2301.09626"
source: "https://arxiv.org/abs/2301.09626"
doi: "10.48550/arxiv.2301.09626"
arxiv: "2301.09626"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ostendorff-2023-efficient-language-model-training-through/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ostendorff-2023-efficient-language-model-training-through/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning

# Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning

**Authors:** [Malte Ostendorff](https://md.research.elamur.ai/free/en/authors/malte-ostendorff/) and [Georg Rehm](https://md.research.elamur.ai/free/en/authors/georg-rehm/) · **Year:** 2023 · **Venue:** arXiv:2301.09626 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2301.09626) · [PDF](https://arxiv.org/pdf/2301.09626) · [DOI 10.48550/arxiv.2301.09626](https://doi.org/10.48550/arxiv.2301.09626) · [arXiv 2301.09626](https://arxiv.org/abs/2301.09626) · [OpenAlex](https://openalex.org/W4317951304)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/ostendorff-2023-efficient-language-model-training-through/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large Transformer language models are mostly pretrained on English. Training equally large models for other languages from scratch requires data and compute that are often unavailable. Ostendorff and Rehm propose CLP-Transfer, which transfers both across languages and across model sizes. To build a large target-language model, they start from two existing models: a large source model (English or multilingual) with the same architecture, and a small model already trained in the target language. All Transformer layer weights come directly from the large source model. Embeddings of tokens present in both vocabularies are copied as well. Each remaining token gets an embedding mixed from shared-token embeddings, weighted by cosine similarity in the small target model's embedding space. The authors tested this on GPT2 and BLOOM decoder models, with German as the target language. CLP-Transfer outperformed WECHSEL and random initialization, matching from-scratch perplexity with half (GPT2) or one fifth (BLOOM) of the training tokens. This makes large models cheaper to build for languages with fewer resources.

## Key points
- **Two assumptions behind the method:**
  - Source and target tokenizers must share vocabulary. English GPT2 and the German tokenizer overlap by 24.04%, and multilingual BLOOM and German by 5.55%.
  - Embedding spaces must look alike across model sizes. English OPT models of 125M and 13B parameters share 54% of their 10 nearest neighbours.
- **Training savings:** at 1.5B parameters (GPT2-XL-CLP), the method saves about 50% of training effort. At 6.4B parameters (BLOOM-CLP), it saves about 80%.
- **Perplexity on German OSCAR validation:** BLOOM-CLP 6.4B reached 44.1, versus 69.3 for BLOOM 6.7B trained from scratch on the same 50B tokens. GPT2-XL-CLP 1.5B reached 46.33, versus 157.95 for GPT2-XL-WECHSEL.
- **Zero-shot results on six German benchmarks** (GermEval 2017/2018, GNAD10, PAWSX, XNLI, X-Stance):
  - Most models perform near the random baseline.
  - BLOOM-CLP 6.4B has the best average score (0.43).
- **Released resources:** checkpoints are on HuggingFace (malteos/bloom-6b4-clp-german), code is on GitHub (malteos/clp-transfer), and a web demo is available.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [German Research Centre for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/german-research-centre-for-artificial-intelligence/) — Research institute, Kaiserslautern, Germany · 77 works · 29,479 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Malte Ostendorff and Georg Rehm. 2023. Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning. arXiv:2301.09626.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ostendorff2023efficient,
  title = {{Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning}},
  author = {Malte Ostendorff and Georg Rehm},
  year = {2023},
  doi = {10.48550/arxiv.2301.09626},
  eprint = {2301.09626},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.09626}
}
```

- [← previous An Emulator for Fine-Tuning Large Language Models using Small Language Models](https://md.research.elamur.ai/free/en/papers/mitchell-2023-emulator-fine-tuning-large-language/)
- [next → Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing](https://md.research.elamur.ai/free/en/papers/jung-2023-impossible-distillation-low-quality-model/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

49 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. [AraGPT2: Pre-trained transformer for Arabic language generation](https://md.research.elamur.ai/free/en/papers/antoun-2021-aragpt2-pre-trained-transformer-arabic/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196–207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
3. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. [GPT-NeoX-20B: An Open-Source Autoregr…](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. Branden Chan, Stefan Schweter, and Timo Möller. 2020. [German’s next language model](https://md.research.elamur.ai/free/en/papers/chan-2020-german-s-next-language-model/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics.
6. Alexandra Chronopoulou, Matthew E. Peters, and Jesse Dodge. 2021. Efficient Hierarchical Domain Adaptation for Pretrained Language Models. [find](https://scholar.google.com/scholar?q=Alexandra+Chronopoulou%2C+Matthew+E%2E+Peters%2C+and+Jesse+Dodge%2E+2021%2E+Efficient+Hierarchical+Domain+Adaptation+for+Pretrained+Language+Models%2E)
7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+Cross%2Dlingu)
8. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computation… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+R%2E+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+)
9. Wietse de Vries and Malvina Nissim. 2021. [As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages](https://md.research.elamur.ai/free/en/papers/vries-2021-as-good-as-new-how/). Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836–846.
10. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
11. Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Efficient training of BERT by progressively stacking. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2337–2346. PMLR. [find](https://scholar.google.com/scholar?q=Linyuan+Gong%2C+Di+He%2C+Zhuohan+Li%2C+Tao+Qin%2C+Liwei+Wang%2C+and+Tieyan+Liu%2E+2019%2E+Efficient+training+of+BERT+by+progressively+stacking%2E+In+Proceedings+of+the+36th+International+Conference+on+Machine+Learnin)
12. Alex Graves. 2016. Adaptive Computation Time for Recurrent Neural Networks. [find](https://scholar.google.com/scholar?q=Alex+Graves%2E+2016%2E+Adaptive+Computation+Time+for+Recurrent+Neural+Networks%2E)
13. Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. 2021. On the Transformer Growth for Progressive BERT Training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5174–5180, O… [find](https://scholar.google.com/scholar?q=Xiaotao+Gu%2C+Liyuan+Liu%2C+Hongkun+Yu%2C+Jing+Li%2C+Chen+Chen%2C+and+Jiawei+Han%2E+2021%2E+On+the+Transformer+Growth+for+Progressive+BERT+Training%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapt)
14. Xu Guo, Boyang Albert Li, and Han Yu. 2022. Improving the sample efficiency of prompt tuning with domain adaptation. ArXiv, abs/2210.02952. [find](https://scholar.google.com/scholar?q=Xu+Guo%2C+Boyang+Albert+Li%2C+and+Han+Yu%2E+2022%2E+Improving+the+sample+efficiency+of+prompt+tuning+with+domain+adaptation%2E+ArXiv%2C+abs%2F2210%2E02952%2E)
15. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
16. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
17. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive Growing of GANs for Improved Quality, Stability, and Variation. [find](https://scholar.google.com/scholar?q=Tero+Karras%2C+Timo+Aila%2C+Samuli+Laine%2C+and+Jaakko+Lehtinen%2E+2017%2E+Progressive+Growing+of+GANs+for+Improved+Quality%2C+Stability%2C+and+Variation%2E)
18. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Daphne+Ippolito%2C+Andrew+Nystrom%2C+Chiyuan+Zhang%2C+Douglas+Eck%2C+Chris+Callison%2DBurch%2C+and+Nicholas+Carlini%2E+2022%2E+Deduplicating+Training+Data+Makes+Language+Models+Better%2E+In+Proceedings+o)
19. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… — [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
20. Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. [Choosing transfer languages for cross-lingual learning](https://md.research.elamur.ai/free/en/papers/lin-2019-choosing-transfer-languages-cross-lingual/). In Proceedings of the 57th Annual Meeting of…
21. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. [CamemBERT: a tasty French language model](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, O…
22. Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2021. [WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models](https://md.research.elamur.ai/free/en/papers/minixhofer-2022-wechsel-effective-initialization-subword-embeddings/).
23. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel… — [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
24. Toan Q. Nguyen and David Chiang. 2017. [Transfer learning across low-resource, related languages for neural machine translation](https://md.research.elamur.ai/free/en/papers/nguyen-2017-transfer-learning-across-low-resource/). In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 296–301, Taipei, Taiwan. Asian Federation of Natu…
25. Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [MASK]? Making Sense of Language-Specific BERT Models. [find](https://scholar.google.com/scholar?q=Debora+Nozza%2C+Federico+Bianchi%2C+and+Dirk+Hovy%2E+2020%2E+What+the+%5BMASK%5D%3F+Making+Sense+of+Language%2DSpecific+BERT+Models%2E)
26. Pedro Javier Ortiz Su’arez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannhei… [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%E2%80%99arez%2C+Benoit+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+pipelines+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+Proceedings+of+the+Workshop+on+Challen)
27. Malte Ostendorff, Till Blume, and Saskia Ostendorff. 2020. Towards an Open Platform for Legal Information. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pages 385–388. [find](https://scholar.google.com/scholar?q=Malte+Ostendorff%2C+Till+Blume%2C+and+Saskia+Ostendorff%2E+2020%2E+Towards+an+Open+Platform+for+Legal+Information%2E+In+Proceedings+of+the+ACM%2FIEEE+Joint+Conference+on+Digital+Libraries+in+2020%2C+pages+385%E2%80%93388%2E)
28. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. [AdapterHub: A Framework for Adapting Transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/). pages 46–54.
29. Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+2022%2E+Train+Short%2C+Test+Long%3A+Attention+with+Linear+Biases+Enables+Input+Length+Extrapolation%2E)
30. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. arXiv. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+Models+are+Unsupervised+Multitask+Learners%2E+arXiv%2E)
31. Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, and Iz Beltagy. 2022. What Languag… [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Thomas+Wang%2C+Daniel+Hesslow%2C+Lucile+Saulnier%2C+Stas+Bekman%2C+M+Saiful+Bari%2C+Stella+Biderman%2C+Hady+Elsahar%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Ofir+Press%2C+Colin+Raffel%2C+Victor+Sanh%2C+Sheng+Sh)
32. Dietmar Schabus, Marcin Skowron, and Martin Trapp. 2017. One million posts: A data set of german online discussions. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1241–1244, Tokyo, Japan. [find](https://scholar.google.com/scholar?q=Dietmar+Schabus%2C+Marcin+Skowron%2C+and+Martin+Trapp%2E+2017%2E+One+million+posts%3A+A+data+set+of+german+online+discussions%2E+In+Proceedings+of+the+40th+International+ACM+SIGIR+Conference+on+Research+and+Devel)
33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
34. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. [mGPT: Few-Shot Learners Go Multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/).
35. Karen Simonyan and Andrew Zisserman. 2014. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). CoRR, abs/1409.1556.
36. Antoine Simoulin and Benoit Crabbé. 2021. Un modèle Transformer Génératif Pré-entrainé pour le ______ français. In Traitement Automatique des Langues Naturelles, pages 246–255, Lille, France. ATALA. [find](https://scholar.google.com/scholar?q=Antoine+Simoulin+and+Benoit+Crabb%C3%A9%2E+2021%2E+Un+mod%C3%A8le+Transformer+G%C3%A9n%C3%A9ratif+Pr%C3%A9%2Dentrain%C3%A9+pour+le+%5F%5F%5F%5F%5F%5F+fran%C3%A7ais%2E+In+Traitement+Automatique+des+Langues+Naturelles%2C+pages+246%E2%80%93255%2C+Lille%2C+France%2E+ATALA%2E)
37. Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. 2018. Meta-transfer learning for few-shot learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 403– 412. [find](https://scholar.google.com/scholar?q=Qianru+Sun%2C+Yaoyao+Liu%2C+Tat%2DSeng+Chua%2C+and+Bernt+Schiele%2E+2018%2E+Meta%2Dtransfer+learning+for+few%2Dshot+learning%2E+2019+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+403%E2%80%93+412)
38. Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. 2018. A survey on deep transfer learning. In International Conference on Artificial Neural Networks. [find](https://scholar.google.com/scholar?q=Chuanqi+Tan%2C+Fuchun+Sun%2C+Tao+Kong%2C+Wenchang+Zhang%2C+Chao+Yang%2C+and+Chunfang+Liu%2E+2018%2E+A+survey+on+deep+transfer+learning%2E+In+International+Conference+on+Artificial+Neural+Networks%2E)
39. Jannis Vamvas and Rico Sennrich. 2020. X-Stance: A multilingual multi-target dataset for stance detection. In Proceedings of the 5th Swiss Text Analytics Conference (SwissText) 16th Conference on Natural Language Processing (KONVENS), Zurich, Switzerland. [find](https://scholar.google.com/scholar?q=Jannis+Vamvas+and+Rico+Sennrich%2E+2020%2E+X%2DStance%3A+A+multilingual+multi%2Dtarget+dataset+for+stance+detection%2E+In+Proceedings+of+the+5th+Swiss+Text+Analytics+Conference+%28SwissText%29+16th+Conference+on+Natu)
40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention Is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Proceedings of the 31st International Conference on Neural Information Processing Systems, (Nips):6000–6010.
41. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. [Multilingual is not enough: Bert for finnish](https://md.research.elamur.ai/free/en/papers/virtanen-2019-multilingual-enough-bert-finnish/).
42. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. [Finetuned Language Models Are Zero-Shot Learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/).
43. Michael Wiegand and Melanie Siegel. 2018. Overview of the germeval 2018 shared task on the identification of offensive language. In Proceedings of the GermEval. [find](https://scholar.google.com/scholar?q=Michael+Wiegand+and+Melanie+Siegel%2E+2018%2E+Overview+of+the+germeval+2018+shared+task+on+the+identification+of+offensive+language%2E+In+Proceedings+of+the+GermEval%2E)
44. Michael Wojatzki, Eugen Ruppert, Sarah Holschneider, Torsten Zesch, and Chris Biemann. 2017. [find](https://scholar.google.com/scholar?q=Michael+Wojatzki%2C+Eugen+Ruppert%2C+Sarah+Holschneider%2C+Torsten+Zesch%2C+and+Chris+Biemann%2E+2017%2E)
45. Germeval 2017: Shared task on aspect-based sentiment in social media customer feedback. Proceedings of the GermEval, pages 1–12. [find](https://scholar.google.com/scholar?q=Germeval+2017%3A+Shared+task+on+aspect%2Dbased+sentiment+in+social+media+customer+feedback%2E+Proceedings+of+the+GermEval%2C+pages+1%E2%80%9312%2E)
46. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. [PAWS-X: A cross-lingual adversarial dataset for paraphrase identification](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin…
47. Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). (i).
48. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. [OPT: Open Pre-trai…](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
49. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Barret+Zoph%2C+Deniz+Yuret%2C+Jonathan+May%2C+and+Kevin+Knight%2E+2016%2E+Transfer+learning+for+low%2Dresource+neural+machine+translation%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Lan)

*Updated 31 October 2026*