---
title: "BLOOM+1 — paper by Yong et al. (2023) — summary and ideas"
description: "The study offers an affordable recipe for extending large models to under-served languages. 73 references, cited by 81 papers here."
authors: ["Zheng Xin Yong", "Hailey Schoelkopf", "Niklas Muennighoff", "Alham Fikri Aji", "David Ifeoluwa Adelani", "Khalid Almubarak", "M Saiful Bari", "Lintang Sutawika", "Jungo Kasai", "Ahmed Baruwa", "Genta Winata", "Stella Biderman", "Edward Raff", "Dragomir Radev", "Vassilina Nikoulina"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.653/"
doi: "10.18653/v1/2023.acl-long.653"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yong-2023-bloom-1-adding-language-support/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yong-2023-bloom-1-adding-language-support/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting

# BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting

**Authors:** [Zheng Xin Yong](https://md.research.elamur.ai/free/en/authors/zheng-xin-yong/), [Hailey Schoelkopf](https://md.research.elamur.ai/free/en/authors/hailey-schoelkopf/), [Niklas Muennighoff](https://md.research.elamur.ai/free/en/authors/niklas-muennighoff/), [Alham Fikri Aji](https://md.research.elamur.ai/free/en/authors/alham-fikri-aji/), [David Ifeoluwa Adelani](https://md.research.elamur.ai/free/en/authors/david-ifeoluwa-adelani/), [Khalid Almubarak](https://md.research.elamur.ai/free/en/authors/khalid-almubarak/), [M Saiful Bari](https://md.research.elamur.ai/free/en/authors/m-saiful-bari/), [Lintang Sutawika](https://md.research.elamur.ai/free/en/authors/lintang-sutawika/), [Jungo Kasai](https://md.research.elamur.ai/free/en/authors/jungo-kasai/), Ahmed Baruwa, [Genta Winata](https://md.research.elamur.ai/free/en/authors/genta-indra-winata/), [Stella Biderman](https://md.research.elamur.ai/free/en/authors/stella-biderman/), Edward Raff, [Dragomir Radev](https://md.research.elamur.ai/free/en/authors/dragomir-radev/) and Vassilina Nikoulina · **Year:** 2023 · **Venue:** [Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2023](https://md.research.elamur.ai/free/en/venues/acl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.653/) · [PDF](https://aclanthology.org/2023.acl-long.653.pdf) · [DOI 10.18653/v1/2023.acl-long.653](https://doi.org/10.18653/v1/2023.acl-long.653) · [OpenAlex](https://openalex.org/W4385570226)

**Citations:** 20 citations (OpenAlex, 2026-10-14) · [81](https://md.research.elamur.ai/free/en/papers/yong-2023-bloom-1-adding-language-support/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
BLOOM is a large, openly released multilingual model, but its pretraining covered only 46 natural languages and omitted widely spoken ones such as Russian and Korean. Retraining it to fix this would be too costly. The authors test whether existing language adaptation techniques can add languages after pretraining, and how this affects zero-shot prompting.

BLOOM checkpoints ranging from 560M to 7.1B parameters are adapted to eight languages absent from pretraining: Guarani, Korean, Greek, Turkish, Thai, Bulgarian, Russian and German. Each language gets at most 100K monolingual samples. They compare continued pretraining, MAD-X adapters and (IA)3 on NLU benchmarks such as XNLI.

Adaptation generally improves zero-shot results. For models around 3B parameters and larger, adapters outperform continued pretraining, which contradicts earlier findings on smaller masked models. Performance depends mainly on the amount of adaptation data, not on script or other language traits. For the instruction-tuned BLOOMZ, monolingual adaptation damages instruction following, while adding the new language to the multitask mixture works best. The study offers an affordable recipe for extending large models to under-served languages.

## Key points
- The base model is BLOOM, pretrained on about 350B tokens of the ROOTS corpus (46 natural and 13 programming languages). Its byte-level BPE tokenizer has a 250,680-token vocabulary, so text in unseen scripts can still be tokenized.
- The three strategies benchmarked are continued pretraining, MAD-X (language and invertible adapters) and (IA)3. The authors recommend adapters over continued pretraining for BLOOM models with at least 3B parameters.
- Effective prompting in a new language required roughly 100 million tokens of adaptation data. Guarani was the only truly low-resource language tested, and such languages often lack data at this scale.
- Two Russian BLOOMZ variants were built:
  - BLOOMZ-7.1B-RU, tuned on Russian task data only, gave just tiny gains, on XStoryCloze.
  - BLOOMZ-7.1B-xP3RU, tuned on xP3 plus Russian data, improved best-prompt results on XNLI and XStoryCloze.
- Limitations include:
  - evaluation on NLU tasks only;
  - catastrophic forgetting of previously seen languages;
  - a sequence length of 1024 used by mistake instead of 2048;
  - no adaptation of the 176B model.

  Code is available at github.com/bigscience-workshop/multilingual-modeling.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations
- [Brown University](https://md.research.elamur.ai/free/en/universities/brown-university/) — University, Providence, United States · 58 works · 25,310 citations
- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 58 works · 23,058 citations
- [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 59 works · 16,463 citations
- [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) — University, New Haven, United States · 54 works · 11,964 citations
- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 73 works · 24,390 citations
- [Hugging Face](https://md.research.elamur.ai/free/en/companies/hugging-face/) — Company, New York, United States · 12 works · 20,193 citations
- [Naver](https://md.research.elamur.ai/free/en/companies/naver/) — Company, Seongnam-si, South Korea · 15 works · 11,528 citations
- [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 47 works · 4,456 citations
- [University of Oregon](https://md.research.elamur.ai/free/en/universities/university-of-oregon/) — University, Eugene, United States · 14 works · 3,167 citations
- [Prince Sattam Bin Abdulaziz University](https://md.research.elamur.ai/free/en/universities/prince-sattam-bin-abdulaziz-university/) — University, Al Kharj, Saudi Arabia · 5 works · 410 citations
- [Booz Allen Hamilton](https://md.research.elamur.ai/free/en/companies/booz-allen-hamilton/) — Company, Tysons Corner, United States · 3 works · 1,829 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev and Vassilina Nikoulina. 2023. BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yong2023bloom,
  title = {{BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting}},
  author = {Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev and Vassilina Nikoulina},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.653},
  url = {https://doi.org/10.18653/v1/2023.acl-long.653}
}
```

- [← previous Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://md.research.elamur.ai/free/en/papers/xia-2023-speculative-decoding-exploiting-speculative-execution/)
- [next → Copyright Violations and Large Language Models](https://md.research.elamur.ai/free/en/papers/karamolegkou-2023-copyright-violations-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

73 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alham Fikri Aji, Nikolay Bogoychev, Kenneth Heafield, and Rico Sennrich. 2020. In neural machine translation, what does transfer learning transfer? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7701–7710. [find](https://scholar.google.com/scholar?q=Alham+Fikri+Aji%2C+Nikolay+Bogoychev%2C+Kenneth+Heafield%2C+and+Rico+Sennrich%2E+2020%2E+In+neural+machine+translation%2C+what+does+transfer+learning+transfer%3F+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Ass)
2. Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. 2022. Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1778–1796, Dublin, Ireland. Association for Computatio… [find](https://scholar.google.com/scholar?q=Alan+Ansell%2C+Edoardo+Ponti%2C+Anna+Korhonen%2C+and+Ivan+Vulic%CC%81%2E+2022%2E+Composable+sparse+fine%2Dtuning+for+cross%2Dlingual+transfer%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computation)
3. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. 2021. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Shruti+Bhosale%2C+Naman+Goyal%2C+Todor+Mihaylov%2C+Myle+Ott%2C+Sam+Shleifer%2C+Xi+Victoria+Lin%2C+Jingfei+Du%2C+Srinivasan+Iyer%2C+Ramakanth+Pasunuru%2C+et+al%2E+2021%2E+Efficient+large+scale+language+modeli)
4. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
5. Mikel Artetxe and Holger Schwenk. 2019. [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/). Transactions of the Association for Computational Linguistics, 7:597– 610.
6. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. [BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, Dublin, Ireland. Asso…
7. Alexandre Berard. 2021. Continual learning in multilingual NMT via language-specific embeddings. In Proceedings of the Sixth Conference on Machine Translation, pages 542–565, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexandre+Berard%2E+2021%2E+Continual+learning+in+multilingual+NMT+via+language%2Dspecific+embeddings%2E+In+Proceedings+of+the+Sixth+Conference+on+Machine+Translation%2C+pages+542%E2%80%93565%2C+Online%2E+Association+for+C)
8. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata, 58. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Leo+Gao%2C+Phil+Wang%2C+Connor+Leahy%2C+and+Stella+Biderman%2E+2021%2E+Gpt%2Dneo%3A+Large+scale+autoregressive+language+modeling+with+mesh%2Dtensorflow%2E+If+you+use+this+software%2C+please+cite+it+using+these)
9. Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. [GPT-NeoX-20B: An open-source autor…](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
10. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. [Parsing with multilingual BERT, a small corpus, and a small treebank](https://md.research.elamur.ai/free/en/papers/chau-2020-parsing-multilingual-bert-small-corpus/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online. Association for Computational Linguistics.
11. Luis Chiruzzo, Santiago Góngora, Aldo Alvarez, Gustavo Giménez-Lugo, Marvin Agüero-Torales, and Yliana Rodríguez. 2022. Jojajovai: A parallel Guarani-Spanish corpus for MT benchmarking. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2098–2107, Marseille, France.… [find](https://scholar.google.com/scholar?q=Luis+Chiruzzo%2C+Santiago+G%C3%B3ngora%2C+Aldo+Alvarez%2C+Gustavo+Gim%C3%A9nez%2DLugo%2C+Marvin+Ag%C3%BCero%2DTorales%2C+and+Yliana+Rodr%C3%ADguez%2E+2022%2E+Jojajovai%3A+A+parallel+Guarani%2DSpanish+corpus+for+MT+benchmarking%2E+In+Proceedings)
12. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. [Scaling instruction-finetuned language models](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/). arXiv preprint arXiv:2210.11416.
13. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
14. Fahim Faisal and Antonios Anastasopoulos. 2022. Phylogeny-inspired adaptation of multilingual models to new languages. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Pro… [find](https://scholar.google.com/scholar?q=Fahim+Faisal+and+Antonios+Anastasopoulos%2E+2022%2E+Phylogeny%2Dinspired+adaptation+of+multilingual+models+to+new+languages%2E+In+Proceedings+of+the+2nd+Conference+of+the+Asia%2DPacific+Chapter+of+the+Associati)
15. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
16. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+2019%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+In+International+Conference+on+Learning+Representations%2E)
17. Javier De la Rosa and Andrés Fernández. 2022. Zero-shot reading comprehension and reasoning for spanish with bertin gpt-j-6b. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022). [find](https://scholar.google.com/scholar?q=Javier+De+la+Rosa+and+Andr%C3%A9s+Fern%C3%A1ndez%2E+2022%2E+Zero%2Dshot+reading+comprehension+and+reasoning+for+spanish+with+bertin+gpt%2Dj%2D6b%2E+Proceedings+of+the+Iberian+Languages+Evaluation+Forum+%28IberLEF+2022%29%2E)
18. Yoshinari Fujinuma, Jordan Boyd-Graber, and Katharina Kann. 2022. Match the script, adapt if multilingual: Analyzing the effect of multilingual pretraining on cross-lingual transferability. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap… [find](https://scholar.google.com/scholar?q=Yoshinari+Fujinuma%2C+Jordan+Boyd%2DGraber%2C+and+Katharina+Kann%2E+2022%2E+Match+the+script%2C+adapt+if+multilingual%3A+Analyzing+the+effect+of+multilingual+pretraining+on+cross%2Dlingual+transferability%2E+In+Proceed)
19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
20. Philipp Dufter and Hinrich Schütze. 2020. [Identifying elements essential for BERT’s multilinguality](https://md.research.elamur.ai/free/en/papers/dufter-2020-identifying-elements-essential-bert-s/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4423–4437, Online. Association for Computational Linguistics.
21. Abteen Ebrahimi and Katharina Kann. 2021. [How to adapt your pretrained multilingual model to 1600 languages](https://md.research.elamur.ai/free/en/papers/ebrahimi-2021-how-adapt-your-pretrained-multilingual/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),…
22. Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022. [AmericasNLI: E…](https://md.research.elamur.ai/free/en/papers/ebrahimi-2022-americasnli-evaluating-zero-shot-natural/)
23. Philip Gage. 1994. A new algorithm for data compression. C Users Journal, 12(2):23–38. [find](https://scholar.google.com/scholar?q=Philip+Gage%2E+1994%2E+A+new+algorithm+for+data+compression%2E+C+Users+Journal%2C+12%282%29%3A23%E2%80%9338%2E)
24. Demi Guo, Alexander Rush, and Yoon Kim. 2021. [Parameter-efficient transfer learning with diff pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages…
25. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. [Towards a unified view of parameter-efficient transfer learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/). In International Conference on Learning Representations.
26. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
27. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
28. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. [The state and fate of linguistic diversity and inclusion in the NLP world](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
29. Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Indobertweet: A pretrained language model for indonesian twitter with effective domain-specific vocabulary initialization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10660–10668. [find](https://scholar.google.com/scholar?q=Fajri+Koto%2C+Jey+Han+Lau%2C+and+Timothy+Baldwin%2E+2021%2E+Indobertweet%3A+A+pretrained+language+model+for+indonesian+twitter+with+effective+domain%2Dspecific+vocabulary+initialization%2E+In+Proceedings+of+the+202)
30. [Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro…](https://md.research.elamur.ai/free/en/papers/kreutzer-2021-quality-glance-audit-web-crawled/)
31. [Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loub…](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-bigscience-roots-corpus-1-6tb/)
32. Teven Le Scao and Alexander Rush. 2021. [How many data points is a prompt worth?](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627–2636, Online. Association for Computational Linguistics.
33. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+2021%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing)
34. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
35. [Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano…](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
36. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). arXiv preprint arXiv:2205.05638.
37. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). arXiv preprint arXiv:2107.13586.
38. Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022. [UniPELT: A unified framework for parameter-efficient language model tuning](https://md.research.elamur.ai/free/en/papers/mao-2022-unipelt-unified-framework-parameter-efficient/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers…
39. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. [Crosslingual generalization through multitask finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/). arXiv preprint arXiv:2211.01786.
40. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. [When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models](https://md.research.elamur.ai/free/en/papers/muller-2021-when-being-unseen-mbert-is/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lingu…
41. Martin Müller and Florian Laurent. 2022. Cedille: A large autoregressive french language model. arXiv preprint arXiv:2202.03371. [find](https://scholar.google.com/scholar?q=Martin+M%C3%BCller+and+Florian+Laurent%2E+2022%2E+Cedille%3A+A+large+autoregressive+french+language+model%2E+arXiv+preprint+arXiv%3A2202%2E03371%2E)
42. Graham Neubig and Junjie Hu. 2018. [Rapid adaptation of neural machine translation to new languages](https://md.research.elamur.ai/free/en/papers/neubig-2018-rapid-adaptation-neural-machine-translation/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 875–880, Brussels, Belgium. Association for Computational Linguistics.
43. NovelAI. 2022. Data efficient language transfer with gpt-j. Accessed: 2023-01-16. [find](https://scholar.google.com/scholar?q=NovelAI%2E+2022%2E+Data+efficient+language+transfer+with+gpt%2Dj%2E+Accessed%3A+2023%2D01%2D16%2E)
44. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannh… [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Beno%C3%AEt+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+pipelines+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+In+Proceedings+of+the+Workshop+on+Chall)
45. Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Junseong Kim, Youngsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim,… [find](https://scholar.google.com/scholar?q=Sungjoon+Park%2C+Jihyung+Moon%2C+Sungdong+Kim%2C+Won+Ik+Cho%2C+Ji+Yoon+Han%2C+Jangwon+Park%2C+Chisung+Song%2C+Junseong+Kim%2C+Youngsook+Song%2C+Taehwan+Oh%2C+Joohong+Lee%2C+Juhyun+Oh%2C+Sungwon+Lyu%2C+Younghoon+Jeong%2C+Inkwon+L)
46. Marinela Parović, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2022. BAD-X: Bilingual adapters improve zero-shot cross-lingual transfer. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1791– 17… [find](https://scholar.google.com/scholar?q=Marinela+Parovic%CC%81%2C+Goran+Glava%C5%A1%2C+Ivan+Vulic%CC%81%2C+and+Anna+Korhonen%2E+2022%2E+BAD%2DX%3A+Bilingual+adapters+improve+zero%2Dshot+cross%2Dlingual+transfer%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+C)
47. Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. [Lifting the curse of multilinguality by pre-training modular transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2022-lifting-curse-multilinguality-pre-training/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human…
48. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021a. [AdapterFusion: Non-destructive task composition for transfer learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487–50…
49. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational L…
50. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021b. [UNKs everywhere: Adapting multilingual language models to new scripts](https://md.research.elamur.ai/free/en/papers/pfeiffer-2021-unks-everywhere-adapting-multilingual-language/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186–10203, Online and Punta Cana, Dominican Republic. Ass…
51. Jerin Philip, Alexandre Berard, Matthias Gallé, and Laurent Besacier. 2020. [Monolingual adapters for zero-shot neural machine translation](https://md.research.elamur.ai/free/en/papers/philip-2020-monolingual-adapters-zero-shot-neural/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4465–4470, Online. Association for Computational Linguis…
52. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. [XCOPA: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association…
53. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+Smith%2C+and+Mike+Lewis%2E+2022%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+International+Conference+on+Learning+Representations%2E)
54. Kunxun Qi, Hai Wan, Jianfeng Du, and Haolan Chen. 2022. Enhancing cross-lingual natural language inference by prompt-learning from cross-lingual templates. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1910–1923, Dublin, Ire… [find](https://scholar.google.com/scholar?q=Kunxun+Qi%2C+Hai+Wan%2C+Jianfeng+Du%2C+and+Haolan+Chen%2E+2022%2E+Enhancing+cross%2Dlingual+natural+language+inference+by+prompt%2Dlearning+from+cross%2Dlingual+templates%2E+In+Proceedings+of+the+60th+Annual+Meeting+of)
55. Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. [AdapterDrop: On the efficiency of adapters in transformers](https://md.research.elamur.ai/free/en/papers/ruckle-2021-adapterdrop-efficiency-adapters-transformers/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7930–7946, Online and Punta C…
56. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
57. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725.
58. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). In The Eleventh International Conference on Learning Representa…
59. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. [mgpt: Few-shot learners go multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/).
60. Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. arXiv preprint arXiv:2206.06522. — [arXiv:2206.06522](https://arxiv.org/abs/2206.06522) [find](https://scholar.google.com/scholar?q=Yi%2DLin+Sung%2C+Jaemin+Cho%2C+and+Mohit+Bansal%2E+2022%2E+Lst%3A+Ladder+side%2Dtuning+for+parameter+and+memory+efficient+transfer+learning%2E+arXiv+preprint+arXiv%3A2206%2E06522%2E+%E2%80%94+%5BarXiv%3A2206%2E06522%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
61. Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Training neural networks with fixed sparse masks. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yi%2DLin+Sung%2C+Varun+Nair%2C+and+Colin+Raffel%2E+2021%2E+Training+neural+networks+with+fixed+sparse+masks%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
62. Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla, and Oskar Van Der Wal. 2022. [You reap what you sow: On the challen…](https://md.research.elamur.ai/free/en/papers/talat-2022-you-reap-what-you-sow/)
63. Alexey Tikhonov and Max Ryabinin. 2021. [It’s All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/tikhonov-2021-it-s-all-heads-attention/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3534–3546, Online. Association for Computational Linguistics.
64. Lifu Tu, Caiming Xiong, and Yingbo Zhou. 2022. Prompt-tuning can be much better than fine-tuning on cross-lingual understanding with multilingual language models. arXiv preprint arXiv:2210.12360. [find](https://scholar.google.com/scholar?q=Lifu+Tu%2C+Caiming+Xiong%2C+and+Yingbo+Zhou%2E+2022%2E+Prompt%2Dtuning+can+be+much+better+than+fine%2Dtuning+on+cross%2Dlingual+understanding+with+multilingual+language+models%2E+arXiv+preprint+arXiv%3A2210%2E12360%2E)
65. Ahmet Üstün, Alexandre Berard, Laurent Besacier, and Matthias Gallé. 2021. Multilingual unsupervised neural machine translation with denoising adapters. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6650–6662, Online and Punta Cana, Dominican Republ… [find](https://scholar.google.com/scholar?q=Ahmet+%C3%9Cst%C3%BCn%2C+Alexandre+Berard%2C+Laurent+Besacier%2C+and+Matthias+Gall%C3%A9%2E+2021%2E+Multilingual+unsupervised+neural+machine+translation+with+denoising+adapters%2E+In+Proceedings+of+the+2021+Conference+on+Empiri)
66. Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A 6 billion parameter autoregressive language model. [find](https://scholar.google.com/scholar?q=Ben+Wang+and+Aran+Komatsuzaki%2E+2021%2E+Gpt%2Dj%2D6b%3A+A+6+billion+parameter+autoregressive+language+model%2E)
67. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649–2656, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zihan+Wang%2C+Karthikeyan+K%2C+Stephen+Mayhew%2C+and+Dan+Roth%2E+2020%2E+Extending+multilingual+BERT+to+low%2Dresource+languages%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+26)
68. Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preotiuc-Pietro. 2022. Cross-lingual few-shot learning on unseen languages. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conferen… [find](https://scholar.google.com/scholar?q=Genta+Winata%2C+Shijie+Wu%2C+Mayank+Kulkarni%2C+Thamar+Solorio%2C+and+Daniel+Preotiuc%2DPietro%2E+2022%2E+Cross%2Dlingual+few%2Dshot+learning+on+unseen+languages%2E+In+Proceedings+of+the+2nd+Conference+of+the+Asia%2DPacifi)
69. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. [PAWS-X: A cross-lingual adversarial dataset for paraphrase identification](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin…
70. Zheng-Xin Yong and Vassilina Nikoulina. 2022. Adapting bigscience multilingual model to unseen languages. arXiv preprint arXiv:2204.04873. — [arXiv:2204.04873](https://arxiv.org/abs/2204.04873) [find](https://scholar.google.com/scholar?q=Zheng%2DXin+Yong+and+Vassilina+Nikoulina%2E+2022%2E+Adapting+bigscience+multilingual+model+to+unseen+languages%2E+arXiv+preprint+arXiv%3A2204%2E04873%2E+%E2%80%94+%5BarXiv%3A2204%2E04873%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2204%2E04873%29)
71. Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, and Todd Ward. 2020. Multi-stage pre-training for low-resource domain adaptation. In Proceedings of the 2020 Conference on Empirical Methods in Natural La… [find](https://scholar.google.com/scholar?q=Rong+Zhang%2C+Revanth+Gangi+Reddy%2C+Md+Arafat+Sultan%2C+Vittorio+Castelli%2C+Anthony+Ferritto%2C+Radu+Florian%2C+Efsun+Sarioglu+Kayi%2C+Salim+Roukos%2C+Avi+Sil%2C+and+Todd+Ward%2E+2020%2E+Multi%2Dstage+pre%2Dtraining+for+low%2D)
72. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
73. Mengjie Zhao and Hinrich Schütze. 2021. Discrete and soft prompting for multilingual models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547–8555, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mengjie+Zhao+and+Hinrich+Sch%C3%BCtze%2E+2021%2E+Discrete+and+soft+prompting+for+multilingual+models%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+8547%E2%80%938555%2C)

*Updated 24 October 2026*