---
title: "Crosslingual Generalization through — Muennighoff 2022"
description: "Multitask prompted finetuning (MTF) helps large language models handle unseen tasks zero-shot, but earlier studies stayed almost entirely within English."
authors: ["Niklas Muennighoff", "Thomas Wang", "Lintang Sutawika", "Adam Roberts", "Stella Biderman", "Teven Le Scao", "M Saiful Bari", "Sheng Shen", "Zheng-Xin Yong", "Hailey Schoelkopf", "Xiangru Tang", "Dragomir Radev", "Alham Fikri Aji", "Khalid Almubarak", "Samuel Albanie", "Zaid Alyafeai", "Albert Webson", "Edward Raff", "Colin Raffel"]
year: 2022
venue: "arXiv:2211.01786"
source: "https://arxiv.org/abs/2211.01786"
doi: "10.48550/arxiv.2211.01786"
arxiv: "2211.01786"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / Crosslingual Generalization through Multitask Finetuning

# Crosslingual Generalization through Multitask Finetuning

**Authors:** [Niklas Muennighoff](https://md.research.elamur.ai/free/en/authors/niklas-muennighoff/), [Thomas Wang](https://md.research.elamur.ai/free/en/authors/thomas-wang/), [Lintang Sutawika](https://md.research.elamur.ai/free/en/authors/lintang-sutawika/), [Adam Roberts](https://md.research.elamur.ai/free/en/authors/adam-roberts/), [Stella Biderman](https://md.research.elamur.ai/free/en/authors/stella-biderman/), [Teven Le Scao](https://md.research.elamur.ai/free/en/authors/teven-le-scao/), [M Saiful Bari](https://md.research.elamur.ai/free/en/authors/m-saiful-bari/), [Sheng Shen](https://md.research.elamur.ai/free/en/authors/sheng-shen/), [Zheng-Xin Yong](https://md.research.elamur.ai/free/en/authors/zheng-xin-yong/), [Hailey Schoelkopf](https://md.research.elamur.ai/free/en/authors/hailey-schoelkopf/), [Xiangru Tang](https://md.research.elamur.ai/free/en/authors/xiangru-tang/), [Dragomir Radev](https://md.research.elamur.ai/free/en/authors/dragomir-radev/) et al. (19 authors) · **Year:** 2022 · **Venue:** arXiv:2211.01786 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2211.01786) · [PDF](https://arxiv.org/pdf/2211.01786) · [DOI 10.48550/arxiv.2211.01786](https://doi.org/10.48550/arxiv.2211.01786) · [arXiv 2211.01786](https://arxiv.org/abs/2211.01786) · [OpenAlex](https://openalex.org/W4308244210)

**Citations:** 28 citations (openalex-snapshot · 2026-10-14) · [664](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Multitask prompted finetuning (MTF) helps large language models handle unseen tasks zero-shot, but earlier studies stayed almost entirely within English. This paper tests whether the approach carries over to multilingual settings. The authors finetune the BLOOM and mT5 model families, producing BLOOMZ and mT0, on three mixtures. The first is the English-only P3. The second is the new xP3, which pairs supervised datasets in 46 languages with English prompts and adds task types such as translation and program synthesis. The third is xP3mt, whose prompts are machine-translated into each dataset's language. Training on English data alone already improves held-out tasks in other languages seen during pretraining. xP3 improves results further and yields several state-of-the-art zero-shot scores. Machine-translated prompts help when users write prompts in non-English languages, though English-prompt performance drops somewhat. The models even solve tasks in languages they were never deliberately trained on, which suggests they acquire skills that do not depend on a specific task or language. All data and models are openly released, giving a practical starting point for zero-shot NLP in low-resource languages.

## Key points
- **Training mixtures:** P3 (English data and prompts), xP3 (46 languages, English prompts) and xP3mt (prompts machine-translated into each dataset's language). A larger follow-up, xP3x, covers 277 languages and is about ten times the size of xP3, but no models had been finetuned on it yet.
- **Scale:** Models range from 560M to 176B parameters. The advantage of finetuned models over pretrained-only models grows with size.
- **Short-answer bias:** Finetuning on billions of short targets pushes models toward brief outputs, which hurts generative tasks. The authors' workaround is to enforce a minimum generation length at inference.
- **Limitations:**
  - xP3 is restricted to BLOOM's languages, although the mT5 corpus spans more than 101.
  - mT0 scored 0.00 on HumanEval, a language (code) it saw only during finetuning.
  - BLOOMZ is sensitive to prompt formatting because inputs and targets are separated by a space. Adding a trailing full stop or newline helps.
- **Resources:** Code, datasets and models are released at github.com/bigscience-workshop/xmtf.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

664 papers of our library cite it (of the 140884 whose bibliographies we read); 61 of them have a page here, 603 not yet.

- [UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu](https://md.research.elamur.ai/free/en/papers/adeeba-2026-urblimp-benchmark-evaluating-linguistic-competence/)
- [Corpora Generation for Urdu Grammatical Error Correction](https://md.research.elamur.ai/free/en/papers/ahad-2026-corpora-generation-urdu-grammatical-error/)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/)
- [MEGA: Multilingual Evaluation of Generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/)
- [Investigating Cultural Alignment of Large Language Models](https://md.research.elamur.ai/free/en/papers/alkhamissi-2024-investigating-cultural-alignment-large-language/)
- [Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models](https://md.research.elamur.ai/free/en/papers/alyafeai-2023-taqyim-evaluating-arabic-nlp-tasks/)
- [Jais 2: A Family of Arabic-Centric Open Large Language Models](https://md.research.elamur.ai/free/en/papers/anwar-2026-jais-2-family-arabic-centric/)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://md.research.elamur.ai/free/en/papers/aryabumi-2024-aya-23-open-weight-releases/)
- [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
- [ALLaM: Large Language Models for Arabic and English](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/)
- [Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants](https://md.research.elamur.ai/free/en/papers/bhatti-2026-beyond-mcq-open-ended-arabic/)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/)
- [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://md.research.elamur.ai/free/en/papers/dang-2024-aya-expanse-combining-research-breakthroughs/)
- [Multilingual and Explainable Text Detoxification with Parallel Corpora](https://md.research.elamur.ai/free/en/papers/dementieva-2025-multilingual-explainable-text-detoxification-parallel/)
- [On the importance of Data Scale in Pretraining Arabic Language Models](https://md.research.elamur.ai/free/en/papers/ghaddar-2024-importance-data-scale-pretraining-arabic/)
- [How well can LLMs Grade Essays in Arabic?](https://md.research.elamur.ai/free/en/papers/ghazawi-2025-well-llms-grade-essays-arabic/)
- [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/)
- [Arabic Multimodal Machine Learning: Datasets, Applications, Approaches, and Challenges](https://md.research.elamur.ai/free/en/papers/haouhat-2025-arabic-multimodal-machine-learning-datasets/)
- [Bridging the Resource Gap: Exploring the Efficacy of English and Multilingual LLMs for Swedish](https://md.research.elamur.ai/free/en/papers/holmstrom-2023-bridging-resource-gap-exploring-efficacy/)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/)
- [AceGPT, Localizing Large Language Models in Arabic](https://md.research.elamur.ai/free/en/papers/huang-2024-acegpt-localizing-large-language-models/)
- [Towards Effective Disambiguation for Machine Translation with Large Language Models](https://md.research.elamur.ai/free/en/papers/iyer-2023-effective-disambiguation-machine-translation-large/)
- [TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties](https://md.research.elamur.ai/free/en/papers/kadaoui-2023-tarjamat-evaluation-bard-chatgpt-machine/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [ALADAN at IWSLT24 Low-resource Arabic Dialectal Speech Translation Task](https://md.research.elamur.ai/free/en/papers/kheder-2024-aladan-iwslt24-low-resource-arabic/)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://md.research.elamur.ai/free/en/papers/lai-2023-okapi-instruction-tuned-large-language/)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://md.research.elamur.ai/free/en/papers/laskar-2023-systematic-study-comprehensive-evaluation-chatgpt/)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/)
- [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
- [SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages](https://md.research.elamur.ai/free/en/papers/liu-2026-siniticmterror-machine-translation-dataset-error/)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/)
- [Adaptive Machine Translation with Large Language Models](https://md.research.elamur.ai/free/en/papers/moslem-2023-adaptive-machine-translation-large-language/)
- [MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/)
- [Generative Representational Instruction Tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/)
- [AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects](https://md.research.elamur.ai/free/en/papers/mustapha-2024-arastem-native-arabic-multiple-choice/)
- [Sentiment Analysis and Language Models for Kwanyama](https://md.research.elamur.ai/free/en/papers/nakashole-2026-sentiment-analysis-language-models-kwanyama/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [Data-Centric Approach at the LoResMT 2026 Turkic Translation Challenge: Russian-Kyrgyz](https://md.research.elamur.ai/free/en/papers/novokshanov-2026-data-centric-approach-loresmt-2026/)
- [HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models](https://md.research.elamur.ai/free/en/papers/oepen-2026-hplt-3-0-very-large/)
- [SynthLLM: An LLM-based Scalable Synthetic Data Generation Pipeline for Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/panahi-2026-synthllm-llm-based-scalable-synthetic/)
- [CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks](https://md.research.elamur.ai/free/en/papers/qian-2024-cameleval-advancing-culturally-aligned-arabic/)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://md.research.elamur.ai/free/en/papers/qin-2023-cross-lingual-prompting-improving-zero/)
- [Can Linguistically Related Languages Guide LLM Translation in Low-Resource Settings?](https://md.research.elamur.ai/free/en/papers/ramasethu-2026-can-linguistically-related-languages-guide/)
- [Normalizing without Modernizing: Keeping Historical Wordforms of Middle French while Reducing Spelling Variants](https://md.research.elamur.ai/free/en/papers/rubino-2024-normalizing-without-modernizing-keeping-historical/)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://md.research.elamur.ai/free/en/papers/sainz-2023-gollie-annotation-guidelines-improve-zero/)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/singh-2024-aya-dataset-open-access-collection/)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://md.research.elamur.ai/free/en/papers/sun-2023-aligning-large-multimodal-models-factually/)
- [When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/tan-2026-when-flores-bloomz-wrong-cross/)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/)
- [Multilingual E5 Text Embeddings: A Technical Report](https://md.research.elamur.ai/free/en/papers/wang-2024-multilingual-e5-text-embeddings-technical/)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/)
- [Instruction Tuning for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://md.research.elamur.ai/free/en/papers/zhang-2024-mgte-generalized-long-context-text/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{muennighoff2022crosslingual,
  title = {{Crosslingual Generalization through Multitask Finetuning}},
  author = {Muennighoff, Niklas and Wang, Thomas and Sutawika, Lintang and Roberts, Adam and Biderman, Stella and Scao, Teven Le and Bari, M Saiful and Shen, Sheng and Yong, Zheng-Xin and Schoelkopf, Hailey and Tang, Xiangru and Radev, Dragomir and Aji, Alham Fikri and Almubarak, Khalid and Albanie, Samuel and Alyafeai, Zaid and Webson, Albert and Raff, Edward and Raffel, Colin},
  year = {2022},
  doi = {10.48550/arxiv.2211.01786},
  eprint = {2211.01786},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.01786}
}
```

- [← previous Cross-document Event Coreference Search: Task, Dataset and Modeling](https://md.research.elamur.ai/free/en/papers/eirew-2022-cross-document-event-coreference-search/)
- [next → CS-UM6P at SemEval-2022 Task 6: Transformer-based Models for Intended Sarcasm Detection in English and Arabic](https://md.research.elamur.ai/free/en/papers/mahdaouy-2022-cs-um6p-semeval-2022-task/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/)

## References

117 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2018. Neural code search evaluation dataset. page arXiv:1908.09804 [cs.SE]. 2020. Wikilingua: A new benchmark dataset for multilingual abstractive summarization. arXiv preprint arXiv:2010.03093. — [arXiv:1908.09804](https://arxiv.org/abs/1908.09804) [find](https://scholar.google.com/scholar?q=2018%2E+Neural+code+search+evaluation+dataset%2E+page+arXiv%3A1908%2E09804+%5Bcs%2ESE%5D%2E+2020%2E+Wikilingua%3A+A+new+benchmark+dataset+for+multilingual+abstractive+summarization%2E+arXiv+preprint+arXiv%3A2010%2E03093%2E+%E2%80%94+%5Bar)
2. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. Santacoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988. [find](https://scholar.google.com/scholar?q=Loubna+Ben+Allal%2C+Raymond+Li%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Christopher+Akiki%2C+Carlos+Munoz+Ferrandis%2C+Niklas+Muennighoff%2C+Mayank+Mishra%2C+Alex+Gu%2C+Manan+Dey%2C+et+al%2E+2023%2E+Santacoder%3A+don%E2%80%99t+reach+for+t) [our card](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/)
3. Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2021. Ext5: Towards extreme multi-task scaling for transfer learning. CoRR, abs/2111.10… [find](https://scholar.google.com/scholar?q=Vamsi+Aribandi%2C+Yi+Tay%2C+Tal+Schuster%2C+Jinfeng+Rao%2C+Huaixiu+Steven+Zheng%2C+Sanket+Vaibhav+Mehta%2C+Honglei+Zhuang%2C+Vinh+Q%2E+Tran%2C+Dara+Bahri%2C+Jianmo+Ni%2C+Jai+Prakash+Gupta%2C+Kai+Hui%2C+Sebastian+Ruder%2C+and+Don)
4. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. CoRR, abs/1910.11856. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Sebastian+Ruder%2C+and+Dani+Yogatama%2E+2019%2E+On+the+cross%2Dlingual+transferability+of+monolingual+representations%2E+CoRR%2C+abs%2F1910%2E11856%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
5. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. [find](https://scholar.google.com/scholar?q=Jacob+Austin%2C+Augustus+Odena%2C+Maxwell+Nye%2C+Maarten+Bosma%2C+Henryk+Michalewski%2C+David+Dohan%2C+Ellen+Jiang%2C+Carrie+Cai%2C+Michael+Terry%2C+Quoc+Le%2C+et+al%2E+2021%2E+Program+synthesis+with+large+language+models%2E+a) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
6. Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaib… [find](https://scholar.google.com/scholar?q=Stephen+H%2E+Bach%2C+Victor+Sanh%2C+Zheng%2DXin+Yong%2C+Albert+Webson%2C+Colin+Raffel%2C+Nihal+V%2E+Nayak%2C+Abheesht+Sharma%2C+Taewoon+Kim%2C+M+Saiful+Bari%2C+Thibault+Fevry%2C+Zaid+Alyafeai%2C+Manan+Dey%2C+Andrea+Santilli%2C+Zhiqi) [our card](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/)
7. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373. [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+2023%2E+Pythia%3A+A+suite)
8. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Stella+Biderman%2C+Eric+Hallahan%2C+Quentin+Anthony%2C+Leo+Gao%2C+Laurence+Golding%2C+Horace+He%2C+Connor+Leahy%2C+Kyle+McDonell%2C+Jason+Phang%2C+et+al%2E+2022%2E+Gpt%2Dneox%2D20b%3A+An+open%2Dsource+autoregressive+lan) [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
9. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata, 58. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Leo+Gao%2C+Phil+Wang%2C+Connor+Leahy%2C+and+Stella+Biderman%2E+2021%2E+Gpt%2Dneo%3A+Large+scale+autoregressive+language+modeling+with+mesh%2Dtensorflow%2E+If+you+use+this+software%2C+please+cite+it+using+these)
10. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. [find](https://scholar.google.com/scholar?q=Rishi+Bommasani%2C+Drew+A+Hudson%2C+Ehsan+Adeli%2C+Russ+Altman%2C+Simran+Arora%2C+Sydney+von+Arx%2C+Michael+S+Bernstein%2C+Jeannette+Bohg%2C+Antoine+Bosselut%2C+Emma+Brunskill%2C+et+al%2E+2021%2E+On+the+opportunities+and+ris) [our card](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
12. Samuel Cahyawijaya, Holy Lovenia, Tiezheng Yu, Willy Chung, and Pascale Fung. 2023. Instructalign: Teaching novel languages with to llms through alignment-based cross-lingual instruction. arXiv preprint arXiv:2305.13627. [find](https://scholar.google.com/scholar?q=Samuel+Cahyawijaya%2C+Holy+Lovenia%2C+Tiezheng+Yu%2C+Willy+Chung%2C+and+Pascale+Fung%2E+2023%2E+Instructalign%3A+Teaching+novel+languages+with+to+llms+through+alignment%2Dbased+cross%2Dlingual+instruction%2E+arXiv+prepri)
13. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. Multieurlex–a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. arXiv preprint arXiv:2109.00904. [find](https://scholar.google.com/scholar?q=Ilias+Chalkidis%2C+Manos+Fergadiotis%2C+and+Ion+Androutsopoulos%2E+2021%2E+Multieurlex%E2%80%93a+multi%2Dlingual+and+multi%2Dlabel+legal+document+classification+dataset+for+zero%2Dshot+cross%2Dlingual+transfer%2E+arXiv+preprin) [our card](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-multieurlex-multi-lingual-multi-label/)
14. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde+de+Oliveira+Pinto%2C+Jared+Kaplan%2C+Harri+Edwards%2C+Yuri+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+et+al%2E+2021%2E+Evaluating+large+language+mode) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
15. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+et+al%2E+2022%2E+Palm%3A+Scaling+language+mode)
16. Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. 2023. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. arXiv preprint arXiv:2304.09151. — [arXiv:2304.09151](https://arxiv.org/abs/2304.09151) [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Noah+Constant%2C+Xavier+Garcia%2C+Adam+Roberts%2C+Yi+Tay%2C+Sharan+Narang%2C+and+Orhan+Firat%2E+2023%2E+Unimax%3A+Fairer+and+more+effective+language+sampling+for+large%2Dscale+multilingual+pretraining%2E)
17. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, A… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+Xiny)
18. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Unsupervised+cross%2Dlingu)
19. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computation… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+R%2E+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+)
20. Yiming Cui, Ting Liu, Li Xiao, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, and Guoping Hu. 2018. A span-extraction dataset for chinese machine reading comprehension. arXiv preprint arXiv:1810.07366. [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Ting+Liu%2C+Li+Xiao%2C+Zhipeng+Chen%2C+Wentao+Ma%2C+Wanxiang+Che%2C+Shijin+Wang%2C+and+Guoping+Hu%2E+2018%2E+A+span%2Dextraction+dataset+for+chinese+machine+reading+comprehension%2E+arXiv+preprint+arXiv%3A1810%2E) [our card](https://md.research.elamur.ai/free/en/papers/cui-2018-span-extraction-dataset-chinese/)
21. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+2023%2E+Qlora%3A+Efficient+finetuning+of+quantized+llms%2E+arXiv+preprint+arXiv%3A2305%2E14314%2E)
22. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
23. Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, et al. 2021. Nl-augmenter: A framework for task-sensitive natural language augmentation. arXiv preprint arXiv:2112.02721. [find](https://scholar.google.com/scholar?q=Kaustubh+D+Dhole%2C+Varun+Gangal%2C+Sebastian+Gehrmann%2C+Aadesh+Gupta%2C+Zhenhao+Li%2C+Saad+Mahamood%2C+Abinaya+Mahendiran%2C+Simon+Mille%2C+Ashish+Srivastava%2C+Samson+Tan%2C+et+al%2E+2021%2E+Nl%2Daugmenter%3A+A+framework+for+) [our card](https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/)
24. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06904. — [arXiv:2203.06904](https://arxiv.org/abs/2203.06904) [find](https://scholar.google.com/scholar?q=Ning+Ding%2C+Yujia+Qin%2C+Guang+Yang%2C+Fuchao+Wei%2C+Zonghan+Yang%2C+Yusheng+Su%2C+Shengding+Hu%2C+Yulin+Chen%2C+Chi%2DMin+Chan%2C+Weize+Chen%2C+et+al%2E+2022%2E+Delta+tuning%3A+A+comprehensive+study+of+parameter+efficient+meth)
25. Philipp Ennen, Po-Chun Hsu, Chan-Jan Hsu, Chang-Le Liu, Yen-Chen Wu, Yin-Hsiang Liao, Chin-Tung Lin, Da-Shan Shiu, and Wei-Yun Ma. 2023. Extending the pre-training of bloom for improved support of traditional chinese: Models, methods and results. arXiv preprint arXiv:2303.04715. — [arXiv:2303.04715](https://arxiv.org/abs/2303.04715) [find](https://scholar.google.com/scholar?q=Philipp+Ennen%2C+Po%2DChun+Hsu%2C+Chan%2DJan+Hsu%2C+Chang%2DLe+Liu%2C+Yen%2DChen+Wu%2C+Yin%2DHsiang+Liao%2C+Chin%2DTung+Lin%2C+Da%2DShan+Shiu%2C+and+Wei%2DYun+Ma%2E+2023%2E+Extending+the+pre%2Dtraining+of+bloom+for+improved+support+of+tra)
26. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22(107):1–48. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+et+al%2E+2021%2E+Beyond+english%2Dcentric+multilingu)
27. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Armen+Aghajanyan%2C+Jessy+Lin%2C+Sida+Wang%2C+Eric+Wallace%2C+Freda+Shi%2C+Ruiqi+Zhong%2C+Wen%2Dtau+Yih%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2022%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synt)
28. Jason Alan Fries, Leon Weber, Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Myungsun Kang, Ruisi Su, Wojciech Kusa, Samuel Cahyawijaya, et al. 2022. Bigbio: A framework for data-centric biomedical natural language processing. arXiv preprint arXiv:2206.15076. — [arXiv:2206.15076](https://arxiv.org/abs/2206.15076) [find](https://scholar.google.com/scholar?q=Jason+Alan+Fries%2C+Leon+Weber%2C+Natasha+Seelam%2C+Gabriel+Altay%2C+Debajyoti+Datta%2C+Samuele+Garda%2C+Myungsun+Kang%2C+Ruisi+Su%2C+Wojciech+Kusa%2C+Samuel+Cahyawijaya%2C+et+al%2E+2022%2E+Bigbio%3A+A+framework+for+data%2Dcentr)
29. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
30. Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021a. Larger-scale transformers for multilingual masked language modeling. arXiv preprint arXiv:2105.00572. [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Jingfei+Du%2C+Myle+Ott%2C+Giri+Anantharaman%2C+and+Alexis+Conneau%2E+2021a%2E+Larger%2Dscale+transformers+for+multilingual+masked+language+modeling%2E+arXiv+preprint+arXiv%3A2105%2E00572%2E) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2021-larger-scale-transformers-multilingual-masked/)
31. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzm’an, and Angela Fan. 2021b. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzm%E2%80%99an%2C+and+Angela+Fan%2E+2021b%2E+The+flores%2D101+evaluation+benchma)
32. Francisco Guzm’an, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. Two new evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. [find](https://scholar.google.com/scholar?q=Francisco+Guzm%E2%80%99an%2C+Peng%2DJen+Chen%2C+Myle+Ott%2C+Juan+Pino%2C+Guillaume+Lample%2C+Philipp+Koehn%2C+Vishrav+Chaudhary%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2019%2E+Two+new+evaluation+datasets+for+low%2Dresource+machine+translat)
33. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pa… [find](https://scholar.google.com/scholar?q=Tahmid+Hasan%2C+Abhik+Bhattacharjee%2C+Md%2E+Saiful+Islam%2C+Kazi+Mubasshir%2C+Yuan%2DFang+Li%2C+Yong%2DBin+Kang%2C+M%2E+Sohel+Rahman%2C+and+Rifat+Shahriyar%2E+2021%2E+XL%2Dsum%3A+Large%2Dscale+multilingual+abstractive+summarization)
34. Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2020. Global relational models of source code. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Vincent+J%2E+Hellendoorn%2C+Charles+Sutton%2C+Rishabh+Singh%2C+Petros+Maniatis%2C+and+David+Bieber%2E+2020%2E+Global+relational+models+of+source+code%2E+In+8th+International+Conference+on+Learning+Representations%2C+IC)
35. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with apps. NeurIPS. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Steven+Basart%2C+Saurav+Kadavath%2C+Mantas+Mazeika%2C+Akul+Arora%2C+Ethan+Guo%2C+Collin+Burns%2C+Samir+Puranik%2C+Horace+He%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+2021%2E+Measuring+coding+challenge+competen) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/)
36. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann%2C+Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Elena+Buchatskaya%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Diego+de+Las+Casas%2C+Lisa+Anne+Hendricks%2C+Johannes+Welbl%2C+Aidan+Clark%2C+et+al%2E+2022%2E+Training+compute)
37. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689. [find](https://scholar.google.com/scholar?q=Or+Honovich%2C+Thomas+Scialom%2C+Omer+Levy%2C+and+Timo+Schick%2E+2022%2E+Unnatural+instructions%3A+Tuning+language+models+with+%28almost%29+no+human+labor%2E+arXiv+preprint+arXiv%3A2212%2E09689%2E) [our card](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/)
38. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. [find](https://scholar.google.com/scholar?q=Edward+J+Hu%2C+Yelong+Shen%2C+Phillip+Wallis%2C+Zeyuan+Allen%2DZhu%2C+Yuanzhi+Li%2C+Shean+Wang%2C+Lu+Wang%2C+and+Weizhu+Chen%2E+2021%2E+Lora%3A+Low%2Drank+adaptation+of+large+language+models%2E+arXiv+preprint+arXiv%3A2106%2E09685%2E)
39. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017. [find](https://scholar.google.com/scholar?q=Srinivasan+Iyer%2C+Xi+Victoria+Lin%2C+Ramakanth+Pasunuru%2C+Todor+Mihaylov%2C+D%C3%A1niel+Simig%2C+Ping+Yu%2C+Kurt+Shuster%2C+Tianlu+Wang%2C+Qing+Liu%2C+Punit+Singh+Koura%2C+et+al%2E+2022%2E+Opt%2Diml%3A+Scaling+language+model+instru) [our card](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/)
40. Joongwon Kim, Mounica Maddela, Reno Kriz, Wei Xu, and Chris Callison-Burch. 2021. BiSECT: Learning to split and rephrase sentences with bitexts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6193– 6209, Online and Punta Cana, Dominican Republic. Ass… [find](https://scholar.google.com/scholar?q=Joongwon+Kim%2C+Mounica+Maddela%2C+Reno+Kriz%2C+Wei+Xu%2C+and+Chris+Callison%2DBurch%2E+2021%2E+BiSECT%3A+Learning+to+split+and+rephrase+sentences+with+bitexts%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Meth)
41. Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2022. Sparse upcycling: Training mixture-of-experts from dense checkpoints. arXiv preprint arXiv:2212.05055. [find](https://scholar.google.com/scholar?q=Aran+Komatsuzaki%2C+Joan+Puigcerver%2C+James+Lee%2DThorp%2C+Carlos+Riquelme+Ruiz%2C+Basil+Mustafa%2C+Joshua+Ainslie%2C+Yi+Tay%2C+Mostafa+Dehghani%2C+and+Neil+Houlsby%2E+2022%2E+Sparse+upcycling%3A+Training+mixture%2Dof%2Dexperts)
42. Matej Kosec, Sheng Fu, and Mario Michael Krell. 2021. Packing: Towards 2x nlp bert acceleration. arXiv preprint arXiv:2107.02027. — [arXiv:2107.02027](https://arxiv.org/abs/2107.02027) [find](https://scholar.google.com/scholar?q=Matej+Kosec%2C+Sheng+Fu%2C+and+Mario+Michael+Krell%2E+2021%2E+Packing%3A+Towards+2x+nlp+bert+acceleration%2E+arXiv+preprint+arXiv%3A2107%2E02027%2E+%E2%80%94+%5BarXiv%3A2107%2E02027%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2107%2E02027%29)
43. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E)
44. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. In Thirty-sixth Conference on Neura… [find](https://scholar.google.com/scholar?q=Hugo+Lauren%C3%A7on%2C+Lucile+Saulnier%2C+Thomas+Wang%2C+Christopher+Akiki%2C+Albert+Villanova+del+Moral%2C+Teven+Le+Scao%2C+Leandro+Von+Werra%2C+Chenghao+Mou%2C+Eduardo+Gonz%C3%A1lez+Ponferrada%2C+Huu+Nguyen%2C+et+al%2E+2022%2E+The+b)
45. Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707–710. Soviet Union. [find](https://scholar.google.com/scholar?q=Vladimir+I+Levenshtein+et+al%2E+1966%2E+Binary+codes+capable+of+correcting+deletions%2C+insertions%2C+and+reversals%2E+In+Soviet+physics+doklady%2C+volume+10%2C+pages+707%E2%80%93710%2E+Soviet+Union%2E)
46. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. Mlqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Barlas+Oguz%2C+Ruty+Rinott%2C+Sebastian+Riedel%2C+and+Holger+Schwenk%2E+2019%2E+Mlqa%3A+Evaluating+cross%2Dlingual+extractive+question+answering%2E+arXiv+preprint+arXiv%3A1910%2E07475%2E) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/)
47. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+et+al%2E+2023%2E+Starcoder%3A+may+the+source+be+with+you%21+arX)
48. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668. [find](https://scholar.google.com/scholar?q=Xi+Victoria+Lin%2C+Todor+Mihaylov%2C+Mikel+Artetxe%2C+Tianlu+Wang%2C+Shuohui+Chen%2C+Daniel+Simig%2C+Myle+Ott%2C+Naman+Goyal%2C+Shruti+Bhosale%2C+Jingfei+Du%2C+et+al%2E+2021%2E+Few%2Dshot+learning+with+multilingual+language+mo) [our card](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
49. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. arXiv preprint arXiv:2205.05638. [find](https://scholar.google.com/scholar?q=Haokun+Liu%2C+Derek+Tam%2C+Mohammed+Muqeeth%2C+Jay+Mohta%2C+Tenghao+Huang%2C+Mohit+Bansal%2C+and+Colin+Raffel%2E+2022%2E+Few%2Dshot+parameter%2Defficient+fine%2Dtuning+is+better+and+cheaper+than+in%2Dcontext+learning%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/)
50. Qian Liu, Fan Zhou, Zhengbao Jiang, Longxu Dou, and Min Lin. 2023. From zero to hero: Examining the power of symbolic tasks in instruction tuning. arXiv preprint arXiv:2304.07995. [find](https://scholar.google.com/scholar?q=Qian+Liu%2C+Fan+Zhou%2C+Zhengbao+Jiang%2C+Longxu+Dou%2C+and+Min+Lin%2E+2023%2E+From+zero+to+hero%3A+Examining+the+power+of+symbolic+tasks+in+instruction+tuning%2E+arXiv+preprint+arXiv%3A2304%2E07995%2E)
51. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactio)
52. Robert L Logan, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2106.13353. [find](https://scholar.google.com/scholar?q=Robert+L+Logan%2C+Ivana+Bala%C5%BEevic%CC%81%2C+Eric+Wallace%2C+Fabio+Petroni%2C+Sameer+Singh%2C+and+Sebastian+Riedel%2E+2021%2E+Cutting+down+on+prompts+and+parameters%3A+Simple+few%2Dshot+learning+with+language+models%2E+arXiv+pr) [our card](https://md.research.elamur.ai/free/en/papers/iv-2022-cutting-down-prompts-parameters-simple/)
53. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023a. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688. [find](https://scholar.google.com/scholar?q=Shayne+Longpre%2C+Le+Hou%2C+Tu+Vu%2C+Albert+Webson%2C+Hyung+Won+Chung%2C+Yi+Tay%2C+Denny+Zhou%2C+Quoc+V+Le%2C+Barret+Zoph%2C+Jason+Wei%2C+et+al%2E+2023a%2E+The+flan+collection%3A+Designing+data+and+methods+for+effective+instru) [our card](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/)
54. Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. 2023b. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv:2305.13169. [find](https://scholar.google.com/scholar?q=Shayne+Longpre%2C+Gregory+Yauney%2C+Emily+Reif%2C+Katherine+Lee%2C+Adam+Roberts%2C+Barret+Zoph%2C+Denny+Zhou%2C+Jason+Wei%2C+Kevin+Robinson%2C+David+Mimno%2C+et+al%2E+2023b%2E+A+pretrainer%E2%80%99s+guide+to+training+data%3A+Measuring)
55. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+2021%2E+Metaicl%3A+Learning+to+learn+in+context%2E+arXiv+preprint+arXiv%3A2110%2E15943%2E)
56. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021a. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+2021a%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+arXiv+preprint+arXiv%3A2104%2E08773%2E)
57. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021b. Natural instructions: Benchmarking generalization to new tasks from natural language instructions. CoRR, abs/2104.08773. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+2021b%2E+Natural+instructions%3A+Benchmarking+generalization+to+new+tasks+from+natural+language+instructions%2E+CoRR%2C+abs%2F2104%2E08773%2E)
58. Niklas Muennighoff. 2022. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2E+2022%2E+Sgpt%3A+Gpt+sentence+embeddings+for+semantic+search%2E+arXiv+preprint+arXiv%3A2202%2E08904%2E) [our card](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-sgpt-gpt-sentence-embeddings-semantic/)
59. Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling data-constrained language models. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Alexander+M%2E+Rush%2C+Boaz+Barak%2C+Teven+Le+Scao%2C+Aleksandra+Piktus%2C+Nouamane+Tazi%2C+Sampo+Pyysalo%2C+Thomas+Wolf%2C+and+Colin+Raffel%2E+2023%2E+Scaling+data%2Dconstrained+language+models%2E)
60. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Nouamane+Tazi%2C+Lo%C3%AFc+Magne%2C+and+Nils+Reimers%2E+2022%2E+Mteb%3A+Massive+text+embedding+benchmark%2E+arXiv+preprint+arXiv%3A2210%2E07316%2E) [our card](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/)
61. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… [find](https://scholar.google.com/scholar?q=NLLB+Team%2C+Marta+R%2E+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+Anna+Sun%2C+Skyler+Wang%2C+Guillaume+We)
62. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeff+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+L+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+2022%2E+Training+language+models+to+follow+instructio)
63. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+annual+meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
64. Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison-Burch. 2022. Bidirectional language models are also few-shot learners. arXiv preprint arXiv:2209.14500. — [arXiv:2209.14500](https://arxiv.org/abs/2209.14500) [find](https://scholar.google.com/scholar?q=Ajay+Patel%2C+Bryan+Li%2C+Mohammad+Sadegh+Rasooli%2C+Noah+Constant%2C+Colin+Raffel%2C+and+Chris+Callison%2DBurch%2E+2022%2E+Bidirectional+language+models+are+also+few%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2209%2E14500%2E+%E2%80%94)
65. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021a. True few-shot learning with language models. Advances in Neural Information Processing Systems, 34:11054–11070. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Douwe+Kiela%2C+and+Kyunghyun+Cho%2E+2021a%2E+True+few%2Dshot+learning+with+language+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A11054%E2%80%9311070%2E) [our card](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)
66. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021b. True few-shot learning with language models. CoRR, abs/2105.11447. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Douwe+Kiela%2C+and+Kyunghyun+Cho%2E+2021b%2E+True+few%2Dshot+learning+with+language+models%2E+CoRR%2C+abs%2F2105%2E11447%2E) [our card](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)
67. Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, and Samuel R Bowman. 2020. English intermediate-task training improves zero-shot cross-lingual transfer too. arXiv preprint arXiv:2005.13013. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Iacer+Calixto%2C+Phu+Mon+Htut%2C+Yada+Pruksachatkun%2C+Haokun+Liu%2C+Clara+Vania%2C+Katharina+Kann%2C+and+Samuel+R+Bowman%2E+2020%2E+English+intermediate%2Dtask+training+improves+zero%2Dshot+cross%2Dlingual+tr)
68. Edoardo M. Ponti, Goran Glavas, Olga Majewska, Qianchu Liu, Ivan Vuli’c, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. arXiv preprint. [find](https://scholar.google.com/scholar?q=Edoardo+M%2E+Ponti%2C+Goran+Glavas%2C+Olga+Majewska%2C+Qianchu+Liu%2C+Ivan+Vuli%E2%80%99c%2C+and+Anna+Korhonen%2E+2020%2E+XCOPA%3A+A+multilingual+dataset+for+causal+commonsense+reasoning%2E+arXiv+preprint%2E) [our card](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/)
69. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
70. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+2021%2E+Scaling+language+models%3A+Methods%2C+a)
71. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+Peter+J+Liu%2C+et+al%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dt)
72. Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, and Mohammad Taher Pilehvar. 2020. Xl-wic: A multilingual benchmark for evaluating semantic contextualization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7193–7206. [find](https://scholar.google.com/scholar?q=Alessandro+Raganato%2C+Tommaso+Pasini%2C+Jose+Camacho%2DCollados%2C+and+Mohammad+Taher+Pilehvar%2E+2020%2E+Xl%2Dwic%3A+A+multilingual+benchmark+for+evaluating+semantic+contextualization%2E+In+Proceedings+of+the+2020+Co)
73. Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashch… [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Hyung+Won+Chung%2C+Anselm+Levskaya%2C+Gaurav+Mishra%2C+James+Bradbury%2C+Daniel+Andor%2C+Sharan+Narang%2C+Brian+Lester%2C+Colin+Gaffney%2C+Afroz+Mohiuddin%2C+Curtis+Hawthorne%2C+Aitor+Lewkowycz%2C+Alex+Salcia)
74. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+2011+AAAI+Spring+Symposium+Series%2E)
75. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Albert+Webson%2C+Colin+Raffel%2C+Stephen+Bach%2C+Lintang+Sutawika%2C+Zaid+Alyafeai%2C+Antoine+Chaffin%2C+Arnaud+Stiegler%2C+Teven+Le+Scao%2C+Arun+Raja%2C+et+al%2E+2022%2E+Multitask+prompted+training+enables+ze) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
76. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022a. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022a%2E+Bloom%3A+A+176b%2Dparamete)
77. Teven Le Scao and Alexander M Rush. 2021. How many data points is a prompt worth? arXiv preprint arXiv:2103.08493. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao+and+Alexander+M+Rush%2E+2021%2E+How+many+data+points+is+a+prompt+worth%3F+arXiv+preprint+arXiv%3A2103%2E08493%2E) [our card](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/)
78. Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. 2022b. What language model to train if you have one million gpu hours? arXiv preprint arXiv:2210.15424. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Thomas+Wang%2C+Daniel+Hesslow%2C+Lucile+Saulnier%2C+Stas+Bekman%2C+M+Saiful+Bari%2C+Stella+Bideman%2C+Hady+Elsahar%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+et+al%2E+2022b%2E+What+language+model+to+train+if+yo)
79. Timo Schick and Hinrich Schütze. 2020. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+2020%2E+Exploiting+cloze+questions+for+few+shot+text+classification+and+natural+language+inference%2E+arXiv+preprint+arXiv%3A2001%2E07676%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)
80. Timo Schick and Hinrich Schütze. 2020. Exploiting cloze questions for few-shot text classification and natural language inference. CoRR, abs/2001.07676. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+2020%2E+Exploiting+cloze+questions+for+few%2Dshot+text+classification+and+natural+language+inference%2E+CoRR%2C+abs%2F2001%2E07676%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)
81. Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. Continual-t0: Progressively instructing 50+ tasks to language models without forgetting. arXiv preprint arXiv:2205.12393. [find](https://scholar.google.com/scholar?q=Thomas+Scialom%2C+Tuhin+Chakrabarty%2C+and+Smaranda+Muresan%2E+2022%2E+Continual%2Dt0%3A+Progressively+instructing+50%2B+tasks+to+language+models+without+forgetting%2E+arXiv+preprint+arXiv%3A2205%2E12393%2E)
82. Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, et al. 2023. Flan-moe: Scaling instruction-finetuned language models with sparse mixture of experts. arXiv preprint arXiv:2305.14705. [find](https://scholar.google.com/scholar?q=Sheng+Shen%2C+Le+Hou%2C+Yanqi+Zhou%2C+Nan+Du%2C+Shayne+Longpre%2C+Jason+Wei%2C+Hyung+Won+Chung%2C+Barret+Zoph%2C+William+Fedus%2C+Xinyun+Chen%2C+et+al%2E+2023%2E+Flan%2Dmoe%3A+Scaling+instruction%2Dfinetuned+language+models+with+s)
83. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. mgpt: Few-shot learners go multilingual. arXiv preprint arXiv:2204.07580. [find](https://scholar.google.com/scholar?q=Oleh+Shliazhko%2C+Alena+Fenogenova%2C+Maria+Tikhonova%2C+Vladislav+Mikhailov%2C+Anastasia+Kozlova%2C+and+Tatiana+Shavrina%2E+2022%2E+mgpt%3A+Few%2Dshot+learners+go+multilingual%2E+arXiv+preprint+arXiv%3A2204%2E07580%2E) [our card](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/)
84. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint ar… [find](https://scholar.google.com/scholar?q=Shaden+Smith%2C+Mostofa+Patwary%2C+Brandon+Norick%2C+Patrick+LeGresley%2C+Samyam+Rajbhandari%2C+Jared+Casper%2C+Zhun+Liu%2C+Shrimai+Prabhumoye%2C+George+Zerveas%2C+Vijay+Korthikanti%2C+et+al%2E+2022%2E+Using+deepspeed+and+me) [our card](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
85. Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gokhan Tur, and Prem Natarajan. 2022. Alexatm 20b: Few-shot learning us… [find](https://scholar.google.com/scholar?q=Saleh+Soltan%2C+Shankar+Ananthakrishnan%2C+Jack+FitzGerald%2C+Rahul+Gupta%2C+Wael+Hamza%2C+Haidar+Khan%2C+Charith+Peris%2C+Stephen+Rawls%2C+Andy+Rosenbaum%2C+Anna+Rumshisky%2C+Chandana+Satya+Prakash%2C+Mukund+Sridhar%2C+Fabi)
86. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615. arXiv preprint… [find](https://scholar.google.com/scholar?q=Aarohi+Srivastava%2C+Abhinav+Rastogi%2C+Abhishek+Rao%2C+Abu+Awal+Md+Shoeb%2C+Abubakar+Abid%2C+Adam+Fisch%2C+Adam+R+Brown%2C+Adam+Santoro%2C+Aditya+Gupta%2C+Adri%C3%A0+Garriga%2DAlonso%2C+et+al%2E+2022%2E+Beyond+the+imitation+game%3A+) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
87. Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2020. Investigating prior knowledge for challenging chinese machine reading comprehension. Trans. Assoc. Comput. Linguistics, 8:141–155. [find](https://scholar.google.com/scholar?q=Kai+Sun%2C+Dian+Yu%2C+Dong+Yu%2C+and+Claire+Cardie%2E+2020%2E+Investigating+prior+knowledge+for+challenging+chinese+machine+reading+comprehension%2E+Trans%2E+Assoc%2E+Comput%2E+Linguistics%2C+8%3A141%E2%80%93155%2E)
88. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261. [find](https://scholar.google.com/scholar?q=Mirac+Suzgun%2C+Nathan+Scales%2C+Nathanael+Sch%C3%A4rli%2C+Sebastian+Gehrmann%2C+Yi+Tay%2C+Hyung+Won+Chung%2C+Aakanksha+Chowdhery%2C+Quoc+V+Le%2C+Ed+H+Chi%2C+Denny+Zhou%2C+et+al%2E+2022%2E+Challenging+big%2Dbench+tasks+and+whether+) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
89. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022a. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Mostafa+Dehghani%2C+Vinh+Q+Tran%2C+Xavier+Garcia%2C+Dara+Bahri%2C+Tal+Schuster%2C+Huaixiu+Steven+Zheng%2C+Neil+Houlsby%2C+and+Donald+Metzler%2E+2022a%2E+Unifying+language+learning+paradigms%2E+arXiv+preprint+arXi) [our card](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/)
90. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. 2022b. Ul2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Mostafa+Dehghani%2C+Vinh+Q+Tran%2C+Xavier+Garcia%2C+Jason+Wei%2C+Xuezhi+Wang%2C+Hyung+Won+Chung%2C+Dara+Bahri%2C+Tal+Schuster%2C+Steven+Zheng%2C+et+al%2E+2022b%2E+Ul2%3A+Unifying+language+learning+paradigms%2E+In+The+E) [our card](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/)
91. Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. 2022c. Transcending scaling laws with 0.1% extra compute. arXiv preprint arXiv:2210.11399. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Jason+Wei%2C+Hyung+Won+Chung%2C+Vinh+Q+Tran%2C+David+R+So%2C+Siamak+Shakeri%2C+Xavier+Garcia%2C+Huaixiu+Steven+Zheng%2C+Jinfeng+Rao%2C+Aakanksha+Chowdhery%2C+et+al%2E+2022c%2E+Transcending+scaling+laws+with+0%2E1%25+ex)
92. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085. [find](https://scholar.google.com/scholar?q=Ross+Taylor%2C+Marcin+Kardas%2C+Guillem+Cucurull%2C+Thomas+Scialom%2C+Anthony+Hartshorn%2C+Elvis+Saravia%2C+Andrew+Poulton%2C+Viktor+Kerkez%2C+and+Robert+Stojnic%2E+2022%2E+Galactica%3A+A+large+language+model+for+science%2E+) [our card](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/)
93. J"org Tiedemann. 2020. The Tatoeba Translation Challenge – Realistic data sets for low resource and multilingual MT. In Proceedings of the Fifth Conference on Machine Translation, pages 1174–1182. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=J%22org+Tiedemann%2E+2020%2E+The+Tatoeba+Translation+Challenge+%E2%80%93+Realistic+data+sets+for+low+resource+and+multilingual+MT%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+pages+1174%E2%80%931182%2E+Ass) [our card](https://md.research.elamur.ai/free/en/papers/tiedemann-2020-tatoeba-translation-challenge-realistic-data/)
94. Alexey Tikhonov and Max Ryabinin. 2021. It’s all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. [find](https://scholar.google.com/scholar?q=Alexey+Tikhonov+and+Max+Ryabinin%2E+2021%2E+It%E2%80%99s+all+in+the+heads%3A+Using+attention+heads+as+a+baseline+for+cross%2Dlingual+transfer+in+commonsense+reasoning%2E)
95. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+2023%2E+Llama%3A+Open+and+efficient+f)
96. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+Advances+in+neural+information+processing+)
97. Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, and Noah Constant. 2022. Overcoming catastrophic forgetting in zero-shot cross-lingual generation. arXiv preprint arXiv:2205.12647. [find](https://scholar.google.com/scholar?q=Tu+Vu%2C+Aditya+Barua%2C+Brian+Lester%2C+Daniel+Cer%2C+Mohit+Iyyer%2C+and+Noah+Constant%2E+2022%2E+Overcoming+catastrophic+forgetting+in+zero%2Dshot+cross%2Dlingual+generation%2E+arXiv+preprint+arXiv%3A2205%2E12647%2E)
98. Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A 6 billion parameter autoregressive language model. [find](https://scholar.google.com/scholar?q=Ben+Wang+and+Aran+Komatsuzaki%2E+2021%2E+Gpt%2Dj%2D6b%3A+A+6+billion+parameter+autoregressive+language+model%2E)
99. Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022a. What language model architecture and pretraining objective work best for zero-shot generalization? arXiv preprint arXiv:2204.05832. [find](https://scholar.google.com/scholar?q=Thomas+Wang%2C+Adam+Roberts%2C+Daniel+Hesslow%2C+Teven+Le+Scao%2C+Hyung+Won+Chung%2C+Iz+Beltagy%2C+Julien+Launay%2C+and+Colin+Raffel%2E+2022a%2E+What+language+model+architecture+and+pretraining+objective+work+best+for+)
100. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022b. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022b%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+pr)
101. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Ma… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)
102. Zhenhailong Wang, Xiaoman Pan, Dian Yu, Dong Yu, Jianshu Chen, and Heng Ji. 2022d. Zemi: Learning zero-shot semi-parametric language models from multiple tasks. arXiv preprint arXiv:2210.00185. [find](https://scholar.google.com/scholar?q=Zhenhailong+Wang%2C+Xiaoman+Pan%2C+Dian+Yu%2C+Dong+Yu%2C+Jianshu+Chen%2C+and+Heng+Ji%2E+2022d%2E+Zemi%3A+Learning+zero%2Dshot+semi%2Dparametric+language+models+from+multiple+tasks%2E+arXiv+preprint+arXiv%3A2210%2E00185%2E)
103. Albert Webson and Ellie Pavlick. 2021. Do prompt-based models really understand the meaning of their prompts? [find](https://scholar.google.com/scholar?q=Albert+Webson+and+Ellie+Pavlick%2E+2021%2E+Do+prompt%2Dbased+models+really+understand+the+meaning+of+their+prompts%3F) [our card](https://md.research.elamur.ai/free/en/papers/webson-2022-do-prompt-based-models-really/)
104. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+2021%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2109)
105. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
106. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844,… [find](https://scholar.google.com/scholar?q=Shijie+Wu+and+Mark+Dredze%2E+2019%2E+Beto%2C+bentz%2C+becas%3A+The+surprising+cross%2Dlingual+effectiveness+of+BERT%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+t) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/)
107. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05986. [find](https://scholar.google.com/scholar?q=Liang+Xu%2C+Hai+Hu%2C+Xuanwei+Zhang%2C+Lu+Li%2C+Chenjie+Cao%2C+Yudong+Li%2C+Yechen+Xu%2C+Kai+Sun%2C+Dian+Yu%2C+Cong+Yu%2C+et+al%2E+2020%2E+Clue%3A+A+chinese+language+understanding+evaluation+benchmark%2E+arXiv+preprint+arXiv%3A200) [our card](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/)
108. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Noah+Constant%2C+Adam+Roberts%2C+Mihir+Kale%2C+Rami+Al%2DRfou%2C+Aditya+Siddhant%2C+Aditya+Barua%2C+and+Colin+Raffel%2E+2020%2E+mt5%3A+A+massively+multilingual+pre%2Dtrained+text%2Dto%2Dtext+transformer%2E+arXiv+pre)
109. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Yinfei+Yang%2C+Yuan+Zhang%2C+Chris+Tar%2C+and+Jason+Baldridge%2E+2019%2E+PAWS%2DX%3A+A+Cross%2Dlingual+Adversarial+Dataset+for+Paraphrase+Identification%2E+In+Proc%2E+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/)
110. Zheng-Xin Yong and Vassilina Nikoulina. 2022. Adapting bigscience multilingual model to unseen languages. arXiv preprint arXiv:2204.04873. — [arXiv:2204.04873](https://arxiv.org/abs/2204.04873) [find](https://scholar.google.com/scholar?q=Zheng%2DXin+Yong+and+Vassilina+Nikoulina%2E+2022%2E+Adapting+bigscience+multilingual+model+to+unseen+languages%2E+arXiv+preprint+arXiv%3A2204%2E04873%2E+%E2%80%94+%5BarXiv%3A2204%2E04873%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2204%2E04873%29)
111. Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, et al. 2022. Bloom+ 1: Adding language support to bloom for zero-shot prompting. arXiv preprint arXiv:2212.09535. [find](https://scholar.google.com/scholar?q=Zheng%2DXin+Yong%2C+Hailey+Schoelkopf%2C+Niklas+Muennighoff%2C+Alham+Fikri+Aji%2C+David+Ifeoluwa+Adelani%2C+Khalid+Almubarak%2C+M+Saiful+Bari%2C+Lintang+Sutawika%2C+Jungo+Kasai%2C+Ahmed+Baruwa%2C+et+al%2E+2022%2E+Bloom%2B+1%3A+Add)
112. Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199. [find](https://scholar.google.com/scholar?q=Elad+Ben+Zaken%2C+Shauli+Ravfogel%2C+and+Yoav+Goldberg%2E+2021%2E+Bitfit%3A+Simple+parameter%2Defficient+fine%2Dtuning+for+transformer%2Dbased+masked+language%2Dmodels%2E+arXiv+preprint+arXiv%3A2106%2E10199%2E) [our card](https://md.research.elamur.ai/free/en/papers/ben-2021-zaken-bitfit-simple-parameter-efficient-fine/)
113. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. [find](https://scholar.google.com/scholar?q=Aohan+Zeng%2C+Xiao+Liu%2C+Zhengxiao+Du%2C+Zihan+Wang%2C+Hanyu+Lai%2C+Ming+Ding%2C+Zhuoyi+Yang%2C+Yifan+Xu%2C+Wendi+Zheng%2C+Xiao+Xia%2C+et+al%2E+2022%2E+Glm%2D130b%3A+An+open+bilingual+pre%2Dtrained+model%2E+arXiv+preprint+arXiv%3A221) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/)
114. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. [find](https://scholar.google.com/scholar?q=Susan+Zhang%2C+Stephen+Roller%2C+Naman+Goyal%2C+Mikel+Artetxe%2C+Moya+Chen%2C+Shuohui+Chen%2C+Christopher+Dewan%2C+Mona+Diab%2C+Xian+Li%2C+Xi+Victoria+Lin%2C+et+al%2E+2022%2E+Opt%3A+Open+pre%2Dtrained+transformer+language+models)
115. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130. [find](https://scholar.google.com/scholar?q=Yuan+Zhang%2C+Jason+Baldridge%2C+and+Luheng+He%2E+2019%2E+Paws%3A+Paraphrase+adversaries+from+word+scrambling%2E+arXiv+preprint+arXiv%3A1904%2E01130%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-paws-paraphrase-adversaries-word-scrambling/)
116. Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Meta-tuning language models to answer prompts better. CoRR, abs/2104.04670. [find](https://scholar.google.com/scholar?q=Ruiqi+Zhong%2C+Kristy+Lee%2C+Zheng+Zhang%2C+and+Dan+Klein%2E+2021%2E+Meta%2Dtuning+language+models+to+answer+prompts+better%2E+CoRR%2C+abs%2F2104%2E04670%2E)
117. Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K. Reddy. 2022. Xlcost: A benchmark dataset for cross-lingual code intelligence. [find](https://scholar.google.com/scholar?q=Ming+Zhu%2C+Aneesh+Jain%2C+Karthik+Suresh%2C+Roshan+Ravindran%2C+Sindhu+Tipirneni%2C+and+Chandan+K%2E+Reddy%2E+2022%2E+Xlcost%3A+A+benchmark+dataset+for+cross%2Dlingual+code+intelligence%2E)