---
title: "mGPT: Few-Shot Learners Go Multilingual — Shliazhko 2022"
description: "Most autoregressive language models capable of few-shot learning target English or a few high-resource languages."
authors: ["Oleh Shliazhko", "Alena Fenogenova", "Maria Tikhonova", "Vladislav Mikhailov", "Anastasia Kozlova", "Tatiana Shavrina"]
year: 2022
venue: "arXiv:2204.07580"
source: "https://arxiv.org/abs/2204.07580"
doi: "10.48550/arxiv.2204.07580"
arxiv: "2204.07580"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / mGPT: Few-Shot Learners Go Multilingual

# mGPT: Few-Shot Learners Go Multilingual

**Authors:** Oleh Shliazhko, [Alena Fenogenova](https://md.research.elamur.ai/free/en/authors/alena-fenogenova/), Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, Tatiana Shavrina · **Year:** 2022 · **Venue:** arXiv:2204.07580 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2204.07580) · [PDF](https://arxiv.org/pdf/2204.07580) · [DOI 10.48550/arxiv.2204.07580](https://doi.org/10.48550/arxiv.2204.07580) · [arXiv 2204.07580](https://arxiv.org/abs/2204.07580) · [OpenAlex](https://openalex.org/W4224247062)

**Citations:** 45 citations (openalex-snapshot · 2026-10-14) · [157](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Most autoregressive language models capable of few-shot learning target English or a few high-resource languages. Many typologically distant and low-resource languages get little support. The authors address this gap with mGPT, a GPT-3-style model released in 1.3B and 13B parameter sizes. It covers 61 languages from 25 families, including Ukrainian, other CIS languages and minority languages of Russia. The training data combines Wikipedia and C4. It was cleaned by hash-based deduplication, compression-rate filtering and a quality classifier that treats Wikipedia as the positive class. Small 163M pilot models were used to compare tokenization schemes before the final design was fixed. The models were tested on per-language perplexity, cross-lingual NLU tasks, and knowledge probing. Their zero- and few-shot results roughly match XGLM while using fewer parameters and covering more languages. Hate speech detection remained near chance level. The code and weights are open under MIT. The work offers a practical generative baseline for under-served languages.

## Key points
- **Scale and coverage:** the models come in 1.3B and 13B parameter sizes and cover 61 languages from 25 families. About 20 of these are CIS or minority languages of Russia, such as Bashkir, Tatar, Yakut, Buryat and Kalmyk.
- **Corpus:** Wikipedia (dump 20201101) and C4 together total about 600GB, made up of 46B characters from Wikipedia and 442B UTF characters from C4. Texts were kept only if their zlib compression ratio fell between ×1.2 and ×8.
- **Tokenization:** five strategies were compared on mGPT 163M: DEFAULT BBPE, CASE, ARITHMETIC, COMBINED and CHAR. None of the alternatives beat DEFAULT on character-normalized perplexity.
- **Evaluation:** language modeling was assessed across all 61 languages, NLU tasks covered 33 languages, and probing of world knowledge was carried out for 23 languages. mGPT 1.3B performed comparably to XGLM 1.7B. Adding more demonstration examples sometimes lowered scores.
- **Release:** weights are available at `hf.co/ai-forever/mGPT` and `hf.co/ai-forever/mGPT-13B`, and code at `github.com/ai-forever/mgpt`. Estimated pretraining emissions are about 15.9k kg CO2 for the 1.3B model and 47.5k kg for the 13B model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

157 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 142 not yet.

- [In-context Examples Selection for Machine Translation](https://md.research.elamur.ai/free/en/papers/agrawal-2023-context-examples-selection-machine-translation/)
- [Multilingual Nonce Dependency Treebanks: Understanding how Language Models Represent and Process Syntactic Structure](https://md.research.elamur.ai/free/en/papers/arps-2024-multilingual-nonce-dependency-treebanks-understanding/)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://md.research.elamur.ai/free/en/papers/bao-2023-fast-detectgpt-efficient-zero-shot/)
- [E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets](https://md.research.elamur.ai/free/en/papers/jallad-2026-e-conan-entailment-contradition-neutral-benchmarks/)
- [E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/jallad-2026-e-conan-entailment-contradition-neutral/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages](https://md.research.elamur.ai/free/en/papers/liu-2026-siniticmterror-machine-translation-dataset-error/)
- [Russian Generative Spelling, Punctuation and Capitalization Correction](https://md.research.elamur.ai/free/en/papers/martynov-2026-russian-generative-spelling-punctuation-capitalization/)
- [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
- [Few-shot Prompting or Supervised Tuning? A Comparative Study of LLMs for Linguistically Distant Language Pairs in BDI](https://md.research.elamur.ai/free/en/papers/naorem-2026-few-shot-prompting-or-supervised/)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/)
- [Reading Time in the Wild: An Assessment of Readability Predictors Based on Naturally-Observed Reading Times](https://md.research.elamur.ai/free/en/papers/vaals-2026-reading-time-wild-assessment-readability/)
- [Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights](https://md.research.elamur.ai/free/en/papers/valero-2026-merge-conquer-instructing-multilingual-models/)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://md.research.elamur.ai/free/en/papers/wendler-2024-do-llamas-work-english-latent/)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://md.research.elamur.ai/free/en/papers/wilcox-2023-testing-predictions-surprisal-theory-11/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shliazhko2022mgpt,
  title = {{mGPT: Few-Shot Learners Go Multilingual}},
  author = {Shliazhko, Oleh and Fenogenova, Alena and Tikhonova, Maria and Mikhailov, Vladislav and Kozlova, Anastasia and Shavrina, Tatiana},
  year = {2022},
  doi = {10.48550/arxiv.2204.07580},
  eprint = {2204.07580},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2204.07580}
}
```

- [← previous Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/)
- [next → MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset](https://md.research.elamur.ai/free/en/papers/fomicheva-2022-mlqe-pe-multilingual-quality-estimation/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

87 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Emre Can Acikgoz, Tilek Chubakov, Muge Kural, Gözde Şahin, and Deniz Yuret. 2022. Transformers on Multilingual Clause-Level Morphology. In Proceedings of the The 2nd Workshop on Multi-lingual Representation Learning (MRL), pages 100–105, Abu Dhabi, United Arab Emirates (Hybrid). Association for Co… [find](https://scholar.google.com/scholar?q=Emre+Can+Acikgoz%2C+Tilek+Chubakov%2C+Muge+Kural%2C+Go%CC%88zde+S%CC%A7ahin%2C+and+Deniz+Yuret%2E+2022%2E+Transformers+on+Multilingual+Clause%2DLevel+Morphology%2E+In+Proceedings+of+the+The+2nd+Workshop+on+Multi%2Dlingual+Repres)
2. Kabir Ahuja, Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. 2022. Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5454–5467, Dublin, Irel… [find](https://scholar.google.com/scholar?q=Kabir+Ahuja%2C+Shanu+Kumar%2C+Sandipan+Dandapat%2C+and+Monojit+Choudhury%2E+2022%2E+Multi+Task+Learning+For+Zero+Shot+Performance+Prediction+of+Multilingual+Models%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+)
3. Mikhail Arkhipov, Maria Trofimova, Yuri Kuratov, and Alexey Sorokin. 2019. Tuning multilingual transformers for language-specific named entity recognition. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 89–93, Florence, Italy. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Mikhail+Arkhipov%2C+Maria+Trofimova%2C+Yuri+Kuratov%2C+and+Alexey+Sorokin%2E+2019%2E+Tuning+multilingual+transformers+for+language%2Dspecific+named+entity+recognition%2E+In+Proceedings+of+the+7th+Workshop+on+Balto%2D) [our card](https://md.research.elamur.ai/free/en/papers/arkhipov-2019-tuning-multilingual-transformers-language-specific/)
4. Giusepppe Attardi. 2015. WikiExtractor. [https://github.com/attardi/](https://github.com/attardi/) wikiextractor. [↗](https://github.com/attardi/)
5. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. [find](https://scholar.google.com/scholar?q=Emily+M+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+2021%2E+On+the+Dangers+of+Stochastic+Parrots%3A+Can+Language+Models+Be+Too+Big%3F+In+Proceedings+of+the+2021+ACM+Conference+)
6. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In International Conferenc… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+2023%2E+Pythia%3A)
7. Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX-20B: An Open-Source Autor… [find](https://scholar.google.com/scholar?q=Sidney+Black%2C+Stella+Biderman%2C+Eric+Hallahan%2C+Quentin+Anthony%2C+Leo+Gao%2C+Laurence+Golding%2C+Horace+He%2C+Connor+Leahy%2C+Kyle+McDonell%2C+Jason+Phang%2C+Michael+Pieler%2C+Usvsn+Sai+Prashanth%2C+Shivanshu+Purohit%2C+L) [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
9. Yiran Chen, Zhenqiao Song, Xianze Wu, Danqing Wang, Jingjing Xu, Jiaze Chen, Hao Zhou, and Lei Li. 2022. MTG: A Benchmark Suite for Multilingual Text Generation. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2508–2527, Seattle, United States. Association for Computa… [find](https://scholar.google.com/scholar?q=Yiran+Chen%2C+Zhenqiao+Song%2C+Xianze+Wu%2C+Danqing+Wang%2C+Jingjing+Xu%2C+Jiaze+Chen%2C+Hao+Zhou%2C+and+Lei+Li%2E+2022%2E+MTG%3A+A+Benchmark+Suite+for+Multilingual+Text+Generation%2E+In+Findings+of+the+Association+for+Com)
10. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating Long Sequences with Sparse Transformers. [find](https://scholar.google.com/scholar?q=Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+and+Ilya+Sutskever%2E+2019%2E+Generating+Long+Sequences+with+Sparse+Transformers%2E)
11. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking Embedding Coupling in Pre-trained Language Models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Thibault+Fevry%2C+Henry+Tsai%2C+Melvin+Johnson%2C+and+Sebastian+Ruder%2E+2021%2E+Rethinking+Embedding+Coupling+in+Pre%2Dtrained+Language+Models%2E+In+International+Conference+on+Learning+Representa) [our card](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/)
12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
13. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
14. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages… [find](https://scholar.google.com/scholar?q=Ryan+Cotterell%2C+Sabrina+J%2E+Mielke%2C+Jason+Eisner%2C+and+Brian+Roark%2E+2018%2E+Are+all+languages+equally+hard+to+language%2Dmodel%3F+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Ass)
15. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11, pages 512–515. [find](https://scholar.google.com/scholar?q=Thomas+Davidson%2C+Dana+Warmsley%2C+Michael+Macy%2C+and+Ingmar+Weber%2E+2017%2E+Automated+Hate+Speech+Detection+and+the+Problem+of+Offensive+Language%2E+In+Proceedings+of+the+International+AAAI+Conference+on+Web+) [our card](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/)
16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
17. Nolan Dey, Gurpreet Gosal, Zhiming, Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, and Joel Hestness. 2023. Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster. [find](https://scholar.google.com/scholar?q=Nolan+Dey%2C+Gurpreet+Gosal%2C+Zhiming%2C+Chen%2C+Hemant+Khachane%2C+William+Marshall%2C+Ribhu+Pathria%2C+Marvin+Tom%2C+and+Joel+Hestness%2E+2023%2E+Cerebras%2DGPT%3A+Open+Compute%2DOptimal+Language+Models+Trained+on+the+Cereb)
18. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+Smith%2E+2020%2E+Fine%2DTuning+Pretrained+Language+Models%3A+Weight+Initializations%2C+Data+Orders%2C+and+Early+Stopping%2E)
19. Fanny Ducel, Karën Fort, Gaël Lejeune, and Yves Lepage. 2022. Do We Name the Languages we Study? The \#BenderRule in LREC and ACL Articles. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 564–573, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Fanny+Ducel%2C+Kare%CC%88n+Fort%2C+Gae%CC%88l+Lejeune%2C+and+Yves+Lepage%2E+2022%2E+Do+We+Name+the+Languages+we+Study%3F+The+%5C%23BenderRule+in+LREC+and+ACL+Articles%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+and+Ev)
20. Erkut Erdem, Menekse Kuyu, Semih Yagcioglu, Anette Frank, Letitia Parcalabescu, Barbara Plank, Andrii Babii, Oleksii Turuta, Aykut Erdem, Iacer Calixto, et al. 2022. Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning. Journal of Artificial In… [find](https://scholar.google.com/scholar?q=Erkut+Erdem%2C+Menekse+Kuyu%2C+Semih+Yagcioglu%2C+Anette+Frank%2C+Letitia+Parcalabescu%2C+Barbara+Plank%2C+Andrii+Babii%2C+Oleksii+Turuta%2C+Aykut+Erdem%2C+Iacer+Calixto%2C+et+al%2E+2022%2E+Neural+Natural+Language+Generation)
21. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori… [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Tosin+Adewumi%2C+Karmanya+Aggarwal%2C+Pawan+Sasanka+Ammanamanchi%2C+Anuoluwapo+Aremu%2C+Antoine+Bosselut%2C+Khyathi+Raghavi+Chandu%2C+Miruna%2DAdriana+Clinciu%2C+Dipanjan+Das%2C+Kaustubh+Dhole%2C+Wany)
22. Omer Goldman, Francesco Tinner, Hila Gonen, Benjamin Muller, Victoria Basmov, Shadrack Kirimi, Lydia Nishimwe, Benoı̂t Sagot, Djamé Seddah, Reut Tsarfaty, and Duygu Ataman. 2022. [find](https://scholar.google.com/scholar?q=Omer+Goldman%2C+Francesco+Tinner%2C+Hila+Gonen%2C+Benjamin+Muller%2C+Victoria+Basmov%2C+Shadrack+Kirimi%2C+Lydia+Nishimwe%2C+Beno%C4%B1%CC%82t+Sagot%2C+Djame%CC%81+Seddah%2C+Reut+Tsarfaty%2C+and+Duygu+Ataman%2E+2022%2E)
23. The MRL 2022 Shared Task on Multilingual Clause-level Morphology. In Proceedings of the The 2nd Workshop on Multi-lingual Representation Learning (MRL), pages 134–146, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=The+MRL+2022+Shared+Task+on+Multilingual+Clause%2Dlevel+Morphology%2E+In+Proceedings+of+the+The+2nd+Workshop+on+Multi%2Dlingual+Representation+Learning+%28MRL%29%2C+pages+134%E2%80%93146%2C+Abu+Dhabi%2C+United+Arab+Emirates+)
24. Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021. A survey on recent approaches for natural language processing in low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: H… [find](https://scholar.google.com/scholar?q=Michael+A%2E+Hedderich%2C+Lukas+Lange%2C+Heike+Adel%2C+Jannik+Stro%CC%88tgen%2C+and+Dietrich+Klakow%2E+2021%2E+A+survey+on+recent+approaches+for+natural+language+processing+in+low%2Dresource+scenarios%2E+In+Proceedings+of+t) [our card](https://md.research.elamur.ai/free/en/papers/hedderich-2021-survey-recent-approaches-natural-language/)
25. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann%2C+Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Elena+Buchatskaya%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Diego+de+Las+Casas%2C+Lisa+Anne+Hendricks%2C+Johannes+Welbl%2C+Aidan+Clark%2C+Tom+Hennigan%2C+Eric+Noland%2C+Ka)
26. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+XTREME%3A+A+Massively+Multilingual+Multi%2Dtask+Benchmark+for+Evaluating+Cross%2Dlingual+Generalisation%2E+In)
27. Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internationa… [find](https://scholar.google.com/scholar?q=Haoyang+Huang%2C+Yaobo+Liang%2C+Nan+Duan%2C+Ming+Gong%2C+Linjun+Shou%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+2019%2E+Unicoder%3A+A+universal+language+encoder+by+pre%2Dtraining+with+multiple+cross%2Dlingual+tasks%2E+In+Proceedings) [our card](https://md.research.elamur.ai/free/en/papers/huang-2019-unicoder-universal-language-encoder-pre/)
28. Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks Lakshmanan, V.S. 2020. Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2296– 2309, Barcelona, Spain (Online). International Committee on Computati… [find](https://scholar.google.com/scholar?q=Ganesh+Jawahar%2C+Muhammad+Abdul%2DMageed%2C+and+Laks+Lakshmanan%2C+V%2ES%2E+2020%2E+Automatic+detection+of+machine+generated+text%3A+A+critical+survey%2E+In+Proceedings+of+the+28th+International+Conference+on+Computat) [our card](https://md.research.elamur.ai/free/en/papers/jawahar-2020-automatic-detection-machine-generated-text/)
29. Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Computation… [find](https://scholar.google.com/scholar?q=Divyanshu+Kakwani%2C+Anoop+Kunchukuttan%2C+Satish+Golla%2C+Gokul+N%2EC%2E%2C+Avik+Bhattacharyya%2C+Mitesh+M%2E+Khapra%2C+and+Pratyush+Kumar%2E+2020%2E+IndicNLPSuite%3A+Monolingual+corpora%2C+evaluation+benchmarks+and+pre%2Dtrain) [our card](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/)
30. Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2021. AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Katikapalli+Subramanyam+Kalyan%2C+Ajit+Rajasekharan%2C+and+Sivanesan+Sangeetha%2E+2021%2E+AMMUS+%3A+A+Survey+of+Transformer%2Dbased+Pretrained+Models+in+Natural+Language+Processing%2E)
31. Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250– 3258, Online. Asso… [find](https://scholar.google.com/scholar?q=Nora+Kassner%2C+Philipp+Dufter%2C+and+Hinrich+Schu%CC%88tze%2E+2021%2E+Multilingual+LAMA%3A+Investigating+knowledge+in+multilingual+pretrained+language+models%2E+In+Proceedings+of+the+16th+Conference+of+the+European+C) [our card](https://md.research.elamur.ai/free/en/papers/kassner-2021-multilingual-lama-investigating-knowledge-multilingual/)
32. Victoria Khurshudyan, Timofey Arkhangelskiy, Misha Daniel, Vladimir Plungian, Dmitri Levonian, Alex Polyakov, and Sergei Rubakov. 2022. Eastern Armenian National Corpus: State of the Art and Perspectives. In Proceedings of the Workshop on Processing Language Variation: Digital Armenian (DigitAm) wit… [find](https://scholar.google.com/scholar?q=Victoria+Khurshudyan%2C+Timofey+Arkhangelskiy%2C+Misha+Daniel%2C+Vladimir+Plungian%2C+Dmitri+Levonian%2C+Alex+Polyakov%2C+and+Sergei+Rubakov%2E+2022%2E+Eastern+Armenian+National+Corpus%3A+State+of+the+Art+and+Perspecti)
33. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoı̂t Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedr… [find](https://scholar.google.com/scholar?q=Julia+Kreutzer%2C+Isaac+Caswell%2C+Lisa+Wang%2C+Ahsan+Wahab%2C+Daan+van+Esch%2C+Nasanbayar+Ulzii%2DOrshikh%2C+Allahsera+Tapo%2C+Nishant+Subramani%2C+Artem+Sokolov%2C+Claytone+Sikasote%2C+Monang+Setyawan%2C+Supheakmungkol+Sar) [our card](https://md.research.elamur.ai/free/en/papers/kreutzer-2021-quality-glance-audit-web-crawled/)
34. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483–4499, Online. Ass… [find](https://scholar.google.com/scholar?q=Anne+Lauscher%2C+Vinit+Ravishankar%2C+Ivan+Vulic%CC%81%2C+and+Goran+Glavas%CC%8C%2E+2020%2E+From+zero+to+hero%3A+On+the+limitations+of+zero%2Dshot+language+transfer+with+multilingual+Transformers%2E+In+Proceedings+of+the+2020+) [our card](https://md.research.elamur.ai/free/en/papers/lauscher-2020-zero-hero-limitations-zero-shot/)
35. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Mi… [find](https://scholar.google.com/scholar?q=Yaobo+Liang%2C+Nan+Duan%2C+Yeyun+Gong%2C+Ning+Wu%2C+Fenfei+Guo%2C+Weizhen+Qi%2C+Ming+Gong%2C+Linjun+Shou%2C+Daxin+Jiang%2C+Guihong+Cao%2C+Xiaodong+Fan%2C+Ruofei+Zhang%2C+Rahul+Agrawal%2C+Edward+Cui%2C+Sining+Wei%2C+Taroon+Bharti%2C+)
36. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… [find](https://scholar.google.com/scholar?q=Xi+Victoria+Lin%2C+Todor+Mihaylov%2C+Mikel+Artetxe%2C+Tianlu+Wang%2C+Shuohui+Chen%2C+Daniel+Simig%2C+Myle+Ott%2C+Naman+Goyal%2C+Shruti+Bhosale%2C+Jingfei+Du%2C+Ramakanth+Pasunuru%2C+Sam+Shleifer%2C+Punit+Singh+Koura%2C+Vishrav) [our card](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
37. Pierre Lison and Jörg Tiedemann. 2016. Open-Subtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia. European Language Resources Association… [find](https://scholar.google.com/scholar?q=Pierre+Lison+and+Jo%CC%88rg+Tiedemann%2E+2016%2E+Open%2DSubtitles2016%3A+Extracting+large+parallel+corpora+from+movie+and+TV+subtitles%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+an)
38. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023a. Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 55(9):1–35. [find](https://scholar.google.com/scholar?q=Pengfei+Liu%2C+Weizhe+Yuan%2C+Jinlan+Fu%2C+Zhengbao+Jiang%2C+Hiroaki+Hayashi%2C+and+Graham+Neubig%2E+2023a%2E+Pre%2DTrain%2C+Prompt%2C+and+Predict%3A+A+Systematic+Survey+of+Prompting+Methods+in+Natural+Language+Processing%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/)
39. Qi Liu, Matt J. Kusner, and Phil Blunsom. 2020a. A Survey on Contextual Embeddings. [find](https://scholar.google.com/scholar?q=Qi+Liu%2C+Matt+J%2E+Kusner%2C+and+Phil+Blunsom%2E+2020a%2E+A+Survey+on+Contextual+Embeddings%2E)
40. Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell, and Mrinmaya Sachan. 2022. Autoregressive Structured Prediction with Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 993–1005, Abu Dhabi, United Arab Emirates. Association for Computat… [find](https://scholar.google.com/scholar?q=Tianyu+Liu%2C+Yuchen+Eleanor+Jiang%2C+Nicholas+Monath%2C+Ryan+Cotterell%2C+and+Mrinmaya+Sachan%2E+2022%2E+Autoregressive+Structured+Prediction+with+Language+Models%2E+In+Findings+of+the+Association+for+Computationa)
41. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023b. GPT Understands, Too. AI Open. [find](https://scholar.google.com/scholar?q=Xiao+Liu%2C+Yanan+Zheng%2C+Zhengxiao+Du%2C+Ming+Ding%2C+Yujie+Qian%2C+Zhilin+Yang%2C+and+Jie+Tang%2E+2023b%2E+GPT+Understands%2C+Too%2E+AI+Open%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/)
42. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020b. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020b%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transacti)
43. H Mann and D Whitney. 1947. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Ann. Math. Stat, 18(1):50–60. [find](https://scholar.google.com/scholar?q=H+Mann+and+D+Whitney%2E+1947%2E+Controlling+the+False+Discovery+Rate%3A+A+Practical+and+Powerful+Approach+to+Multiple+Testing%2E+Ann%2E+Math%2E+Stat%2C+18%281%29%3A50%E2%80%9360%2E)
44. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–721… [find](https://scholar.google.com/scholar?q=Louis+Martin%2C+Benjamin+Muller%2C+Pedro+Javier+Ortiz+Sua%CC%81rez%2C+Yoann+Dupont%2C+Laurent+Romary%2C+E%CC%81ric+de+la+Clergerie%2C+Djame%CC%81+Seddah%2C+and+Beno%C4%B1%CC%82t+Sagot%2E+2020%2E+CamemBERT%3A+a+tasty+French+language+model%2E+In+Pro) [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
45. Mihai Masala, Stefan Ruseti, and Mihai Dascalu. 2020. RoBERT – a Romanian BERT model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6626– 6637, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mihai+Masala%2C+Stefan+Ruseti%2C+and+Mihai+Dascalu%2E+2020%2E+RoBERT+%E2%80%93+a+Romanian+BERT+model%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+pages+6626%E2%80%93+6637%2C+Barcelona%2C+Spa)
46. R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2020. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 217–2… [find](https://scholar.google.com/scholar?q=R%2E+Thomas+McCoy%2C+Junghyun+Min%2C+and+Tal+Linzen%2E+2020%2E+BERTs+of+a+feather+do+not+generalize+together%3A+Large+variability+in+generalization+across+models+with+similar+test+set+performance%2E+In+Proceedings+)
47. Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoı̂t Sagot, and Samson Tan. 2021. Between Words and Characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP. [find](https://scholar.google.com/scholar?q=Sabrina+J%2E+Mielke%2C+Zaid+Alyafeai%2C+Elizabeth+Salesky%2C+Colin+Raffel%2C+Manan+Dey%2C+Matthias+Galle%CC%81%2C+Arun+Raja%2C+Chenglei+Si%2C+Wilson+Y%2E+Lee%2C+Beno%C4%B1%CC%82t+Sagot%2C+and+Samson+Tan%2E+2021%2E+Between+Words+and+Characters%3A)
48. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computation… [find](https://scholar.google.com/scholar?q=Aakanksha+Naik%2C+Abhilasha+Ravichander%2C+Norman+Sadeh%2C+Carolyn+Rose%2C+and+Graham+Neubig%2E+2018%2E+Stress+test+evaluation+for+natural+language+inference%2E+In+Proceedings+of+the+27th+International+Conference+o)
49. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Timothy+Niven+and+Hung%2DYu+Kao%2E+2019%2E+Probing+neural+network+comprehension+of+natural+language+arguments%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pag)
50. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth I… [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+de+Marneffe%2C+Filip+Ginter%2C+Yoav+Goldberg%2C+Jan+Hajic%CC%8C%2C+Christopher+D%2E+Manning%2C+Ryan+McDonald%2C+Slav+Petrov%2C+Sampo+Pyysalo%2C+Natalia+Silveira%2C+Reut+Tsarfaty%2C+and+Daniel+Zeman) [our card](https://md.research.elamur.ai/free/en/papers/ud-2016-nivre-universal-dependencies-v1-multilingual-treebank/)
51. Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2021. Investigating the Limitations of Transformers with Simple Arithmetic Tasks. the Royal Society of London, 58(347-352):240– 242. [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira%2C+Zhiying+Jiang%2C+and+Jimmy+Lin%2E+2021%2E+Investigating+the+Limitations+of+Transformers+with+Simple+Arithmetic+Tasks%2E+the+Royal+Society+of+London%2C+58%28347%2D352%29%3A240%E2%80%93+242%2E)
52. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True Few-shot Learning with Language Models. Advances in Neural Information Processing Systems, 34. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Douwe+Kiela%2C+and+Kyunghyun+Cho%2E+2021%2E+True+Few%2Dshot+Learning+with+Language+Models%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%2E) [our card](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)
53. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Tim+Rockta%CC%88schel%2C+Sebastian+Riedel%2C+Patrick+Lewis%2C+Anton+Bakhtin%2C+Yuxiang+Wu%2C+and+Alexander+Miller%2E+2019%2E+Language+models+as+knowledge+bases%3F+In+Proceedings+of+the+2019+Conference+on+Em) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
54. Boris Orekhov, I Krylova, I Popov, E Stepanova, and L Zaydelman. 2016. Russian Minority Languages on the Web: Descriptive Statistics. In Vladimir Selegey (chief ed.), Computational linguistics and intellectual technologies: Proceedings of the international conference “Dialogue, pages 498–508. [find](https://scholar.google.com/scholar?q=Boris+Orekhov%2C+I+Krylova%2C+I+Popov%2C+E+Stepanova%2C+and+L+Zaydelman%2E+2016%2E+Russian+Minority+Languages+on+the+Web%3A+Descriptive+Statistics%2E+In+Vladimir+Selegey+%28chief+ed%2E%29%2C+Computational+linguistics+and+int)
55. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021. UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186–10203, Online and Punta Cana, Dominican Republic. Asso… [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Iryna+Gurevych%2C+and+Sebastian+Ruder%2E+2021%2E+UNKs+everywhere%3A+Adapting+multilingual+language+models+to+new+scripts%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Method) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2021-unks-everywhere-adapting-multilingual-language/)
56. Pedro Javier Ortiz Suárez, Benoı̂t Sagot, and Laurent Romary. 2019. Asynchronous Pipelines for Processing Huge Corpora on Medium to Low Resource Infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannhe… [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Sua%CC%81rez%2C+Beno%C4%B1%CC%82t+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+Pipelines+for+Processing+Huge+Corpora+on+Medium+to+Low+Resource+Infrastructures%2E+Proceedings+of+the+Workshop+on+Challe)
57. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Associatio… [find](https://scholar.google.com/scholar?q=Edoardo+Maria+Ponti%2C+Goran+Glavas%CC%8C%2C+Olga+Majewska%2C+Qianchu+Liu%2C+Ivan+Vulic%CC%81%2C+and+Anna+Korhonen%2E+2020%2E+XCOPA%3A+A+multilingual+dataset+for+causal+commonsense+reasoning%2E+In+Proceedings+of+the+2020+Confere) [our card](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/)
58. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Sergey+Edunov%2C+Alexei+Baevski%2C+Angela+Fan%2C+Sam+Gross%2C+Nathan+Ng%2C+David+Grangier%2C+and+Michael+Auli%2E+2019%2E+fairseq%3A+A+fast%2C+extensible+toolkit+for+sequence+modeling%2E+In+Proceedings+of+the+2019) [our card](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)
59. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+Models+are+Unsupervised+Multitask+Learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
60. David Paper. 2021. TensorFlow Datasets. State-ofthe-Art Deep Learning Models in TensorFlow: Modern Machine Learning in the Google Colab Ecosystem, pages 65–91. [find](https://scholar.google.com/scholar?q=David+Paper%2E+2021%2E+TensorFlow+Datasets%2E+State%2Dofthe%2DArt+Deep+Learning+Models+in+TensorFlow%3A+Modern+Machine+Learning+in+the+Google+Colab+Ecosystem%2C+pages+65%E2%80%9391%2E)
61. Karl Pearson. 1895. Note on Regression and Inheritance in the Case of Two Parents. Proceedings of Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text T… [find](https://scholar.google.com/scholar?q=Karl+Pearson%2E+1895%2E+Note+on+Regression+and+Inheritance+in+the+Case+of+Two+Parents%2E+Proceedings+of+Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+We)
62. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. DeepSpeed: System Pptimizations Enable Training Deep Learning Models with over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+DeepSpeed%3A+System+Pptimizations+Enable+Training+Deep+Learning+Models+with+over+100+Billion+Parameters%2E+In+Proceedings+of+the+26t)
63. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join… [find](https://scholar.google.com/scholar?q=Phillip+Rust%2C+Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Sebastian+Ruder%2C+and+Iryna+Gurevych%2E+2021%2E+How+good+is+your+tokenizer%3F+on+the+monolingual+performance+of+multilingual+language+models%2E+In+Proceedings+of+the+) [our card](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)
64. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoı̂t Sagot, Niklas Muenn… [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagne%CC%81%2C+Alexandra+Sasha+Luccioni%2C+Franc%CC%A7ois+Yvon%2C+Matthias+Galle%CC%81%2C+Jonathan+Tow%2C+Alexander+M%2E+Rush%2C+)
65. Timo Schick and Hinrich Schütze. 2021. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Assoc… [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Schu%CC%88tze%2E+2021%2E+It%E2%80%99s+not+just+size+that+matters%3A+Small+language+models+are+also+few%2Dshot+learners%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+As) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/)
66. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+2020%2E+Megatron%2DLM%3A+Training+Multi%2DBillion+Parameter+Language+Models+Using+Model+Parallelism%2E)
67. Emma Strubell, Ananya Ganesh, and Andrew Mc-Callum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emma+Strubell%2C+Ananya+Ganesh%2C+and+Andrew+Mc%2DCallum%2E+2019%2E+Energy+and+policy+considerations+for+deep+learning+in+NLP%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Ling) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/)
68. Alexey Tikhonov and Max Ryabinin. 2021. It’s All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3534–3546, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexey+Tikhonov+and+Max+Ryabinin%2E+2021%2E+It%E2%80%99s+All+in+the+Heads%3A+Using+Attention+Heads+as+a+Baseline+for+Cross%2DLingual+Transfer+in+Commonsense+Reasoning%2E+In+Findings+of+the+Association+for+Computational)
69. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. Advances in Neural Information Processing Systems, 30. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+All+You+Need%2E+Advances+in+Neural+Information+Processing+)
70. Sanh Victor, Webson Albert, Raffel Colin, Bach Stephen, Sutawika Lintang, Alyafeai Zaid, Chaffin Antoine, Stiegler Arnaud, Raja Arun, Dey Manan, et al. 2022. Multitask Prompted Training Enables Zero-shot Task Generalization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sanh+Victor%2C+Webson+Albert%2C+Raffel+Colin%2C+Bach+Stephen%2C+Sutawika+Lintang%2C+Alyafeai+Zaid%2C+Chaffin+Antoine%2C+Stiegler+Arnaud%2C+Raja+Arun%2C+Dey+Manan%2C+et+al%2E+2022%2E+Multitask+Prompted+Training+Enables+Zero%2Ds) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
71. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-purpose Language Understanding Systems. Advances in Neural Information Processing Systems, 32. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2019%2E+SuperGLUE%3A+A+Stickier+Benchmark+for+General%2Dpurpose+Language+Understandin) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
72. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [find](https://scholar.google.com/scholar?q=Ben+Wang+and+Aran+Komatsuzaki%2E+2021%2E+GPT%2DJ%2D6B%3A+A+6+Billion+Parameter+Autoregressive+Language+Model%2E)
73. Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2020. Neural Machine Translation with Byte-Level Subwords. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9154–9160. [find](https://scholar.google.com/scholar?q=Changhan+Wang%2C+Kyunghyun+Cho%2C+and+Jiatao+Gu%2E+2020%2E+Neural+Machine+Translation+with+Byte%2DLevel+Subwords%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+34%2C+pages+9154%E2%80%939160%2E)
74. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? GPT-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195–4205, Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shuohang+Wang%2C+Yang+Liu%2C+Yichong+Xu%2C+Chenguang+Zhu%2C+and+Michael+Zeng%2E+2021%2E+Want+to+reduce+labeling+cost%3F+GPT%2D3+can+help%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2021%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/wang-2021-want-reduce-labeling-cost-gpt/)
75. Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empi… [find](https://scholar.google.com/scholar?q=Funtowicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara+Ma%2C+Yacine+Jernite%2C+Julien+Plu%2C+Canwen+Xu%2C+Teven+Le+Scao%2C+Sylvain+Gugger%2C+Mariama+Drame%2C+Quentin+Lhoest%2C+and+Alexander+Rush%2E+2020%2E+Trans) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
76. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shijie+Wu+and+Mark+Dredze%2E+2020%2E+Are+all+languages+created+equal+in+multilingual+BERT%3F+In+Proceedings+of+the+5th+Workshop+on+Representation+Learning+for+NLP%2C+pages+120%E2%80%93130%2C+Online%2E+Association+for+Com) [our card](https://md.research.elamur.ai/free/en/papers/wu-2020-are-all-languages-created-equal/)
77. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. 2023. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+Liang%2C+Quoc+V%2E+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+2023%2E+DoReMi%3A+Optimizing+Data+Mixtures+Speeds+Up+Language+Model+Pretrain)
78. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022. Taxonomy of Risks Posed by Language Models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, page… [find](https://scholar.google.com/scholar?q=Laura+Weidinger%2C+Jonathan+Uesato%2C+Maribeth+Rauh%2C+Conor+Griffin%2C+Po%2DSen+Huang%2C+John+Mellor%2C+Amelia+Glaese%2C+Myra+Cheng%2C+Borja+Balle%2C+Atoosa+Kasirzadeh%2C+et+al%2E+2022%2E+Taxonomy+of+Risks+Posed+by+Language+M)
79. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Noah+Constant%2C+Adam+Roberts%2C+Mihir+Kale%2C+Rami+Al%2DRfou%2C+Aditya+Siddhant%2C+Aditya+Barua%2C+and+Colin+Raffel%2E+2021%2E+mT5%3A+A+massively+multilingual+pre%2Dtrained+text%2Dto%2Dtext+transformer%2E+In+Procee)
80. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003– 4012, Mars… [find](https://scholar.google.com/scholar?q=Guillaume+Wenzek%2C+Marie%2DAnne+Lachaux%2C+Alexis+Conneau%2C+Vishrav+Chaudhary%2C+Francisco+Guzma%CC%81n%2C+Armand+Joulin%2C+and+Edouard+Grave%2E+2020%2E+CCNet%3A+Extracting+high+quality+monolingual+datasets+from+web+crawl+d) [our card](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/)
81. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019a. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… [find](https://scholar.google.com/scholar?q=Yinfei+Yang%2C+Yuan+Zhang%2C+Chris+Tar%2C+and+Jason+Baldridge%2E+2019a%2E+PAWS%2DX%3A+A+cross%2Dlingual+adversarial+dataset+for+paraphrase+identification%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/)
82. Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021. Language models are few-shot multilingual learners. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 1–15, Punta Cana, Dominican Republic. Association for Computat… [find](https://scholar.google.com/scholar?q=Genta+Indra+Winata%2C+Andrea+Madotto%2C+Zhaojiang+Lin%2C+Rosanne+Liu%2C+Jason+Yosinski%2C+and+Pascale+Fung%2E+2021%2E+Language+models+are+few%2Dshot+multilingual+learners%2E+In+Proceedings+of+the+1st+Workshop+on+Multil) [our card](https://md.research.elamur.ai/free/en/papers/winata-2021-language-models-are-few-shot/)
83. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019b. XLNet: Generalized Autoregressive Pre-training for Language Understanding. Advances in Neural Information Processing Systems, 32. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Russ+R+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019b%2E+XLNet%3A+Generalized+Autoregressive+Pre%2Dtraining+for+Language+Understanding%2E+Advances+in+Neural+Informa)
84. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, Chen Li, Ziyan Gong, Yifan Yao, Xinjing Huang… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+Remi+Louf%2C+Morgan+Wei+Zeng%2C+Xiaozhe+Ren%2C+Teng+Su%2C+Hui+Wang%2C+Yi+Liao%2C+Zhiwei+Wang%2C+X)
85. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trai… [find](https://scholar.google.com/scholar?q=Susan+Zhang%2C+Stephen+Roller%2C+Naman+Goyal%2C+Mikel+Artetxe%2C+Moya+Chen%2C+Shuohui+Chen%2C+Christopher+Dewan%2C+Mona+Diab%2C+Xian+Li%2C+Xi+Victoria+Lin%2C+Todor+Mihaylov%2C+Myle+Ott%2C+Sam+Shleifer%2C+Kurt+Shuster%2C+Daniel+S)
86. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate Before Use: Improving Few-shot Performance of Language Models. In International Conference on Machine Learning, pages 12697–12706. PMLR. [find](https://scholar.google.com/scholar?q=Zihao+Zhao%2C+Eric+Wallace%2C+Shi+Feng%2C+Dan+Klein%2C+and+Sameer+Singh%2E+2021%2E+Calibrate+Before+Use%3A+Improving+Few%2Dshot+Performance+of+Language+Models%2E+In+International+Conference+on+Machine+Learning%2C+pages+1) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/)
87. Dmitry Zmitrovich, Alexander Abramov, Andrey Kalmykov, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergey Markov, Vladislav Mikhailov, and Alena Fenogenova. 2023. A Family of Pretrained Transformer Language Models for Russian. [find](https://scholar.google.com/scholar?q=Dmitry+Zmitrovich%2C+Alexander+Abramov%2C+Andrey+Kalmykov%2C+Maria+Tikhonova%2C+Ekaterina+Taktasheva%2C+Danil+Astafurov%2C+Mark+Baushenko%2C+Artem+Snegirev%2C+Tatiana+Shavrina%2C+Sergey+Markov%2C+Vladislav+Mikhailov%2C+and)