---
title: "Scaling Laws for Generative Mixed-Modal Language Models — Aghajanyan 2023"
description: "This offers practical guidance for planning unified multimodal generative models. 42 references, cited by 32 papers here."
authors: ["Armen Aghajanyan", "Lili Yu", "Alexis Conneau", "Wei-Ning Hsu", "Karen Hambardzumyan", "Susan Zhang", "Stephen Roller", "Naman Goyal", "Omer Levy", "Luke Zettlemoyer"]
year: 2023
venue: "arXiv:2301.03728"
source: "https://arxiv.org/abs/2301.03728"
doi: "10.48550/arxiv.2301.03728"
arxiv: "2301.03728"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aghajanyan-2023-scaling-laws-generative-mixed-modal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aghajanyan-2023-scaling-laws-generative-mixed-modal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Scaling Laws for Generative Mixed-Modal Language Models

# Scaling Laws for Generative Mixed-Modal Language Models

**Authors:** [Armen Aghajanyan](https://md.research.elamur.ai/free/en/authors/armen-aghajanyan/), Lili Yu, [Alexis Conneau](https://md.research.elamur.ai/free/en/authors/alexis-conneau/), [Wei-Ning Hsu](https://md.research.elamur.ai/free/en/authors/wei-ning-hsu/), [Karen Hambardzumyan](https://md.research.elamur.ai/free/en/authors/karen-hambardzumyan/), [Susan Zhang](https://md.research.elamur.ai/free/en/authors/susan-zhang/), [Stephen Roller](https://md.research.elamur.ai/free/en/authors/stephen-roller/), [Naman Goyal](https://md.research.elamur.ai/free/en/authors/naman-goyal/), [Omer Levy](https://md.research.elamur.ai/free/en/authors/omer-levy/) and [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2023 · **Venue:** arXiv:2301.03728 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2301.03728) · [PDF](https://arxiv.org/pdf/2301.03728) · [DOI 10.48550/arxiv.2301.03728](https://doi.org/10.48550/arxiv.2301.03728) · [arXiv 2301.03728](https://arxiv.org/abs/2301.03728) · [OpenAlex](https://openalex.org/W4315705838)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [32](https://md.research.elamur.ai/free/en/papers/aghajanyan-2023-scaling-laws-generative-mixed-modal/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Single generative models can treat images, speech, code and text as token streams. It is unclear, however, whether added scale overcomes the interference that arises when several modalities share one network. The authors ran more than 250 training experiments covering seven modalities: Text, Image, Image-Text, Speech, Speech-Text, Code and Molecules. Model sizes ranged from 8M to 30B parameters, with 5–100B training tokens. Modalities were distinguished empirically: two datasets count as separate if cross-perplexity exceeds a threshold (σ = 3). All data were converted into discrete tokens, using a Make-A-Scene VQGAN for images and HuBERT for speech. The paper starts from the Hoffmann et al. (2022) uni-modal law and adds an extra term for each pair of modalities that captures whether they compete or help each other. The fitted laws explain observed training behaviour and yield hyper-parameter recipes. They also correctly predicted that a 30B speech-text model trained on 45B tokens would leave competition behind and beat its uni-modal counterparts. This offers practical guidance for planning unified multimodal generative models.

## Key points
- Over 250 runs, seven modalities, models from 8M to 30B parameters, 5–100B tokens each.
- Data sources include:
  - the OPT text corpus (180B tokens);
  - images from a 600M-image subset of LAION (Schuhmann et al.) plus Common Crawl (614B tokens; 690B with captions);
  - LibriSpeech, CommonVoice, VoxPopuli, Spotify Podcast and People's Speech for speech;
  - InCoder data for code;
  - Zinc SMILES strings for molecules.
- Images become 1024 tokens drawn from an 8192-entry codebook. Speech is encoded by HuBERT at 50 Hz with 2000 units, roughly 117× compression and little loss in WER.
- The interaction term links to observed phenomena: models alternate their focus between modalities in a coordinate-ascent-like way, and competition relates to training instability and optimal batch size.
- A 30B speech-text model trained in the predicted regime eliminated speech–text competition and outperformed the uni-modal models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 206 works · 158,423 citations

💡 Concepts (1): [📈 Scaling laws](https://md.research.elamur.ai/free/en/concepts/scaling-laws/) · 30 works

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 7,249 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 7,084 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 7,398 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 7,340 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,025 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,079 works

## Cite

Cite as: Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy and Luke Zettlemoyer. 2023. Scaling Laws for Generative Mixed-Modal Language Models. arXiv:2301.03728.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{aghajanyan2023scaling,
  title = {{Scaling Laws for Generative Mixed-Modal Language Models}},
  author = {Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy and Luke Zettlemoyer},
  year = {2023},
  doi = {10.48550/arxiv.2301.03728},
  eprint = {2301.03728},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.03728}
}
```

- [← previous Findings of the Second WMT Shared Task on Sign Language Translation (WMT-SLT23)](https://md.research.elamur.ai/free/en/papers/muller-2023-findings-second-wmt-shared-task/)
- [next → xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://md.research.elamur.ai/free/en/papers/khan-2023-xcodeeval-large-scale-multilingual-multitask/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

42 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520, 2022. — [arXiv:2201.07520](https://arxiv.org/abs/2201.07520) [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Bernie+Huang%2C+Candace+Ross%2C+Vladimir+Karpukhin%2C+Hu+Xu%2C+Naman+Goyal%2C+Dmytro+Okhonko%2C+Mandar+Joshi%2C+Gargi+Ghosh%2C+Mike+Lewis%2C+et+al%2E+Cm3%3A+A+causal+masked+multimodal+model+of+the+interne)
2. Gor Arakelyan, Gevorg Soghomonyan, and The Aim team. //github.com/aimhubio/aim. Aim, 6 2020. [find](https://scholar.google.com/scholar?q=Gor+Arakelyan%2C+Gevorg+Soghomonyan%2C+and+The+Aim+team%2E+%2F%2Fgithub%2Ecom%2Faimhubio%2Faim%2E+Aim%2C+6+2020%2E)
3. URL https: Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019. [our card](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/)
4. Mandeep Baines, Shruti Bhosale, Vittorio Caggiano, Naman Goyal, Siddharth Goyal, Myle Ott, Benjamin Lefaudeux, Vitaliy Liptchinsky, Mike Rabbat, Sam Sheiffer, Anjali Sridhar, and Min Xu. FairScale: A general purpose modular PyTorch library for high performance and large scale training. https://githu… [find](https://scholar.google.com/scholar?q=Mandeep+Baines%2C+Shruti+Bhosale%2C+Vittorio+Caggiano%2C+Naman+Goyal%2C+Siddharth+Goyal%2C+Myle+Ott%2C+Benjamin+Lefaudeux%2C+Vitaliy+Liptchinsky%2C+Mike+Rabbat%2C+Sam+Sheiffer%2C+Anjali+Sridhar%2C+and+Min+Xu%2E+FairScale%3A+A+)
5. Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255, 2022. [our card](https://md.research.elamur.ai/free/en/papers/bavarian-2022-efficient-training-language-models-fill/)
6. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020. [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. arXiv preprint arXiv:2212.07143, 2022. [our card](https://md.research.elamur.ai/free/en/papers/cherti-2022-reproducible-scaling-laws-contrastive-language/)
9. Gayane Chilingaryan, Hovhannes Tamoyan, Ani Tevosyan, Nelly Babayan, Lusine Khond-karyan, Karen Hambardzumyan, Zaven Navoyan, Hrant Khachatrian, and Armen Aghajanyan. Bartsmiles: Generative masked language models for molecular representations. arXiv preprint arXiv:2211.16349, 2022. — [arXiv:2211.16349](https://arxiv.org/abs/2211.16349) [find](https://scholar.google.com/scholar?q=Gayane+Chilingaryan%2C+Hovhannes+Tamoyan%2C+Ani+Tevosyan%2C+Nelly+Babayan%2C+Lusine+Khond%2Dkaryan%2C+Karen+Hambardzumyan%2C+Zaven+Navoyan%2C+Hrant+Khachatrian%2C+and+Armen+Aghajanyan%2E+Bartsmiles%3A+Generative+masked+lan)
10. Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning, pp. 4057– 4086. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Aidan+Clark%2C+Diego+de+Las+Casas%2C+Aurelia+Guy%2C+Arthur+Mensch%2C+Michela+Paganini%2C+Jordan+Hoffmann%2C+Bogdan+Damoc%2C+Blake+Hechtman%2C+Trevor+Cai%2C+Sebastian+Borgeaud%2C+et+al%2E+Unified+scaling+laws+for+routed+lan)
11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Unsupervised+cross%2Dlingual+re)
12. Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws, 2022. URL https://arxiv.org/abs/2212.09720. [↗](https://arxiv.org/abs/2212.09720) [our card](https://md.research.elamur.ai/free/en/papers/dettmers-2022-case-4-bit-precision-k/)
13. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+arXiv+preprint+arXiv%3A2208%2E07339%2C+2022%2E)
14. Jasha Droppo and Oguz Elibol. Scaling laws for acoustic models. In Interspeech 2021, 2021. URL [https://www.amazon.science/publications/](https://www.amazon.science/publications/) scaling-laws-for-acoustic-models. [↗](https://www.amazon.science/publications/)
15. Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020. [find](https://scholar.google.com/scholar?q=Patrick+Esser%2C+Robin+Rombach%2C+and+Bjo%CC%88rn+Ommer%2E+Taming+transformers+for+high%2Dresolution+image+synthesis%2C+2020%2E)
16. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999, 2022. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Armen+Aghajanyan%2C+Jessy+Lin%2C+Sida+Wang%2C+Eric+Wallace%2C+Freda+Shi%2C+Ruiqi+Zhong%2C+Wen%2Dtau+Yih%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synthesis%2E)
17. Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2203.13131, 2022. — [arXiv:2203.13131](https://arxiv.org/abs/2203.13131) [find](https://scholar.google.com/scholar?q=Oran+Gafni%2C+Adam+Polyak%2C+Oron+Ashual%2C+Shelly+Sheynin%2C+Devi+Parikh%2C+and+Yaniv+Taigman%2E+Make%2Da%2Dscene%3A+Scene%2Dbased+text%2Dto%2Dimage+generation+with+human+priors%2E+arXiv+preprint+arXiv%3A2203%2E13131%2C+2022%2E+%E2%80%94+%5Bar)
18. Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740, 2021. — [arXiv:2109.07740](https://arxiv.org/abs/2109.07740) [find](https://scholar.google.com/scholar?q=Behrooz+Ghorbani%2C+Orhan+Firat%2C+Markus+Freitag%2C+Ankur+Bapna%2C+Maxim+Krikun%2C+Xavier+Garcia%2C+Ciprian+Chelba%2C+and+Colin+Cherry%2E+Scaling+laws+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A2109%2E07740%2C)
19. Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5915–5922, Online and Punta Cana, Dominican Republic, November 2021. Association for Computat… [find](https://scholar.google.com/scholar?q=Mitchell+A+Gordon%2C+Kevin+Duh%2C+and+Jared+Kaplan%2E+Data+and+parameter+scaling+laws+for+neural+machine+translation%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
20. Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. Larger-scale transformers for multilingual masked language modeling. arXiv preprint arXiv:2105.00572, 2021. [our card](https://md.research.elamur.ai/free/en/papers/goyal-2021-larger-scale-transformers-multilingual-masked/)
21. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. [our card](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/)
22. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. [our card](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/)
23. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
24. Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022. doi: 10.1101/2022.04.10.487779. URL [https://www.biorxiv.org/content/early/](https://www.biorxiv.org/content/early/) 2022/04/10/2022.04.10.487779. — [doi:10.1101/2022.04.10.487779](https://doi.org/10.1101/2022.04.10.487779) [↗](https://www.biorxiv.org/content/early/)
25. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021. doi: 1… [our card](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/)
26. Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, B… — [arXiv:1905.12322](https://arxiv.org/abs/1905.12322) [find](https://scholar.google.com/scholar?q=Dhiraj+Kalamkar%2C+Dheevatsa+Mudigere%2C+Naveen+Mellempudi%2C+Dipankar+Das%2C+Kunal+Banerjee%2C+Sasikanth+Avancha%2C+Dharma+Teja+Vooturi%2C+Nataraj+Jammalamadaka%2C+Jianyu+Huang%2C+Hector+Yuen%2C+Jiyan+Yang%2C+Jongsoo+Park)
27. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
28. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR%2C+2015%2E)
29. Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, et al. Direct speech-to-speech translation with discrete units. arXiv preprint arXiv:2107.05604, 2021. [find](https://scholar.google.com/scholar?q=Ann+Lee%2C+Peng%2DJen+Chen%2C+Changhan+Wang%2C+Jiatao+Gu%2C+Xutai+Ma%2C+Adam+Polyak%2C+Yossi+Adi%2C+Qing+He%2C+Yun+Tang%2C+Juan+Pino%2C+et+al%2E+Direct+speech%2Dto%2Dspeech+translation+with+discrete+units%2E+arXiv+preprint+arXiv%3A2)
30. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019. [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
31. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411, 2020. [our card](https://md.research.elamur.ai/free/en/papers/pratap-2020-mls-large-scale-multilingual-dataset/)
32. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021. [our card](https://md.research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/)
33. Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar,… [find](https://scholar.google.com/scholar?q=Scott+Reed%2C+Konrad+Zolna%2C+Emilio+Parisotto%2C+Sergio+Go%CC%81mez+Colmenarejo%2C+Alexander+Novikov%2C+Gabriel+Barth%2Dmaron%2C+Mai+Gime%CC%81nez%2C+Yury+Sulsky%2C+Jackie+Kay%2C+Jost+Tobias+Springenberg%2C+Tom+Eccles%2C+Jake+Bruce%2C+)
34. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022. [our card](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/)
35. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725, Berlin, Germany, August 2016. Association for Computationa… [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
36. Uri Shaham, Maha Elbayad, Vedanuj Goswami, Omer Levy, and Shruti Bhosale. Causes and cures for interference in multilingual translation. arXiv preprint arXiv:2212.07530, 2022. [find](https://scholar.google.com/scholar?q=Uri+Shaham%2C+Maha+Elbayad%2C+Vedanuj+Goswami%2C+Omer+Levy%2C+and+Shruti+Bhosale%2E+Causes+and+cures+for+interference+in+multilingual+translation%2E+arXiv+preprint+arXiv%3A2212%2E07530%2C+2022%2E)
37. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019. URL https://arxiv.org/abs/1909.08053. [↗](https://arxiv.org/abs/1909.08053)
38. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00390, 2021. [our card](https://md.research.elamur.ai/free/en/papers/wang-2021-voxpopuli-large-scale-multilingual-speech/)
39. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework, 2022. URL https://arxiv.org/abs/ 2202.03052. [↗](https://arxiv.org/abs/) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/)
40. Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-augmented multimodal language modeling. arXiv preprint arXiv:2211.12561, 2022. [find](https://scholar.google.com/scholar?q=Michihiro+Yasunaga%2C+Armen+Aghajanyan%2C+Weijia+Shi%2C+Rich+James%2C+Jure+Leskovec%2C+Percy+Liang%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Wen%2Dtau+Yih%2E+Retrieval%2Daugmented+multimodal+language+modeling%2E+arXiv+preprin)
41. Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Jiasen+Lu%2C+Ximing+Lu%2C+Youngjae+Yu%2C+Yanpeng+Zhao%2C+Mohammadreza+Salehi%2C+Aditya+Kusupati%2C+Jack+Hessel%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Merlot+reserve%3A+Neural+script+knowledge+through+vision+a)
42. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)

*Updated 23 October 2026*