---
title: "OPT: Open Pre-trained Transformer Language Models — Zhang 2022"
description: "The report candidly describes hardware failures and loss divergences, as well as the manual interventions used… 98 references, cited by 1,709 papers here."
authors: ["Susan Zhang", "Stephen Roller", "Naman Goyal", "Mikel Artetxe", "Moya Chen", "Shuohui Chen", "Christopher Dewan", "Mona Diab", "Xian Li", "Xi Victoria Lin", "Todor Mihaylov", "Myle Ott", "Sam Shleifer", "Kurt Shuster", "Daniel Simig", "Punit Singh Koura", "Anjali Sridhar", "Tianlu Wang", "Luke Zettlemoyer"]
year: 2022
venue: "arXiv:2205.01068"
source: "https://arxiv.org/abs/2205.01068"
doi: "10.48550/arxiv.2205.01068"
arxiv: "2205.01068"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / OPT: Open Pre-trained Transformer Language Models

# OPT: Open Pre-trained Transformer Language Models

**Authors:** [Susan Zhang](https://md.research.elamur.ai/free/en/authors/susan-zhang/), [Stephen Roller](https://md.research.elamur.ai/free/en/authors/stephen-roller/), [Naman Goyal](https://md.research.elamur.ai/free/en/authors/naman-goyal/), [Mikel Artetxe](https://md.research.elamur.ai/free/en/authors/mikel-artetxe/), [Moya Chen](https://md.research.elamur.ai/free/en/authors/moya-chen/), [Shuohui Chen](https://md.research.elamur.ai/free/en/authors/shuohui-chen/), Christopher Dewan, [Mona Diab](https://md.research.elamur.ai/free/en/authors/mona-diab/), [Xian Li](https://md.research.elamur.ai/free/en/authors/xian-li/), [Xi Victoria Lin](https://md.research.elamur.ai/free/en/authors/xi-victoria-lin/), [Todor Mihaylov](https://md.research.elamur.ai/free/en/authors/todor-mihaylov/), [Myle Ott](https://md.research.elamur.ai/free/en/authors/myle-ott/), [Sam Shleifer](https://md.research.elamur.ai/free/en/authors/sam-shleifer/), [Kurt Shuster](https://md.research.elamur.ai/free/en/authors/kurt-shuster/), [Daniel Simig](https://md.research.elamur.ai/free/en/authors/daniel-simig/), Punit Singh Koura, Anjali Sridhar, [Tianlu Wang](https://md.research.elamur.ai/free/en/authors/tianlu-wang/) and [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2022 · **Venue:** arXiv:2205.01068 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2205.01068) · [PDF](https://arxiv.org/pdf/2205.01068) · [DOI 10.48550/arxiv.2205.01068](https://doi.org/10.48550/arxiv.2205.01068) · [arXiv 2205.01068](https://arxiv.org/abs/2205.01068)

**Citations:** [1709](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The largest language models are very expensive to train, so only a few labs can reproduce them. Models available through paid APIs do not provide access to their weights, which makes them hard to study. The authors from Meta AI created OPT, a suite of autoregressive decoder-only transformers ranging from 125M to 175B parameters. The architecture and hyperparameters largely follow GPT-3. The pretraining corpus of approximately 180B tokens was assembled from parts of the RoBERTa corpus, the Pile, and PushShift.io Reddit. Duplicates were removed, and the text was tokenized with the GPT-2 BPE. The largest model, OPT-175B, was trained on 992 GPUs. The report candidly describes hardware failures and loss divergences, as well as the manual interventions used to overcome them. On standard zero- and few-shot benchmarks, OPT-175B roughly matches GPT-3, while the carbon footprint of its development is only about one-seventh that of GPT-3. The work is significant because it was the first to give the broad research community a GPT-3-scale model together with code and a detailed training logbook. This makes it possible to reproduce results and to study bias, toxicity, and robustness in LLMs.

## Key points
- **Model sizes:** 125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B, and 175B parameters. Models up to 66B are released openly, while researchers from academia, civil society, government organizations, and industry labs can obtain access to OPT-175B upon request.
- **Training:** 992 A100 GPUs (80GB), Fully Sharded Data Parallel combined with Megatron-LM tensor parallelism, up to 147 TFLOP/s per GPU. AdamW optimizer, sequence length 2048, ReLU activation.
- **Data:** BookCorpus, Stories, CC-News v2, selected subsets of the Pile (including CommonCrawl, Wikipedia, Project Gutenberg, OpenWebText2, USPTO), and PushShift.io Reddit. For Reddit, only the longest comment chain from each thread was kept. Duplicates were removed using MinhashLSH with a Jaccard similarity threshold of ≥ 0.95. The authors note that the Pile contains a particularly large number of duplicates.
- **Training instability:** over approximately 2 months there were at least 35 manual and more than 70 automatic restarts, and more than 100 hosts were replaced. Loss divergences were handled by lowering the learning rate, rolling back to an earlier checkpoint, and reducing the gradient clipping threshold from 1.0 to 0.3.
- **Evaluation and resources:** evaluation covered 16 NLP tasks (HellaSwag, StoryCloze, PIQA, ARC, OpenBookQA, WinoGrande, SuperGLUE, etc.) following the GPT-3 protocol. In the zero-shot setting, OPT-175B roughly matched GPT-3 on 10 tasks. The metaseq codebase and the training logbook were released together with the models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

Cite as: Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01068.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2022opt,
  title = {{OPT: Open Pre-trained Transformer Language Models}},
  author = {Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang and Luke Zettlemoyer},
  year = {2022},
  doi = {10.48550/arxiv.2205.01068},
  eprint = {2205.01068},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.01068}
}
```

- [← previous BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://md.research.elamur.ai/free/en/papers/grootendorst-2022-bertopic-neural-topic-modeling-class/)
- [next → Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

98 references, 45 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977. [our card](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/)
2. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeff Wang, Luke Zettle… [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Shruti+Bhosale%2C+Naman+Goyal%2C+Todor+Mihaylov%2C+Myle+Ott%2C+Sam+Shleifer%2C+Xi+Victoria+Lin%2C+Jingfei+Du%2C+Srinivasan+Iyer%2C+Ramakanth+Pasunuru%2C+Giri+Anantharaman%2C+Xian+Li%2C+Shuohui+Chen%2C+Halil+Ak)
3. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. CoRR, abs/2001.08435. — [arXiv:2001.08435](https://arxiv.org/abs/2001.08435) [find](https://scholar.google.com/scholar?q=Jason+Baumgartner%2C+Savvas+Zannettou%2C+Brian+Keegan%2C+Megan+Squire%2C+and+Jeremy+Blackburn%2E+2020%2E+The+pushshift+reddit+dataset%2E+CoRR%2C+abs%2F2001%2E08435%2E+%E2%80%94+%5BarXiv%3A2001%2E08435%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2001%2E08435%29)
4. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. [find](https://scholar.google.com/scholar?q=Emily+M+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+2021%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference+)
5. Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439. [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
6. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neox-20b: An open-source autoregr… [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
7. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… [find](https://scholar.google.com/scholar?q=Su+Lin+Blodgett%2C+Gilsinia+Lopez%2C+Alexandra+Olteanu%2C+Robert+Sim%2C+and+Hanna+Wallach%2E+2021%2E+Stereotyping+Norwegian+salmon%3A+An+inventory+of+pitfalls+in+fairness+benchmark+datasets%2E+In+Proceedings+of+the+5)
8. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Doro… [our card](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
9. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2021. Improving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04426. [find](https://scholar.google.com/scholar?q=Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Jordan+Hoffmann%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Katie+Millican%2C+George+van+den+Driessche%2C+Jean%2DBaptiste+Lespiau%2C+Bogdan+Damoc%2C+Aidan+Clark%2C+et+al%2E+2021%2E+Improving+lan)
10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
11. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789–797, Columbus, Ohio. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nathanael+Chambers+and+Dan+Jurafsky%2E+2008%2E+Unsupervised+learning+of+narrative+event+chains%2E+In+Proceedings+of+ACL%2D08%3A+HLT%2C+pages+789%E2%80%93797%2C+Columbus%2C+Ohio%2E+Association+for+Computational+Linguistics%2E)
12. Ke-Li Chiu and Rohan Alexander. 2021. Detecting hate speech with gpt-3. arXiv preprint arXiv:2103.12407. — [arXiv:2103.12407](https://arxiv.org/abs/2103.12407) [find](https://scholar.google.com/scholar?q=Ke%2DLi+Chiu+and+Rohan+Alexander%2E+2021%2E+Detecting+hate+speech+with+gpt%2D3%2E+arXiv+preprint+arXiv%3A2103%2E12407%2E+%E2%80%94+%5BarXiv%3A2103%2E12407%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2103%2E12407%29)
13. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… [our card](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
14. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+AI2+reasoning+challenge%2E+CoR)
15. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164. [our card](https://md.research.elamur.ai/free/en/papers/dathathri-2019-plug-play-language-models-simple/)
16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Association for Computational Linguistics (NAACL). [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
17. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2020b. The second conversationa… [our card](https://md.research.elamur.ai/free/en/papers/dinan-2019-second-conversational-intelligence-challenge-convai2/)
18. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019b. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Emily+Dinan%2C+Stephen+Roller%2C+Kurt+Shuster%2C+Angela+Fan%2C+Michael+Auli%2C+and+Jason+Weston%2E+2019b%2E+Wizard+of+Wikipedia%3A+Knowledge%2Dpowered+conversational+agents%2E+In+Proceedings+of+the+International+Conferen)
19. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021a. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2021a%2E+The+pile%3A+An+800gb+data)
20. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021b. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume… [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/)
21. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Commun. ACM, 64(12):86–92. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daum%C3%A9+III%2C+and+Kate+Crawford%2E+2021%2E+Datasheets+for+datasets%2E+Commun%2E+ACM%2C+64%2812%29%3A86%E2%80%9392%2E)
22. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 862–872… [find](https://scholar.google.com/scholar?q=Jwala+Dhamala%2C+Tony+Sun%2C+Varun+Kumar%2C+Satyapriya+Krishna%2C+Yada+Pruksachatkun%2C+Kai%2DWei+Chang%2C+and+Rahul+Gupta%2E+2021%2E+Bold%3A+Dataset+and+metrics+for+measuring+biases+in+open%2Dended+language+generation%2E+In)
23. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti… [our card](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/)
24. Emily Dinan, Gavin Abercrombie, A Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2021. Anticipating safety issues in e2e conversational ai: Framework and tooling. arXiv preprint arXiv:2107.03451. — [arXiv:2107.03451](https://arxiv.org/abs/2107.03451) [find](https://scholar.google.com/scholar?q=Emily+Dinan%2C+Gavin+Abercrombie%2C+A+Stevie+Bergman%2C+Shannon+Spruit%2C+Dirk+Hovy%2C+Y%2DLan+Boureau%2C+and+Verena+Rieser%2E+2021%2E+Anticipating+safety+issues+in+e2e+conversational+ai%3A+Framework+and+tooling%2E+arXiv+p)
25. Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. 2021. Chasing carbon: The elusive environmental footprint of computing. IEEE International Symposium on High-Performance Computer Architecture (HPCA 2021). [find](https://scholar.google.com/scholar?q=Udit+Gupta%2C+Young+Geun+Kim%2C+Sylvia+Lee%2C+Jordan+Tse%2C+Hsien%2DHsin+S+Lee%2C+Gu%2DYeon+Wei%2C+David+Brooks%2C+and+Carole%2DJean+Wu%2E+2021%2E+Chasing+carbon%3A+The+elusive+environmental+footprint+of+computing%2E+IEEE+Intern)
26. Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020a. Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8173–8188, Online. Association… [our card](https://md.research.elamur.ai/free/en/papers/dinan-2020-queens-are-powerful-too-mitigating/)
27. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019a. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. arXiv preprint arXiv:1908.06083. [our card](https://md.research.elamur.ai/free/en/papers/dinan-2019-build-it-break-it-fix/)
28. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+770%E2%80%93+)
29. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
30. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. ArXiv, abs/1904.09751. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2020%2E+The+curious+case+of+neural+text+degeneration%2E+ArXiv%2C+abs%2F1904%2E09751%2E)
31. Abigail Z. Jacobs and Hanna Wallach. 2021. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 375–385, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Abigail+Z%2E+Jacobs+and+Hanna+Wallach%2E+2021%2E+Measurement+and+fairness%2E+In+Proceedings+of+the+2021+ACM+Conference+on+Fairness%2C+Accountability%2C+and+Transparency%2C+FAccT+%E2%80%9921%2C+page+375%E2%80%93385%2C+New+York%2C+NY%2C+USA)
32. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of language agents. CoRR, abs/2103.14659. [find](https://scholar.google.com/scholar?q=Zachary+Kenton%2C+Tom+Everitt%2C+Laura+Weidinger%2C+Iason+Gabriel%2C+Vladimir+Mikulik%2C+and+Geoffrey+Irving%2E+2021%2E+Alignment+of+language+agents%2E+CoRR%2C+abs%2F2103%2E14659%2E)
33. Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021. Internet-augmented dialogue generation. CoRR, abs/2107.07566. [our card](https://md.research.elamur.ai/free/en/papers/komeili-2022-internet-augmented-dialogue-generation/)
34. Ben Krause, Akhilesh Deepak Gotmare, Bryan Mc-Cann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020. GEDI: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367. [our card](https://md.research.elamur.ai/free/en/papers/krause-2021-gedi-generative-discriminator-guided-sequence/)
35. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. CoRR, abs/2104.08691. [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+2021%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+CoRR%2C+abs%2F2104%2E08691%2E)
36. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, page 47. [find](https://scholar.google.com/scholar?q=Hector+J+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2011%2E+The+Winograd+schema+challenge%2E+In+AAAI+Spring+Symposium%3A+Logical+Formalizations+of+Commonsense+Reasoning%2C+volume+46%2C+page+47%2E)
37. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ethan+Perez%2C+Aleksandra+Piktus%2C+Fabio+Petroni%2C+Vladimir+Karpukhin%2C+Naman+Goyal%2C+Heinrich+K%C3%BCttler%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Tim+Rockt%C3%A4schel%2C+et+al%2E+2020%2E+Retrieval%2Daugmented+generation+f)
38. Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. pages 4582–4597. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2DTuning%3A+Optimizing+Continuous+Prompts+for+Generation%2E+pages+4582%E2%80%934597%2E)
39. Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565–6576. PMLR. [our card](https://md.research.elamur.ai/free/en/papers/liang-2021-towards-understanding-mitigating-social-biases/)
40. Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic-1: Technical details and evaluation. Technical report, AI21 Labs. [find](https://scholar.google.com/scholar?q=Opher+Lieber%2C+Or+Sharir%2C+Barak+Lenz%2C+and+Yoav+Shoham%2E+2021%2E+Jurassic%2D1%3A+Technical+details+and+evaluation%2E+Technical+report%2C+AI21+Labs%2E)
41. Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2019a. Does gender matter? towards fairness in dialogue systems. arXiv preprint arXiv:1910.10486. [find](https://scholar.google.com/scholar?q=Haochen+Liu%2C+Jamell+Dacon%2C+Wenqi+Fan%2C+Hui+Liu%2C+Zitao+Liu%2C+and+Jiliang+Tang%2E+2019a%2E+Does+gender+matter%3F+towards+fairness+in+dialogue+systems%2E+arXiv+preprint+arXiv%3A1910%2E10486%2E)
42. Haokun Liu, William Huang, Dhara Mungra, and Samuel R. Bowman. 2020. Precise task formalization matters in Winograd schema evaluations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8275–8280, Online. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Haokun+Liu%2C+William+Huang%2C+Dhara+Mungra%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+Precise+task+formalization+matters+in+Winograd+schema+evaluations%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Na)
43. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? CoRR, abs/2101.06804. [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/)
44. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+)
45. Ilya Loshchilov and Frank Hutter. 2017. weight decay regularization in adam. abs/1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+weight+decay+regularization+in+adam%2E+abs%2F1711%2E05101%2E)
46. Fixing CoRR, Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/)
47. Clara Meister, Tim Vieira, and Ryan Cotterell. 2020. Best-first beam search. Transactions of the Association for Computational Linguistics, 8:795–809. [find](https://scholar.google.com/scholar?q=Clara+Meister%2C+Tim+Vieira%2C+and+Ryan+Cotterell%2E+2020%2E+Best%2Dfirst+beam+search%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A795%E2%80%93809%2E)
48. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2017. Mixed precision training. arXiv preprint arXiv:1710.03740. [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Sharan+Narang%2C+Jonah+Alben%2C+Gregory+Diamos%2C+Erich+Elsen%2C+David+Garcia%2C+Boris+Ginsburg%2C+Michael+Houston%2C+Oleksii+Kuchaiev%2C+Ganesh+Venkatesh%2C+et+al%2E+2017%2E+Mixed+precision+training%2E)
49. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. CoRR, abs/1809.02789. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+A+new+dataset+for+open+book+question+answering%2E+CoRR%2C+abs%2F1809%2E02789%2E)
50. Tomas Mikolov, Jiri Kopecky, Lukas Burget, Ondrej Glembek, et al. 2009. Neural network based language models for highly inflective languages. In 2009 IEEE international conference on acoustics, speech and signal processing, pages 4725–4728. IEEE. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Jiri+Kopecky%2C+Lukas+Burget%2C+Ondrej+Glembek%2C+et+al%2E+2009%2E+Neural+network+based+language+models+for+highly+inflective+languages%2E+In+2009+IEEE+international+conference+on+acoustics%2C+speech)
51. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+2021%2E+Metaicl%3A+Learning+to+learn+in+context%2E)
52. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837. [our card](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/)
53. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2018. Model cards for model reporting. CoRR, abs/1810.03993. [find](https://scholar.google.com/scholar?q=Margaret+Mitchell%2C+Simone+Wu%2C+Andrew+Zaldivar%2C+Parker+Barnes%2C+Lucy+Vasserman%2C+Ben+Hutchinson%2C+Elena+Spitzer%2C+Inioluwa+Deborah+Raji%2C+and+Timnit+Gebru%2E+2018%2E+Model+cards+for+model+reporting%2E+CoRR%2C+abs%2F1)
54. Ioannis Mollas, Zoe Chrysopoulou, Stamatis Kar-los, and Grigorios Tsoumakas. 2020. ETHOS: an online hate speech detection dataset. CoRR, abs/2006.08328. — [arXiv:2006.08328](https://arxiv.org/abs/2006.08328) [find](https://scholar.google.com/scholar?q=Ioannis+Mollas%2C+Zoe+Chrysopoulou%2C+Stamatis+Kar%2Dlos%2C+and+Grigorios+Tsoumakas%2E+2020%2E+ETHOS%3A+an+online+hate+speech+detection+dataset%2E+CoRR%2C+abs%2F2006%2E08328%2E+%E2%80%94+%5BarXiv%3A2006%2E08328%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006)
55. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2C+OpenAI%2E)
56. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. CoRR, abs/1604.01696. [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+F%2E+Allen%2E+2016%2E+A+corpus+and+evaluation+framework+for+deeper+understanding+o)
57. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-S… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+H%2E+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick)
58. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pre-trained language models. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Moin+Nadeem%2C+Anna+Bethke%2C+and+Siva+Reddy%2E+2021%2E+StereoSet%3A+Measuring+stereotypical+bias+in+pre%2Dtrained+language+models%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E)
59. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
60. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. arXiv preprint arXiv:2010.00133. [our card](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/)
61. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A conversational paradigm for program synthesis. arXiv preprint. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2022%2E+A+conversational+paradigm+for+program+synthesis%2E+arXiv+preprint%2E)
62. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. [our card](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
63. David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350. [our card](https://md.research.elamur.ai/free/en/papers/patterson-2021-carbon-emissions-large-neural-network/)
64. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Advances in Neural Information Processing Systems, 34. [our card](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)
65. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
66. Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Time+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+with+unsupervised+learning%2E+Technical+report%2C+OpenAI%2E)
67. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research (JMLR), 21:1–67. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
68. Anand Rajaraman and Jeffrey David Ullman. 2011. Mining of massive datasets. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Anand+Rajaraman+and+Jeffrey+David+Ullman%2E+2011%2E+Mining+of+massive+datasets%2E+Cambridge+University+Press%2E)
69. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Com… [find](https://scholar.google.com/scholar?q=Hannah+Rashkin%2C+Eric+Michael+Smith%2C+Margaret+Li%2C+and+Y%2DLan+Boureau%2E+2019%2E+Towards+empathetic+open%2Ddomain+conversation+models%3A+A+new+benchmark+and+dataset%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+)
70. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational… [our card](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/)
71. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAA… [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+In+The+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intellige)
72. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Da… [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
73. Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. arXiv preprint arXiv:2203.13224. [find](https://scholar.google.com/scholar?q=Kurt+Shuster%2C+Mojtaba+Komeili%2C+Leonard+Adolphs%2C+Stephen+Roller%2C+Arthur+Szlam%2C+and+Jason+Weston%2E+2022%2E+Language+models+that+seek+for+knowledge%3A+Modular+search+%26+generation+for+dialogue+and+prompt+compl)
74. Eric Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. Can you put it all together: Evaluating conversational agents’ ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL. [our card](https://md.research.elamur.ai/free/en/papers/smith-2020-can-you-put-it-all/)
75. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwa… [our card](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
76. Teven Le Scao and Alexander M. Rush. 2021. How many data points is a prompt worth? pages 2627– 2636. [our card](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/)
77. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239. [our card](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
78. Timo Schick and Hinrich Schütze. 2020. It’s not just size that matters: Small language models are also few-shot learners. CoRR, abs/2009.07118. [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/)
79. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. CoRR, abs/1806.02847. [find](https://scholar.google.com/scholar?q=Trieu+H%2E+Trinh+and+Quoc+V%2E+Le%2E+2018%2E+A+simple+method+for+commonsense+reasoning%2E+CoRR%2C+abs%2F1806%2E02847%2E)
80. Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguistics, 9:1408–1424. [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/)
81. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
82. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326. [our card](https://md.research.elamur.ai/free/en/papers/sheng-2019-woman-worked-as-babysitter-biases/)
83. Megan Ung, Jing Xu, and Y-Lan Boureau. 2021. Saferdialogues: Taking feedback gracefully after conversational safety failures. ArXiv, abs/2110.07518. [find](https://scholar.google.com/scholar?q=Megan+Ung%2C+Jing+Xu%2C+and+Y%2DLan+Boureau%2E+2021%2E+Saferdialogues%3A+Taking+feedback+gracefully+after+conversational+safety+failures%2E+ArXiv%2C+abs%2F2110%2E07518%2E)
84. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
85. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537. [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
86. Albert Webson and Ellie Pavlick. 2021. Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247. [our card](https://md.research.elamur.ai/free/en/papers/webson-2022-do-prompt-based-models-really/)
87. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. pages 4222– 4235. [our card](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/)
88. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. CoRR, abs/2109.01652. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y%2E+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M%2E+Dai%2C+and+Quoc+V%2E+Le%2E+2021%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+CoRR%2C+abs%2F2109%2E01652%2E)
89. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+2019%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+p)
90. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean L… [our card](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)
91. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021b. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. [our card](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)
92. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In International Conference on Learning Representations. [our card](https://md.research.elamur.ai/free/en/papers/welleck-2019-neural-text-generation-unlikelihood-training/)
93. Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin S. Lee, Bugra Akyi… [find](https://scholar.google.com/scholar?q=Carole%2DJean+Wu%2C+Ramya+Raghavendra%2C+Udit+Gupta%2C+Bilge+Acun%2C+Newsha+Ardalani%2C+Kiwan+Maeng%2C+Gloria+Chang%2C+Fiona+Aga+Behram%2C+James+Huang%2C+Charles+Bai%2C+Michael+Gschwind%2C+Anurag+Gupta%2C+Myle+Ott%2C+Anastasia+M)
94. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079. [find](https://scholar.google.com/scholar?q=Jing+Xu%2C+Da+Ju%2C+Margaret+Li%2C+Y%2DLan+Boureau%2C+Jason+Weston%2C+and+Emily+Dinan%2E+2020%2E+Recipes+for+safety+in+open%2Ddomain+chatbots%2E+arXiv+preprint+arXiv%3A2010%2E07079%2E)
95. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021a. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950–296… [find](https://scholar.google.com/scholar?q=Jing+Xu%2C+Da+Ju%2C+Margaret+Li%2C+Y%2DLan+Boureau%2C+Jason+Weston%2C+and+Emily+Dinan%2E+2021a%2E+Bot%2Dadversarial+dialogue+for+safe+conversational+agents%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+C)
96. Jing Xu, Arthur Szlam, and Jason Weston. 2021b. Beyond goldfish memory: Long-term open-domain conversation. arXiv preprint arXiv:2107.07567. [find](https://scholar.google.com/scholar?q=Jing+Xu%2C+Arthur+Szlam%2C+and+Jason+Weston%2E+2021b%2E+Beyond+goldfish+memory%3A+Long%2Dterm+open%2Ddomain+conversation%2E+arXiv+preprint+arXiv%3A2107%2E07567%2E)
97. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Conference+of+the+Association+for+Computa)
98. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. CoRR, abs/1506.06724. [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)