---
title: "Scaling Instruction-Finetuned Language Models — Chung 2022"
description: "The authors from Google seek a way to teach pretrained language models to better handle tasks they have not seen before."
authors: ["Hyung Won Chung", "Le Hou", "Shayne Longpre", "Barret Zoph", "Yi Tay", "William Fedus", "Yunxuan Li", "Xuezhi Wang", "Mostafa Dehghani", "Siddhartha Brahma", "Albert Webson", "Shixiang Shane Gu", "Zhuyun Dai", "Mirac Suzgun", "Xinyun Chen", "Aakanksha Chowdhery", "Alex Castro-Ros", "Marie Pellat", "Kevin Robinson", "Dasha Valter", "Sharan Narang", "Gaurav Mishra", "Adams Yu", "Vincent Zhao", "Yanping Huang", "Andrew Dai", "Hongkun Yu", "Slav Petrov", "Ed H. Chi", "Jeff Dean", "Jacob Devlin", "Adam Roberts", "Denny Zhou", "Quoc V. Le", "Jason Wei"]
year: 2022
venue: "arXiv:2210.11416"
source: "https://arxiv.org/abs/2210.11416"
doi: "10.48550/arxiv.2210.11416"
arxiv: "2210.11416"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / Scaling Instruction-Finetuned Language Models

# Scaling Instruction-Finetuned Language Models

**Authors:** [Hyung Won Chung](https://md.research.elamur.ai/free/en/authors/hyung-won-chung/), [Le Hou](https://md.research.elamur.ai/free/en/authors/le-hou/), [Shayne Longpre](https://md.research.elamur.ai/free/en/authors/shayne-longpre/), [Barret Zoph](https://md.research.elamur.ai/free/en/authors/barret-zoph/), [Yi Tay](https://md.research.elamur.ai/free/en/authors/yi-tay/), [William Fedus](https://md.research.elamur.ai/free/en/authors/william-fedus/), [Yunxuan Li](https://md.research.elamur.ai/free/en/authors/yunxuan-li/), [Xuezhi Wang](https://md.research.elamur.ai/free/en/authors/xuezhi-wang/), [Mostafa Dehghani](https://md.research.elamur.ai/free/en/authors/mostafa-dehghani/), [Siddhartha Brahma](https://md.research.elamur.ai/free/en/authors/siddhartha-brahma/), [Albert Webson](https://md.research.elamur.ai/free/en/authors/albert-webson/), [Shixiang Shane Gu](https://md.research.elamur.ai/free/en/authors/shixiang-shane-gu/) et al. (35 authors) · **Year:** 2022 · **Venue:** arXiv:2210.11416 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2210.11416) · [PDF](https://arxiv.org/pdf/2210.11416) · [DOI 10.48550/arxiv.2210.11416](https://doi.org/10.48550/arxiv.2210.11416) · [arXiv 2210.11416](https://arxiv.org/abs/2210.11416)

**Citations:** [2176](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The authors from Google seek a way to teach pretrained language models to better handle tasks they have not seen before. To this end, the models are finetuned on a large collection of tasks phrased as instructions (the Flan procedure), and three scaling directions are examined: the number of tasks, the model size, and the addition of examples with step-by-step reasoning (CoT). The collection was assembled from four sets: Muffin, T0-SF, NIV2 and CoT. Together, these amount to 1 836 tasks from 473 datasets. The examples are presented in four formats: with and without exemplars, and with and without reasoning. The method was applied to T5, PaLM and U-PaLM ranging in size from 80M to 540B parameters. Results improve with both the number of tasks and the model size. Just nine CoT datasets in the mixture prevent degradation of reasoning while also improving results across all evaluations. Flan-PaLM 540B achieves 75.2% on MMLU (5-shot) and outperforms PaLM by 9.4% on average. The paper demonstrates a cheap, general-purpose way to improve any model, and the open Flan-T5 checkpoints can be used in one's own projects.

## Key points
- **Mixture composition:** Muffin (80 tasks), T0-SF (193), NIV2 (1 554) and CoT (9). In total, 1 836 tasks in 146 categories. 44 tasks related to MMLU were removed from NIV2 to keep the evaluation fair.
- **Flan-PaLM 540B records:**
  - 75.2% on MMLU with CoT and self-consistency (PaLM: 69.3%);
  - +14.9% on one-shot TyDiQA;
  - +8.1% on MGSM for languages that are underrepresented in the data.
- **Low cost:** finetuning PaLM 540B requires only about 0.2% of the compute spent on pretraining.
- **A smaller finetuned model can outperform a larger one without finetuning:**
  - Flan-PaLM 62B surpasses PaLM 540B on TyDiQA (58.7 vs. 52.9 EM);
  - Flan-T5 11B surpasses PaLM 62B on BBH-direct (43.7 vs. 37.5).
- **Open resources and compatibility:** Flan-T5 checkpoints ranging from 80M to 11B have been released. The method combines well with UL2R: Flan-U-PaLM achieved the best results among all models in the paper.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chung2022scaling,
  title = {{Scaling Instruction-Finetuned Language Models}},
  author = {Chung, Hyung Won and Hou, Le and Longpre, Shayne and Zoph, Barret and Tay, Yi and Fedus, William and Li, Yunxuan and Wang, Xuezhi and Dehghani, Mostafa and Brahma, Siddhartha and Webson, Albert and Gu, Shixiang Shane and Dai, Zhuyun and Suzgun, Mirac and Chen, Xinyun and Chowdhery, Aakanksha and Castro-Ros, Alex and Pellat, Marie and Robinson, Kevin and Valter, Dasha and Narang, Sharan and Mishra, Gaurav and Yu, Adams and Zhao, Vincent and Huang, Yanping and Dai, Andrew and Yu, Hongkun and Petrov, Slav and Chi, Ed H. and Dean, Jeff and Devlin, Jacob and Roberts, Adam and Zhou, Denny and Le, Quoc V. and Wei, Jason},
  year = {2022},
  doi = {10.48550/arxiv.2210.11416},
  eprint = {2210.11416},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.11416}
}
```

- [← previous Emergent Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
- [next → Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

96 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. Explanations for CommonsenseQA: New Dataset and Models. In ACL, 2021. URL https://aclanthology.org/2021.acl-long.238. [↗](https://aclanthology.org/2021.acl-long.238) [our card](https://md.research.elamur.ai/free/en/papers/aggarwal-2021-explanations-commonsenseqa-new-dataset-models/)
2. Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In EMNLP, 2021. URL https://aclanthology.org/2021.emnlp-main.468. [↗](https://aclanthology.org/2021.emnlp-main.468) [our card](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-muppet-massive-multi-task-representations/)
3. Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. Open-domain question answering goes conversational via question rewriting. In NAACL, 2021. URL https://aclanthology.org/2021.naacl-main.44/. [↗](https://aclanthology.org/2021.naacl-main.44/) [our card](https://md.research.elamur.ai/free/en/papers/anantha-2021-open-domain-question-answering-goes/)
4. Anja Austermann, Michelle Linch, Romina Stella, and Kellie Webster. Translated wikipeda biographies. [https://research.google/tools/datasets/translated-wikipedia-biographies/](https://research.google/tools/datasets/translated-wikipedia-biographies/), 2021. Accessed: 2022-11-17. [↗](https://research.google/tools/datasets/translated-wikipedia-biographies/)
5. Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanp… — [arXiv:2205.03983](https://arxiv.org/abs/2205.03983) [find](https://scholar.google.com/scholar?q=Ankur+Bapna%2C+Isaac+Caswell%2C+Julia+Kreutzer%2C+Orhan+Firat%2C+Daan+van+Esch%2C+Aditya+Siddhant%2C+Mengmeng+Niu%2C+Pallavi+Baljekar%2C+Xavier+Garcia%2C+Wolfgang+Macherey%2C+Theresa+Breiner%2C+Vera+Axelrod%2C+Jason+Riesa%2C+Y)
6. Connor Baumler and Rachel Rudinger. Recognition of they/them as singular personal pronouns in coreference resolution. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3426–3432, Seattle, United St… [find](https://scholar.google.com/scholar?q=Connor+Baumler+and+Rachel+Rudinger%2E+Recognition+of+they%2Fthem+as+singular+personal+pronouns+in+coreference+resolution%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Associa) [our card](https://md.research.elamur.ai/free/en/papers/baumler-2022-recognition-they-them-as-singular/)
7. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA, 2021. Associat… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference+on+Fa)
8. Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. Re-contextualizing fairness in nlp: The case of india, 2022. URL https://arxiv.org/abs/2209.12226. [↗](https://arxiv.org/abs/2209.12226)
9. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. [find](https://scholar.google.com/scholar?q=Rishi+Bommasani%2C+Drew+A+Hudson%2C+Ehsan+Adeli%2C+Russ+Altman%2C+Simran+Arora%2C+Sydney+von+Arx%2C+Michael+S+Bernstein%2C+Jeannette+Bohg%2C+Antoine+Bosselut%2C+Emma+Brunskill%2C+et+al%2E+On+the+opportunities+and+risks+of+) [our card](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
10. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification, 2019a. URL https://arxiv.org/abs/ 1903.04561. [↗](https://arxiv.org/abs/)
11. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification, 2019b. URL https://arxiv.org/abs/ 1903.04561. [↗](https://arxiv.org/abs/)
12. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL [http://github.com/google/jax](http://github.com/google/jax). [↗](http://github.com/google/jax)
13. Philip A.E. Brey. Anticipatory ethics for emerging technologies. NanoEthics, 6(1):1–13, 2012. ISSN 1871-4757. doi: 10.1007/s11569-012-0141-7. — [doi:10.1007/s11569-012-0141-7](https://doi.org/10.1007/s11569-012-0141-7) [find](https://scholar.google.com/scholar?q=Philip+A%2EE%2E+Brey%2E+Anticipatory+ethics+for+emerging+technologies%2E+NanoEthics%2C+6%281%29%3A1%E2%80%9313%2C+2012%2E+ISSN+1871%2D4757%2E+doi%3A+10%2E1007%2Fs11569%2D012%2D0141%2D7%2E+%E2%80%94+%5Bdoi%3A10%2E1007%2Fs11569%2D012%2D0141%2D7%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1007%2F)
14. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chr… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D%2E+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche) [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
15. Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. Taskmaster-1: Toward a realistic and diverse dialog dataset. In EMNLP, 2019. doi: 10.18653/v1/D19-1459. URL [https://aclanthology.or…](https://aclanthology.or…) [↗](https://aclanthology.or…) [our card](https://md.research.elamur.ai/free/en/papers/byrne-2019-taskmaster-1-realistic-diverse-dialog/)
16. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-SNLI: Natural language inference with natural language explanations. NeurIPS, 2018. URL [https://arxiv.org/pdf/1812.01193](https://arxiv.org/pdf/1812.01193). pdf. [↗](https://arxiv.org/pdf/1812.01193) [our card](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/)
17. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. Make up your mind! Adversarial generation of inconsistent natural language explanations. In ACL, 2020. URL https://aclanthology.org/2020.acl-main.382. [↗](https://aclanthology.org/2020.acl-main.382)
18. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. [↗](https://arxiv.org/abs/2107.03374) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
19. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, et al. PaLM: Scaling language modeling with Pathways. arXiv preprint arXiv:2204.02311, 2022. URL https://arxiv.org/abs/2204.02311. [↗](https://arxiv.org/abs/2204.02311) [our card](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
20. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDiQA: A benchmark for information-seeking question answering in typologically diverse languages. TACL, 2020. URL https://aclanthology.org/2020.tacl-1.30. [↗](https://aclanthology.org/2020.tacl-1.30) [our card](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)
21. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
22. Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Y Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, and Kelvin Guu. Dialog inpainting: Turning documents into dialogs. In ICML, 2022. URL [https://proceedings.mlr.press/v162/dai22a.html](https://proceedings.mlr.press/v162/dai22a.html). [↗](https://proceedings.mlr.press/v162/dai22a.html)
23. Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff M Phillips, and Kai-Wei Chang. Harms of gender exclusivity and challenges in non-binary representation in language technologies. 2021a. doi: 10.48550/ARXIV.2108.12084. URL https://arxiv.org/abs/2108.12084. [↗](https://arxiv.org/abs/2108.12084) [our card](https://md.research.elamur.ai/free/en/papers/dev-2021-harms-gender-exclusivity-challenges-non/)
24. Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, and Kai-Wei Chang. On measures of biases and harms in nlp, 2021b. URL https://arxiv.org/abs/2108.03362. [↗](https://arxiv.org/abs/2108.03362)
25. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas… [find](https://scholar.google.com/scholar?q=Nan+Du%2C+Yanping+Huang%2C+Andrew+M%2E+Dai%2C+Simon+Tong%2C+Dmitry+Lepikhin%2C+Yuanzhong+Xu%2C+Maxim+Krikun%2C+Yanqi+Zhou%2C+Adams+Wei+Yu%2C+Orhan+Firat%2C+Barret+Zoph%2C+Liam+Fedus%2C+Maarten+Bosma%2C+Zongwei+Zhou%2C+Tao+Wang%2C+Yu) [our card](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
26. Tanmay Garg, Sarah Masud, Tharun Suresh, and Tanmoy Chakraborty. Handling bias in toxic speech detection: A survey, 2022. URL https://arxiv.org/abs/2202.00126. — [arXiv:2202.00126](https://arxiv.org/abs/2202.00126) [↗](https://arxiv.org/abs/2202.00126)
27. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models, 2020. [find](https://scholar.google.com/scholar?q=Samuel+Gehman%2C+Suchin+Gururangan%2C+Maarten+Sap%2C+Yejin+Choi%2C+and+Noah+A%2E+Smith%2E+Realtoxicityprompts%3A+Evaluating+neural+toxic+degeneration+in+language+models%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/)
28. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. TACL, 2021. doi: 10.1162/ tacl_a_00370. URL https://aclanthology.org/2021.tacl-1.21. [↗](https://aclanthology.org/2021.tacl-1.21)
29. Nitesh Goyal, Ian Kivlichan, Rachel Rosen, and Lucy Vasserman. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation, 2022. URL https://arxiv.org/abs/2205.00501. — [arXiv:2205.00501](https://arxiv.org/abs/2205.00501) [↗](https://arxiv.org/abs/2205.00501)
30. Saga Hansson, Konstantinos Mavromatakis, Yvonne Adesam, Gerlof Bouma, and Dana Dannélls. The Swedish Winogender dataset. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 452–459, Reykjavik, Iceland (Online), May 31–2 June 2021. Linköping University Electron… [find](https://scholar.google.com/scholar?q=Saga+Hansson%2C+Konstantinos+Mavromatakis%2C+Yvonne+Adesam%2C+Gerlof+Bouma%2C+and+Dana+Dann%C3%A9lls%2E+The+Swedish+Winogender+dataset%2E+In+Proceedings+of+the+23rd+Nordic+Conference+on+Computational+Linguistics+%28NoDa)
31. Ali Hasan, Shea Brown, Jovana Davidovic, Benjamin Lange, and Mitt Regan. Algorithmic bias and risk assessments: Lessons from practice. Digital Society, 1(1):1–15, 2022. [find](https://scholar.google.com/scholar?q=Ali+Hasan%2C+Shea+Brown%2C+Jovana+Davidovic%2C+Benjamin+Lange%2C+and+Mitt+Regan%2E+Algorithmic+bias+and+risk+assessments%3A+Lessons+from+practice%2E+Digital+Society%2C+1%281%29%3A1%E2%80%9315%2C+2022%2E)
32. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ICLR, 2020. URL [https://openreview.net/](https://openreview.net/) forum?id=d7KBjmI3GmQ. [↗](https://openreview.net/) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
33. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann%2C+Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Elena+Buchatskaya%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Diego+de+Las+Casas%2C+Lisa+Anne+Hendricks%2C+Johannes+Welbl%2C+Aidan+Clark%2C+et+al%2E+Training+compute%2Doptim) [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
34. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In arxiv, 2022. URL [https://openreview.net/forum?id=NiEtU7blzN](https://openreview.net/forum?id=NiEtU7blzN). [↗](https://openreview.net/forum?id=NiEtU7blzN) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/)
35. Norman P Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63(7):67–78, 2020. [find](https://scholar.google.com/scholar?q=Norman+P+Jouppi%2C+Doe+Hyun+Yoon%2C+George+Kurian%2C+Sheng+Li%2C+Nishant+Patil%2C+James+Laudon%2C+Cliff+Young%2C+and+David+Patterson%2E+A+domain%2Dspecific+supercomputer+for+training+deep+neural+networks%2E+Communication)
36. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. [↗](https://arxiv.org/abs/2001.08361) [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
37. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UnifiedQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020. URL https://aclanthology.org/2020.findings-e… [↗](https://aclanthology.org/2020.findings-e…) [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/)
38. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. QASC: A dataset for question answering via sentence composition. In AAAI, 2020. URL https://arxiv.org/abs/1910.11473. [↗](https://arxiv.org/abs/1910.11473)
39. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. NeurIPS, 2022. URL https://arxiv.org/abs/2205.11916. [↗](https://arxiv.org/abs/2205.11916) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
40. Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. QED: A framework and dataset for explanations in question answering. TACL, 9:790–806, 2021. URL https://arxiv.org/abs/2009.06354. [↗](https://arxiv.org/abs/2009.06354)
41. Chelsea Lee. Welcome, singular "they". [https://apastyle.apa.org/blog/singular-they](https://apastyle.apa.org/blog/singular-they), 2019. Accessed: 2022-11-18. [↗](https://apastyle.apa.org/blog/singular-they)
42. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. EMNLP, 2021. doi: 10.18653/v1/2021.emnlp-main.243. URL https://aclanthology.org/2021. emnlp-main.243. [↗](https://aclanthology.org/2021)
43. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019. URL [https://arxiv.…](https://arxiv.…) [↗](https://arxiv.…) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
44. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. URL https://arxiv.org/abs/ 2206.14858… [↗](https://arxiv.org/abs/) [our card](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/)
45. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814, 2022. URL http://go/arxiv/2203.07814. [find](https://scholar.google.com/scholar?q=Yujia+Li%2C+David+Choi%2C+Junyoung+Chung%2C+Nate+Kushman%2C+Julian+Schrittwieser%2C+R%C3%A9mi+Leblond%2C+Tom+Eccles%2C+James+Keeling%2C+Felix+Gimeno%2C+Agustin+Dal+Lago%2C+et+al%2E+Competition%2Dlevel+code+generation+with+alphaco) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
46. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. ACL, 2017. URL https://aclanthology.org/ P17-1015. [↗](https://aclanthology.org/)
47. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. URL https://arxiv.org/abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
48. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICL: Learning to learn in context. In NAACL, 2022. URL https://aclanthology.org/2022.naacl-main.201. [↗](https://aclanthology.org/2022.naacl-main.201)
49. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. ACL, 2021. URL https://arxiv.org/abs/2104.08773. [↗](https://arxiv.org/abs/2104.08773)
50. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019. [find](https://scholar.google.com/scholar?q=Margaret+Mitchell%2C+Simone+Wu%2C+Andrew+Zaldivar%2C+Parker+Barnes%2C+Lucy+Vasserman%2C+Ben+Hutchinson%2C+Elena+Spitzer%2C+Inioluwa+Deborah+Raji%2C+and+Timnit+Gebru%2E+Model+cards+for+model+reporting%2E+In+Proceedings+of)
51. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. URL [http://arxiv.o…](http://arxiv.o…) [↗](http://arxiv.o…) [our card](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/)
52. Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. CREAK: A dataset for commonsense reasoning over entity knowledge. In NeurIPS Datasets and Benchmarks Track (Round 2), 2021. URL https://arxiv.org/abs/2109.01653. — [arXiv:2109.01653](https://arxiv.org/abs/2109.01653) [↗](https://arxiv.org/abs/2109.01653)
53. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. URL https://arxiv.org/abs/2203.02155. [↗](https://arxiv.org/abs/2203.02155) [our card](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
54. Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, and George Karypis. Exploring the role of task transferability in large-scale multi-task learning. arXiv preprint arXiv:2204.11117, 2022. URL https://aclanthology.org/2022.naacl-main.183/. [↗](https://aclanthology.org/2022.naacl-main.183/)
55. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Comp… [find](https://scholar.google.com/scholar?q=Denis+Paperno%2C+Germ%C3%A1n+Kruszewski%2C+Angeliki+Lazaridou%2C+Ngoc+Quan+Pham%2C+Raffaella+Bernardi%2C+Sandro+Pezzelle%2C+Marco+Baroni%2C+Gemma+Boleda%2C+and+Raquel+Fern%C3%A1ndez%2E+The+LAMBADA+dataset%3A+Word+prediction+requir) [our card](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
56. Ravsehaj Singh Puri, Swaroop Mishra, Mihir Parmar, and Chitta Baral. How many data samples is an additional instruction worth? arXiv preprint arXiv:2203.09161, 2022. URL https://arxiv.org/abs/2203. 09161. [↗](https://arxiv.org/abs/2203)
57. Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. Data cards: Purposeful and transparent dataset documentation for responsible ai. In Data-centric AI Workshop at NeurIPS, 2021. URL https://arxiv.org/abs/2204.01075. — [arXiv:2204.01075](https://arxiv.org/abs/2204.01075) [↗](https://arxiv.org/abs/2204.01075)
58. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. URL [https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf). [↗](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
59. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446, 2021a. URL [https://arxiv.org/ab…](https://arxiv.org/ab…) [↗](https://arxiv.org/ab…)
60. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-S… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+H%2E+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick)
61. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67, 2020. URL https://arxiv.org/abs/1910.10683. [↗](https://arxiv.org/abs/1910.10683) [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
62. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! Leveraging language models for commonsense reasoning. ACL, 2019. doi: 10.18653/v1/P19-1487. URL https://aclanthology.org/P19-1487. [↗](https://aclanthology.org/P19-1487) [our card](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/)
63. Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashch… [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Hyung+Won+Chung%2C+Anselm+Levskaya%2C+Gaurav+Mishra%2C+James+Bradbury%2C+Daniel+Andor%2C+Sharan+Narang%2C+Brian+Lester%2C+Colin+Gaffney%2C+Afroz+Mohiuddin%2C+Curtis+Hawthorne%2C+Aitor+Lewkowycz%2C+Alex+Salcia)
64. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New… [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Jason+Naradowsky%2C+Brian+Leonard%2C+and+Benjamin+Van+Durme%2E+Gender+bias+in+coreference+resolution%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+) [our card](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/)
65. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. ICLR 2022, 2021. URL https://arxiv.org/abs/2110.08207. [↗](https://arxiv.org/abs/2110.08207) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
66. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection, 2021. URL https://arxiv.org/abs/2111.07997. [↗](https://arxiv.org/abs/2111.07997) [our card](https://md.research.elamur.ai/free/en/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/)
67. Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. CoRR, abs/2103.00453, 2021. URL https://arxiv.org/abs/2103.00453. [↗](https://arxiv.org/abs/2103.00453) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/)
68. Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Continual-T0: Progressively instructing 50+ tasks to language models without forgetting. arXiv preprint arXiv:2205.12393, 2022. URL https://arxiv.org/abs/2205.12393. [↗](https://arxiv.org/abs/2205.12393)
69. Nikil Roashan Selvam, Sunipa Dev, Daniel Khashabi, Tushar Khot, and Kai-Wei Chang. The tail wagging the dog: Dataset construction biases of social bias benchmarks, 2022. URL https://arxiv.org/abs/2210. 10040. [↗](https://arxiv.org/abs/2210)
70. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML, 2018. URL https://arxiv.org/abs/1804.04235. [↗](https://arxiv.org/abs/1804.04235)
71. Renee Shelby, Shalaleh Rismani, Kathryn Henne, AJung Moon, Negar Rostamzadeh, Paul Nicholas, N’Mah Yilla, Jess Gallegos, Andrew Smart, Emilio Garcia, and Gurleen Virk. Sociotechnical harms: Scoping a taxonomy for harm reduction, 2022. URL https://arxiv.org/abs/2210.05791. — [arXiv:2210.05791](https://arxiv.org/abs/2210.05791) [↗](https://arxiv.org/abs/2210.05791)
72. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057, 2022. URL https://arxiv.org/abs/2210… [↗](https://arxiv.org/abs/2210…) [our card](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/)
73. Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. "i’m sorry to hear that": Finding new biases in language models with a holistic descriptor dataset, 2022. URL https://arxiv.org/abs/2205.09209. [↗](https://arxiv.org/abs/2205.09209) [our card](https://md.research.elamur.ai/free/en/papers/smith-2022-i-m-sorry-hear-that/)
74. David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. Searching for efficient transformers for language modeling. Advances in Neural Information Processing Systems, 2021. URL https://arxiv.org/abs/2109.08668. — [arXiv:2109.08668](https://arxiv.org/abs/2109.08668) [↗](https://arxiv.org/abs/2109.08668)
75. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.… [find](https://scholar.google.com/scholar?q=Aarohi+Srivastava%2C+Abhinav+Rastogi%2C+Abhishek+Rao%2C+Abu+Awal+Md+Shoeb%2C+Abubakar+Abid%2C+Adam+Fisch%2C+Adam+R+Brown%2C+Adam+Santoro%2C+Aditya+Gupta%2C+Adri%C3%A0+Garriga%2DAlonso%2C+et+al%2E+Beyond+the+imitation+game%3A+Quanti) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
76. Jacob Steinhardt. Updates and lessons from ai forecasting. Blog post., 2021. URL [https://bounded-regret.ghost.io/ai-forecasting/](https://bounded-regret.ghost.io/ai-forecasting/). See https://prod.hypermind.com/ngdp/en/showcase2/showcase.html? sc=JSAI. [↗](https://bounded-regret.ghost.io/ai-forecasting/)
77. Mirac Suzgun, Nathan Scales, Nathaneal Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny ZHou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. URL [https://arxiv.org/ab…](https://arxiv.org/ab…) [↗](https://arxiv.org/ab…) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
78. Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge. NeurIPS, 2020. URL https://arxiv.org/abs/2006.06609. [↗](https://arxiv.org/abs/2006.06609)
79. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131, 2022a. URL https://arxiv.org/abs/2205.05131. [↗](https://arxiv.org/abs/2205.05131) [our card](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/)
80. Yi Tay, Jason Wei, Hyung Won Chung, David R. So, Siamak Shakeri, Xavier Garcia, Vinh Q. Tran, Hauixiu Steven Zheng, Jinfeng Rao, Denny Zhou, Donald Metzler, Neil Houlsby, Quoc V. Le, and Mostafa Dehghani. Transcending scaling laws with 0.1% extra compute. In arxiv, 2022b. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Jason+Wei%2C+Hyung+Won+Chung%2C+David+R%2E+So%2C+Siamak+Shakeri%2C+Xavier+Garcia%2C+Vinh+Q%2E+Tran%2C+Hauixiu+Steven+Zheng%2C+Jinfeng+Rao%2C+Denny+Zhou%2C+Donald+Metzler%2C+Neil+Houlsby%2C+Quoc+V%2E+Le%2C+and+Mostafa+Dehgh) [our card](https://md.research.elamur.ai/free/en/papers/tay-2022-transcending-scaling-laws-0-1/)
81. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGlue: A stickier benchmark for general-purpose language understanding systems. NeurIPS, 2019a. URL http://go/arxiv/1905.00537. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+SuperGlue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understanding+syst) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
82. Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. Does it make sense? And why? A pilot study for sense making and explanation. In ACL, 2019b. URL https://arxiv.org/abs/1906.00363. [↗](https://arxiv.org/abs/1906.00363)
83. Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for zero-shot generalization? ICML, 2022a. URL https://arxiv.org/abs/2204.05832. [↗](https://arxiv.org/abs/2204.05832)
84. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022b. URL https://arxiv.org/abs/2203.11171. [↗](https://arxiv.org/abs/2203.11171) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
85. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2022c. U… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+et+al%2E+Benchmarking+generaliza)
86. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. ICLR 2022, 2021. URL [https://openreview.net/forum?id=gEZrGCozdqR](https://openreview.net/forum?id=gEZrGCozdqR). [↗](https://openreview.net/forum?id=gEZrGCozdqR)
87. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. TMLR, 2022a. URL [https://openreview.net/forum?id=yzkSU5zdwD](https://openreview.net/forum?id=yzkSU5zdwD). [↗](https://openreview.net/forum?id=yzkSU5zdwD) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
88. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. NeurIPS, 2022b. URL https://arxiv.org/abs/2201.11903. [↗](https://arxiv.org/abs/2201.11903) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
89. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sea… — [doi:10.1145/3531146.3533088](https://doi.org/10.1145/3531146.3533088) [find](https://scholar.google.com/scholar?q=Laura+Weidinger%2C+Jonathan+Uesato%2C+Maribeth+Rauh%2C+Conor+Griffin%2C+Po%2DSen+Huang%2C+John+Mellor%2C+Amelia+Glaese%2C+Myra+Cheng%2C+Borja+Balle%2C+Atoosa+Kasirzadeh%2C+Courtney+Biles%2C+Sasha+Brown%2C+Zac+Kenton%2C+Will+Hawk)
90. Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. Detoxifying language models risks marginalizing minority voices, 2021. URL https://arxiv.org/abs/2104.06390. [↗](https://arxiv.org/abs/2104.06390)
91. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. TACL, 2022. URL https://aclanthology.org/2022.tacl-1.17. [↗](https://aclanthology.org/2022.tacl-1.17)
92. Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. In ICML, 2020. URL http://go/arxiv/2005.10636. [find](https://scholar.google.com/scholar?q=Michihiro+Yasunaga+and+Percy+Liang%2E+Graph%2Dbased%2C+self%2Dsupervised+program+repair+from+diagnostic+feedback%2E+In+ICML%2C+2020%2E+URL+http%3A%2F%2Fgo%2Farxiv%2F2005%2E10636%2E)
93. Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in NLP. In EMNLP, 2021. URL https://arxiv.org/abs/2104.08835. [↗](https://arxiv.org/abs/2104.08835)
94. Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann. Synthbio: A case study in human-ai collaborative curation of text datasets, 2021. URL https://arxiv.org/abs/2111.06467. — [arXiv:2111.06467](https://arxiv.org/abs/2111.06467) [↗](https://arxiv.org/abs/2111.06467)
95. Eric Zelikman, Yuhuai Wu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203.14465, 2022. URL https://arxiv.org/abs/2203.14465. [↗](https://arxiv.org/abs/2203.14465) [our card](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/)
96. Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. EMNLP Findings, 2021. URL https://aclanthology.org/2021.findings-emnlp.244/. [↗](https://aclanthology.org/2021.findings-emnlp.244/)