---
title: "Transcending Scaling Laws with 0.1% Extra Compute — Tay 2022"
description: "Making large language models better usually means scaling them up, which costs a great deal of compute. 52 references, cited by 34 papers here."
authors: ["Yi Tay", "Jason Wei", "Hyung Won Chung", "Vinh Q. Tran", "David R. So", "Siamak Shakeri", "Xavier Garcia", "Huaixiu Steven Zheng", "Jinfeng Rao", "Aakanksha Chowdhery", "Denny Zhou", "Donald Metzler", "Slav Petrov", "Neil Houlsby", "Quoc V. Le", "Mostafa Dehghani"]
year: 2022
venue: "arXiv:2210.11399"
source: "https://arxiv.org/abs/2210.11399"
doi: "10.48550/arxiv.2210.11399"
arxiv: "2210.11399"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tay-2022-transcending-scaling-laws-0-1/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tay-2022-transcending-scaling-laws-0-1/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Transcending Scaling Laws with 0.1% Extra Compute

# Transcending Scaling Laws with 0.1% Extra Compute

**Authors:** [Yi Tay](https://md.research.elamur.ai/free/en/authors/yi-tay/), [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/), [Hyung Won Chung](https://md.research.elamur.ai/free/en/authors/hyung-won-chung/), [Vinh Q. Tran](https://md.research.elamur.ai/free/en/authors/vinh-q-tran/), [David R. So](https://md.research.elamur.ai/free/en/authors/david-r-so/), [Siamak Shakeri](https://md.research.elamur.ai/free/en/authors/siamak-shakeri/), [Xavier Garcia](https://md.research.elamur.ai/free/en/authors/xavier-garcia/), [Huaixiu Steven Zheng](https://md.research.elamur.ai/free/en/authors/huaixiu-steven-zheng/), [Jinfeng Rao](https://md.research.elamur.ai/free/en/authors/jinfeng-rao/), [Aakanksha Chowdhery](https://md.research.elamur.ai/free/en/authors/aakanksha-chowdhery/), [Denny Zhou](https://md.research.elamur.ai/free/en/authors/denny-zhou/), [Donald Metzler](https://md.research.elamur.ai/free/en/authors/donald-metzler/), [Slav Petrov](https://md.research.elamur.ai/free/en/authors/slav-petrov/), Neil Houlsby, [Quoc V. Le](https://md.research.elamur.ai/free/en/authors/quoc-v-le/) and [Mostafa Dehghani](https://md.research.elamur.ai/free/en/authors/mostafa-dehghani/) · **Year:** 2022 · **Venue:** arXiv:2210.11399 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2210.11399) · [PDF](https://arxiv.org/pdf/2210.11399) · [DOI 10.48550/arxiv.2210.11399](https://doi.org/10.48550/arxiv.2210.11399) · [arXiv 2210.11399](https://arxiv.org/abs/2210.11399) · [OpenAlex](https://openalex.org/W4307079190)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/tay-2022-transcending-scaling-laws-0-1/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Making large language models better usually means scaling them up, which costs a great deal of compute. The authors ask whether an already trained causal model can be improved cheaply instead. Their method, UL2R (UL2Restore), takes existing PaLM checkpoints and trains them for a small number of extra steps using the UL2 mixture-of-denoisers objective. This objective combines prefix language modeling with short and long span corruption, and mode tokens select the denoiser. No new data is involved: training reuses PaLM's original data mixture, and the added cost is roughly 0.1–1% of the original FLOPs. The resulting U-PaLM models (8B, 62B, 540B) score higher than PaLM on English few-shot NLP tasks, chain-of-thought reasoning, multilingual benchmarks, MMLU and difficult BIG-Bench tasks. At 540B, U-PaLM matches final PaLM quality with about half the compute. The models can also fill in several blanks within a prompt. The work matters because it shows that diverse training objectives, not only scale, can shift scaling curves and unlock emergent abilities.

## Key points
- At 540B scale, U-PaLM reaches the final PaLM 540B quality with about 2x less compute, saving roughly 4.4 million TPUv4 hours. Quality is measured as the average over 20+ zero/few-shot NLP tasks.
- Reported gains cover commonsense reasoning, question answering, GSM8K (chain-of-thought), MGSM, TydiQA, MMLU and BIG-Bench.
- On "emergent" BIG-Bench tasks, U-PaLM is clearly stronger where PaLM struggles (navigate, geometric shapes, hyperbaton). It also shows emergence at smaller scales of 62B or 8B (crass ai, vitaminc fact verification).
- The mode tokens [NLU], [NLG] and [S2S] correspond to the R-, X- and S-denoisers and can be chosen at inference time. Varying the mode token and infill setup gives more diverse outputs without sampling and can fix certain prompts, such as answering in Vietnamese.
- Single- and multi-span infilling appears after only about 0.1% extra tokens/FLOPs. Combining UL2R with FLAN-style instruction tuning is covered in follow-up work (Chung et al., 2022).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

💡 Concepts (1): [📈 Scaling laws](https://md.research.elamur.ai/free/en/concepts/scaling-laws/) · 30 works

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 7,249 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 7,084 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 7,398 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 7,340 works

## Cite

Cite as: Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q. Tran, David R. So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc V. Le and Mostafa Dehghani. 2022. Transcending Scaling Laws with 0.1% Extra Compute. arXiv:2210.11399.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tay2022transcending,
  title = {{Transcending Scaling Laws with 0.1\% Extra Compute}},
  author = {Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q. Tran, David R. So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc V. Le and Mostafa Dehghani},
  year = {2022},
  doi = {10.48550/arxiv.2210.11399},
  eprint = {2210.11399},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.11399}
}
```

- [← previous PropBank Comes of Age—Larger, Smarter, and more Diverse](https://md.research.elamur.ai/free/en/papers/pradhan-2022-propbank-comes-age-larger-smarter/)
- [next → Semeval-2022 Task 1: CODWOE – Comparing Dictionaries and Word Embeddings](https://md.research.elamur.ai/free/en/papers/mickus-2022-codwoe-dictionaries-embeddings/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

52 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. arXiv preprint arXiv:2101.11038, 2021. [our card](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-muppet-massive-multi-task-representations/)
2. Philip W Anderson. More is different: broken symmetry and the nature of the hierarchical structure of science. Science, 177(4047):393–396, 1972. [find](https://scholar.google.com/scholar?q=Philip+W+Anderson%2E+More+is+different%3A+broken+symmetry+and+the+nature+of+the+hierarchical+structure+of+science%2E+Science%2C+177%284047%29%3A393%E2%80%93396%2C+1972%2E)
3. Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. ICLR, 2022. URL https:/… [find](https://scholar.google.com/scholar?q=Vamsi+Aribandi%2C+Yi+Tay%2C+Tal+Schuster%2C+Jinfeng+Rao%2C+Huaixiu+Steven+Zheng%2C+Sanket+Vaibhav+Mehta%2C+Honglei+Zhuang%2C+Vinh+Q%2E+Tran%2C+Dara+Bahri%2C+Jianmo+Ni%2C+Jai+Prakash+Gupta%2C+Kai+Hui%2C+Sebastian+Ruder%2C+and+Don)
4. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020. [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
5. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. [our card](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
7. Hyung Won Chung, Longpre Shayne Hou, Le and, Barret Zoph, Yi Tay, William Fedus, and et al. Scaling instruction-finetuned language models. arXiv preprint, 2022. [our card](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/)
8. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019. [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
9. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470, 2020a… [our card](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)
10. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020b. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+arXiv+preprint+arXiv%3A2003%2E10555%2C+2020b%2E)
11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
12. Mostafa Dehghani, Yi Tay, Anurag Arnab, Lucas Beyer, and Ashish Vaswani. The efficiency misnomer. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Yi+Tay%2C+Anurag+Arnab%2C+Lucas+Beyer%2C+and+Ashish+Vaswani%2E+The+efficiency+misnomer%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
14. Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. arXiv preprint arXiv:2209.15003, 2022. [find](https://scholar.google.com/scholar?q=Andrew+Drozdov%2C+Nathanael+Sch%C3%A4rli%2C+Ekin+Aky%C3%BCrek%2C+Nathan+Scales%2C+Xinying+Song%2C+Xinyun+Chen%2C+Olivier+Bousquet%2C+and+Denny+Zhou%2E+Compositional+semantic+parsing+with+large+language+models%2E+arXiv+preprint+a)
15. Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova DasSarma, Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, Ben Mann, Amanda Askell, et al. Predictability and surprise in large generative models. arXiv preprint arXiv:2202.07785, 2022. URL https://arxiv.org/abs/2202.07785. [↗](https://arxiv.org/abs/2202.07785)
16. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361, 2021. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+the+As)
17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
18. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487, 2022. [our card](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/)
19. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
20. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
21. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
22. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questi… [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
23. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021. [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+arXiv+preprint+arXiv%3A2104%2E08691%2C+2021%2E)
24. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. [our card](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/)
25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. URL https://arxiv.org/abs/2203.02155. [↗](https://arxiv.org/abs/2203.02155) [our card](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
26. Denis Paperno, Germ’an Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern’andez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Co… [our card](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
27. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+Scaling+language+models%3A+Methods%2C+analysi)
28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
29. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+arXiv+preprint+arXiv%3A2002%2E08910%2C+2020%2E)
30. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+arXiv+preprint+arXiv%3A1907%2E10641%2C+2019%2E)
31. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. ICLR, 2022. URL [https://openreview.net/forum?id=9Vrb9D0WI4](https://openreview.net/forum?id=9Vrb9D0WI4). [↗](https://openreview.net/forum?id=9Vrb9D0WI4) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
32. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057, 2022. [our card](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/)
33. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
34. Jacob Steinhardt. Future ml systems will be qualitatively different, 2022. URL [https://bounded-regret.ghost.io/future-ml-systems-will-be-qualitatively-different/](https://bounded-regret.ghost.io/future-ml-systems-will-be-qualitatively-different/). Accessed May 20, 2022. [↗](https://bounded-regret.ghost.io/future-ml-systems-will-be-qualitatively-different/)
35. Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint, 2022. [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
36. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… [our card](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/)
37. Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109.10686, 2021. — [arXiv:2109.10686](https://arxiv.org/abs/2109.10686) [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Mostafa+Dehghani%2C+Jinfeng+Rao%2C+William+Fedus%2C+Samira+Abnar%2C+Hyung+Won+Chung%2C+Sharan+Narang%2C+Dani+Yogatama%2C+Ashish+Vaswani%2C+and+Donald+Metzler%2E+Scale+efficiently%3A+Insights+from+pre%2Dtraining+and)
38. Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551, 2022a. [our card](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/)
39. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131, 2022b. [our card](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/)
40. Yi Tay, Vinh Q Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as a differentiable search index. arXiv preprint arXiv:2202.06991, 2022c. [our card](https://md.research.elamur.ai/free/en/papers/tay-2022-transformer-memory-differentiable-search-index/)
41. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. [our card](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
42. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019. [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
43. Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for zero-shot generalization? arXiv preprint arXiv:2204.05832, 2022a. [find](https://scholar.google.com/scholar?q=Thomas+Wang%2C+Adam+Roberts%2C+Daniel+Hesslow%2C+Teven+Le+Scao%2C+Hyung+Won+Chung%2C+Iz+Beltagy%2C+Julien+Launay%2C+and+Colin+Raffel%2E+What+language+model+architecture+and+pretraining+objective+work+best+for+zero%2Dsh)
44. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022b. [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
45. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2109%2E01652)
46. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research (TMLR), 2022a. [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. Conference on Neural Information Processing Systems (NeurIPS), 2022b. [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
48. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019. [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
49. Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2022. — [arXiv:2206.10789](https://arxiv.org/abs/2206.10789) [find](https://scholar.google.com/scholar?q=Jiahui+Yu%2C+Yuanzhong+Xu%2C+Jing+Yu+Koh%2C+Thang+Luong%2C+Gunjan+Baid%2C+Zirui+Wang%2C+Vijay+Vasudevan%2C+Alexander+Ku%2C+Yinfei+Yang%2C+Burcu+Karagol+Ayan%2C+et+al%2E+Scaling+autoregressive+models+for+content%2Drich+text%2Dt)
50. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
51. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. [our card](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/)
52. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL https://arxiv.org/abs/2202.08906. [↗](https://arxiv.org/abs/2202.08906)

*Updated 23 October 2026*