---
title: "No Train No Gain — paper by Kaddour et al. (2023)"
description: "Some compare checkpoints partway through training. 113 references, cited by 23 papers here."
authors: ["Jean Kaddour", "Oscar Key", "Piotr Nawrot", "Pasquale Minervini", "Matt J. Kusner"]
year: 2023
venue: "arXiv:2307.06440"
source: "https://arxiv.org/abs/2307.06440"
doi: "10.48550/arxiv.2307.06440"
arxiv: "2307.06440"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kaddour-2023-no-train-no-gain-revisiting/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kaddour-2023-no-train-no-gain-revisiting/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

# No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

**Authors:** [Jean Kaddour](https://md.research.elamur.ai/free/en/authors/jean-kaddour/), Oscar Key, [Piotr Nawrot](https://md.research.elamur.ai/free/en/authors/piotr-nawrot/), [Pasquale Minervini](https://md.research.elamur.ai/free/en/authors/pasquale-minervini/) and Matt J. Kusner · **Year:** 2023 · **Venue:** arXiv:2307.06440 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.06440) · [PDF](https://arxiv.org/pdf/2307.06440) · [DOI 10.48550/arxiv.2307.06440](https://doi.org/10.48550/arxiv.2307.06440) · [arXiv 2307.06440](https://arxiv.org/abs/2307.06440) · [OpenAlex](https://openalex.org/W4384389803)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/kaddour-2023-no-train-no-gain-revisiting/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Efficient training methods for Transformer language models often claim speed-ups, but these claims tend to rest on uneven comparisons. Some compare checkpoints partway through training. Others count iterations even when the cost per iteration differs between methods. Kaddour et al. retest six such methods under strict, equal compute budgets: layer stacking and layer dropping (dynamic architectures), selective backprop and RHO loss (batch selection), and the Lion and Sophia optimizers. They pre-train BERT and T5-Base from scratch on a single GPU with budgets of 6, 12, and 24 hours. The baseline's learning rate is always decayed to completion within the given budget. To make timings comparable across different hardware, they introduce reference system time (RST). RST counts the iterations a run performs and multiplies them by the per-iteration time measured on one fixed reference machine. Against a properly tuned baseline, the reported gains mostly disappear. Downstream, no method beats the baseline at 24 hours. This matters for anyone with limited compute, because a well-scheduled standard setup is a strong default and new claims need careful budgeting.

## Key points
- **Models and benchmarks:** A crammed BERT model (16 layers, ~120M parameters, MLM objective) was evaluated on GLUE and SuperGLUE. T5v1.1-Base was evaluated on Super-Natural-Instructions (SNI) using ROUGE-L. The batch selection methods were additionally tested on C4, Wikipedia + BookCorpus, and MiniPile.
- **Training loss:** Layer stacking was the only method that consistently achieved lower training loss than the baseline, and its advantage shrank as the budget grew. Lion matched it only for some BERT budgets.
- **Costlier methods:** Batch selection carried heavy overhead. RHO loss, using a mega-batch of 15,360, took about 5.3× the baseline time, and selective backprop took about 1.7×. Even with that overhead set aside, neither method beat the baseline.
- **T5 on SNI at 24h:** The baseline scored 39.5, compared with 38.9 for layer stacking, 34.1 for Sophia, and 33.7 for Lion. The optimizer runs required BF16 for BERT, because FP16 produced NaN losses, and RMS scaling for T5.
- **Availability:** The code is released at github.com/JeanKaddour/NoTrainNoGain, and the paper appeared at NeurIPS 2023.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 396 works · 82,279 citations
- [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 119 works · 23,546 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini and Matt J. Kusner. 2023. No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models. arXiv:2307.06440.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kaddour2023no,
  title = {{No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models}},
  author = {Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini and Matt J. Kusner},
  year = {2023},
  doi = {10.48550/arxiv.2307.06440},
  eprint = {2307.06440},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.06440}
}
```

- [← previous N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition](https://md.research.elamur.ai/free/en/papers/talafha-2023-n-shot-benchmarking-whisper-diverse/)
- [next → Positional Description Matters for Transformers Arithmetic](https://md.research.elamur.ai/free/en/papers/shen-2023-positional-description-matters-transformers-arithmetic/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

113 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abbas, A., Tirumala, K., Simig, D., Ganguli, S., and Morcos, A. S. [Semdedup: Data-efficient learning at web-scale through semantic deduplication](https://md.research.elamur.ai/free/en/papers/abbas-2023-semdedup-data-efficient-learning-web/). arXiv preprint arXiv:2303.09540, 2023.
2. Ainslie, J., Lei, T., de Jong, M., Ontan’on, S., Brahma, S., Zemlyanskiy, Y., Uthus, D. C., Guo, M., Lee-Thorp, J., Tay, Y., Sung, Y.-H., and Sanghai, S. K. [Colt5: Faster long-range transformers with conditional computation](https://md.research.elamur.ai/free/en/papers/ainslie-2023-colt5-faster-long-range-transformers/). ArXiv, abs/2303.09752, 2023.
3. Alabdulmohsin, I. M., Neyshabur, B., and Zhai, X. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300–22312, 2022. [find](https://scholar.google.com/scholar?q=Alabdulmohsin%2C+I%2E+M%2E%2C+Neyshabur%2C+B%2E%2C+and+Zhai%2C+X%2E+Revisiting+neural+scaling+laws+in+language+and+vision%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A22300%E2%80%9322312%2C+2022%2E)
4. Alain, G., Lamb, A., Sankar, C., Courville, A. C., and Bengio, Y. Variance reduction in SGD by distributed importance sampling. CoRR, abs/1511.06481, 2015. — [arXiv:1511.06481](https://arxiv.org/abs/1511.06481) [find](https://scholar.google.com/scholar?q=Alain%2C+G%2E%2C+Lamb%2C+A%2E%2C+Sankar%2C+C%2E%2C+Courville%2C+A%2E+C%2E%2C+and+Bengio%2C+Y%2E+Variance+reduction+in+SGD+by+distributed+importance+sampling%2E+CoRR%2C+abs%2F1511%2E06481%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E06481%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1)
5. Anil, R., Gupta, V., Koren, T., Regan, K., and Singer, Y. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020. — [arXiv:2002.09018](https://arxiv.org/abs/2002.09018) [find](https://scholar.google.com/scholar?q=Anil%2C+R%2E%2C+Gupta%2C+V%2E%2C+Koren%2C+T%2E%2C+Regan%2C+K%2E%2C+and+Singer%2C+Y%2E+Scalable+second+order+optimization+for+deep+learning%2E+arXiv+preprint+arXiv%3A2002%2E09018%2C+2020%2E+%E2%80%94+%5BarXiv%3A2002%2E09018%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2002%2E0)
6. Arora, S., Cohen, N., and Hazan, E. On the optimization of deep networks: Implicit acceleration by overparameterization. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 244–253. PMLR, 10… [find](https://scholar.google.com/scholar?q=Arora%2C+S%2E%2C+Cohen%2C+N%2E%2C+and+Hazan%2C+E%2E+On+the+optimization+of+deep+networks%3A+Implicit+acceleration+by+overparameterization%2E+In+Dy%2C+J%2E+and+Krause%2C+A%2E+%28eds%2E%29%2C+Proceedings+of+the+35th+International+Conferen)
7. Bachlechner, T., Majumder, B. P., Mao, H., Cottrell, G., and McAuley, J. [Rezero is all you need: Fast convergence at large depth](https://md.research.elamur.ai/free/en/papers/bachlechner-2020-rezero-all-need-fast-convergence/). In Uncertainty in Artificial Intelligence, pp. 1352–1361. PMLR, 2021.
8. Bartoldson, B. R., Kailkhura, B., and Blalock, D. Compute-efficient deep learning: Algorithmic trends and opportunities. Journal of Machine Learning Research, 24:1–77, 2023. [find](https://scholar.google.com/scholar?q=Bartoldson%2C+B%2E+R%2E%2C+Kailkhura%2C+B%2E%2C+and+Blalock%2C+D%2E+Compute%2Defficient+deep+learning%3A+Algorithmic+trends+and+opportunities%2E+Journal+of+Machine+Learning+Research%2C+24%3A1%E2%80%9377%2C+2023%2E)
9. Brock, A., Lim, T., Ritchie, J. M., and Weston, N. FreezeOut: Accelerate Training by Progressively Freezing Layers, June 2017. URL http://arxiv.org/abs/1706.04983. arXiv:1706.04983 [cs, stat]. — [arXiv:1706.04983](https://arxiv.org/abs/1706.04983) [↗](http://arxiv.org/abs/1706.04983)
10. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
11. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
12. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
13. Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., et al. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675, 2023. [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Liang%2C+C%2E%2C+Huang%2C+D%2E%2C+Real%2C+E%2E%2C+Wang%2C+K%2E%2C+Liu%2C+Y%2E%2C+Pham%2C+H%2E%2C+Dong%2C+X%2E%2C+Luong%2C+T%2E%2C+Hsieh%2C+C%2E%2DJ%2E%2C+et+al%2E+Symbolic+discovery+of+optimization+algorithms%2E+arXiv+preprint+arXiv%3A2302%2E06675%2C+2023%2E)
14. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Aust… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
15. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044, 2019.
16. Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=HJg2b0VYDr. [↗](https://openreview.net/forum?)
17. Dahl, G. E., Schneider, F., Nado, Z., Agarwal, N., Sastry, C. S., Hennig, P., Medapati, S., Eschenhagen, R., Kasimbeg, P., Suo, D., et al. Benchmarking neural network training algorithms. arXiv preprint arXiv:2306.07179, 2023. — [arXiv:2306.07179](https://arxiv.org/abs/2306.07179) [find](https://scholar.google.com/scholar?q=Dahl%2C+G%2E+E%2E%2C+Schneider%2C+F%2E%2C+Nado%2C+Z%2E%2C+Agarwal%2C+N%2E%2C+Sastry%2C+C%2E+S%2E%2C+Hennig%2C+P%2E%2C+Medapati%2C+S%2E%2C+Eschenhagen%2C+R%2E%2C+Kasimbeg%2C+P%2E%2C+Suo%2C+D%2E%2C+et+al%2E+Benchmarking+neural+network+training+algorithms%2E+arXiv+prepri)
18. Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 16344–16359, 2022. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Fu%2C+D%2E%2C+Ermon%2C+S%2E%2C+Rudra%2C+A%2E%2C+and+R%C3%A9%2C+C%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A+16344%E2%80%9316359%2C+2022%2E)
19. Dasgupta, I., Lampinen, A. K., Chan, S. C., Creswell, A., Kumaran, D., McClelland, J. L., and Hill, F. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.[07051, 2022](https://md.research.elamur.ai/free/en/papers/dasgupta-2022-language-models-show-human-like/).
20. Dehghani, M., Tay, Y., Arnab, A., Beyer, L., and Vaswani, A. The efficiency misnomer. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=iulEMLYh1uR](https://openreview.net/forum?id=iulEMLYh1uR). [↗](https://openreview.net/forum?id=iulEMLYh1uR)
21. Dettmers, T. and Zettlemoyer, L. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019. [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E+and+Zettlemoyer%2C+L%2E+Sparse+networks+from+scratch%3A+Faster+training+without+losing+performance%2E+arXiv+preprint+arXiv%3A1907%2E04840%2C+2019%2E)
22. Devlin, J., Chang, M., Lee, K., and Toutanova, K. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL-HLT (1), pp. 4171–4186. Association for Computational Linguistics, 2019.
23. Dozat, T. Incorporating nesterov momentum into adam. ICLR Workshop track, 2016. [find](https://scholar.google.com/scholar?q=Dozat%2C+T%2E+Incorporating+nesterov+momentum+into+adam%2E+ICLR+Workshop+track%2C+2016%2E)
24. Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943–2952. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Evci%2C+U%2E%2C+Gale%2C+T%2E%2C+Menick%2C+J%2E%2C+Castro%2C+P%2E+S%2E%2C+and+Elsen%2C+E%2E+Rigging+the+lottery%3A+Making+all+tickets+winners%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+2943%E2%80%932952%2E+PMLR%2C+2020%2E)
25. Fahlman, S. and Lebiere, C. The cascade-correlation learning architecture. Advances in neural information processing systems, 2, 1989. [find](https://scholar.google.com/scholar?q=Fahlman%2C+S%2E+and+Lebiere%2C+C%2E+The+cascade%2Dcorrelation+learning+architecture%2E+Advances+in+neural+information+processing+systems%2C+2%2C+1989%2E)
26. Fan, A., Grave, E., and Joulin, A. [Reducing transformer depth on demand with structured dropout](https://md.research.elamur.ai/free/en/papers/fan-2019-reducing-transformer-depth-demand-structured/), 2019.
27. Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020. — [arXiv:2010.01412](https://arxiv.org/abs/2010.01412) [find](https://scholar.google.com/scholar?q=Foret%2C+P%2E%2C+Kleiner%2C+A%2E%2C+Mobahi%2C+H%2E%2C+and+Neyshabur%2C+B%2E+Sharpness%2Daware+minimization+for+efficiently+improving+generalization%2E+arXiv+preprint+arXiv%3A2010%2E01412%2C+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E01412%5D%28https%3A%2F%2Farxiv%2Eo)
28. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+Golding%2C+L%2E%2C+Hoppe%2C+T%2E%2C+Foster%2C+C%2E%2C+Phang%2C+J%2E%2C+He%2C+H%2E%2C+Thite%2C+A%2E%2C+Nabeshima%2C+N%2E%2C+et+al%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+language+modeling%2E+arXiv+prepri)
29. Geiping, J. and Goldstein, T. Cramming: Training a Language Model on a Single GPU in One Day, December 2022. URL http://arxiv.org/abs/2212.14034. arXiv:2212.14034 [cs]. [↗](http://arxiv.org/abs/2212.14034)
30. Gong, L., He, D., Li, Z., Qin, T., Wang, L., and Liu, T. Efficient Training of BERT by Progressively Stacking. In Proceedings of the 36th International Conference on Machine Learning, pp. 2337–2346. PMLR, May 2019. URL [https://proceedings.mlr.press/](https://proceedings.mlr.press/) v97/gong19a.html. ISSN: 2640-3498. [↗](https://proceedings.mlr.press/)
31. Guo, C., Zhao, B., and Bai, Y. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181–195. Springer, 2022. [find](https://scholar.google.com/scholar?q=Guo%2C+C%2E%2C+Zhao%2C+B%2E%2C+and+Bai%2C+Y%2E+Deepcore%3A+A+comprehensive+library+for+coreset+selection+in+deep+learning%2E+In+International+Conference+on+Database+and+Expert+Systems+Applications%2C+pp%2E+181%E2%80%93195%2E+Springer%2C)
32. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.-W. [Realm: Retrieval-augmented language model pre-training](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/), 2020.
33. He, H., Huang, G., and Yuan, Y. Asymmetric valleys: Beyond sharp and flat local minima. Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=He%2C+H%2E%2C+Huang%2C+G%2E%2C+and+Yuan%2C+Y%2E+Asymmetric+valleys%3A+Beyond+sharp+and+flat+local+minima%2E+Advances+in+neural+information+processing+systems%2C+32%2C+2019%2E)
34. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
35. Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. [Deep networks with stochastic depth](https://md.research.elamur.ai/free/en/papers/huang-2016-deep-networks-stochastic-depth/). In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pp. 646–661. Springer, 2016.
36. Irandoust, S., Durand, T., Rakhmangulova, Y., Zi, W., and Hajimirsadeghi, H. Training a Vision Transformer from scratch in less than 24 hours with 1 GPU, November 2022. URL http://arxiv.org/abs/2211.05187. arXiv:2211.05187 [cs]. [↗](http://arxiv.org/abs/2211.05187)
37. Izsak, P., Berchansky, M., and Levy, O. How to Train BERT with an Academic Budget, September 2021. URL http://arxiv.org/abs/2104.07705. arXiv:2104.07705 [cs]. [↗](http://arxiv.org/abs/2104.07705)
38. Jiang, A. H., Wong, D. L.-K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G., Kaminksy, M., Kozuch, M., Lipton, Z. C., and Pillai, P. Accelerating Deep Learning by Focusing on the Biggest Losers, October 2019. URL http://arxiv.org/abs/1910.00762. arXiv:1910.00762 [cs, stat]. — [arXiv:1910.00762](https://arxiv.org/abs/1910.00762) [↗](http://arxiv.org/abs/1910.00762)
39. Kaddour, J. Stop wasting my time! saving days of imagenet and bert training with latest weight averaging. arXiv preprint arXiv:2209.14981, 2022. [find](https://scholar.google.com/scholar?q=Kaddour%2C+J%2E+Stop+wasting+my+time%21+saving+days+of+imagenet+and+bert+training+with+latest+weight+averaging%2E+arXiv+preprint+arXiv%3A2209%2E14981%2C+2022%2E)
40. Kaddour, J. [The minipile challenge for data-efficient language models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-minipile-challenge-data-efficient-language/). arXiv preprint arXiv:2304.08442, 2023.
41. Kaddour, J., Liu, L., Silva, R., and Kusner, M. J. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577–16595, 2022. [find](https://scholar.google.com/scholar?q=Kaddour%2C+J%2E%2C+Liu%2C+L%2E%2C+Silva%2C+R%2E%2C+and+Kusner%2C+M%2E+J%2E+When+do+flat+minima+optimizers+work%3F+Advances+in+Neural+Information+Processing+Systems%2C+35%3A16577%E2%80%9316595%2C+2022%2E)
42. Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., and McHardy, R. [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/). arXiv preprint arXiv:2307.10169, July 2023. doi: 10.48550/arXiv.2307.10169. URL http://arxiv.org/abs/2307.10169. [↗](http://arxiv.org/abs/2307.10169)
43. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
44. Katharopoulos, A. and Fleuret, F. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525–2534. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Katharopoulos%2C+A%2E+and+Fleuret%2C+F%2E+Not+all+samples+are+created+equal%3A+Deep+learning+with+importance+sampling%2E+In+International+conference+on+machine+learning%2C+pp%2E+2525%E2%80%932534%2E+PMLR%2C+2018%2E)
45. Kawaguchi, K. and Lu, H. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pp. 669–679. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Kawaguchi%2C+K%2E+and+Lu%2C+H%2E+Ordered+sgd%3A+A+new+stochastic+optimization+framework+for+empirical+risk+minimization%2E+In+International+Conference+on+Artificial+Intelligence+and+Statistics%2C+pp%2E+669%E2%80%93679%2E+PMLR%2C)
46. Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104.14337, 2021. [find](https://scholar.google.com/scholar?q=Kiela%2C+D%2E%2C+Bartolo%2C+M%2E%2C+Nie%2C+Y%2E%2C+Kaushik%2C+D%2E%2C+Geiger%2C+A%2E%2C+Wu%2C+Z%2E%2C+Vidgen%2C+B%2E%2C+Prasad%2C+G%2E%2C+Singh%2C+A%2E%2C+Ringshia%2C+P%2E%2C+et+al%2E+Dynabench%3A+Rethinking+benchmarking+in+nlp%2E+arXiv+preprint+arXiv%3A2104%2E14337%2C+20)
47. Kingma, D. P. and Ba, J. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014.
48. Kleinberg, B., Li, Y., and Yuan, Y. An alternative view: When does sgd escape local minima? In International conference on machine learning, pp. 2698–2707. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Kleinberg%2C+B%2E%2C+Li%2C+Y%2E%2C+and+Yuan%2C+Y%2E+An+alternative+view%3A+When+does+sgd+escape+local+minima%3F+In+International+conference+on+machine+learning%2C+pp%2E+2698%E2%80%932707%2E+PMLR%2C+2018%2E)
49. Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big Transfer (BiT): General Visual Representation Learning, May 2020. URL http://arxiv.org/abs/1912.11370. arXiv:1912.11370 [cs]. — [arXiv:1912.11370](https://arxiv.org/abs/1912.11370) [↗](http://arxiv.org/abs/1912.11370)
50. Komatsuzaki, A. One epoch is all you need. arXiv preprint arXiv:1906.06669, 2019. — [arXiv:1906.06669](https://arxiv.org/abs/1906.06669) [find](https://scholar.google.com/scholar?q=Komatsuzaki%2C+A%2E+One+epoch+is+all+you+need%2E+arXiv+preprint+arXiv%3A1906%2E06669%2C+2019%2E+%E2%80%94+%5BarXiv%3A1906%2E06669%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1906%2E06669%29)
51. Kudo, T. and Richardson, J. [Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). arXiv preprint arXiv:1808.06226, 2018.
52. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021. [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%2C+Ippolito%2C+D%2E%2C+Nystrom%2C+A%2E%2C+Zhang%2C+C%2E%2C+Eck%2C+D%2E%2C+Callison%2DBurch%2C+C%2E%2C+and+Carlini%2C+N%2E+Deduplicating+training+data+makes+language+models+better%2E+arXiv+preprint+arXiv%3A2107%2E06499%2C+2021%2E)
53. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating Training Data Makes Language Models Better, March 2022. URL http://arxiv.org/abs/2107.06499. arXiv:2107.06499 [cs]. [↗](http://arxiv.org/abs/2107.06499)
54. Lengellé, R. and Denoeux, T. Training mlps layer by layer using an objective function for internal representations. Neural Networks, 9(1):83–97, 1996. [find](https://scholar.google.com/scholar?q=Lengell%C3%A9%2C+R%2E+and+Denoeux%2C+T%2E+Training+mlps+layer+by+layer+using+an+objective+function+for+internal+representations%2E+Neural+Networks%2C+9%281%29%3A83%E2%80%9397%2C+1996%2E)
55. Li, C., Zhuang, B., Wang, G., Liang, X., Chang, X., and Yang, Y. Automated Progressive Learning for Efficient Training of Vision Transformers. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12476–12486, New Orleans, LA, USA, June 2022. IEEE. ISBN 978-1-66546-946-3… — [doi:10.1109/cvpr52688.2022.01216](https://doi.org/10.1109/cvpr52688.2022.01216) [find](https://scholar.google.com/scholar?q=Li%2C+C%2E%2C+Zhuang%2C+B%2E%2C+Wang%2C+G%2E%2C+Liang%2C+X%2E%2C+Chang%2C+X%2E%2C+and+Yang%2C+Y%2E+Automated+Progressive+Learning+for+Efficient+Training+of+Vision+Transformers%2E+In+2022+IEEE%2FCVF+Conference+on+Computer+Vision+and+Patter)
56. Liu, H. Does this work with 16-mixed precision. [https://github.com/Liuhong99/](https://github.com/Liuhong99/) Sophia/issues/16, 2023. GitHub repository issue. [↗](https://github.com/Liuhong99/)
57. Liu, H., Li, Z., Hall, D., Liang, P., and Ma, T. [Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training](https://md.research.elamur.ai/free/en/papers/liu-2023-sophia-scalable-stochastic-second-order/), May 2023. URL http://arxiv.org/abs/ 2305.14342. arXiv:2305.14342 [cs, math]. [↗](http://arxiv.org/abs/)
58. Liu, S. and Wang, Z. Ten lessons we have learned in the new" sparseland": A short handbook for sparse neural network researchers. arXiv preprint arXiv:2302.02596, 2023. — [arXiv:2302.02596](https://arxiv.org/abs/2302.02596) [find](https://scholar.google.com/scholar?q=Liu%2C+S%2E+and+Wang%2C+Z%2E+Ten+lessons+we+have+learned+in+the+new%22+sparseland%22%3A+A+short+handbook+for+sparse+neural+network+researchers%2E+arXiv+preprint+arXiv%3A2302%2E02596%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E02596%5D%28https%3A%2F%2Far)
59. Liu, Y., Agarwal, S., and Venkataraman, S. Autofreeze: Automatically freezing model blocks to accelerate fine-tuning. CoRR, abs/2102.01386, 2021. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Agarwal%2C+S%2E%2C+and+Venkataraman%2C+S%2E+Autofreeze%3A+Automatically+freezing+model+blocks+to+accelerate+fine%2Dtuning%2E+CoRR%2C+abs%2F2102%2E01386%2C+2021%2E)
60. Longpre, S., Yauney, G., Reif, E., Lee, K., Roberts, A., Zoph, B., Zhou, D., Wei, J., Robinson, K., Mimno, D., and Ippolito, D. A Pretrainer’s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity, May 2023. URL — [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://md.research.elamur.ai/free/en/papers/longpre-2023-pretrainer-s-guide-training-data/) [http://arxiv.org/](http://arxiv.org/) abs/2305.13169. arXiv:2305.1… [↗](http://arxiv.org/)
61. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2C+2017%2E)
62. Loshchilov, I. and Hutter, F. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL [https://openreview.net/forum?id=Skq89Scxx](https://openreview.net/forum?id=Skq89Scxx). [↗](https://openreview.net/forum?id=Skq89Scxx)
63. Menghani, G. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Computing Surveys, 55(12):1–37, 2023. [find](https://scholar.google.com/scholar?q=Menghani%2C+G%2E+Efficient+deep+learning%3A+A+survey+on+making+deep+learning+models+smaller%2C+faster%2C+and+better%2E+ACM+Computing+Surveys%2C+55%2812%29%3A1%E2%80%9337%2C+2023%2E)
64. Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., and Wu, H. Mixed precision training. In International Conference on Learning Representations, 2018. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=r1gs9JgRZ. [↗](https://openreview.net/forum?)
65. Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630–15649. PMLR, 2… [find](https://scholar.google.com/scholar?q=Mindermann%2C+S%2E%2C+Brauner%2C+J%2E+M%2E%2C+Razzak%2C+M%2E+T%2E%2C+Sharma%2C+M%2E%2C+Kirsch%2C+A%2E%2C+Xu%2C+W%2E%2C+H%C3%B6ltgen%2C+B%2E%2C+Gomez%2C+A%2E+N%2E%2C+Morisot%2C+A%2E%2C+Farquhar%2C+S%2E%2C+et+al%2E+Prioritized+training+on+points+that+are+learnable%2C+worth+lea)
66. Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., and Liotta, A. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):2383, 2018. [find](https://scholar.google.com/scholar?q=Mocanu%2C+D%2E+C%2E%2C+Mocanu%2C+E%2E%2C+Stone%2C+P%2E%2C+Nguyen%2C+P%2E+H%2E%2C+Gibescu%2C+M%2E%2C+and+Liotta%2C+A%2E+Scalable+training+of+artificial+neural+networks+with+adaptive+sparse+connectivity+inspired+by+network+science%2E+Nature+c)
67. Narang, S., Chung, H. W., Tay, Y., Fedus, W., Fevry, T., Matena, M., Malkan, K., Fiedel, N., Shazeer, N., Lan, Z., et al. Do transformer modifications transfer across implementations and applications? arXiv preprint arXiv:2102.11972, 2021. [find](https://scholar.google.com/scholar?q=Narang%2C+S%2E%2C+Chung%2C+H%2E+W%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Fevry%2C+T%2E%2C+Matena%2C+M%2E%2C+Malkan%2C+K%2E%2C+Fiedel%2C+N%2E%2C+Shazeer%2C+N%2E%2C+Lan%2C+Z%2E%2C+et+al%2E+Do+transformer+modifications+transfer+across+implementations+and+applications%3F)
68. Nawrot, P. nanot5: A pytorch framework for pre-training and fine-tuning t5-style models with limited resources. ArXiv, abs/2309.02373, 2023. [find](https://scholar.google.com/scholar?q=Nawrot%2C+P%2E+nanot5%3A+A+pytorch+framework+for+pre%2Dtraining+and+fine%2Dtuning+t5%2Dstyle+models+with+limited+resources%2E+ArXiv%2C+abs%2F2309%2E02373%2C+2023%2E)
69. Nawrot, P., Tworkowski, S., Tyrolski, M., Kaiser, L., Wu, Y., Szegedy, C., and Michalewski, H. Hierarchical transformers are more efficient language models. ArXiv, abs/2110.13711, 2021. [find](https://scholar.google.com/scholar?q=Nawrot%2C+P%2E%2C+Tworkowski%2C+S%2E%2C+Tyrolski%2C+M%2E%2C+Kaiser%2C+L%2E%2C+Wu%2C+Y%2E%2C+Szegedy%2C+C%2E%2C+and+Michalewski%2C+H%2E+Hierarchical+transformers+are+more+efficient+language+models%2E+ArXiv%2C+abs%2F2110%2E13711%2C+2021%2E)
70. Nawrot, P., Chorowski, J., La’ncucki, A., and Ponti, E. Efficient transformers with dynamic token pooling. ArXiv, abs/2211.09761, 2022. [find](https://scholar.google.com/scholar?q=Nawrot%2C+P%2E%2C+Chorowski%2C+J%2E%2C+La%E2%80%99ncucki%2C+A%2E%2C+and+Ponti%2C+E%2E+Efficient+transformers+with+dynamic+token+pooling%2E+ArXiv%2C+abs%2F2211%2E09761%2C+2022%2E)
71. Park, D., Papailiopoulos, D., and Lee, K. Active learning is a strong baseline for data subset selection. In Has it Trained Yet? NeurIPS 2022 Workshop, 2022. [find](https://scholar.google.com/scholar?q=Park%2C+D%2E%2C+Papailiopoulos%2C+D%2E%2C+and+Lee%2C+K%2E+Active+learning+is+a+strong+baseline+for+data+subset+selection%2E+In+Has+it+Trained+Yet%3F+NeurIPS+2022+Workshop%2C+2022%2E)
72. Patterson, D., Gonzalez, J., Hölzle, U., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D. R., Texier, M., and Dean, J. The carbon footprint of machine learning training will plateau, then shrink. Computer, 55(7):18–28, 2022. doi: 10.1109/MC.2022.3148714. — [doi:10.1109/mc.2022.3148714](https://doi.org/10.1109/mc.2022.3148714) [find](https://scholar.google.com/scholar?q=Patterson%2C+D%2E%2C+Gonzalez%2C+J%2E%2C+H%C3%B6lzle%2C+U%2E%2C+Le%2C+Q%2E%2C+Liang%2C+C%2E%2C+Munguia%2C+L%2E%2DM%2E%2C+Rothchild%2C+D%2E%2C+So%2C+D%2E+R%2E%2C+Texier%2C+M%2E%2C+and+Dean%2C+J%2E+The+carbon+footprint+of+machine+learning+training+will+plateau%2C+then+shri)
73. Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on a data diet: Finding important examples early in training. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in neural information processing systems, volume 34, pp. 20596–20607. Curran Associates… [find](https://scholar.google.com/scholar?q=Paul%2C+M%2E%2C+Ganguli%2C+S%2E%2C+and+Dziugaite%2C+G%2E+K%2E+Deep+learning+on+a+data+diet%3A+Finding+important+examples+early+in+training%2E+In+Ranzato%2C+M%2E%2C+Beygelzimer%2C+A%2E%2C+Dauphin%2C+Y%2E%2C+Liang%2C+P%2E%2C+and+Vaughan%2C+J%2E+W%2E+%28eds)
74. Portes, J., Blalock, D., Stephenson, C., and Frankle, J. Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates, November 2022. URL http://arxiv.org/abs/ 2206.00832. arXiv:2206.00832 [cs]. — [arXiv:2206.00832](https://arxiv.org/abs/2206.00832) [↗](http://arxiv.org/abs/)
75. Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. [find](https://scholar.google.com/scholar?q=Rae%2C+J%2E+W%2E%2C+Borgeaud%2C+S%2E%2C+Cai%2C+T%2E%2C+Millican%2C+K%2E%2C+Hoffmann%2C+J%2E%2C+Song%2C+F%2E%2C+Aslanides%2C+J%2E%2C+Henderson%2C+S%2E%2C+Ring%2C+R%2E%2C+Young%2C+S%2E%2C+et+al%2E+Scaling+language+models%3A+Methods%2C+analysis+%26+insights+from+training+g)
76. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67, 2020.
77. Sandler, M., Zhmoginov, A., Vladymyrov, M., and Miller, N. Training trajectories, mini-batch losses and the curious role of the learning rate. arXiv preprint arXiv:2301.02312, 2023. — [arXiv:2301.02312](https://arxiv.org/abs/2301.02312) [find](https://scholar.google.com/scholar?q=Sandler%2C+M%2E%2C+Zhmoginov%2C+A%2E%2C+Vladymyrov%2C+M%2E%2C+and+Miller%2C+N%2E+Training+trajectories%2C+mini%2Dbatch+losses+and+the+curious+role+of+the+learning+rate%2E+arXiv+preprint+arXiv%3A2301%2E02312%2C+2023%2E+%E2%80%94+%5BarXiv%3A2301%2E0231)
78. Sanyal, S., Kaddour, J., Kumar, A., and Sanghavi, S. Understanding the effectiveness of early weight averaging for training large language models. arXiv preprint arXiv:2306.03241, 2023. — [arXiv:2306.03241](https://arxiv.org/abs/2306.03241) [find](https://scholar.google.com/scholar?q=Sanyal%2C+S%2E%2C+Kaddour%2C+J%2E%2C+Kumar%2C+A%2E%2C+and+Sanghavi%2C+S%2E+Understanding+the+effectiveness+of+early+weight+averaging+for+training+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E03241%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E)
79. Schmidt, R. M., Schneider, F., and Hennig, P. Descending through a crowded valley -benchmarking deep learning optimizers. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9367–9376. PM… [find](https://scholar.google.com/scholar?q=Schmidt%2C+R%2E+M%2E%2C+Schneider%2C+F%2E%2C+and+Hennig%2C+P%2E+Descending+through+a+crowded+valley+%2Dbenchmarking+deep+learning+optimizers%2E+In+Meila%2C+M%2E+and+Zhang%2C+T%2E+%28eds%2E%29%2C+Proceedings+of+the+38th+International+Confe)
80. Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. Green ai, 2019. [find](https://scholar.google.com/scholar?q=Schwartz%2C+R%2E%2C+Dodge%2C+J%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Etzioni%2C+O%2E+Green+ai%2C+2019%2E)
81. Shazeer, N. Glu variants improve transformer, 2020. [find](https://scholar.google.com/scholar?q=Shazeer%2C+N%2E+Glu+variants+improve+transformer%2C+2020%2E)
82. Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. [find](https://scholar.google.com/scholar?q=Shazeer%2C+N%2E+and+Stern%2C+M%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2C+2018%2E)
83. Shazeer, N. M., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ArXiv, abs/1701.06538, 2017. [find](https://scholar.google.com/scholar?q=Shazeer%2C+N%2E+M%2E%2C+Mirhoseini%2C+A%2E%2C+Maziarz%2C+K%2E%2C+Davis%2C+A%2E%2C+Le%2C+Q%2E+V%2E%2C+Hinton%2C+G%2E+E%2E%2C+and+Dean%2C+J%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+ArXiv%2C+abs%2F1701%2E06538%2C+2)
84. Shen, L., Sun, Y., Yu, Z., Ding, L., Tian, X., and Tao, D. On efficient training of large-scale deep learning models: A literature review. arXiv preprint arXiv:2304.03589, 2023. [find](https://scholar.google.com/scholar?q=Shen%2C+L%2E%2C+Sun%2C+Y%2E%2C+Yu%2C+Z%2E%2C+Ding%2C+L%2E%2C+Tian%2C+X%2E%2C+and+Tao%2C+D%2E+On+efficient+training+of+large%2Dscale+deep+learning+models%3A+A+literature+review%2E+arXiv+preprint+arXiv%3A2304%2E03589%2C+2023%2E)
85. Shen, S., Walsh, P., Keutzer, K., Dodge, J., Peters, M., and Beltagy, I. Staged Training for Transformer Language Models. In Proceedings of the 39th International Conference on Machine Learning, pp. 19893–19908. PMLR, June 2022. URL [https://proceedings.mlr.press/v162/shen22f.html](https://proceedings.mlr.press/v162/shen22f.html). ISSN: 2640-3498. [↗](https://proceedings.mlr.press/v162/shen22f.html)
86. Smith, L. N. and Topin, N. Super-convergence: Very fast training of neural networks using large learning rates, 2018. [find](https://scholar.google.com/scholar?q=Smith%2C+L%2E+N%2E+and+Topin%2C+N%2E+Super%2Dconvergence%3A+Very+fast+training+of+neural+networks+using+large+learning+rates%2C+2018%2E)
87. Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. S. [Beyond neural scaling laws: beating power law scaling via data pruning](https://md.research.elamur.ai/free/en/papers/sorscher-2022-neural-scaling-laws-beating-power/). arXiv preprint arXiv:2206.14486, 2022.
88. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. [Dropout: a simple way to prevent neural networks from overfitting](https://md.research.elamur.ai/free/en/papers/srivastava-2014-dropout-simple-way-prevent/). The journal of machine learning research, 15(1):1929–1958, 2014.
89. Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., and Metzler, D. [Scaling laws vs model architectures: How does inductive bias influence scaling?](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/), 2022.
90. Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. [Efficient transformers: A survey](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/). ACM Computing Surveys, 55(6):1–28, 2022.
91. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), 2023.
92. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
93. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461, 2018.
94. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Advances in neural information processing systems, 32, 2019.
95. Wang, P., Panda, R., Hennigen, L. T., Greengard, P., Karlinsky, L., Feris, R., Cox, D. D., Wang, Z., and Kim, Y. Learning to grow pretrained models for efficient transformer training. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=cDY…](https://openreview.net/forum?id=cDY…) [↗](https://openreview.net/forum?id=cDY…)
96. Wang, X., Chen, Y., and Zhu, W. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555–4576, 2021. [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Chen%2C+Y%2E%2C+and+Zhu%2C+W%2E+A+survey+on+curriculum+learning%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+44%289%29%3A4555%E2%80%934576%2C+2021%2E)
97. Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran, A. S., Naik, A., Stap, D., Pathak, E., Karamanolakis, G., Lai, H. G., Purohit, I., Mondal, I., Anderson, J., Kuznia, K., Doshi, K., Patel, M., Pal, K. K., Moradshahi, M., Parmar, M., Purohit,… [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Mishra%2C+S%2E%2C+Alipoormolabashi%2C+P%2E%2C+Kordi%2C+Y%2E%2C+Mirzaei%2C+A%2E%2C+Arunkumar%2C+A%2E%2C+Ashok%2C+A%2E%2C+Dhanasekaran%2C+A%2E+S%2E%2C+Naik%2C+A%2E%2C+Stap%2C+D%2E%2C+Pathak%2C+E%2E%2C+Karamanolakis%2C+G%2E%2C+Lai%2C+H%2E+G%2E%2C+Purohit%2C+I%2E%2C+Mondal%2C+I)
98. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.[11903, 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/).
99. Wu, X., Dyer, E., and Neyshabur, B. arXiv:2012.03107, 2020. When do curricula work? arXiv preprint [find](https://scholar.google.com/scholar?q=Wu%2C+X%2E%2C+Dyer%2C+E%2E%2C+and+Neyshabur%2C+B%2E+arXiv%3A2012%2E03107%2C+2020%2E+When+do+curricula+work%3F+arXiv+preprint)
100. Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining, May 2023. URL http://arxiv.org/abs/2305.10429. arXiv:2305.10429 [cs]. [↗](http://arxiv.org/abs/2305.10429)
101. Xie, S. M., Santurkar, S., Ma, T., and Liang, P. Data selection for language models via importance resampling, 2023. URL https://arxiv.org/abs/2302.03169. [↗](https://arxiv.org/abs/2302.03169)
102. Xing, C., Arpit, D., Tsirigotis, C., and Bengio, Y. A walk with sgd. arXiv preprint arXiv:1802.08770, 2018. — [arXiv:1802.08770](https://arxiv.org/abs/1802.08770) [find](https://scholar.google.com/scholar?q=Xing%2C+C%2E%2C+Arpit%2C+D%2E%2C+Tsirigotis%2C+C%2E%2C+and+Bengio%2C+Y%2E+A+walk+with+sgd%2E+arXiv+preprint+arXiv%3A1802%2E08770%2C+2018%2E+%E2%80%94+%5BarXiv%3A1802%2E08770%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1802%2E08770%29)
103. Xue, F., Fu, Y., Zhou, W., Zheng, Z., and You, Y. To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis, May 2023. URL http://arxiv.org/abs/2305.13230. arXiv:2305.13230 [cs]. [↗](http://arxiv.org/abs/2305.13230)
104. Yao, X., Zheng, Y., Yang, X., and Yang, Z. Nlp from scratch without large-scale pretraining: A simple and efficient framework. In International Conference on Machine Learning, pp. 25438–25451. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Yao%2C+X%2E%2C+Zheng%2C+Y%2E%2C+Yang%2C+X%2E%2C+and+Yang%2C+Z%2E+Nlp+from+scratch+without+large%2Dscale+pretraining%3A+A+simple+and+efficient+framework%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+25438%E2%80%9325451%2E+PMLR%2C+2)
105. Zhang, C., Bengio, S., and Singer, Y. Are all layers created equal? arXiv:1902.01996, 2019. arXiv preprint — [arXiv:1902.01996](https://arxiv.org/abs/1902.01996) [find](https://scholar.google.com/scholar?q=Zhang%2C+C%2E%2C+Bengio%2C+S%2E%2C+and+Singer%2C+Y%2E+Are+all+layers+created+equal%3F+arXiv%3A1902%2E01996%2C+2019%2E+arXiv+preprint+%E2%80%94+%5BarXiv%3A1902%2E01996%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1902%2E01996%29)
106. Zhang, M. and He, Y. Accelerating training of transformer-based language models with progressive layer dropping. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in neural information processing systems, volume 33, pp. 14011–14023. Curran Associates, Inc., 2020.… [find](https://scholar.google.com/scholar?q=Zhang%2C+M%2E+and+He%2C+Y%2E+Accelerating+training+of+transformer%2Dbased+language+models+with+progressive+layer+dropping%2E+In+Larochelle%2C+H%2E%2C+Ranzato%2C+M%2E%2C+Hadsell%2C+R%2E%2C+Balcan%2C+M%2E%2C+and+Lin%2C+H%2E+%28eds%2E%29%2C+Advances+i)
107. Zhang, M. R., Lucas, J., Hinton, G., and Ba, J. Lookahead Optimizer: k steps forward, 1 step back, December 2019. URL http://arxiv.org/abs/1907.08610. arXiv:1907.08610 [cs, stat]. [↗](http://arxiv.org/abs/1907.08610)
108. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. [OPT: Open Pre-trained Transformer Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/), June 2022. URL http://a…
109. Zhu, C., Ni, R., Xu, Z., Kong, K., Huang, W. R., and Goldstein, T. Gradinit: Learning to initialize neural networks for stable and efficient training. Advances in Neural Information Processing Systems, 34:16410–16422, 2021. [find](https://scholar.google.com/scholar?q=Zhu%2C+C%2E%2C+Ni%2C+R%2E%2C+Xu%2C+Z%2E%2C+Kong%2C+K%2E%2C+Huang%2C+W%2E+R%2E%2C+and+Goldstein%2C+T%2E+Gradinit%3A+Learning+to+initialize+neural+networks+for+stable+and+efficient+training%2E+Advances+in+Neural+Information+Processing+Systems)
110. Zhu, Y., Kiros, R., Zemel, R. S., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In ICCV, pp. 19–27. IEEE Computer Society, 2015.
111. Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., and Shen, C. A survey on efficient training of transformers. arXiv preprint arXiv:2302.01107, 2023. — [arXiv:2302.01107](https://arxiv.org/abs/2302.01107) [find](https://scholar.google.com/scholar?q=Zhuang%2C+B%2E%2C+Liu%2C+J%2E%2C+Pan%2C+Z%2E%2C+He%2C+H%2E%2C+Weng%2C+Y%2E%2C+and+Shen%2C+C%2E+A+survey+on+efficient+training+of+transformers%2E+arXiv+preprint+arXiv%3A2302%2E01107%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E01107%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2302%2E0110)
112. Zhuang, J., Tang, T., Ding, Y., Tatikonda, S. C., Dvornek, N., Papademetris, X., and Duncan, J. AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients. In Advances in Neural Information Processing Systems, volume 33, pp. 18795–18806. Curran Associates, Inc., 2020. URL https://pr… [find](https://scholar.google.com/scholar?q=Zhuang%2C+J%2E%2C+Tang%2C+T%2E%2C+Ding%2C+Y%2E%2C+Tatikonda%2C+S%2E+C%2E%2C+Dvornek%2C+N%2E%2C+Papademetris%2C+X%2E%2C+and+Duncan%2C+J%2E+AdaBelief+Optimizer%3A+Adapting+Stepsizes+by+the+Belief+in+Observed+Gradients%2E+In+Advances+in+Neural+Infor)
113. zhuofan xia, Pan, X., Jin, X., He, Y., Xue’, H., Song, S., and Huang, G. Budgeted training for vision transformer. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=sVzBN-DlJRi](https://openreview.net/forum?id=sVzBN-DlJRi). [↗](https://openreview.net/forum?id=sVzBN-DlJRi)

*Updated 3 November 2026*