---
title: "ReZero is All You Need: Fast Convergence at Large Depth (2020)"
description: "The authors propose ReZero, a minimal change to the architecture. 85% accuracy. 44 references, cited by 33 papers here."
authors: ["Thomas Bachlechner", "Bodhisattwa Prasad Majumder", "Huanru Henry Mao", "Garrison W. Cottrell", "Julian McAuley"]
year: 2020
venue: "arXiv:2003.04887"
source: "https://arxiv.org/abs/2003.04887"
doi: "10.48550/arxiv.2003.04887"
arxiv: "2003.04887"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bachlechner-2020-rezero-all-need-fast-convergence/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bachlechner-2020-rezero-all-need-fast-convergence/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / ReZero is All You Need: Fast Convergence at Large Depth

# ReZero is All You Need: Fast Convergence at Large Depth

**Authors:** Thomas Bachlechner, [Bodhisattwa Prasad Majumder](https://md.research.elamur.ai/free/en/authors/bodhisattwa-prasad-majumder/), Huanru Henry Mao, Garrison W. Cottrell and [Julian McAuley](https://md.research.elamur.ai/free/en/authors/julian-mcauley/) · **Year:** 2020 · **Venue:** arXiv:2003.04887 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2003.04887) · [PDF](https://arxiv.org/pdf/2003.04887) · [DOI 10.48550/arxiv.2003.04887](https://doi.org/10.48550/arxiv.2003.04887) · [arXiv 2003.04887](https://arxiv.org/abs/2003.04887) · [OpenAlex](https://openalex.org/W3010768098)

**Citations:** 133 citations (OpenAlex, 2026-10-14) · [33](https://md.research.elamur.ai/free/en/papers/bachlechner-2020-rezero-all-need-fast-convergence/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Deep neural networks are hard to train because signals and gradients vanish or explode as they pass through many layers. The authors propose ReZero, a minimal change to the architecture. The output of each residual block is multiplied by its own scalar parameter α, which is initialized to zero. As a result, the network acts as the identity map at the start of training and automatically satisfies dynamical isometry. Earlier theory links this property to a network's ability to train efficiently. During optimization, the α parameters gradually reach the values they need. The method was tested on fully connected networks, on ResNet with the CIFAR-10 dataset, and on a Transformer for language modeling on enwiki8. ReZero makes it possible to train networks with thousands of layers and Transformers with more than 100 layers. It needs no LayerNorm, no warm-up and no auxiliary losses. Standard-depth models also converge noticeably faster. In practice, the method takes only a few lines of code, works with any architecture that has residual connections, and lowers the compute cost of training.

## Key points

- Layer update rule: x_{i+1} = x_i + α_i·F(x_i), where α_i is learned and initialized to 0. The code is published in the repository github.com/majumderb/rezero.
- A 12-layer Transformer on enwiki8 reaches 1.2 BPB in 8,800 iterations. Post-Norm with warm-up needs 13,690 iterations to get there, so ReZero is about 1.56 times faster. Without warm-up, Post-Norm diverges. ReZero with α = 1 at initialization needs 14,506 iterations. The final quality (1.17 BPB) matches that of the standard Transformer.
- A standard 64-layer Transformer diverges even with warm-up, while a ReZero model of the same depth reaches 1.11 BPB. A 128-layer version (101M parameters) reaches 1.08 BPB. This is close to the Character Transformer (64 layers, 219M parameters, 1.06 BPB), which needs auxiliary losses to get there.
- ReZero made it possible to train a fully connected network with 10,000 layers. ResNet on CIFAR-10 reaches 85% accuracy 32% faster.
- Early in training the α weights grow quickly and then slowly decrease. By the end, their mean is about 1/L: 0.0898 for 12 layers and 0.0185 for 64 layers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 101 works · 54,571 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

## Cite

Cite as: Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell and Julian McAuley. 2020. ReZero is All You Need: Fast Convergence at Large Depth. arXiv:2003.04887.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bachlechner2020rezero,
  title = {{ReZero is All You Need: Fast Convergence at Large Depth}},
  author = {Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell and Julian McAuley},
  year = {2020},
  doi = {10.48550/arxiv.2003.04887},
  eprint = {2003.04887},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.04887}
}
```

- [← previous Neural Syntactic Preordering for Controlled Paraphrase Generation](https://md.research.elamur.ai/free/en/papers/goyal-2020-neural-syntactic-preordering-controlled-paraphrase/)
- [next → ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation](https://md.research.elamur.ai/free/en/papers/xiao-2020-ernie-gen-enhanced-multi/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

44 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553), 2015. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Yoshua+Bengio%2C+and+Geoffrey+Hinton%2E+Deep+learning%2E+Nature%2C+521%287553%29%2C+2015%2E)
2. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In CVPR, 2016.
3. Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In NIPS, 2017. [find](https://scholar.google.com/scholar?q=G%C3%BCnter+Klambauer%2C+Thomas+Unterthiner%2C+Andreas+Mayr%2C+and+Sepp+Hochreiter%2E+Self%2Dnormalizing+neural+networks%2E+In+NIPS%2C+2017%2E)
4. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2C+2019%2E)
5. Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. In NIPS, pages 3360–3368, 2016. [find](https://scholar.google.com/scholar?q=Ben+Poole%2C+Subhaneil+Lahiri%2C+Maithra+Raghu%2C+Jascha+Sohl%2DDickstein%2C+and+Surya+Ganguli%2E+Exponential+expressivity+in+deep+neural+networks+through+transient+chaos%2E+In+NIPS%2C+pages+3360%E2%80%933368%2C+2016%2E)
6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+Delving+deep+into+rectifiers%3A+Surpassing+human%2Dlevel+performance+on+imagenet+classification%2E+In+ICCV%2C+2015%2E)
7. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. [Highway networks](https://md.research.elamur.ai/free/en/papers/srivastava-2015-highway-networks/). arXiv preprint arXiv:1505.00387, 2015.
8. Dmytro Mishkin and Jiri Matas. All you need is a good init. arXiv preprint arXiv:1511.06422, 2015. — [arXiv:1511.06422](https://arxiv.org/abs/1511.06422) [find](https://scholar.google.com/scholar?q=Dmytro+Mishkin+and+Jiri+Matas%2E+All+you+need+is+a+good+init%2E+arXiv+preprint+arXiv%3A1511%2E06422%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E06422%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E06422%29)
9. Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. arXiv preprint arXiv:1806.05393, 2018. — [arXiv:1806.05393](https://arxiv.org/abs/1806.05393) [find](https://scholar.google.com/scholar?q=Lechao+Xiao%2C+Yasaman+Bahri%2C+Jascha+Sohl%2DDickstein%2C+Samuel+S+Schoenholz%2C+and+Jeffrey+Pennington%2E+Dynamical+isometry+and+a+mean+field+theory+of+cnns%3A+How+to+train+10%2C000%2Dlayer+vanilla+convolutional+neur)
10. Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321, 2019. [find](https://scholar.google.com/scholar?q=Hongyi+Zhang%2C+Yann+N+Dauphin%2C+and+Tengyu+Ma%2E+Fixup+initialization%3A+Residual+learning+without+normalization%2E+arXiv+preprint+arXiv%3A1901%2E09321%2C+2019%2E)
11. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. [find](https://scholar.google.com/scholar?q=Sergey+Ioffe+and+Christian+Szegedy%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+arXiv+preprint+arXiv%3A1502%2E03167%2C+2015%2E)
12. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450, 2016.
13. Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In NIPS, 2017. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Samuel+Schoenholz%2C+and+Surya+Ganguli%2E+Resurrecting+the+sigmoid+in+deep+learning+through+dynamical+isometry%3A+theory+and+practice%2E+In+NIPS%2C+2017%2E)
14. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. [Character-level language modeling with deeper self-attention](https://md.research.elamur.ai/free/en/papers/al-rfou-2018-character-level-language-modeling-deeper/). In AAAI, volume 33, 2019.
15. Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In NIPS, 2014. [find](https://scholar.google.com/scholar?q=Guido+F+Montufar%2C+Razvan+Pascanu%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+On+the+number+of+linear+regions+of+deep+neural+networks%2E+In+NIPS%2C+2014%2E)
16. Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. arXiv preprint arXiv:1802.09979, 2018. — [arXiv:1802.09979](https://arxiv.org/abs/1802.09979) [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Samuel+S+Schoenholz%2C+and+Surya+Ganguli%2E+The+emergence+of+spectral+universality+in+deep+networks%2E+arXiv+preprint+arXiv%3A1802%2E09979%2C+2018%2E+%E2%80%94+%5BarXiv%3A1802%2E09979%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1)
17. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In NIPS, 2010. [find](https://scholar.google.com/scholar?q=Xavier+Glorot+and+Yoshua+Bengio%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+NIPS%2C+2010%2E)
18. Andrew M Saxe, James L McClelland, and Surya Ganguli. [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://md.research.elamur.ai/free/en/papers/saxe-2013-exact-solutions-nonlinear-dynamics-learning/). arXiv preprint arXiv:1312.6120, 2013.
19. Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. arXiv preprint arXiv:1611.01232, 2016. — [arXiv:1611.01232](https://arxiv.org/abs/1611.01232) [find](https://scholar.google.com/scholar?q=Samuel+S+Schoenholz%2C+Justin+Gilmer%2C+Surya+Ganguli%2C+and+Jascha+Sohl%2DDickstein%2E+Deep+information+propagation%2E+arXiv+preprint+arXiv%3A1611%2E01232%2C+2016%2E+%E2%80%94+%5BarXiv%3A1611%2E01232%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1611%2E01232)
20. Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In NIPS, 2017. [find](https://scholar.google.com/scholar?q=Ge+Yang+and+Samuel+Schoenholz%2E+Mean+field+residual+networks%3A+On+the+edge+of+chaos%2E+In+NIPS%2C+2017%2E)
21. Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S Schoenholz, Ed H Chi, and Jeffrey Pennington. Dynamical isometry and a mean field theory of lstms and grus. arXiv preprint arXiv:1901.08987, 2019. — [arXiv:1901.08987](https://arxiv.org/abs/1901.08987) [find](https://scholar.google.com/scholar?q=Dar+Gilboa%2C+Bo+Chang%2C+Minmin+Chen%2C+Greg+Yang%2C+Samuel+S+Schoenholz%2C+Ed+H+Chi%2C+and+Jeffrey+Pennington%2E+Dynamical+isometry+and+a+mean+field+theory+of+lstms+and+grus%2E+arXiv+preprint+arXiv%3A1901%2E08987%2C+2019)
22. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision. Springer, 2016. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+Identity+mappings+in+deep+residual+networks%2E+In+European+conference+on+computer+vision%2E+Springer%2C+2016%2E)
23. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. [find](https://scholar.google.com/scholar?q=Priya+Goyal%2C+Piotr+Doll%C3%A1r%2C+Ross+Girshick%2C+Pieter+Noordhuis%2C+Lukasz+Wesolowski%2C+Aapo+Kyrola%2C+Andrew+Tulloch%2C+Yangqing+Jia%2C+and+Kaiming+He%2E+Accurate%2C+large+minibatch+sgd%3A+Training+imagenet+in+1+hour%2E+ar)
24. Moritz Hardt and Tengyu Ma. arXiv:1611.04231, 2016. — [arXiv:1611.04231](https://arxiv.org/abs/1611.04231) [find](https://scholar.google.com/scholar?q=Moritz+Hardt+and+Tengyu+Ma%2E+arXiv%3A1611%2E04231%2C+2016%2E+%E2%80%94+%5BarXiv%3A1611%2E04231%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1611%2E04231%29)
25. Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Tong+He%2C+Zhi+Zhang%2C+Hang+Zhang%2C+Zhongyue+Zhang%2C+Junyuan+Xie%2C+and+Mu+Li%2E+Bag+of+tricks+for+image+classification+with+convolutional+neural+networks%2E+In+CVPR%2C+2019%2E)
26. Soham De and Samuel L Smith. Batch normalization biases deep residual networks towards shallow paths. arXiv preprint arXiv:2002.10444, 2020. — [arXiv:2002.10444](https://arxiv.org/abs/2002.10444) [find](https://scholar.google.com/scholar?q=Soham+De+and+Samuel+L+Smith%2E+Batch+normalization+biases+deep+residual+networks+towards+shallow+paths%2E+arXiv+preprint+arXiv%3A2002%2E10444%2C+2020%2E+%E2%80%94+%5BarXiv%3A2002%2E10444%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2002%2E10444%29)
27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS, 2017.
28. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(Jul), 2011. [find](https://scholar.google.com/scholar?q=John+Duchi%2C+Elad+Hazan%2C+and+Yoram+Singer%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+Journal+of+machine+learning+research%2C+12%28Jul%29%2C+2011%2E)
29. Pedro HP Savarese, Leonardo O Mazza, and Daniel R Figueiredo. Learning identity mappings with residual gates. arXiv preprint arXiv:1611.01260, 2016. — [arXiv:1611.01260](https://arxiv.org/abs/1611.01260) [find](https://scholar.google.com/scholar?q=Pedro+HP+Savarese%2C+Leonardo+O+Mazza%2C+and+Daniel+R+Figueiredo%2E+Learning+identity+mappings+with+residual+gates%2E+arXiv+preprint+arXiv%3A1611%2E01260%2C+2016%2E+%E2%80%94+%5BarXiv%3A1611%2E01260%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1611%2E012)
30. Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, volume 11006, page 1100612. International Society for Optics and Photonics, 2019. [find](https://scholar.google.com/scholar?q=Leslie+N+Smith+and+Nicholay+Topin%2E+Super%2Dconvergence%3A+Very+fast+training+of+neural+networks+using+large+learning+rates%2E+In+Artificial+Intelligence+and+Machine+Learning+for+Multi%2DDomain+Operations+Appl)
31. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Jill Burstein, Christy Doran, and Thamar Solorio, editors, NAACL, 2019.
32. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). In ICLR, 2017.
33. Matt Mahoney. Large text compression benchmark, 2009. [find](https://scholar.google.com/scholar?q=Matt+Mahoney%2E+Large+text+compression+benchmark%2C+2009%2E)
34. Toan Q. Nguyen and Julian Salazar. [Transformers without tears: Improving the normalization of self-attention](https://md.research.elamur.ai/free/en/papers/nguyen-2019-transformers-tears-improving-normalization-self/). CoRR, abs/1910.05895, 2019. Identity matters in deep learning. arXiv preprint
35. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). CoRR, abs/1910.10683, 2019.
36. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. arXiv preprint arXiv:2002.04745, 2020. [find](https://scholar.google.com/scholar?q=Ruibin+Xiong%2C+Yunchang+Yang%2C+Di+He%2C+Kai+Zheng%2C+Shuxin+Zheng%2C+Chen+Xing%2C+Huishuai+Zhang%2C+Yanyan+Lan%2C+Liwei+Wang%2C+and+Tie%2DYan+Liu%2E+On+layer+normalization+in+the+transformer+architecture%2E+arXiv+preprint+)
37. Microsoft. Turing-nlg: A 17-billion-parameter language model, 2020. [find](https://scholar.google.com/scholar?q=Microsoft%2E+Turing%2Dnlg%3A+A+17%2Dbillion%2Dparameter+language+model%2C+2020%2E)
38. Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tie-Yan Liu. Efficient training of BERT by progressively stacking. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, volume 97 of Proceedings of Machine Learning Research, 2019. [find](https://scholar.google.com/scholar?q=Linyuan+Gong%2C+Di+He%2C+Zhuohan+Li%2C+Tao+Qin%2C+Liwei+Wang%2C+and+Tie%2DYan+Liu%2E+Efficient+training+of+BERT+by+progressively+stacking%2E+In+Kamalika+Chaudhuri+and+Ruslan+Salakhutdinov%2C+editors%2C+ICML%2C+volume+97+of)
39. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
40. Emma Strubell, Ananya Ganesh, and Andrew McCallum. [Energy and policy considerations for deep learning in NLP](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August…
41. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. abs/1907.[10597, 2019](https://md.research.elamur.ai/free/en/papers/schwartz-2019-green-ai/).
42. Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, 2016. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+Bridging+nonlinearities+and+stochastic+regularizers+with+gaussian+error+linear+units%2E+CoRR%2C+abs%2F1606%2E08415%2C+2016%2E)
43. Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Reducing BERT pre-training time from 3 days to 76 minutes. CoRR, abs/1904.00962, 2019. [find](https://scholar.google.com/scholar?q=Yang+You%2C+Jing+Li%2C+Jonathan+Hseu%2C+Xiaodan+Song%2C+James+Demmel%2C+and+Cho%2DJui+Hsieh%2E+Reducing+BERT+pre%2Dtraining+time+from+3+days+to+76+minutes%2E+CoRR%2C+abs%2F1904%2E00962%2C+2019%2E)
44. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8), 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%2C+1997%2E)

*Updated 3 November 2026*