---
title: "Deep Networks with Stochastic Depth — Huang 2016"
description: "Huang et al. propose stochastic depth, a way to train a short network while keeping a… 0.2 accuracy. 35 references, cited by 168 papers here."
authors: ["Gao Huang", "Yu Sun", "Zhuang Liu", "Daniel Sedra", "Kilian Weinberger"]
year: 2016
venue: "arXiv:1603.09382"
source: "https://arxiv.org/abs/1603.09382"
doi: "10.48550/arxiv.1603.09382"
arxiv: "1603.09382"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/huang-2016-deep-networks-stochastic-depth/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/huang-2016-deep-networks-stochastic-depth/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / Deep Networks with Stochastic Depth

# Deep Networks with Stochastic Depth

**Authors:** [Gao Huang](https://md.research.elamur.ai/free/en/authors/gao-ming-huang/), [Yu Sun](https://md.research.elamur.ai/free/en/authors/yu-sun/), [Zhuang Liu](https://md.research.elamur.ai/free/en/authors/zhuang-liu/), Daniel Sedra and [Kilian Weinberger](https://md.research.elamur.ai/free/en/authors/kilian-q-weinberger/) · **Year:** 2016 · **Venue:** [arXiv:1603.09382](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/) ([Lecture notes in computer science 2016](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/2016/)) · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1603.09382) · [PDF](https://arxiv.org/pdf/1603.09382) · [DOI 10.48550/arxiv.1603.09382](https://doi.org/10.48550/arxiv.1603.09382) · [arXiv 1603.09382](https://arxiv.org/abs/1603.09382) · [OpenAlex](https://openalex.org/W2331143823)

**Citations:** 1,992 citations (OpenAlex, 2026-10-14) · [168](https://md.research.elamur.ai/free/en/papers/huang-2016-deep-networks-stochastic-depth/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Very deep convolutional networks are hard to train. Gradients weaken on their way back to early layers, input features fade as they pass forward, and training cost grows with every added layer. Huang et al. propose stochastic depth, a way to train a short network while keeping a deep one for inference. Starting from the ResNet architecture of He et al., each residual block is switched off at random for every mini-batch, so only its identity shortcut remains. The chance that a block stays active falls linearly from 1 near the input to a final value p_L (0.5 by default) at the last block. At test time the complete network is used.

Because the expected depth during training is much smaller, training is faster and gradients reaching early layers are stronger. The method also works as a regularizer, behaving much like a hidden combination of many networks that vary in depth. On CIFAR-10 it made a ResNet with over 1200 layers trainable, reaching 4.91% test error. This matters because it is simple, works alongside Batch Normalization, and pushes usable depth further.

## Key points
- Each ResBlock gets a Bernoulli gate. Survival probability decays linearly to p_L, and the linear schedule consistently beat a uniform one on a 110-layer ResNet (CIFAR-10).
- Results stay robust for p_L between roughly 0.4 and 0.8. Even at p_L = 0.2, accuracy remains competitive while training time drops by about 40%.
- A ResNet deeper than 1200 layers still improved, reaching 4.91% test error on CIFAR-10.
- Gradient magnitudes at the first convolutional layer were consistently larger than in constant-depth training, especially after learning-rate drops. By contrast, Dropout gave practically no gain on a 110-layer ResNet with Batch Normalization.
- On ImageNet with a 152-layer ResNet there was no immediate benefit. The authors suggest that larger models may be needed there.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 134 works · 65,159 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra and Kilian Weinberger. 2016. Deep Networks with Stochastic Depth. arXiv:1603.09382.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{huang2016deep,
  title = {{Deep Networks with Stochastic Depth}},
  author = {Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra and Kilian Weinberger},
  year = {2016},
  doi = {10.48550/arxiv.1603.09382},
  eprint = {1603.09382},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1603.09382}
}
```

- [← previous Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/)
- [next → Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1](https://md.research.elamur.ai/free/en/papers/courbariaux-2016-binarized-neural-networks-training-deep/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

35 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images (2009) [find](https://scholar.google.com/scholar?q=Krizhevsky%2C+A%2E%2C+Hinton%2C+G%2E%3A+Learning+multiple+layers+of+features+from+tiny+images+%282009%29)
2. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, IEEE (2009) 248–255 [find](https://scholar.google.com/scholar?q=Deng%2C+J%2E%2C+Dong%2C+W%2E%2C+Socher%2C+R%2E%2C+Li%2C+L%2EJ%2E%2C+Li%2C+K%2E%2C+Fei%2DFei%2C+L%2E%3A+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In%3A+Computer+Vision+and+Pattern+Recognition%2C+2009%2E+CVPR+2009%2E+IEEE+Conference+on%2C+IE)
3. Krizhevsky, A., Sutskever, I., Hinton, G.E.: [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In: Advances in neural information processing systems. (2012) 1097–1105
4. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: [Overfeat: Integrated recognition, localization and detection using convolutional networks](https://md.research.elamur.ai/free/en/papers/sermanet-2013-overfeat-integrated-recognition-localization-detection/). arXiv preprint arXiv:1312.6229 (2013)
5. Simonyan, K., Zisserman, A.: [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). arXiv preprint arXiv:1409.1556 (2014)
6. Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014) — [arXiv:1412.6806](https://arxiv.org/abs/1412.6806) [find](https://scholar.google.com/scholar?q=Springenberg%2C+J%2ET%2E%2C+Dosovitskiy%2C+A%2E%2C+Brox%2C+T%2E%2C+Riedmiller%2C+M%2E%3A+Striving+for+simplicity%3A+The+all+convolutional+net%2E+arXiv+preprint+arXiv%3A1412%2E6806+%282014%29+%E2%80%94+%5BarXiv%3A1412%2E6806%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1412%2E)
7. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 1–9 [find](https://scholar.google.com/scholar?q=Szegedy%2C+C%2E%2C+Liu%2C+W%2E%2C+Jia%2C+Y%2E%2C+Sermanet%2C+P%2E%2C+Reed%2C+S%2E%2C+Anguelov%2C+D%2E%2C+Erhan%2C+D%2E%2C+Vanhoucke%2C+V%2E%2C+Rabinovich%2C+A%2E%3A+Going+deeper+with+convolutions%2E+In%3A+Proceedings+of+the+IEEE+Conference+on+Computer+Vision)
8. He, K., Zhang, X., Ren, S., Sun, J.: [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). arXiv preprint arXiv:1512.03385 (2015)
9. Håstad, J., Goldmann, M.: On the power of small-depth threshold circuits. Computational Complexity 1(2) (1991) 113–129 [find](https://scholar.google.com/scholar?q=Ha%CC%8Astad%2C+J%2E%2C+Goldmann%2C+M%2E%3A+On+the+power+of+small%2Ddepth+threshold+circuits%2E+Computational+Complexity+1%282%29+%281991%29+113%E2%80%93129)
10. Håstad, J.: Computational limitations of small-depth circuits. (1987) [find](https://scholar.google.com/scholar?q=Ha%CC%8Astad%2C+J%2E%3A+Computational+limitations+of+small%2Ddepth+circuits%2E+%281987%29)
11. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on 5(2) (1994) 157–166 [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E%2C+Simard%2C+P%2E%2C+Frasconi%2C+P%2E%3A+Learning+long%2Dterm+dependencies+with+gradient+descent+is+difficult%2E+Neural+Networks%2C+IEEE+Transactions+on+5%282%29+%281994%29+157%E2%80%93166)
12. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: International conference on artificial intelligence and statistics. (2010) 249–256 [find](https://scholar.google.com/scholar?q=Glorot%2C+X%2E%2C+Bengio%2C+Y%2E%3A+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In%3A+International+conference+on+artificial+intelligence+and+statistics%2E+%282010%29+249%E2%80%93256)
13. Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. arXiv preprint arXiv:1409.5185 (2014) [find](https://scholar.google.com/scholar?q=Lee%2C+C%2EY%2E%2C+Xie%2C+S%2E%2C+Gallagher%2C+P%2E%2C+Zhang%2C+Z%2E%2C+Tu%2C+Z%2E%3A+Deeply%2Dsupervised+nets%2E+arXiv+preprint+arXiv%3A1409%2E5185+%282014%29)
14. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015) [find](https://scholar.google.com/scholar?q=Ioffe%2C+S%2E%2C+Szegedy%2C+C%2E%3A+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+arXiv+preprint+arXiv%3A1502%2E03167+%282015%29)
15. Srivastava, R.K., Greff, K., Schmidhuber, J.: [Highway networks](https://md.research.elamur.ai/free/en/papers/srivastava-2015-highway-networks/). arXiv preprint arXiv:1505.00387 (2015)
16. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: [Dropout: A simple way to prevent neural networks from overfitting](https://md.research.elamur.ai/free/en/papers/srivastava-2014-dropout-simple-way-prevent/). The Journal of Machine Learning Research 15(1) (2014) 1929–1958
17. Fahlman, S.E., Lebiere, C.: The cascade-correlation learning architecture. (1989) [find](https://scholar.google.com/scholar?q=Fahlman%2C+S%2EE%2E%2C+Lebiere%2C+C%2E%3A+The+cascade%2Dcorrelation+learning+architecture%2E+%281989%29)
18. Erhan, D., Bengio, Y., Courville, A., Manzagol, P.A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? The Journal of Machine Learning Research 11 (2010) 625–660 [find](https://scholar.google.com/scholar?q=Erhan%2C+D%2E%2C+Bengio%2C+Y%2E%2C+Courville%2C+A%2E%2C+Manzagol%2C+P%2EA%2E%2C+Vincent%2C+P%2E%2C+Bengio%2C+S%2E%3A+Why+does+unsupervised+pre%2Dtraining+help+deep+learning%3F+The+Journal+of+Machine+Learning+Research+11+%282010%29+625%E2%80%93660)
19. Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10). (2010) 807–814 [find](https://scholar.google.com/scholar?q=Nair%2C+V%2E%2C+Hinton%2C+G%2EE%2E%3A+Rectified+linear+units+improve+restricted+boltzmann+machines%2E+In%3A+Proceedings+of+the+27th+International+Conference+on+Machine+Learning+%28ICML%2D10%29%2E+%282010%29+807%E2%80%93814)
20. Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using dropconnect. In Dasgupta, S., Mcallester, D., eds.: Proceedings of the 30th International Conference on Machine Learning (ICML-13). Volume 28., JMLR Workshop and Conference Proceedings (May 2013) 1058–1066 [find](https://scholar.google.com/scholar?q=Wan%2C+L%2E%2C+Zeiler%2C+M%2E%2C+Zhang%2C+S%2E%2C+Cun%2C+Y%2EL%2E%2C+Fergus%2C+R%2E%3A+Regularization+of+neural+networks+using+dropconnect%2E+In+Dasgupta%2C+S%2E%2C+Mcallester%2C+D%2E%2C+eds%2E%3A+Proceedings+of+the+30th+International+Conference+on+M)
21. Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. arXiv preprint arXiv:1302.4389 (2013) — [arXiv:1302.4389](https://arxiv.org/abs/1302.4389) [find](https://scholar.google.com/scholar?q=Goodfellow%2C+I%2EJ%2E%2C+Warde%2DFarley%2C+D%2E%2C+Mirza%2C+M%2E%2C+Courville%2C+A%2E%2C+Bengio%2C+Y%2E%3A+Maxout+networks%2E+arXiv+preprint+arXiv%3A1302%2E4389+%282013%29+%E2%80%94+%5BarXiv%3A1302%2E4389%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1302%2E4389%29)
22. Smith, L.N., Hand, E.M., Doster, T.: Gradual dropin of layers to train very deep neural networks. CVPR (2016) [find](https://scholar.google.com/scholar?q=Smith%2C+L%2EN%2E%2C+Hand%2C+E%2EM%2E%2C+Doster%2C+T%2E%3A+Gradual+dropin+of+layers+to+train+very+deep+neural+networks%2E+CVPR+%282016%29)
23. Zagoruyko, S.: 92.45% on cifar-10 in torch (2015) [find](https://scholar.google.com/scholar?q=Zagoruyko%2C+S%2E%3A+92%2E45%25+on+cifar%2D10+in+torch+%282015%29)
24. Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013) — [arXiv:1312.4400](https://arxiv.org/abs/1312.4400) [find](https://scholar.google.com/scholar?q=Lin%2C+M%2E%2C+Chen%2C+Q%2E%2C+Yan%2C+S%2E%3A+Network+in+network%2E+arXiv+preprint+arXiv%3A1312%2E4400+%282013%29+%E2%80%94+%5BarXiv%3A1312%2E4400%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1312%2E4400%29)
25. Graham, B.: Fractional max-pooling. arXiv preprint arXiv:1412.6071 (2014) — [arXiv:1412.6071](https://arxiv.org/abs/1412.6071) [find](https://scholar.google.com/scholar?q=Graham%2C+B%2E%3A+Fractional+max%2Dpooling%2E+arXiv+preprint+arXiv%3A1412%2E6071+%282014%29+%E2%80%94+%5BarXiv%3A1412%2E6071%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1412%2E6071%29)
26. Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P.: Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014) — [arXiv:1412.6830](https://arxiv.org/abs/1412.6830) [find](https://scholar.google.com/scholar?q=Agostinelli%2C+F%2E%2C+Hoffman%2C+M%2E%2C+Sadowski%2C+P%2E%2C+Baldi%2C+P%2E%3A+Learning+activation+functions+to+improve+deep+neural+networks%2E+arXiv+preprint+arXiv%3A1412%2E6830+%282014%29+%E2%80%94+%5BarXiv%3A1412%2E6830%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F14)
27. Liang, M., Hu, X.: Recurrent convolutional neural network for object recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 3367–3375 [find](https://scholar.google.com/scholar?q=Liang%2C+M%2E%2C+Hu%2C+X%2E%3A+Recurrent+convolutional+neural+network+for+object+recognition%2E+In%3A+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2E+%282015%29+3367%E2%80%933375)
28. Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Ali, M., Adams, R.P., et al.: Scalable bayesian optimization using deep neural networks. arXiv preprint arXiv:1502.05700 (2015) — [arXiv:1502.05700](https://arxiv.org/abs/1502.05700) [find](https://scholar.google.com/scholar?q=Snoek%2C+J%2E%2C+Rippel%2C+O%2E%2C+Swersky%2C+K%2E%2C+Kiros%2C+R%2E%2C+Satish%2C+N%2E%2C+Sundaram%2C+N%2E%2C+Patwary%2C+M%2E%2C+Ali%2C+M%2E%2C+Adams%2C+R%2EP%2E%2C+et+al%2E%3A+Scalable+bayesian+optimization+using+deep+neural+networks%2E+arXiv+preprint+arXiv%3A1502)
29. Srivastava, R.K., Greff, K., Schmidhuber, J.: Training very deep networks. In: Advances in Neural Information Processing Systems. (2015) 2368–2376 [find](https://scholar.google.com/scholar?q=Srivastava%2C+R%2EK%2E%2C+Greff%2C+K%2E%2C+Schmidhuber%2C+J%2E%3A+Training+very+deep+networks%2E+In%3A+Advances+in+Neural+Information+Processing+Systems%2E+%282015%29+2368%E2%80%932376)
30. Lee, C.Y., Gallagher, P.W., Tu, Z.: Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. arXiv preprint arXiv:1509.08985 (2015) — [arXiv:1509.08985](https://arxiv.org/abs/1509.08985) [find](https://scholar.google.com/scholar?q=Lee%2C+C%2EY%2E%2C+Gallagher%2C+P%2EW%2E%2C+Tu%2C+Z%2E%3A+Generalizing+pooling+functions+in+convolutional+neural+networks%3A+Mixed%2C+gated%2C+and+tree%2E+arXiv+preprint+arXiv%3A1509%2E08985+%282015%29+%E2%80%94+%5BarXiv%3A1509%2E08985%5D%28https%3A%2F%2Farxiv%2Eo)
31. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS workshop on deep learning and unsupervised feature learning. Volume 2011., Granada, Spain (2011) 4 [find](https://scholar.google.com/scholar?q=Netzer%2C+Y%2E%2C+Wang%2C+T%2E%2C+Coates%2C+A%2E%2C+Bissacco%2C+A%2E%2C+Wu%2C+B%2E%2C+Ng%2C+A%2EY%2E%3A+Reading+digits+in+natural+images+with+unsupervised+feature+learning%2E+In%3A+NIPS+workshop+on+deep+learning+and+unsupervised+feature+learn)
32. Collobert, R., Kavukcuoglu, K., Farabet, C.: Torch7: A matlab-like environment for machine learning. In: BigLearn, NIPS Workshop. (2011) [find](https://scholar.google.com/scholar?q=Collobert%2C+R%2E%2C+Kavukcuoglu%2C+K%2E%2C+Farabet%2C+C%2E%3A+Torch7%3A+A+matlab%2Dlike+environment+for+machine+learning%2E+In%3A+BigLearn%2C+NIPS+Workshop%2E+%282011%29)
33. Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: Proceedings of the 30th international conference on machine learning (ICML-13). (2013) 1139–1147 [find](https://scholar.google.com/scholar?q=Sutskever%2C+I%2E%2C+Martens%2C+J%2E%2C+Dahl%2C+G%2E%2C+Hinton%2C+G%2E%3A+On+the+importance+of+initialization+and+momentum+in+deep+learning%2E+In%3A+Proceedings+of+the+30th+international+conference+on+machine+learning+%28ICML%2D13%29%2E)
34. Gross, S., Wilber, M.: Training and investigating residual nets (2016) [find](https://scholar.google.com/scholar?q=Gross%2C+S%2E%2C+Wilber%2C+M%2E%3A+Training+and+investigating+residual+nets+%282016%29)
35. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027 (2016) [find](https://scholar.google.com/scholar?q=He%2C+K%2E%2C+Zhang%2C+X%2E%2C+Ren%2C+S%2E%2C+Sun%2C+J%2E%3A+Identity+mappings+in+deep+residual+networks%2E+arXiv+preprint+arXiv%3A1603%2E05027+%282016%29)

*Updated 2 November 2026*