---
title: "Deep Learning Scaling is Predictable, Empirically — Hestness 2017"
description: "Practitioners widely assume that more data and bigger models improve accuracy, but there was no reliable way to… 40 references, cited by 97 papers here."
authors: ["Joel Hestness", "Sharan Narang", "Newsha Ardalani", "Gregory Diamos", "Heewoo Jun", "Hassan Kianinejad", "Md. Mostofa Ali Patwary", "Yang Yang", "Yanqi Zhou"]
year: 2017
venue: "arXiv:1712.00409"
source: "https://arxiv.org/abs/1712.00409"
doi: "10.48550/arxiv.1712.00409"
arxiv: "1712.00409"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/) / Deep Learning Scaling is Predictable, Empirically

# Deep Learning Scaling is Predictable, Empirically

**Authors:** [Joel Hestness](https://md.research.elamur.ai/free/en/authors/joel-hestness/), [Sharan Narang](https://md.research.elamur.ai/free/en/authors/sharan-narang/), Newsha Ardalani, [Gregory Diamos](https://md.research.elamur.ai/free/en/authors/greg-diamos/), [Heewoo Jun](https://md.research.elamur.ai/free/en/authors/heewoo-jun/), Hassan Kianinejad, Md. Mostofa Ali Patwary, [Yang Yang](https://md.research.elamur.ai/free/en/authors/yang-yang/) and [Yanqi Zhou](https://md.research.elamur.ai/free/en/authors/yanqi-zhou/) · **Year:** 2017 · **Venue:** arXiv:1712.00409 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1712.00409) · [PDF](https://arxiv.org/pdf/1712.00409) · [DOI 10.48550/arxiv.1712.00409](https://doi.org/10.48550/arxiv.1712.00409) · [arXiv 1712.00409](https://arxiv.org/abs/1712.00409) · [OpenAlex](https://openalex.org/W2775461895)

**Citations:** 400 citations (OpenAlex, 2026-10-14) · [97](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Practitioners widely assume that more data and bigger models improve accuracy, but there was no reliable way to forecast how much gain a given increase in training data would bring. Classical learning theory predicts power-law learning curves with exponents of −0.5 or −1, yet these values fail to describe real deep learning systems. The Baidu Research authors ran a large empirical study. Using ever-larger portions of the training data, they trained six deep neural network models spanning four application areas: speech recognition, image classification, language modeling and machine translation. At each data scale they searched for the best-fitting model size, then fitted curves for generalization error and model size. In every domain, error falls as a power law of training set size, but the slopes are much shallower than theory suggests. Better architectures and optimizers lower the curve without changing its steepness, and the parameter count needed grows sublinearly with data. This predictability helps teams estimate the data and compute needed to reach a target accuracy, debug models, decide whether to grow datasets, and plan computing systems.

## Key points
- Empirical learning-curve exponents fall roughly between −0.07 and −0.35, while prior theory predicts −0.5, −1 or −2. The paper says these observed values are still unexplained.
- The study covers six models in four domains and finds that the power-law behaviour holds across many models, optimizers, regularizers and loss functions.
- Within one domain, model improvements shift only the intercept of the learning curve, not its exponent.
- Learning curves have three regions:
  - a small-data region dominated by best guessing,
  - a power-law region,
  - a final plateau set by irreducible error (e.g., Bayes error).
- Required model size grows sublinearly with data. The authors also re-estimate an exponent of about 0.72 for the Banko and Brill (2001) models. Appendix B proves that a simple counting model for a coin flip converges with exponent −0.5.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cite

Cite as: Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang and Yanqi Zhou. 2017. Deep Learning Scaling is Predictable, Empirically. arXiv:1712.00409.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hestness2017deep,
  title = {{Deep Learning Scaling is Predictable, Empirically}},
  author = {Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang and Yanqi Zhou},
  year = {2017},
  doi = {10.48550/arxiv.1712.00409},
  eprint = {1712.00409},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1712.00409}
}
```

- [← previous Toward Controlled Generation of Text](https://md.research.elamur.ai/free/en/papers/hu-2017-toward-controlled-generation-text/)
- [next → A Convolutional Encoder Model for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-encoder-model-neural-machine/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/)

## References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Amari. A Universal Theorem on Learning Curves. Neural Networks, 6:161–166, 1993. [find](https://scholar.google.com/scholar?q=S%2E+Amari%2E+A+Universal+Theorem+on+Learning+Curves%2E+Neural+Networks%2C+6%3A161%E2%80%93166%2C+1993%2E)
2. S. Amari and N. Murata. Statistical Theory of Learning Curves under Entropic Loss Criterion. Neural Computation, 5(1):140–153, 1993. [find](https://scholar.google.com/scholar?q=S%2E+Amari+and+N%2E+Murata%2E+Statistical+Theory+of+Learning+Curves+under+Entropic+Loss+Criterion%2E+Neural+Computation%2C+5%281%29%3A140%E2%80%93153%2C+1993%2E)
3. S. Amari, N. Fujita, and S. Shinomoto. Four Types of Learning Curves. Neural Computation, 4(4):605–618, 1992. [find](https://scholar.google.com/scholar?q=S%2E+Amari%2C+N%2E+Fujita%2C+and+S%2E+Shinomoto%2E+Four+Types+of+Learning+Curves%2E+Neural+Computation%2C+4%284%29%3A605%E2%80%93618%2C+1992%2E)
4. D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, et al. Deep Speech 2: End-to-End Speech Recognition in English and Mandarin. In Proceedings of The International Conference on Machine Learning (ICML), pages 173–182, 2016. [our card](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/)
5. D. Arpit, S. Jastrz˛ebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, and S. Lacoste-Julien. A Closer Look at Memorization in Deep Networks. In Proceedings of the International Conference on Machine Learning, August 2017. [find](https://scholar.google.com/scholar?q=D%2E+Arpit%2C+S%2E+Jastrz%CB%9Bebski%2C+N%2E+Ballas%2C+D%2E+Krueger%2C+E%2E+Bengio%2C+M%2E+S%2E+Kanwal%2C+T%2E+Maharaj%2C+A%2E+Fischer%2C+A%2E+Courville%2C+Y%2E+Bengio%2C+and+S%2E+Lacoste%2DJulien%2E+A+Closer+Look+at+Memorization+in+Deep+Networks%2E+In+Pr)
6. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio. End-to-end Attention-based Large Vocabulary Speech Recognition. arXiv preprint arXiv:1508.04395v2, 2016. — [arXiv:1508.04395](https://arxiv.org/abs/1508.04395) [find](https://scholar.google.com/scholar?q=D%2E+Bahdanau%2C+J%2E+Chorowski%2C+D%2E+Serdyuk%2C+P%2E+Brakel%2C+and+Y%2E+Bengio%2E+End%2Dto%2Dend+Attention%2Dbased+Large+Vocabulary+Speech+Recognition%2E+arXiv+preprint+arXiv%3A1508%2E04395v2%2C+2016%2E+%E2%80%94+%5BarXiv%3A1508%2E04395%5D%28https%3A%2F%2Fa)
7. M. Banko and E. Brill. Scaling to Very Very Large Corpora for Natural Language Disambiguation. In Proceedings of Association of Computational Linguistics (ACL), January 2001. [our card](https://md.research.elamur.ai/free/en/papers/banko-2001-scaling-very-very-large-corpora/)
8. P. L. Bartlett and S. Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. In Journal of Machine Learning Research 3, pages 463–482, November 2002. [find](https://scholar.google.com/scholar?q=P%2E+L%2E+Bartlett+and+S%2E+Mendelson%2E+Rademacher+and+Gaussian+Complexities%3A+Risk+Bounds+and+Structural+Results%2E+In+Journal+of+Machine+Learning+Research+3%2C+pages+463%E2%80%93482%2C+November+2002%2E)
9. E. Battenberg, J. Chen, R. Child, A. Coates, Y. Gaur, Y. Li, H. Liu, S. Satheesh, D. Seetapun, A. Sriram, and Z. Zhu. Exploring Neural Transducers for End-to-end Speech Recognition. arXiv preprint arXiv:1707.07413, 2017. — [arXiv:1707.07413](https://arxiv.org/abs/1707.07413) [find](https://scholar.google.com/scholar?q=E%2E+Battenberg%2C+J%2E+Chen%2C+R%2E+Child%2C+A%2E+Coates%2C+Y%2E+Gaur%2C+Y%2E+Li%2C+H%2E+Liu%2C+S%2E+Satheesh%2C+D%2E+Seetapun%2C+A%2E+Sriram%2C+and+Z%2E+Zhu%2E+Exploring+Neural+Transducers+for+End%2Dto%2Dend+Speech+Recognition%2E+arXiv+preprint+arX)
10. A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability and the Vapnik-Chervonenkis Dimension. Journal of the ACM (JACM), 36(4):929–965, October 1989. [find](https://scholar.google.com/scholar?q=A%2E+Blumer%2C+A%2E+Ehrenfeucht%2C+D%2E+Haussler%2C+and+M%2E+K%2E+Warmuth%2E+Learnability+and+the+Vapnik%2DChervonenkis+Dimension%2E+Journal+of+the+ACM+%28JACM%29%2C+36%284%29%3A929%E2%80%93965%2C+October+1989%2E)
11. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. arXiv preprint arXiv:1312.3005, 2013. [our card](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/)
12. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio. Attention-based Models for Speech Recognition. In Advances in Neural Information Processing Systems (NIPS), pages 577–585, 2015. [our card](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/)
13. J. Collins, J. Sohl-Dickstein, and D. Sussillo. Capacity and Trainability in Recurrent Neural Networks. In Proceedings of The International Conference on Learning Representations (ICLR), 2017. [find](https://scholar.google.com/scholar?q=J%2E+Collins%2C+J%2E+Sohl%2DDickstein%2C+and+D%2E+Sussillo%2E+Capacity+and+Trainability+in+Recurrent+Neural+Networks%2E+In+Proceedings+of+The+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2017%2E)
14. G. K. Dziugaite and D. M. Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. In Proceedings of The International Conference on Machine Learning (ICML), 2017. [find](https://scholar.google.com/scholar?q=G%2E+K%2E+Dziugaite+and+D%2E+M%2E+Roy%2E+Computing+Nonvacuous+Generalization+Bounds+for+Deep+%28Stochastic%29+Neural+Networks+with+Many+More+Parameters+than+Training+Data%2E+In+Proceedings+of+The+International+Confer)
15. A. Ehrenfeucht, D. Haussler, M. Kearns, and L. Valiant. A General Lower Bound on the Number of Examples Needed for Learning. Information and Computation, 82:247–261, 1989. Workshop on Computing Learning Theory, 1988. [find](https://scholar.google.com/scholar?q=A%2E+Ehrenfeucht%2C+D%2E+Haussler%2C+M%2E+Kearns%2C+and+L%2E+Valiant%2E+A+General+Lower+Bound+on+the+Number+of+Examples+Needed+for+Learning%2E+Information+and+Computation%2C+82%3A247%E2%80%93261%2C+1989%2E+Workshop+on+Computing+Learni)
16. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In Proceedings of the International Conference on Machine Learning (ICML), 2006. [find](https://scholar.google.com/scholar?q=A%2E+Graves%2C+S%2E+Fern%C3%A1ndez%2C+F%2E+Gomez%2C+and+J%2E+Schmidhuber%2E+Connectionist+Temporal+Classification%3A+Labelling+Unsegmented+Sequence+Data+with+Recurrent+Neural+Networks%2E+In+Proceedings+of+the+International+Co)
17. G. Györgyi and N. Tishby. Statistical Theory of Learning a Rule. Neural Networks and Spin Glasses, pages 3–36, 1990. [find](https://scholar.google.com/scholar?q=G%2E+Gy%C3%B6rgyi+and+N%2E+Tishby%2E+Statistical+Theory+of+Learning+a+Rule%2E+Neural+Networks+and+Spin+Glasses%2C+pages+3%E2%80%9336%2C+1990%2E)
18. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, et al. Deep Speech: Scaling Up End-to-End Speech Recognition. arXiv preprint arXiv:1412.5567, 2014. [our card](https://md.research.elamur.ai/free/en/papers/hannun-2014-deep-speech-scaling-up-end/)
19. N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight VC-dimension Bounds for Piecewise Linear Neural Networks. In Proceedings of Machine Learning Research, volume 65, pages 1––12, 2017. [find](https://scholar.google.com/scholar?q=N%2E+Harvey%2C+C%2E+Liaw%2C+and+A%2E+Mehrabian%2E+Nearly%2Dtight+VC%2Ddimension+Bounds+for+Piecewise+Linear+Neural+Networks%2E+In+Proceedings+of+Machine+Learning+Research%2C+volume+65%2C+pages+1%E2%80%93%E2%80%9312%2C+2017%2E)
20. D. Haussler. Quantifying Inductive Bias: AI Learning Algorithms and Valiant’s Learning Framework. Artificial Intelligence, 36(2):177–221, 1988. [find](https://scholar.google.com/scholar?q=D%2E+Haussler%2E+Quantifying+Inductive+Bias%3A+AI+Learning+Algorithms+and+Valiant%E2%80%99s+Learning+Framework%2E+Artificial+Intelligence%2C+36%282%29%3A177%E2%80%93221%2C+1988%2E)
21. D. Haussler, M. Kearns, H. S. Seung, and N. Tishby. Rigorous Learning Curve Bounds from Statistical Mechanics. Machine Learning, 25(2):195–236, November 1996. [find](https://scholar.google.com/scholar?q=D%2E+Haussler%2C+M%2E+Kearns%2C+H%2E+S%2E+Seung%2C+and+N%2E+Tishby%2E+Rigorous+Learning+Curve+Bounds+from+Statistical+Mechanics%2E+Machine+Learning%2C+25%282%29%3A195%E2%80%93236%2C+November+1996%2E)
22. K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, June 2016. [find](https://scholar.google.com/scholar?q=K%2E+He%2C+X%2E+Zhang%2C+S%2E+Ren%2C+and+J%2E+Sun%2E+Deep+Residual+Learning+for+Image+Recognition%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+770%E2%80%93778%2C+June+2016%2E)
23. R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu. Exploring the Limits of Language Modeling. arXiv preprint arXiv:1602.02410v2, 2016. [find](https://scholar.google.com/scholar?q=R%2E+Jozefowicz%2C+O%2E+Vinyals%2C+M%2E+Schuster%2C+N%2E+Shazeer%2C+and+Y%2E+Wu%2E+Exploring+the+Limits+of+Language+Modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410v2%2C+2016%2E)
24. K. Kawaguchi, L. P. Kaelbling, and Y. Bengio. arXiv:1710.05468v1, October 2017. — [arXiv:1710.05468](https://arxiv.org/abs/1710.05468) [find](https://scholar.google.com/scholar?q=K%2E+Kawaguchi%2C+L%2E+P%2E+Kaelbling%2C+and+Y%2E+Bengio%2E+arXiv%3A1710%2E05468v1%2C+October+2017%2E+%E2%80%94+%5BarXiv%3A1710%2E05468%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1710%2E05468%29)
25. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of the Association for Computational Linguistics (ACL), 2017. [our card](https://md.research.elamur.ai/free/en/papers/klein-2017-opennmt-open-source-toolkit-neural/)
26. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the Association of Computational Linguistics, Interactiv… [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2C+H%2E+Hoang%2C+A%2E+Birch%2C+C%2E+Callison%2DBurch%2C+M%2E+Federico%2C+N%2E+Bertoldi%2C+B%2E+Cowan%2C+W%2E+Shen%2C+C%2E+Moran%2C+R%2E+Zens%2C+C%2E+Dyer%2C+O%2E+Bojar%2C+A%2E+Constantin%2C+and+E%2E+Herbst%2E+Moses%3A+Open+Source+Toolkit+for+Statist)
27. M. Luong, E. Brevdo, and R. Zhao. [https://github.com/tensorflow/nmt](https://github.com/tensorflow/nmt), 2017. [↗](https://github.com/tensorflow/nmt)
28. T. Luong, H. Pham, and C. D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421, 2015. [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/)
29. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv preprint arXiv:1409.0575, January 2015. [our card](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
30. H. Schwarze and J. Hertz. Statistical Mechanics of Learning in a Large Committee Machine. In Advances in Neural Information Processing Systems, (NIPS), pages 523–530, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=H%2E+Schwarze+and+J%2E+Hertz%2E+Statistical+Mechanics+of+Learning+in+a+Large+Committee+Machine%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+%28NIPS%29%2C+pages+523%E2%80%93530%2C+San+Francisco%2C+CA%2C+USA%2C+1993%2E+Mor)
31. R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units. arXiv preprint arXiv:1508.07909, 2016a. [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
32. R. Sennrich, B. Haddow, and A. Birch. Edinburgh Neural Machine Translation Systems for WMT 16. arXiv preprint arXiv:1606.02891, 2016b. [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/)
33. H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical Mechanics of Learning from Examples. Physical Review A, 45:6056–6091, April 1992. [find](https://scholar.google.com/scholar?q=H%2E+S%2E+Seung%2C+H%2E+Sompolinsky%2C+and+N%2E+Tishby%2E+Statistical+Mechanics+of+Learning+from+Examples%2E+Physical+Review+A%2C+45%3A6056%E2%80%936091%2C+April+1992%2E)
34. S. L. Smith and Q. V. Le. A Bayesian Perspective on Generalization and Stochastic Gradient Descent. arXiv preprint arXiv:1710.06451v2, October 2017. — [arXiv:1710.06451](https://arxiv.org/abs/1710.06451) [find](https://scholar.google.com/scholar?q=S%2E+L%2E+Smith+and+Q%2E+V%2E+Le%2E+A+Bayesian+Perspective+on+Generalization+and+Stochastic+Gradient+Descent%2E+arXiv+preprint+arXiv%3A1710%2E06451v2%2C+October+2017%2E+%E2%80%94+%5BarXiv%3A1710%2E06451%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1710%2E064)
35. C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. In Proceedings of the International Conference on Computer Vision (ICCV), 2017. [find](https://scholar.google.com/scholar?q=C%2E+Sun%2C+A%2E+Shrivastava%2C+S%2E+Singh%2C+and+A%2E+Gupta%2E+Revisiting+Unreasonable+Effectiveness+of+Data+in+Deep+Learning+Era%2E+In+Proceedings+of+the+International+Conference+on+Computer+Vision+%28ICCV%29%2C+2017%2E)
36. H. G. Tucker. A Generalization of the Glivenko-Cantelli Theorem. The Annals of Mathematical Statistics, 30 (3):828–830, September 1959. [find](https://scholar.google.com/scholar?q=H%2E+G%2E+Tucker%2E+A+Generalization+of+the+Glivenko%2DCantelli+Theorem%2E+The+Annals+of+Mathematical+Statistics%2C+30+%283%29%3A828%E2%80%93830%2C+September+1959%2E)
37. V. Vapnik. An Overview of Statistical Learning Theory. In IEEE Transactions on Neural Networks, volume 10, pages 988–999, September 1998. [find](https://scholar.google.com/scholar?q=V%2E+Vapnik%2E+An+Overview+of+Statistical+Learning+Theory%2E+In+IEEE+Transactions+on+Neural+Networks%2C+volume+10%2C+pages+988%E2%80%93999%2C+September+1998%2E)
38. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding Deep Learning Requires Rethinking Generalization. arXiv preprint arXiv:1611.03530v2, 2017. [find](https://scholar.google.com/scholar?q=C%2E+Zhang%2C+S%2E+Bengio%2C+M%2E+Hardt%2C+B%2E+Recht%2C+and+O%2E+Vinyals%2E+Understanding+Deep+Learning+Requires+Rethinking+Generalization%2E+arXiv+preprint+arXiv%3A1611%2E03530v2%2C+2017%2E)
39. J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber. Recurrent Highway Networks. In Proceedings of The International Conference on Machine Learning (ICML), 2017. [find](https://scholar.google.com/scholar?q=J%2E+G%2E+Zilly%2C+R%2E+K%2E+Srivastava%2C+J%2E+Koutn%C3%ADk%2C+and+J%2E+Schmidhuber%2E+Recurrent+Highway+Networks%2E+In+Proceedings+of+The+International+Conference+on+Machine+Learning+%28ICML%29%2C+2017%2E)
40. Generalization in Deep Learning. arXiv preprint Neural Machine Translation (seq2seq) Tutorial. [find](https://scholar.google.com/scholar?q=Generalization+in+Deep+Learning%2E+arXiv+preprint+Neural+Machine+Translation+%28seq2seq%29+Tutorial%2E)