---
title: "Regularizing Neural Networks by Penalizing — Pereyra 2017"
description: "Common regularizers such as dropout, weight decay and batch normalization act on a network's weights or hidden activations."
authors: ["Gabriel Pereyra", "George Tucker", "Jan Chorowski", "Łukasz Kaiser", "Geoffrey Hinton"]
year: 2017
venue: "arXiv:1701.06548"
source: "https://arxiv.org/abs/1701.06548"
doi: "10.48550/arxiv.1701.06548"
arxiv: "1701.06548"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/) / Regularizing Neural Networks by Penalizing Confident Output Distributions

# Regularizing Neural Networks by Penalizing Confident Output Distributions

**Authors:** Gabriel Pereyra, George Tucker, [Jan Chorowski](https://md.research.elamur.ai/free/en/authors/jan-chorowski/), [Łukasz Kaiser](https://md.research.elamur.ai/free/en/authors/lukasz-kaiser/), [Geoffrey Hinton](https://md.research.elamur.ai/free/en/authors/geoffrey-hinton/) · **Year:** 2017 · **Venue:** arXiv:1701.06548 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1701.06548) · [PDF](https://arxiv.org/pdf/1701.06548) · [DOI 10.48550/arxiv.1701.06548](https://doi.org/10.48550/arxiv.1701.06548) · [arXiv 1701.06548](https://arxiv.org/abs/1701.06548) · [OpenAlex](https://openalex.org/W2581377246)

**Citations:** 326 citations (openalex-snapshot · 2026-10-14) · [156](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Common regularizers such as dropout, weight decay and batch normalization act on a network's weights or hidden activations. This paper instead regularizes the output distribution. The authors add a confidence penalty to the training objective: the standard negative log-likelihood minus β times the entropy of the softmax prediction. This discourages the overly peaked outputs that often signal overfitting. They also describe annealed and hinge-thresholded variants. In addition, they show that uniform label smoothing differs from this penalty only in the direction of the KL divergence to the uniform distribution. Both regularizers, including a unigram form of label smoothing, are evaluated on six benchmarks: MNIST, CIFAR-10, Penn Treebank, WMT'14 English-to-German, TIMIT and WSJ, using TensorFlow implementations. When added to strong existing models without retuning their hyperparameters, both methods improved results across tasks. The practical value is a cheap loss term with a single hyperparameter. It does not depend on how the network is parameterized and transfers across vision, language and speech.

## Key points
- **Objective:** loss = −log p(y|x) − β·H(p). The entropy gradient with respect to each logit has a simple closed form: each class probability multiplies how far that class's negative log-probability lies from the entropy. A thresholded version penalizes only outputs whose entropy falls below Γ. It converges faster but adds a hyperparameter, so most experiments use the plain β version.
- **Link to label smoothing:** uniform label smoothing corresponds to adding D_KL(u‖p). Swapping the arguments to D_KL(p‖u) gives the confidence penalty, which suggests other target distributions could be used.
- **MNIST (permutation-invariant, 2 hidden layers × 1024 ReLU units):** test error was 1.17 ± 0.06% with the confidence penalty, 1.23% with label smoothing and 1.28% with dropout. The best settings were β = 1.0 and a smoothing value of 0.1.
- **Training behaviour:** both output regularizers produced smoother softmax distributions than dropout, which pushes probabilities toward 0 or 1. They also yielded smaller gradient norms and converged faster.
- **Excluded baseline:** virtual adversarial training (VAT) was not compared because its multiple hyperparameters and extra forward/backward passes make it much more expensive.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

156 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 145 not yet.

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
- [mixup: Beyond Empirical Risk Minimization](https://md.research.elamur.ai/free/en/papers/zhang-2017-mixup-empirical-risk-minimization/)
- [Categorical Reparameterization with Gumbel-Softmax](https://md.research.elamur.ai/free/en/papers/jang-2016-categorical-reparameterization-gumbel-softmax/)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/)
- [Scaling Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://md.research.elamur.ai/free/en/papers/wu-2019-pay-less-attention-lightweight-dynamic/)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://md.research.elamur.ai/free/en/papers/henderson-2020-convert-efficient-accurate-conversational-representations/)
- [Classical Structured Prediction Losses for Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/edunov-2018-classical-structured-prediction-losses-sequence/)
- [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://md.research.elamur.ai/free/en/papers/wieting-2019-beyond-bleu-training-neural-machine/)
- [Locally Typical Sampling](https://md.research.elamur.ai/free/en/papers/meister-2023-locally-typical-sampling/)
- [Document-level Neural MT: A Systematic Comparison](https://md.research.elamur.ai/free/en/papers/lopes-2020-document-level-neural-mt-systematic/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{pereyra2017regularizing,
  title = {{Regularizing Neural Networks by Penalizing Confident Output Distributions}},
  author = {Pereyra, Gabriel and Tucker, George and Chorowski, Jan and Kaiser, Łukasz and Hinton, Geoffrey},
  year = {2017},
  doi = {10.48550/arxiv.1701.06548},
  eprint = {1701.06548},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1701.06548}
}
```

- [← previous Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison](https://md.research.elamur.ai/free/en/papers/onto-2017-raganato-word-sense-disambiguation-unified-evaluation/)
- [next → Learned in Translation: Contextualized Word Vectors](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/)

## References

47 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016. [find](https://scholar.google.com/scholar?q=Mart%C4%B1n+Abadi%2C+Ashish+Agarwal%2C+Paul+Barham%2C+Eugene+Brevdo%2C+Zhifeng+Chen%2C+Craig+Citro%2C+Greg+S+Corrado%2C+Andy+Davis%2C+Jeffrey+Dean%2C+Matthieu+Devin%2C+et+al%2E+Tensorflow%3A+Large%2Dscale+machine+learning+on+hetero) [our card](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-large-scale-machine-learning/)
2. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio. End-to-end attention-based large vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4945–4949, March 2016. doi: 10.1109/ICASSP.2016.7472618. — [doi:10.1109/icassp.2016.7472618](https://doi.org/10.1109/icassp.2016.7472618) [find](https://scholar.google.com/scholar?q=D%2E+Bahdanau%2C+J%2E+Chorowski%2C+D%2E+Serdyuk%2C+P%2E+Brakel%2C+and+Y%2E+Bengio%2E+End%2Dto%2Dend+attention%2Dbased+large+vocabulary+speech+recognition%2E+In+2016+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+P)
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+arXiv+preprint+arXiv%3A1409%2E0473%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
4. Adam L Berger, Vincent J Della Pietra, and Stephen A Della Pietra. A maximum entropy approach to natural language processing. Computational linguistics, 22(1):39–71, 1996. [find](https://scholar.google.com/scholar?q=Adam+L+Berger%2C+Vincent+J+Della+Pietra%2C+and+Stephen+A+Della+Pietra%2E+A+maximum+entropy+approach+to+natural+language+processing%2E+Computational+linguistics%2C+22%281%29%3A39%E2%80%9371%2C+1996%2E) [our card](https://md.research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/)
5. Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535–541. ACM, 2006. [find](https://scholar.google.com/scholar?q=Cristian+Bucilu%2C+Rich+Caruana%2C+and+Alexandru+Niculescu%2DMizil%2E+Model+compression%2E+In+Proceedings+of+the+12th+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining%2C+pp%2E+535%E2%80%93541%2E+ACM)
6. Christian Buck, Kenneth Heafield, and Bas Van Ooyen. N-gram counts and language models from the common crawl. In LREC, volume 2, pp. 4. Citeseer, 2014. [find](https://scholar.google.com/scholar?q=Christian+Buck%2C+Kenneth+Heafield%2C+and+Bas+Van+Ooyen%2E+N%2Dgram+counts+and+language+models+from+the+common+crawl%2E+In+LREC%2C+volume+2%2C+pp%2E+4%2E+Citeseer%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/buck-2014-n-gram-counts-language-models/)
7. Parameters Validation Test 26.5M 5.7M 5.1M 5.9M -- - - 27.3 18.6 14.7 12.9 6.6M 6.6M 6.6M 6.6M 17.9 14.7 14.0 ± 0.25 17.2 14.2 11.3 11.0 ± 0.35 12.7 William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint arXiv:1508.01211, 2015. — [arXiv:1508.01211](https://arxiv.org/abs/1508.01211) [find](https://scholar.google.com/scholar?q=Parameters+Validation+Test+26%2E5M+5%2E7M+5%2E1M+5%2E9M+%2D%2D+%2D+%2D+27%2E3+18%2E6+14%2E7+12%2E9+6%2E6M+6%2E6M+6%2E6M+6%2E6M+17%2E9+14%2E7+14%2E0+%C2%B1+0%2E25+17%2E2+14%2E2+11%2E3+11%2E0+%C2%B1+0%2E35+12%2E7+William+Chan%2C+Navdeep+Jaitly%2C+Quoc+V+Le%2C+and+Oriol+)
8. William Chan, Yu Zhang, Quoc Le, and Navdeep Jaitly. Latent sequence decompositions. arXiv preprint arXiv:1610.03035, 2016. — [arXiv:1610.03035](https://arxiv.org/abs/1610.03035) [find](https://scholar.google.com/scholar?q=William+Chan%2C+Yu+Zhang%2C+Quoc+Le%2C+and+Navdeep+Jaitly%2E+Latent+sequence+decompositions%2E+arXiv+preprint+arXiv%3A1610%2E03035%2C+2016%2E+%E2%80%94+%5BarXiv%3A1610%2E03035%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1610%2E03035%29)
9. Sébastien Jean Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. 2015. [find](https://scholar.google.com/scholar?q=Se%CC%81bastien+Jean+Kyunghyun+Cho%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+On+using+very+large+target+vocabulary+for+neural+machine+translation%2E+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/)
10. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems, pp. 577–585, 2015. [find](https://scholar.google.com/scholar?q=Jan+K+Chorowski%2C+Dzmitry+Bahdanau%2C+Dmitriy+Serdyuk%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+Attention%2Dbased+models+for+speech+recognition%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+577%E2%80%93585%2C+) [our card](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/)
11. Yarin Gal. A theoretically grounded application of dropout in recurrent neural networks. arXiv preprint arXiv:1512.05287, 2015. [find](https://scholar.google.com/scholar?q=Yarin+Gal%2E+A+theoretically+grounded+application+of+dropout+in+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1512%2E05287%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/)
12. Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML, volume 14, pp. 1764–1772, 2014. [find](https://scholar.google.com/scholar?q=Alex+Graves+and+Navdeep+Jaitly%2E+Towards+end%2Dto%2Dend+speech+recognition+with+recurrent+neural+networks%2E+In+ICML%2C+volume+14%2C+pp%2E+1764%E2%80%931772%2C+2014%2E)
13. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369–376. ACM, 2006. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Santiago+Ferna%CC%81ndez%2C+Faustino+Gomez%2C+and+Ju%CC%88rgen+Schmidhuber%2E+Connectionist+temporal+classification%3A+labelling+unsegmented+sequence+data+with+recurrent+neural+networks%2E+In+Proceedings+of+)
14. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645–6649. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Abdel%2Drahman+Mohamed%2C+and+Geoffrey+Hinton%2E+Speech+recognition+with+deep+recurrent+neural+networks%2E+In+2013+IEEE+international+conference+on+acoustics%2C+speech+and+signal+processing%2C+pp%2E+66)
15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+Deep+residual+learning+for+image+recognition%2E+arXiv+preprint+arXiv%3A1512%2E03385%2C+2015%2E)
16. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
17. Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016a. — [arXiv:1608.06993](https://arxiv.org/abs/1608.06993) [find](https://scholar.google.com/scholar?q=Gao+Huang%2C+Zhuang+Liu%2C+and+Kilian+Q+Weinberger%2E+Densely+connected+convolutional+networks%2E+arXiv+preprint+arXiv%3A1608%2E06993%2C+2016a%2E+%E2%80%94+%5BarXiv%3A1608%2E06993%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1608%2E06993%29)
18. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382, 2016b. [find](https://scholar.google.com/scholar?q=Gao+Huang%2C+Yu+Sun%2C+Zhuang+Liu%2C+Daniel+Sedra%2C+and+Kilian+Weinberger%2E+Deep+networks+with+stochastic+depth%2E+arXiv+preprint+arXiv%3A1603%2E09382%2C+2016b%2E)
19. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. [find](https://scholar.google.com/scholar?q=Sergey+Ioffe+and+Christian+Szegedy%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+arXiv+preprint+arXiv%3A1502%2E03167%2C+2015%2E)
20. Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957. [find](https://scholar.google.com/scholar?q=Edwin+T+Jaynes%2E+Information+theory+and+statistical+mechanics%2E+Physical+review%2C+106%284%29%3A620%2C+1957%2E)
21. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2C+2016%2E)
22. Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648, 2016. — [arXiv:1605.07648](https://arxiv.org/abs/1605.07648) [find](https://scholar.google.com/scholar?q=Gustav+Larsson%2C+Michael+Maire%2C+and+Gregory+Shakhnarovich%2E+Fractalnet%3A+Ultra%2Ddeep+neural+networks+without+residuals%2E+arXiv+preprint+arXiv%3A1605%2E07648%2C+2016%2E+%E2%80%94+%5BarXiv%3A1605%2E07648%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F16)
23. Yuping Luo, Chung-Cheng Chiu, Navdeep Jaitly, and Ilya Sutskever. Learning online alignments with continuous rewards policy gradient. arXiv preprint arXiv:1608.01281, 2016. — [arXiv:1608.01281](https://arxiv.org/abs/1608.01281) [find](https://scholar.google.com/scholar?q=Yuping+Luo%2C+Chung%2DCheng+Chiu%2C+Navdeep+Jaitly%2C+and+Ilya+Sutskever%2E+Learning+online+alignments+with+continuous+rewards+policy+gradient%2E+arXiv+preprint+arXiv%3A1608%2E01281%2C+2016%2E+%E2%80%94+%5BarXiv%3A1608%2E01281%5D%28https%3A)
24. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025, 2015. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D+Manning%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1508%2E04025%2C+2015%2E)
25. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993. [find](https://scholar.google.com/scholar?q=Mitchell+P+Marcus%2C+Mary+Ann+Marcinkiewicz%2C+and+Beatrice+Santorini%2E+Building+a+large+annotated+corpus+of+english%3A+The+penn+treebank%2E+Computational+linguistics%2C+19%282%29%3A313%E2%80%93330%2C+1993%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
26. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Caiming+Xiong%2C+James+Bradbury%2C+and+Richard+Socher%2E+Pointer+sentinel+mixture+models%2E+arXiv+preprint+arXiv%3A1609%2E07843%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/)
27. David Miller, Ajit V Rao, Kenneth Rose, and Allen Gersho. A global optimization technique for statistical classifier design. IEEE Transactions on Signal Processing, 44(12):3108–3122, 1996. [find](https://scholar.google.com/scholar?q=David+Miller%2C+Ajit+V+Rao%2C+Kenneth+Rose%2C+and+Allen+Gersho%2E+A+global+optimization+technique+for+statistical+classifier+design%2E+IEEE+Transactions+on+Signal+Processing%2C+44%2812%29%3A3108%E2%80%933122%2C+1996%2E)
28. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing by virtual adversarial examples. arXiv preprint arXiv:1507.00677, 2015. — [arXiv:1507.00677](https://arxiv.org/abs/1507.00677) [find](https://scholar.google.com/scholar?q=Takeru+Miyato%2C+Shin%2Dichi+Maeda%2C+Masanori+Koyama%2C+Ken+Nakae%2C+and+Shin+Ishii%2E+Distributional+smoothing+by+virtual+adversarial+examples%2E+arXiv+preprint+arXiv%3A1507%2E00677%2C+2015%2E+%E2%80%94+%5BarXiv%3A1507%2E00677%5D%28https%3A)
29. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. arXiv preprint arXiv:1602.01783, 2016. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Adria+Puigdomenech+Badia%2C+Mehdi+Mirza%2C+Alex+Graves%2C+Timothy+P+Lillicrap%2C+Tim+Harley%2C+David+Silver%2C+and+Koray+Kavukcuoglu%2E+Asynchronous+methods+for+deep+reinforcement+learning%2E+arXiv+pr)
30. Abdel-rahman Mohamed, George E Dahl, and Geoffrey Hinton. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):14–22, 2012. [find](https://scholar.google.com/scholar?q=Abdel%2Drahman+Mohamed%2C+George+E+Dahl%2C+and+Geoffrey+Hinton%2E+Acoustic+modeling+using+deep+belief+networks%2E+IEEE+Transactions+on+Audio%2C+Speech%2C+and+Language+Processing%2C+20%281%29%3A14%E2%80%9322%2C+2012%2E)
31. Mohammad Norouzi, Dale Schuurmans, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, and Yonghui Wu. Reward augmented maximum likelihood for neural structured prediction. In Advances in Neural Information Processing Systems, 2016. [find](https://scholar.google.com/scholar?q=Mohammad+Norouzi%2C+Dale+Schuurmans%2C+Samy+Bengio%2C+Zhifeng+Chen%2C+Navdeep+Jaitly%2C+Mike+Schuster%2C+and+Yonghui+Wu%2E+Reward+augmented+maximum+likelihood+for+neural+structured+prediction%2E+In+Advances+in+Neural)
32. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. The kaldi speech recognition toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Unders… [find](https://scholar.google.com/scholar?q=Daniel+Povey%2C+Arnab+Ghoshal%2C+Gilles+Boulianne%2C+Lukas+Burget%2C+Ondrej+Glembek%2C+Nagendra+Goel%2C+Mirko+Hannemann%2C+Petr+Motlicek%2C+Yanmin+Qian%2C+Petr+Schwarz%2C+Jan+Silovsky%2C+Georg+Stemmer%2C+and+Karel+Vesely%2E+Th)
33. IEEE Signal Processing Society, December 2011. IEEE Catalog No.: CFP11SRW-USB. [find](https://scholar.google.com/scholar?q=IEEE+Signal+Processing+Society%2C+December+2011%2E+IEEE+Catalog+No%2E%3A+CFP11SRW%2DUSB%2E)
34. Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859, 2016. [find](https://scholar.google.com/scholar?q=Ofir+Press+and+Lior+Wolf%2E+Using+the+output+embedding+to+improve+language+models%2E+arXiv+preprint+arXiv%3A1608%2E05859%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/)
35. Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014. — [arXiv:1412.6596](https://arxiv.org/abs/1412.6596) [find](https://scholar.google.com/scholar?q=Scott+Reed%2C+Honglak+Lee%2C+Dragomir+Anguelov%2C+Christian+Szegedy%2C+Dumitru+Erhan%2C+and+Andrew+Rabinovich%2E+Training+deep+neural+networks+on+noisy+labels+with+bootstrapping%2E+arXiv+preprint+arXiv%3A1412%2E6596%2C+2)
36. Kenneth Rose. Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proceedings of the IEEE, 86(11):2210–2239, 1998. [find](https://scholar.google.com/scholar?q=Kenneth+Rose%2E+Deterministic+annealing+for+clustering%2C+compression%2C+classification%2C+regression%2C+and+related+optimization+problems%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2210%E2%80%932239%2C+1998%2E)
37. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+E+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+Journal+of+Machine+Learning+Research)
38. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9, 2015a. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wei+Liu%2C+Yangqing+Jia%2C+Pierre+Sermanet%2C+Scott+Reed%2C+Dragomir+Anguelov%2C+Dumitru+Erhan%2C+Vincent+Vanhoucke%2C+and+Andrew+Rabinovich%2E+Going+deeper+with+convolutions%2E+In+Proceedings+of+the)
39. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. arXiv preprint arXiv:1512.00567, 2015b. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Vincent+Vanhoucke%2C+Sergey+Ioffe%2C+Jonathon+Shlens%2C+and+Zbigniew+Wojna%2E+Rethinking+the+inception+architecture+for+computer+vision%2E+arXiv+preprint+arXiv%3A1512%2E00567%2C+2015b%2E)
40. László Tóth. Combining time-and frequency-domain convolution in convolutional neural network-based phone recognition. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 190–194. [find](https://scholar.google.com/scholar?q=La%CC%81szlo%CC%81+To%CC%81th%2E+Combining+time%2Dand+frequency%2Ddomain+convolution+in+convolutional+neural+network%2Dbased+phone+recognition%2E+In+2014+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processin)
41. Li Wan, Matthew Zeiler, Sixin Zhang, Yann L Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 1058–1066, 2013. [find](https://scholar.google.com/scholar?q=Li+Wan%2C+Matthew+Zeiler%2C+Sixin+Zhang%2C+Yann+L+Cun%2C+and+Rob+Fergus%2E+Regularization+of+neural+networks+using+dropconnect%2E+In+Proceedings+of+the+30th+International+Conference+on+Machine+Learning+%28ICML%2D13%29%2C)
42. Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268, 1991. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams+and+Jing+Peng%2E+Function+optimization+using+connectionist+reinforcement+learning+algorithms%2E+Connection+Science%2C+3%283%29%3A241%E2%80%93268%2C+1991%2E)
43. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridging+th)
44. Lingxi Xie, Jingdong Wang, Zhen Wei, Meng Wang, and Qi Tian. Disturblabel: Regularizing cnn on the loss layer. arXiv preprint arXiv:1605.00055, 2016. — [arXiv:1605.00055](https://arxiv.org/abs/1605.00055) [find](https://scholar.google.com/scholar?q=Lingxi+Xie%2C+Jingdong+Wang%2C+Zhen+Wei%2C+Meng+Wang%2C+and+Qi+Tian%2E+Disturblabel%3A+Regularizing+cnn+on+the+loss+layer%2E+arXiv+preprint+arXiv%3A1605%2E00055%2C+2016%2E+%E2%80%94+%5BarXiv%3A1605%2E00055%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1605%2E00)
45. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. [find](https://scholar.google.com/scholar?q=Wojciech+Zaremba%2C+Ilya+Sutskever%2C+and+Oriol+Vinyals%2E+Recurrent+neural+network+regularization%2E+arXiv+preprint+arXiv%3A1409%2E2329%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/)
46. Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016. [find](https://scholar.google.com/scholar?q=Jie+Zhou%2C+Ying+Cao%2C+Xuguang+Wang%2C+Peng+Li%2C+and+Wei+Xu%2E+Deep+recurrent+models+with+fast%2Dforward+connections+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1606%2E04199%2C+2016%2E)
47. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnı́k, and Jürgen Schmidhuber. Recurrent highway networks. arXiv preprint arXiv:1607.03474, 2016. [find](https://scholar.google.com/scholar?q=Julian+Georg+Zilly%2C+Rupesh+Kumar+Srivastava%2C+Jan+Koutn%C4%B1%CC%81k%2C+and+Ju%CC%88rgen+Schmidhuber%2E+Recurrent+highway+networks%2E+arXiv+preprint+arXiv%3A1607%2E03474%2C+2016%2E)