---
title: "Recurrent Neural Network Regularization — Zaremba 2014"
description: "Dropout works well for regularizing feedforward networks but had largely failed for recurrent ones."
authors: ["Wojciech Zaremba", "Ilya Sutskever", "Oriol Vinyals"]
year: 2014
venue: "arXiv:1409.2329"
source: "https://arxiv.org/abs/1409.2329"
doi: "10.48550/arxiv.1409.2329"
arxiv: "1409.2329"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / Recurrent Neural Network Regularization

# Recurrent Neural Network Regularization

**Authors:** [Wojciech Zaremba](https://md.research.elamur.ai/free/en/authors/wojciech-zaremba/), [Ilya Sutskever](https://md.research.elamur.ai/free/en/authors/ilya-sutskever/), [Oriol Vinyals](https://md.research.elamur.ai/free/en/authors/oriol-vinyals/) · **Year:** 2014 · **Venue:** arXiv:1409.2329 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1409.2329) · [PDF](https://arxiv.org/pdf/1409.2329) · [DOI 10.48550/arxiv.1409.2329](https://doi.org/10.48550/arxiv.1409.2329) · [arXiv 1409.2329](https://arxiv.org/abs/1409.2329) · [OpenAlex](https://openalex.org/W1591801644)

**Citations:** 2123 citations (openalex-snapshot · 2026-10-14) · [256](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Dropout works well for regularizing feedforward networks but had largely failed for recurrent ones. Earlier work argued that noise injected into the recurrence builds up over time and disrupts learning. Without effective regularization, practitioners kept RNNs small to avoid overfitting. Zaremba, Sutskever and Vinyals propose a simple fix for multilayer LSTMs: apply dropout only to the vertical, non-recurrent connections (input to layer, layer to layer, and top layer to output). The hidden-to-hidden connections through time are left untouched. A signal is then corrupted only L+1 times, where L is the number of layers, no matter how many timesteps it travels. This leaves the LSTM's long-range memory intact. The authors test the method on word-level language modeling (Penn Tree Bank), acoustic modeling (Icelandic speech), English→French translation and image captioning. On PTB, it allows much larger networks and clearly lowers perplexity. The recipe became a standard baseline for regularizing LSTMs, and the code was released publicly (github.com/wojzaremba/lstm).

## Key points
- **Method:** dropout is applied only to non-recurrent connections, so memory carried through time is never dropped. Pham et al. (2013) independently found the same technique for handwriting recognition.
- **Penn Tree Bank (929k/73k/82k words, 10k vocabulary):** the best non-regularized LSTM (2×200 units) reaches a test perplexity of 114.5. The medium regularized LSTM (2×650 units, 50% dropout) reaches 82.7, and the large one (2×1500 units, 65% dropout) reaches 78.4. An ensemble of 38 large models gets 68.7.
- **Training cost:** on one NVIDIA K20 GPU, the medium model trains in about half a day and the large model in about a full day. Gradient norms are clipped at 5 and 10 respectively.
- **Speech (Icelandic Speech Dataset, 93k utterances):** dropout lowers training frame accuracy from 71.6 to 69.4 but raises validation accuracy from 68.9 to 70.5, a sign of less overfitting.
- **English→French MT:** test perplexity falls from 5.8 to 5.0 and BLEU rises from 25.9 to 29.03. This is still below the phrase-based LIUM system at 33.30.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

256 papers of our library cite it (of the 140884 whose bibliographies we read); 29 of them have a page here, 227 not yet.

- [Semi-supervised Contextual Historical Text Normalization](https://md.research.elamur.ai/free/en/papers/acl-2020-makarov-semi-supervised-contextual-historical-text/)
- [Contextual String Embeddings for Sequence Labeling](https://md.research.elamur.ai/free/en/papers/akbik-2018-contextual-string-embeddings-sequence-labeling/)
- [Evaluating prose style transfer with the Bible](https://md.research.elamur.ai/free/en/papers/carlson-2017-evaluating-prose-style-transfer-bible/)
- [TVM: An Automated End-to-End Optimizing Compiler for Deep Learning](https://md.research.elamur.ai/free/en/papers/chen-2018-tvm-automated-end-end-optimizing/)
- [常识问答研究综述](https://md.research.elamur.ai/free/en/papers/fan-2023-commonsense-question-answering/)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/)
- [Sarcasm Analysis Using Conversation Context](https://md.research.elamur.ai/free/en/papers/ghosh-2018-sarcasm-analysis-conversation-context/)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/)
- [Dialogue Act Classification in Domain-Independent Conversations Using a Deep Recurrent Neural Network](https://md.research.elamur.ai/free/en/papers/khanpour-2016-dialogue-act-classification-domain-independent/)
- [Character-Aware Neural Language Models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://md.research.elamur.ai/free/en/papers/kiros-2014-unifying-visual-semantic-embeddings-multimodal/)
- [Gender Bias in Neural Natural Language Processing](https://md.research.elamur.ai/free/en/papers/lu-2018-gender-bias-neural-natural-language/)
- [Addressing the Rare Word Problem in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/)
- [Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models](https://md.research.elamur.ai/free/en/papers/luong-2016-achieving-open-vocabulary-neural-machine/)
- [What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment](https://md.research.elamur.ai/free/en/papers/mei-2016-what-talk-about-how-selective/)
- [Pointer Sentinel Mixture Models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/)
- [Regularizing and Optimizing LSTM Language Models](https://md.research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://md.research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/)
- [Language Models as Knowledge Bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
- [Using the Output Embedding to Improve Language Models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/)
- [Unsupervised Pretraining for Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/ramachandran-2017-unsupervised-pretraining-sequence-sequence-learning/)
- [Reasoning about Entailment with Neural Attention](https://md.research.elamur.ai/free/en/papers/rocktaschel-2015-reasoning-about-entailment-neural-attention/)
- [Show and Tell: A Neural Image Caption Generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/)
- [Learning Global Features for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/wiseman-2016-learning-global-features-coreference-resolution/)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)
- [Sentence Simplification with Deep Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/zhang-2017-sentence-simplification-deep-reinforcement-learning/)
- [Multi-Source Neural Translation](https://md.research.elamur.ai/free/en/papers/zoph-2016-multi-source-neural-translation/)
- [Neural Architecture Search with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zaremba2014recurrent,
  title = {{Recurrent Neural Network Regularization}},
  author = {Zaremba, Wojciech and Sutskever, Ilya and Vinyals, Oriol},
  year = {2014},
  doi = {10.48550/arxiv.1409.2329},
  eprint = {1409.2329},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1409.2329}
}
```

- [← previous Recall Error Analysis for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/martschat-2014-recall-error-analysis-coreference-resolution/)
- [next → Recursive Deep Models for Discourse Parsing](https://md.research.elamur.ai/free/en/papers/li-2014-recursive-deep-models-discourse-parsing/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/)

## References

33 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bayer, Justin, Osendorfer, Christian, Chen, Nutan, Urban, Sebastian, and van der Smagt, Patrick. On fast dropout and its applicability to recurrent networks. arXiv preprint arXiv:1311.0701, 2013. — [arXiv:1311.0701](https://arxiv.org/abs/1311.0701) [find](https://scholar.google.com/scholar?q=Bayer%2C+Justin%2C+Osendorfer%2C+Christian%2C+Chen%2C+Nutan%2C+Urban%2C+Sebastian%2C+and+van+der+Smagt%2C+Patrick%2E+On+fast+dropout+and+its+applicability+to+recurrent+networks%2E+arXiv+preprint+arXiv%3A1311%2E0701%2C+2013%2E+%E2%80%94+%5Ba)
2. Bourlard, H. and Morgan, N. Connectionist Speech Recognition: A Hybrid Approach. Kluwer Academic Publishers, 1993. [find](https://scholar.google.com/scholar?q=Bourlard%2C+H%2E+and+Morgan%2C+N%2E+Connectionist+Speech+Recognition%3A+A+Hybrid+Approach%2E+Kluwer+Academic+Publishers%2C+1993%2E)
3. Cheng, Wei-Chen, Kok, Stanley, Pham, Hoai Vu, Chieu, Hai Leong, and Chai, Kian Ming A. Language modeling with sum-product networks. Cho, Kyunghyun, van Merrienboer, Bart, Gulcehre, Caglar, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder… [find](https://scholar.google.com/scholar?q=Cheng%2C+Wei%2DChen%2C+Kok%2C+Stanley%2C+Pham%2C+Hoai+Vu%2C+Chieu%2C+Hai+Leong%2C+and+Chai%2C+Kian+Ming+A%2E+Language+modeling+with+sum%2Dproduct+networks%2E+Cho%2C+Kyunghyun%2C+van+Merrienboer%2C+Bart%2C+Gulcehre%2C+Caglar%2C+Bougares%2C+F)
4. Chow, Y, Dunham, M, Kimball, O, Krasner, M, Kubala, G, Makhoul, J, Price, P, Roucos, S, and Schwartz, R. Byblos: The bbn continuous speech recognition system. In Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP’87., volume 12, pp. 89–92. IEEE, 1987. [find](https://scholar.google.com/scholar?q=Chow%2C+Y%2C+Dunham%2C+M%2C+Kimball%2C+O%2C+Krasner%2C+M%2C+Kubala%2C+G%2C+Makhoul%2C+J%2C+Price%2C+P%2C+Roucos%2C+S%2C+and+Schwartz%2C+R%2E+Byblos%3A+The+bbn+continuous+speech+recognition+system%2E+In+Acoustics%2C+Speech%2C+and+Signal+Processi)
5. Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. Fast and robust neural network joint models for statistical machine translation. In ACL, 2014. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Zbib%2C+R%2E%2C+Huang%2C+Z%2E%2C+Lamar%2C+T%2E%2C+Schwartz%2C+R%2E%2C+and+Makhoul%2C+J%2E+Fast+and+robust+neural+network+joint+models+for+statistical+machine+translation%2E+In+ACL%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2014-fast-robust-neural-network-joint/)
6. Graves, Alex. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2E+Generating+sequences+with+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1308%2E0850%2C+2013%2E)
7. Graves, Alex, Liwicki, Marcus, Fernández, Santiago, Bertolami, Roman, Bunke, Horst, and Schmidhuber, Jürgen. A novel connectionist system for unconstrained handwriting recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 31(5):855–868, 2009. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2C+Liwicki%2C+Marcus%2C+Ferna%CC%81ndez%2C+Santiago%2C+Bertolami%2C+Roman%2C+Bunke%2C+Horst%2C+and+Schmidhuber%2C+Ju%CC%88rgen%2E+A+novel+connectionist+system+for+unconstrained+handwriting+recognition%2E+Pattern+Analysis+)
8. Graves, Alex, Mohamed, Abdel-rahman, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6645–6649. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2C+Mohamed%2C+Abdel%2Drahman%2C+and+Hinton%2C+Geoffrey%2E+Speech+recognition+with+deep+recurrent+neural+networks%2E+In+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+2013+IEEE+International+Conferen)
9. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+Sepp+and+Schmidhuber%2C+Ju%CC%88rgen%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
10. Jaeger, Herbert, Lukoševičius, Mantas, Popovici, Dan, and Siewert, Udo. Optimization and applications of echo state networks with leaky-integrator neurons. Neural Networks, 20(3):335–352, 2007. [find](https://scholar.google.com/scholar?q=Jaeger%2C+Herbert%2C+Lukos%CC%8Cevic%CC%8Cius%2C+Mantas%2C+Popovici%2C+Dan%2C+and+Siewert%2C+Udo%2E+Optimization+and+applications+of+echo+state+networks+with+leaky%2Dintegrator+neurons%2E+Neural+Networks%2C+20%283%29%3A335%E2%80%93352%2C+2007%2E)
11. Kalchbrenner, N. and Blunsom, P. Recurrent continuous translation models. In EMNLP, 2013. [find](https://scholar.google.com/scholar?q=Kalchbrenner%2C+N%2E+and+Blunsom%2C+P%2E+Recurrent+continuous+translation+models%2E+In+EMNLP%2C+2013%2E) [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/)
12. Koutnı́k, Jan, Greff, Klaus, Gomez, Faustino, and Schmidhuber, Jürgen. A clockwork rnn. arXiv preprint arXiv:1402.3511, 2014. — [arXiv:1402.3511](https://arxiv.org/abs/1402.3511) [find](https://scholar.google.com/scholar?q=Koutn%C4%B1%CC%81k%2C+Jan%2C+Greff%2C+Klaus%2C+Gomez%2C+Faustino%2C+and+Schmidhuber%2C+Ju%CC%88rgen%2E+A+clockwork+rnn%2E+arXiv+preprint+arXiv%3A1402%2E3511%2C+2014%2E+%E2%80%94+%5BarXiv%3A1402%2E3511%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1402%2E3511%29)
13. Lin, Tsung-Yi, Maire, Michael, Belongie, Serge, Hays, James, Perona, Pietro, Ramanan, Deva, Dollár, Piotr, and Zitnick, C Lawrence. Microsoft coco: Common objects in context. arXiv preprint arXiv:1405.0312, 2014. [find](https://scholar.google.com/scholar?q=Lin%2C+Tsung%2DYi%2C+Maire%2C+Michael%2C+Belongie%2C+Serge%2C+Hays%2C+James%2C+Perona%2C+Pietro%2C+Ramanan%2C+Deva%2C+Dolla%CC%81r%2C+Piotr%2C+and+Zitnick%2C+C+Lawrence%2E+Microsoft+coco%3A+Common+objects+in+context%2E+arXiv+preprint+arXiv%3A140) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
14. Marcus, Mitchell P, Marcinkiewicz, Mary Ann, and Santorini, Beatrice. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993. [find](https://scholar.google.com/scholar?q=Marcus%2C+Mitchell+P%2C+Marcinkiewicz%2C+Mary+Ann%2C+and+Santorini%2C+Beatrice%2E+Building+a+large+annotated+corpus+of+english%3A+The+penn+treebank%2E+Computational+linguistics%2C+19%282%29%3A313%E2%80%93330%2C+1993%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
15. Mikolov, Tomáš. Statistical language models based on neural networks. PhD thesis, Ph. D. thesis, Brno University of Technology, 2012. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Toma%CC%81s%CC%8C%2E+Statistical+language+models+based+on+neural+networks%2E+PhD+thesis%2C+Ph%2E+D%2E+thesis%2C+Brno+University+of+Technology%2C+2012%2E)
16. Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. In SLT, pp. 234–239, 2012. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas+and+Zweig%2C+Geoffrey%2E+Context+dependent+recurrent+neural+network+language+model%2E+In+SLT%2C+pp%2E+234%E2%80%93239%2C+2012%2E)
17. Test perplexity 8.47 7.99 7.5 Test BLEU score 23.5 24.3 24.4 Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH, pp. 1045–1048, 2010. [find](https://scholar.google.com/scholar?q=Test+perplexity+8%2E47+7%2E99+7%2E5+Test+BLEU+score+23%2E5+24%2E3+24%2E4+Mikolov%2C+Tomas%2C+Karafia%CC%81t%2C+Martin%2C+Burget%2C+Lukas%2C+Cernocky%CC%80%2C+Jan%2C+and+Khudanpur%2C+Sanjeev%2E+Recurrent+neural+network+based+language+model%2E+In)
18. Mikolov, Tomas, Deoras, Anoop, Povey, Daniel, Burget, Lukas, and Cernocky, Jan. Strategies for training large scale neural network language models. In Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on, pp. 196–201. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas%2C+Deoras%2C+Anoop%2C+Povey%2C+Daniel%2C+Burget%2C+Lukas%2C+and+Cernocky%2C+Jan%2E+Strategies+for+training+large+scale+neural+network+language+models%2E+In+Automatic+Speech+Recognition+and+Understanding+%28A)
19. Mikolov, Tomas, Le, Quoc V, and Sutskever, Ilya. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168, 2013. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas%2C+Le%2C+Quoc+V%2C+and+Sutskever%2C+Ilya%2E+Exploiting+similarities+among+languages+for+machine+translation%2E+arXiv+preprint+arXiv%3A1309%2E4168%2C+2013%2E)
20. Pachitariu, Marius and Sahani, Maneesh. Regularization and nonlinearities for neural language models: when are they needed? arXiv preprint arXiv:1301.5650, 2013. — [arXiv:1301.5650](https://arxiv.org/abs/1301.5650) [find](https://scholar.google.com/scholar?q=Pachitariu%2C+Marius+and+Sahani%2C+Maneesh%2E+Regularization+and+nonlinearities+for+neural+language+models%3A+when+are+they+needed%3F+arXiv+preprint+arXiv%3A1301%2E5650%2C+2013%2E+%E2%80%94+%5BarXiv%3A1301%2E5650%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
21. Pascanu, Razvan, Gulcehre, Caglar, Cho, Kyunghyun, and Bengio, Yoshua. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026, 2013. [find](https://scholar.google.com/scholar?q=Pascanu%2C+Razvan%2C+Gulcehre%2C+Caglar%2C+Cho%2C+Kyunghyun%2C+and+Bengio%2C+Yoshua%2E+How+to+construct+deep+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1312%2E6026%2C+2013%2E)
22. Pham, Vu, Kermorvant, Christopher, and Louradour, Jérôme. Dropout improves recurrent neural networks for handwriting recognition. arXiv preprint arXiv:1312.4569, 2013. — [arXiv:1312.4569](https://arxiv.org/abs/1312.4569) [find](https://scholar.google.com/scholar?q=Pham%2C+Vu%2C+Kermorvant%2C+Christopher%2C+and+Louradour%2C+Je%CC%81ro%CC%82me%2E+Dropout+improves+recurrent+neural+networks+for+handwriting+recognition%2E+arXiv+preprint+arXiv%3A1312%2E4569%2C+2013%2E+%E2%80%94+%5BarXiv%3A1312%2E4569%5D%28https%3A%2F%2Far)
23. Robinson, Tony, Hochberg, Mike, and Renals, Steve. The use of recurrent neural networks in continuous speech recognition. In Automatic speech and speaker recognition, pp. 233–258. Springer, 1996. [find](https://scholar.google.com/scholar?q=Robinson%2C+Tony%2C+Hochberg%2C+Mike%2C+and+Renals%2C+Steve%2E+The+use+of+recurrent+neural+networks+in+continuous+speech+recognition%2E+In+Automatic+speech+and+speaker+recognition%2C+pp%2E+233%E2%80%93258%2E+Springer%2C+1996%2E)
24. Sak, H., Vinyals, O., Heigold, G., Senior, A., McDermott, E., Monga, R., and Mao, M. Sequence discriminative distributed training of long short-term memory recurrent neural networks. In Interspeech, 2014. [find](https://scholar.google.com/scholar?q=Sak%2C+H%2E%2C+Vinyals%2C+O%2E%2C+Heigold%2C+G%2E%2C+Senior%2C+A%2E%2C+McDermott%2C+E%2E%2C+Monga%2C+R%2E%2C+and+Mao%2C+M%2E+Sequence+discriminative+distributed+training+of+long+short%2Dterm+memory+recurrent+neural+networks%2E+In+Interspeech%2C+2)
25. Schwenk, Holger. University le mans, 2014. [http://www-lium.univ-lemans.fr/˜schwenk/cslm_joint/paper](http://www-lium.univ-lemans.fr/˜schwenk/cslm_joint/paper). [↗](http://www-lium.univ-lemans.fr/˜schwenk/cslm_joint/paper)
26. Schwenk, Holger, Lambert, Patrik, Barrault, Loı̈c, Servan, Christophe, Afli, Haithem, Abdul-Rauf, Sadaf, and Shah, Kashif. Lium’s smt machine translation systems for wmt 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pp. 464–469. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Schwenk%2C+Holger%2C+Lambert%2C+Patrik%2C+Barrault%2C+Lo%C4%B1%CC%88c%2C+Servan%2C+Christophe%2C+Afli%2C+Haithem%2C+Abdul%2DRauf%2C+Sadaf%2C+and+Shah%2C+Kashif%2E+Lium%E2%80%99s+smt+machine+translation+systems+for+wmt+2011%2E+In+Proceedings+of+the+Si)
27. Srivastava, Nitish. Improving neural networks with dropout. PhD thesis, University of Toronto, 2013. [find](https://scholar.google.com/scholar?q=Srivastava%2C+Nitish%2E+Improving+neural+networks+with+dropout%2E+PhD+thesis%2C+University+of+Toronto%2C+2013%2E)
28. Sundermeyer, Martin, Schlüter, Ralf, and Ney, Hermann. Lstm neural networks for language modeling. In INTERSPEECH, 2012. [find](https://scholar.google.com/scholar?q=Sundermeyer%2C+Martin%2C+Schlu%CC%88ter%2C+Ralf%2C+and+Ney%2C+Hermann%2E+Lstm+neural+networks+for+language+modeling%2E+In+INTERSPEECH%2C+2012%2E)
29. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014. [find](https://scholar.google.com/scholar?q=Sutskever%2C+Ilya%2C+Vinyals%2C+Oriol%2C+and+Le%2C+Quoc+VV%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+3104%E2%80%933112%2C+2014%2E)
30. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) [find](https://scholar.google.com/scholar?q=Szegedy%2C+Christian%2C+Liu%2C+Wei%2C+Jia%2C+Yangqing%2C+Sermanet%2C+Pierre%2C+Reed%2C+Scott%2C+Anguelov%2C+Dragomir%2C+Erhan%2C+Dumitru%2C+Vanhoucke%2C+Vincent%2C+and+Rabinovich%2C+Andrew%2E+Going+deeper+with+convolutions%2E+arXiv+prepri)
31. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. arXiv preprint arXiv:1411.4555, 2014. [find](https://scholar.google.com/scholar?q=Vinyals%2C+Oriol%2C+Toshev%2C+Alexander%2C+Bengio%2C+Samy%2C+and+Erhan%2C+Dumitru%2E+Show+and+tell%3A+A+neural+image+caption+generator%2E+arXiv+preprint+arXiv%3A1411%2E4555%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
32. Wan, Li, Zeiler, Matthew, Zhang, Sixin, Cun, Yann L, and Fergus, Rob. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 1058–1066, 2013. [find](https://scholar.google.com/scholar?q=Wan%2C+Li%2C+Zeiler%2C+Matthew%2C+Zhang%2C+Sixin%2C+Cun%2C+Yann+L%2C+and+Fergus%2C+Rob%2E+Regularization+of+neural+networks+using+dropconnect%2E+In+Proceedings+of+the+30th+International+Conference+on+Machine+Learning+%28ICML)
33. Wang, Sida and Manning, Christopher. Fast dropout training. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 118–126, 2013. [find](https://scholar.google.com/scholar?q=Wang%2C+Sida+and+Manning%2C+Christopher%2E+Fast+dropout+training%2E+In+Proceedings+of+the+30th+International+Conference+on+Machine+Learning+%28ICML%2D13%29%2C+pp%2E+118%E2%80%93126%2C+2013%2E)