--- title: "Regularizing and Optimizing LSTM Language — Merity 2017" description: "The paper tackles a practical gap in word-level language modeling." authors: ["Stephen Merity", "Nitish Shirish Keskar", "Richard Socher"] year: 2017 venue: "arXiv:1708.02182" source: "https://arxiv.org/abs/1708.02182" doi: "10.48550/arxiv.1708.02182" arxiv: "1708.02182" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / Regularizing and Optimizing LSTM Language Models # Regularizing and Optimizing LSTM Language Models **Authors:** [Stephen Merity](https://md.research.elamur.ai/free/en/authors/stephen-merity/), [Nitish Shirish Keskar](https://md.research.elamur.ai/free/en/authors/nitish-shirish-keskar/), [Richard Socher](https://md.research.elamur.ai/free/en/authors/richard-socher/) · **Year:** 2017 · **Venue:** arXiv:1708.02182 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1708.02182) · [PDF](https://arxiv.org/pdf/1708.02182) · [DOI 10.48550/arxiv.1708.02182](https://doi.org/10.48550/arxiv.1708.02182) · [arXiv 1708.02182](https://arxiv.org/abs/1708.02182) · [OpenAlex](https://openalex.org/W2743945814) **Citations:** 427 citations (openalex-snapshot · 2026-10-14) · [178](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper tackles a practical gap in word-level language modeling. Standard regularizers such as dropout and batch normalization transfer poorly to recurrent networks, and many RNN-specific fixes require custom cells that cannot run on fast libraries like NVIDIA cuDNN. The authors build AWD-LSTM, which relies only on techniques that leave the LSTM implementation untouched. The core idea is the weight-dropped LSTM: a single DropConnect mask on the hidden-to-hidden matrices, reused across all timesteps. They pair it with NT-ASGD, an averaged SGD variant that starts averaging automatically once validation perplexity stops improving over several checks. Additional regularizers include: - randomized BPTT lengths - dropout applied to the word embedding matrix - AR and TAR, penalties on the size of activations and on how much hidden states change from one timestep to the next - weight decay - tied, smaller embeddings Evaluated on Penn Treebank and WikiText-2, the model sets new state-of-the-art perplexities, and adding a neural cache lowers them further. The work matters because carefully combined simple methods beat elaborate architectures while staying compatible with optimized black-box LSTM code. AWD-LSTM became a widely used baseline. ## Key points - AWD-LSTM (tied) reaches test perplexity 57.3 on Penn Treebank and 65.8 on WikiText-2. With a continuous cache pointer, these fall to 52.8 and 52.0. - NT-ASGD uses a constant learning rate and a non-monotone trigger with logging interval L = one epoch and n = 5, values that worked across models and datasets. Removing it raises PTB test perplexity from 57.3 to 63.7. - In ablations, dropping weight-dropping hurts most, adding up to about 11 perplexity points (PTB validation 60.0 → 71.1). - Making embeddings full-sized grows the model to 43M parameters (PTB) and 70M (WT2) and worsens perplexity by roughly 8 points, which the authors attribute to overfitting. Fixed BPTT length costs about 1 point. - On WikiText-2, the cache helps most on `` and rare names such as Meridian, which occurs only 161 times. It hurts frequent tokens like punctuation and stop words. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 178 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 167 not yet. - [Story Realization: Expanding Plot Events into Sentences](https://md.research.elamur.ai/free/en/papers/ammanabrolu-2019-story-realization-expanding-plot-events/) - [Identifying and Reducing Gender Bias in Word-Level Language Models](https://md.research.elamur.ai/free/en/papers/bordia-2019-identifying-reducing-gender-bias-word/) - [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://md.research.elamur.ai/free/en/papers/carlini-2018-secret-sharer-evaluating-testing-unintended/) - [Representation Degeneration Problem in Training Natural Language Generation Models](https://md.research.elamur.ai/free/en/papers/gao-2019-representation-degeneration-problem-training-natural/) - [Learning to Write with Cooperative Discriminators](https://md.research.elamur.ai/free/en/papers/holtzman-2018-learning-write-cooperative-discriminators/) - [How Can We Know What Language Models Know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) - [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/) - [Politeness Transfer: A Tag and Generate Approach](https://md.research.elamur.ai/free/en/papers/madaan-2020-politeness-transfer-tag-generate-approach/) - [MTNT: A Testbed for Machine Translation of Noisy Text](https://md.research.elamur.ai/free/en/papers/michel-2018-mtnt-testbed-machine-translation-noisy/) - [Dissecting Contextual Word Embeddings: Architecture and Representation](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/) - [ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training](https://md.research.elamur.ai/free/en/papers/qi-2020-prophetnet-predicting-future-n-gram/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{merity2017regularizing, title = {{Regularizing and Optimizing LSTM Language Models}}, author = {Merity, Stephen and Keskar, Nitish Shirish and Socher, Richard}, year = {2017}, doi = {10.48550/arxiv.1708.02182}, eprint = {1708.02182}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1708.02182} } ``` - [← previous Reflexive verbs in Hebrew: Deep unaccusativity meets lexical semantics](https://md.research.elamur.ai/free/en/papers/lingbulk-2017-kastner-reflexive-verbs-hebrew-deep-unaccusativity/) - [next → Regularizing Neural Networks by Penalizing Confident Output Distributions](https://md.research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) ## References 44 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Arjovsky, M., Shah, A., and Bengio, Y. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pp. 1120–1128, 2016. [find](https://scholar.google.com/scholar?q=Arjovsky%2C+M%2E%2C+Shah%2C+A%2E%2C+and+Bengio%2C+Y%2E+Unitary+evolution+recurrent+neural+networks%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+1120%E2%80%931128%2C+2016%2E) 2. Ba, J., Kiros, J., and Hinton, G. E. Layer normalization. CoRR, abs/1607.06450, 2016. [find](https://scholar.google.com/scholar?q=Ba%2C+J%2E%2C+Kiros%2C+J%2E%2C+and+Hinton%2C+G%2E+E%2E+Layer+normalization%2E+CoRR%2C+abs%2F1607%2E06450%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/) 3. Balduzzi, D. and Ghifary, M. Strongly-typed recurrent neural networks. arXiv preprint arXiv:1602.02218, 2016. — [arXiv:1602.02218](https://arxiv.org/abs/1602.02218) [find](https://scholar.google.com/scholar?q=Balduzzi%2C+D%2E+and+Ghifary%2C+M%2E+Strongly%2Dtyped+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1602%2E02218%2C+2016%2E+%E2%80%94+%5BarXiv%3A1602%2E02218%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1602%2E02218%29) 4. Bottou, L., Curtis, F. E., and Nocedal, J. Optimization methods for large-scale machine learning. arXiv preprint arXiv:1606.04838, 2016. — [arXiv:1606.04838](https://arxiv.org/abs/1606.04838) [find](https://scholar.google.com/scholar?q=Bottou%2C+L%2E%2C+Curtis%2C+F%2E+E%2E%2C+and+Nocedal%2C+J%2E+Optimization+methods+for+large%2Dscale+machine+learning%2E+arXiv+preprint+arXiv%3A1606%2E04838%2C+2016%2E+%E2%80%94+%5BarXiv%3A1606%2E04838%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1606%2E04838%29) 5. Bradbury, J., Merity, S., Xiong, C., and Socher, R. Quasi-Recurrent Neural Networks. arXiv preprint arXiv:1611.01576, 2016. — [arXiv:1611.01576](https://arxiv.org/abs/1611.01576) [find](https://scholar.google.com/scholar?q=Bradbury%2C+J%2E%2C+Merity%2C+S%2E%2C+Xiong%2C+C%2E%2C+and+Socher%2C+R%2E+Quasi%2DRecurrent+Neural+Networks%2E+arXiv+preprint+arXiv%3A1611%2E01576%2C+2016%2E+%E2%80%94+%5BarXiv%3A1611%2E01576%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1611%2E01576%29) 6. Cooijmans, T., Ballas, N., Laurent, C., and Courville, A. C. Recurrent batch normalization. CoRR, abs/1603.09025, 2016. — [arXiv:1603.09025](https://arxiv.org/abs/1603.09025) [find](https://scholar.google.com/scholar?q=Cooijmans%2C+T%2E%2C+Ballas%2C+N%2E%2C+Laurent%2C+C%2E%2C+and+Courville%2C+A%2E+C%2E+Recurrent+batch+normalization%2E+CoRR%2C+abs%2F1603%2E09025%2C+2016%2E+%E2%80%94+%5BarXiv%3A1603%2E09025%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1603%2E09025%29) 7. Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121– 2159, 2011. [find](https://scholar.google.com/scholar?q=Duchi%2C+J%2E%2C+Hazan%2C+E%2E%2C+and+Singer%2C+Y%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+Journal+of+Machine+Learning+Research%2C+12%28Jul%29%3A2121%E2%80%93+2159%2C+2011%2E) 8. Földiák, P. Learning invariance from transformation sequences. Neural Computation, 3(2):194–200, 1991. [find](https://scholar.google.com/scholar?q=F%C3%B6ldi%C3%A1k%2C+P%2E+Learning+invariance+from+transformation+sequences%2E+Neural+Computation%2C+3%282%29%3A194%E2%80%93200%2C+1991%2E) 9. Gal, Y. and Ghahramani, Z. A theoretically grounded application of dropout in recurrent neural networks. In NIPS, 2016. [find](https://scholar.google.com/scholar?q=Gal%2C+Y%2E+and+Ghahramani%2C+Z%2E+A+theoretically+grounded+application+of+dropout+in+recurrent+neural+networks%2E+In+NIPS%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/) 10. Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. [find](https://scholar.google.com/scholar?q=Goyal%2C+P%2E%2C+Doll%C3%A1r%2C+P%2E%2C+Girshick%2C+R%2E%2C+Noordhuis%2C+P%2E%2C+Wesolowski%2C+L%2E%2C+Kyrola%2C+A%2E%2C+Tulloch%2C+A%2E%2C+Jia%2C+Y%2E%2C+and+He%2C+K%2E+Accurate%2C+large+minibatch+sgd%3A+Training+imagenet+in+1+hour%2E+arXiv+preprint+arXiv%3A1706%2E0) 11. Grave, E., Joulin, A., and Usunier, N. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426, 2016. [find](https://scholar.google.com/scholar?q=Grave%2C+E%2E%2C+Joulin%2C+A%2E%2C+and+Usunier%2C+N%2E+Improving+neural+language+models+with+a+continuous+cache%2E+arXiv+preprint+arXiv%3A1612%2E04426%2C+2016%2E) 12. Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint arXiv:1509.01240, 2015. — [arXiv:1509.01240](https://arxiv.org/abs/1509.01240) [find](https://scholar.google.com/scholar?q=Hardt%2C+M%2E%2C+Recht%2C+B%2E%2C+and+Singer%2C+Y%2E+Train+faster%2C+generalize+better%3A+Stability+of+stochastic+gradient+descent%2E+arXiv+preprint+arXiv%3A1509%2E01240%2C+2015%2E+%E2%80%94+%5BarXiv%3A1509%2E01240%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1509%2E0) 13. Hinton, G. E. Connectionist learning procedures. Artificial intelligence, 40(1-3):185–234, 1989. [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2E+E%2E+Connectionist+learning+procedures%2E+Artificial+intelligence%2C+40%281%2D3%29%3A185%E2%80%93234%2C+1989%2E) 14. Inan, H., Khosravi, K., and Socher, R. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. arXiv preprint arXiv:1611.01462, 2016. [find](https://scholar.google.com/scholar?q=Inan%2C+H%2E%2C+Khosravi%2C+K%2E%2C+and+Socher%2C+R%2E+Tying+Word+Vectors+and+Word+Classifiers%3A+A+Loss+Framework+for+Language+Modeling%2E+arXiv+preprint+arXiv%3A1611%2E01462%2C+2016%2E) 15. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. [find](https://scholar.google.com/scholar?q=Ioffe%2C+S%2E+and+Szegedy%2C+C%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+In+ICML%2C+2015%2E) 16. Jing, L., Shen, Y., Dubček, T., Peurifoy, J., Skirlo, S., Tegmark, M., and Soljačić, M. Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNN. arXiv preprint arXiv:1612.05231, 2016. — [arXiv:1612.05231](https://arxiv.org/abs/1612.05231) [find](https://scholar.google.com/scholar?q=Jing%2C+L%2E%2C+Shen%2C+Y%2E%2C+Dubc%CC%8Cek%2C+T%2E%2C+Peurifoy%2C+J%2E%2C+Skirlo%2C+S%2E%2C+Tegmark%2C+M%2E%2C+and+Soljac%CC%8Cic%CC%81%2C+M%2E+Tunable+Efficient+Unitary+Neural+Networks+%28EUNN%29+and+their+application+to+RNN%2E+arXiv+preprint+arXiv%3A1612%2E0523) 17. Jonschkowski, R. and Brock, O. Learning state representations with robotic priors. Auton. Robots, 39:407–428, 2015. [find](https://scholar.google.com/scholar?q=Jonschkowski%2C+R%2E+and+Brock%2C+O%2E+Learning+state+representations+with+robotic+priors%2E+Auton%2E+Robots%2C+39%3A407%E2%80%93428%2C+2015%2E) 18. Keskar, N. and Saon, G. A nonmonotone learning rate strategy for sgd training of deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 4974–4978. [find](https://scholar.google.com/scholar?q=Keskar%2C+N%2E+and+Saon%2C+G%2E+A+nonmonotone+learning+rate+strategy+for+sgd+training+of+deep+neural+networks%2E+In+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+2015+IEEE+International+Conference+on%2C+pp%2E+4) 19. Kim, Y., Jernite, Y., Sontag, D., and Rush, A. M. Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence, 2016. [find](https://scholar.google.com/scholar?q=Kim%2C+Y%2E%2C+Jernite%2C+Y%2E%2C+Sontag%2C+D%2E%2C+and+Rush%2C+A%2E+M%2E+Character%2Daware+neural+language+models%2E+In+Thirtieth+AAAI+Conference+on+Artificial+Intelligence%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/) 20. Kingma, D. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [find](https://scholar.google.com/scholar?q=Kingma%2C+D%2E+and+Ba%2C+J%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2C+2014%2E) 21. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. Moses: Open source toolkit for statistical machine translation. In ACL, 2007. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E%2C+Hoang%2C+H%2E%2C+Birch%2C+A%2E%2C+Callison%2DBurch%2C+C%2E%2C+Federico%2C+M%2E%2C+Bertoldi%2C+N%2E%2C+Cowan%2C+B%2E%2C+Shen%2C+W%2E%2C+Moran%2C+C%2E%2C+Zens%2C+R%2E%2C+Dyer%2C+C%2E%2C+Bojar%2C+O%2E%2C+Constantin%2C+A%2E%2C+and+Herbst%2C+E%2E+Moses%3A+Open+source+toolki) 22. Krueger, D., Maharaj, T., Kramár, J., Pezeshki, M., Ballas, N., Ke, N., Goyal, A., Bengio, Y., Larochelle, H., Courville, A., et al. Zoneout: Regularizing RNNss by randomly preserving hidden activations. arXiv preprint arXiv:1606.01305, 2016. — [arXiv:1606.01305](https://arxiv.org/abs/1606.01305) [find](https://scholar.google.com/scholar?q=Krueger%2C+D%2E%2C+Maharaj%2C+T%2E%2C+Kram%C3%A1r%2C+J%2E%2C+Pezeshki%2C+M%2E%2C+Ballas%2C+N%2E%2C+Ke%2C+N%2E%2C+Goyal%2C+A%2E%2C+Bengio%2C+Y%2E%2C+Larochelle%2C+H%2E%2C+Courville%2C+A%2E%2C+et+al%2E+Zoneout%3A+Regularizing+RNNss+by+randomly+preserving+hidden+activatio) 23. Luciw, M. and Schmidhuber, J. Low complexity proto-value function learning from sensory observations with incremental slow feature analysis. Artificial Neural Networks and Machine Learning–ICANN 2012, pp. 279– 287, 2012. [find](https://scholar.google.com/scholar?q=Luciw%2C+M%2E+and+Schmidhuber%2C+J%2E+Low+complexity+proto%2Dvalue+function+learning+from+sensory+observations+with+incremental+slow+feature+analysis%2E+Artificial+Neural+Networks+and+Machine+Learning%E2%80%93ICANN+2012%2C) 24. Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017. — [arXiv:1704.04289](https://arxiv.org/abs/1704.04289) [find](https://scholar.google.com/scholar?q=Mandt%2C+S%2E%2C+Hoffman%2C+M%2E+D%2E%2C+and+Blei%2C+D%2E+M%2E+Stochastic+gradient+descent+as+approximate+bayesian+inference%2E+arXiv+preprint+arXiv%3A1704%2E04289%2C+2017%2E+%E2%80%94+%5BarXiv%3A1704%2E04289%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1704%2E04289%29) 25. Melis, G., Dyer, C., and Blunsom, P. On the State of the Art of Evaluation in Neural Language Models. arXiv preprint arXiv:1707.05589, 2017. [find](https://scholar.google.com/scholar?q=Melis%2C+G%2E%2C+Dyer%2C+C%2E%2C+and+Blunsom%2C+P%2E+On+the+State+of+the+Art+of+Evaluation+in+Neural+Language+Models%2E+arXiv+preprint+arXiv%3A1707%2E05589%2C+2017%2E) 26. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer Sentinel Mixture Models. arXiv preprint arXiv:1609.07843, 2016. [find](https://scholar.google.com/scholar?q=Merity%2C+S%2E%2C+Xiong%2C+C%2E%2C+Bradbury%2C+J%2E%2C+and+Socher%2C+R%2E+Pointer+Sentinel+Mixture+Models%2E+arXiv+preprint+arXiv%3A1609%2E07843%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/) 27. Merity, S., McCann, B., and Socher, R. Revisiting activation regularization for language rnns. arXiv preprint arXiv:1708.01009, 2017. — [arXiv:1708.01009](https://arxiv.org/abs/1708.01009) [find](https://scholar.google.com/scholar?q=Merity%2C+S%2E%2C+McCann%2C+B%2E%2C+and+Socher%2C+R%2E+Revisiting+activation+regularization+for+language+rnns%2E+arXiv+preprint+arXiv%3A1708%2E01009%2C+2017%2E+%E2%80%94+%5BarXiv%3A1708%2E01009%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1708%2E01009%29) 28. Mikolov, T. and Zweig, G. Context dependent recurrent neural network language model. SLT, 12:234–239, 2012. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E+and+Zweig%2C+G%2E+Context+dependent+recurrent+neural+network+language+model%2E+SLT%2C+12%3A234%E2%80%93239%2C+2012%2E) 29. Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., and Khudanpur, S. Recurrent neural network based language model. In INTERSPEECH, 2010. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Karafi%C3%A1t%2C+M%2E%2C+Burget%2C+L%2E%2C+Cernock%C3%BD%2C+J%2E%2C+and+Khudanpur%2C+S%2E+Recurrent+neural+network+based+language+model%2E+In+INTERSPEECH%2C+2010%2E) 30. Panageas, I. and Piliouras, G. Gradient descent converges to minimizers: The case of non-isolated critical points. CoRR, abs/1605.00405, 2016. — [arXiv:1605.00405](https://arxiv.org/abs/1605.00405) [find](https://scholar.google.com/scholar?q=Panageas%2C+I%2E+and+Piliouras%2C+G%2E+Gradient+descent+converges+to+minimizers%3A+The+case+of+non%2Disolated+critical+points%2E+CoRR%2C+abs%2F1605%2E00405%2C+2016%2E+%E2%80%94+%5BarXiv%3A1605%2E00405%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1605%2E00405%29) 31. Polyak, B. and Juditsky, A. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992. [find](https://scholar.google.com/scholar?q=Polyak%2C+B%2E+and+Juditsky%2C+A%2E+Acceleration+of+stochastic+approximation+by+averaging%2E+SIAM+Journal+on+Control+and+Optimization%2C+30%284%29%3A838%E2%80%93855%2C+1992%2E) 32. Press, O. and Wolf, L. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859, 2016. [find](https://scholar.google.com/scholar?q=Press%2C+O%2E+and+Wolf%2C+L%2E+Using+the+output+embedding+to+improve+language+models%2E+arXiv+preprint+arXiv%3A1608%2E05859%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/) 33. Semeniuta, S., Severyn, A., and Barth, E. Recurrent dropout without memory loss. In COLING, 2016. [find](https://scholar.google.com/scholar?q=Semeniuta%2C+S%2E%2C+Severyn%2C+A%2E%2C+and+Barth%2C+E%2E+Recurrent+dropout+without+memory+loss%2E+In+COLING%2C+2016%2E) 34. Seo, M., Min, S., Farhadi, A., and Hajishirzi, H. Query-Reduction Networks for Question Answering. arXiv preprint arXiv:1606.04582, 2016. — [arXiv:1606.04582](https://arxiv.org/abs/1606.04582) [find](https://scholar.google.com/scholar?q=Seo%2C+M%2E%2C+Min%2C+S%2E%2C+Farhadi%2C+A%2E%2C+and+Hajishirzi%2C+H%2E+Query%2DReduction+Networks+for+Question+Answering%2E+arXiv+preprint+arXiv%3A1606%2E04582%2C+2016%2E+%E2%80%94+%5BarXiv%3A1606%2E04582%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1606%2E04582%29) 35. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014. [find](https://scholar.google.com/scholar?q=Srivastava%2C+N%2E%2C+Hinton%2C+G%2E%2C+Krizhevsky%2C+A%2E%2C+Sutskever%2C+I%2E%2C+and+Salakhutdinov%2C+R%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+Journal+of+Machine+Learning+Research%2C+15%3A1929%E2%80%931958%2C+) 36. Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139–1147, 2013. [find](https://scholar.google.com/scholar?q=Sutskever%2C+I%2E%2C+Martens%2C+J%2E%2C+Dahl%2C+G%2E%2C+and+Hinton%2C+G%2E+On+the+importance+of+initialization+and+momentum+in+deep+learning%2E+In+International+conference+on+machine+learning%2C+pp%2E+1139%E2%80%931147%2C+2013%2E) 37. Tallec, C. and Ollivier, Y. Unbiasing truncated backpropagation through time. arXiv preprint arXiv:1705.08209, 2017. — [arXiv:1705.08209](https://arxiv.org/abs/1705.08209) [find](https://scholar.google.com/scholar?q=Tallec%2C+C%2E+and+Ollivier%2C+Y%2E+Unbiasing+truncated+backpropagation+through+time%2E+arXiv+preprint+arXiv%3A1705%2E08209%2C+2017%2E+%E2%80%94+%5BarXiv%3A1705%2E08209%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1705%2E08209%29) 38. Tieleman, T. and Hinton, G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26–31, 2012. [find](https://scholar.google.com/scholar?q=Tieleman%2C+T%2E+and+Hinton%2C+G%2E+Lecture+6%2E5%2Drmsprop%3A+Divide+the+gradient+by+a+running+average+of+its+recent+magnitude%2E+COURSERA%3A+Neural+networks+for+machine+learning%2C+4%282%29%3A26%E2%80%9331%2C+2012%2E) 39. Wan, L., Zeiler, M., Zhang, S., LeCun, Y, and Fergus, R. Regularization of neural networks using dropconnect. In Proceedings of the 30th international conference on machine learning (ICML-13), pp. 1058–1066, 2013. [find](https://scholar.google.com/scholar?q=Wan%2C+L%2E%2C+Zeiler%2C+M%2E%2C+Zhang%2C+S%2E%2C+LeCun%2C+Y%2C+and+Fergus%2C+R%2E+Regularization+of+neural+networks+using+dropconnect%2E+In+Proceedings+of+the+30th+international+conference+on+machine+learning+%28ICML%2D13%29%2C+pp%2E+105) 40. Wilson, A. C, Roelofs, R., Stern, M., Srebro, N., and Recht, B. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:1705.08292, 2017. — [arXiv:1705.08292](https://arxiv.org/abs/1705.08292) [find](https://scholar.google.com/scholar?q=Wilson%2C+A%2E+C%2C+Roelofs%2C+R%2E%2C+Stern%2C+M%2E%2C+Srebro%2C+N%2E%2C+and+Recht%2C+B%2E+The+marginal+value+of+adaptive+gradient+methods+in+machine+learning%2E+arXiv+preprint+arXiv%3A1705%2E08292%2C+2017%2E+%E2%80%94+%5BarXiv%3A1705%2E08292%5D%28https%3A%2F) 41. Wisdom, S., Powers, T., Hershey, J., Le Roux, J., and Atlas, L. Full-capacity unitary recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 4880–4888, 2016. [find](https://scholar.google.com/scholar?q=Wisdom%2C+S%2E%2C+Powers%2C+T%2E%2C+Hershey%2C+J%2E%2C+Le+Roux%2C+J%2E%2C+and+Atlas%2C+L%2E+Full%2Dcapacity+unitary+recurrent+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+4880%E2%80%934888%2C+2016%2E) 42. Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. [find](https://scholar.google.com/scholar?q=Zaremba%2C+W%2E%2C+Sutskever%2C+I%2E%2C+and+Vinyals%2C+O%2E+Recurrent+neural+network+regularization%2E+arXiv+preprint+arXiv%3A1409%2E2329%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/) 43. Zilly, J. G., Srivastava, R. K., Koutník, J., and Schmidhuber, J. Recurrent highway networks. arXiv preprint arXiv:1607.03474, 2016. [find](https://scholar.google.com/scholar?q=Zilly%2C+J%2E+G%2E%2C+Srivastava%2C+R%2E+K%2E%2C+Koutn%C3%ADk%2C+J%2E%2C+and+Schmidhuber%2C+J%2E+Recurrent+highway+networks%2E+arXiv+preprint+arXiv%3A1607%2E03474%2C+2016%2E) 44. Zoph, B. and Le, Q. V. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016. [find](https://scholar.google.com/scholar?q=Zoph%2C+B%2E+and+Le%2C+Q%2E+V%2E+Neural+architecture+search+with+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1611%2E01578%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/)