--- title: "ADADELTA: An Adaptive Learning Rate Method — Zeiler 2012" description: "Plain stochastic gradient descent needs a hand-tuned global learning rate." authors: ["Matthew D. Zeiler"] year: 2012 venue: "arXiv:1212.5701" source: "https://arxiv.org/abs/1212.5701" doi: "10.48550/arxiv.1212.5701" arxiv: "1212.5701" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/) / ADADELTA: An Adaptive Learning Rate Method # ADADELTA: An Adaptive Learning Rate Method **Authors:** [Matthew D. Zeiler](https://md.research.elamur.ai/free/en/authors/matthew-d-zeiler/) · **Year:** 2012 · **Venue:** arXiv:1212.5701 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1212.5701) · [PDF](https://arxiv.org/pdf/1212.5701) · [DOI 10.48550/arxiv.1212.5701](https://doi.org/10.48550/arxiv.1212.5701) · [arXiv 1212.5701](https://arxiv.org/abs/1212.5701) · [OpenAlex](https://openalex.org/W6908809) **Citations:** 5122 citations (openalex-snapshot · 2026-10-14) · [526](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Plain stochastic gradient descent needs a hand-tuned global learning rate. ADAGRAD adapts the rate for each parameter, but it still has a global step size, and its sum of squared gradients keeps growing, so steps eventually shrink to almost nothing. Zeiler's ADADELTA fixes both problems with two changes. First, it replaces ADAGRAD's full history of squared gradients with an exponentially decaying average, so the denominator reflects only recent gradients. Second, it scales each step by a decaying RMS of earlier parameter updates. This gives updates the same units as the parameters, like a diagonal Newton-style step, and removes the global learning rate. It costs very little more computation than SGD. The author tests it on MNIST digit classification on a single machine and on a large speech-recognition network trained on Google's distributed system. ADADELTA converges quickly and beats ADAGRAD on speech frame accuracy, using the same two hyperparameters throughout. Because it largely removes learning-rate tuning, it became a widely used optimizer for deep networks. ## Key points - The update keeps two running averages with decay ρ, one for squared gradients and one for squared updates. Each step is the gradient times the ratio RMS[Δx]ₜ₋₁ / RMS[g]ₜ, and a small ε stabilizes both RMS terms. - The same settings (ρ = 0.95, ε = 1e-6) were used for MNIST and speech without retuning. - The speech model had 4 hidden layers of 2560 units each, using logistic or ReLU activations. Its input was 26 frames of 40 log-energy filter-bank features, and its output was 8,000 senone labels. It was trained on several hundred hours of US English from Voice Search, Voice IME and read speech. - With 100 replicas, ADADELTA was ahead of ADAGRAD for the whole run. With 200 replicas (ReLU), the gradients were much noisier, but it still quickly reached the same frame accuracy as ADAGRAD and momentum. - The MNIST analysis shows three things. Lower layers get larger effective steps, which offsets vanishing gradients. Late in training, the step sizes approach 1 while the actual updates shrink, which works like built-in annealing. Well-tuned momentum can still pull slightly ahead late in training. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 526 papers of our library cite it (of the 140673 whose bibliographies we read); 71 of them have a page here, 455 not yet. - [Semi-supervised Contextual Historical Text Normalization](https://md.research.elamur.ai/free/en/papers/acl-2020-makarov-semi-supervised-contextual-historical-text/) - [Morphological Word Embeddings for Arabic Neural Machine Translation in Low-Resource Settings](https://md.research.elamur.ai/free/en/papers/aclbulk-2018-shapiro-morphological-word-embeddings-arabic-neural/) - [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/) - [Morphological Inflection Generation with Hard Monotonic Attention](https://md.research.elamur.ai/free/en/papers/aharoni-2017-morphological-inflection-generation-hard-monotonic/) - [Towards String-To-Tree Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2017-string-tree-neural-machine-translation/) - [Neural Module Networks](https://md.research.elamur.ai/free/en/papers/andreas-2015-neural-module-networks/) - [Learning to Compose Neural Networks for Question Answering](https://md.research.elamur.ai/free/en/papers/andreas-2016-learning-compose-neural-networks-question/) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) - [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://md.research.elamur.ai/free/en/papers/carlini-2018-secret-sharer-evaluating-testing-unintended/) - [Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/chen-2015-event-extraction-dynamic-multi-pooling/) - [Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder](https://md.research.elamur.ai/free/en/papers/chen-2017-improved-neural-machine-translation-syntax/) - [Named Entity Recognition with Bidirectional LSTM-CNNs](https://md.research.elamur.ai/free/en/papers/chiu-2015-named-entity-recognition-bidirectional-lstm/) - [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/) - [Attention-Based Models for Speech Recognition](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/) - [Simple and Effective Multi-Paragraph Reading Comprehension](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/) - [Structural Scaffolds for Citation Intent Classification in Scientific Publications](https://md.research.elamur.ai/free/en/papers/cohan-2019-structural-scaffolds-citation-intent-classification/) - [Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection](https://md.research.elamur.ai/free/en/papers/corkery-2019-are-we-there-yet-encoder/) - [Stronger Baselines for Trustable Results in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/denkowski-2017-stronger-baselines-trustable-results-neural/) - [Multi-Task Learning for Multiple Language Translation](https://md.research.elamur.ai/free/en/papers/dong-2015-multi-task-learning-multiple-language/) - [Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions](https://md.research.elamur.ai/free/en/papers/dowmunt-2016-is-neural-machine-translation-ready/) - [Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing](https://md.research.elamur.ai/free/en/papers/dowmunt-2016-log-linear-combinations-monolingual-bilingual/) - [Question Generation for Question Answering](https://md.research.elamur.ai/free/en/papers/duan-2017-question-generation-question-answering/) - [A Language-Independent Neural Network for Event Detection](https://md.research.elamur.ai/free/en/papers/feng-2016-language-independent-neural-network-event/) - [Linguistic realisation as machine translation: Comparing different MT models for AMR-to-text generation](https://md.research.elamur.ai/free/en/papers/ferreira-2017-linguistic-realisation-as-machine-translation/) - [Style Transfer in Text: Exploration and Evaluation](https://md.research.elamur.ai/free/en/papers/fu-2017-style-transfer-text-exploration-evaluation/) - [On Using Monolingual Corpora in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/) - [Pointing the Unknown Words](https://md.research.elamur.ai/free/en/papers/gulcehre-2016-pointing-unknown-words/) - [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://md.research.elamur.ai/free/en/papers/ha-2016-multilingual-neural-machine-translation-universal/) - [Deep Semantic Role Labeling: What Works and What's Next](https://md.research.elamur.ai/free/en/papers/he-2017-deep-semantic-role-labeling-what/) - [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://md.research.elamur.ai/free/en/papers/henderson-2020-convert-efficient-accurate-conversational-representations/) - [Learning to Understand Phrases by Embedding the Dictionary](https://md.research.elamur.ai/free/en/papers/hill-2016-learning-understand-phrases-embedding-dictionary/) - [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://md.research.elamur.ai/free/en/papers/hokamp-2017-lexically-constrained-decoding-sequence-generation/) - [Harnessing Deep Neural Networks with Logic Rules](https://md.research.elamur.ai/free/en/papers/hu-2016-harnessing-deep-neural-networks-logic/) - [Intra-Sentential Subject Zero Anaphora Resolution using Multi-Column Convolutional Neural Network](https://md.research.elamur.ai/free/en/papers/iida-2016-intra-sentential-subject-zero-anaphora/) - [Convolutional Neural Networks for Sentence Classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/) - [Frame-Semantic Role Labeling with Heterogeneous Annotations](https://md.research.elamur.ai/free/en/papers/kshirsagar-2015-frame-semantic-role-labeling-heterogeneous/) - [Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/lee-2016-sequential-short-text-classification-recurrent/) - [Discourse Parsing with Attention-based Hierarchical Neural Networks](https://md.research.elamur.ai/free/en/papers/li-2016-discourse-parsing-attention-based-hierarchical/) - [Neural Net Models of Open-domain Discourse Coherence](https://md.research.elamur.ai/free/en/papers/li-2017-neural-net-models-open-domain/) - [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://md.research.elamur.ai/free/en/papers/li-2018-delete-retrieve-generate-simple-approach/) - [Leveraging FrameNet to Improve Automatic Event Detection](https://md.research.elamur.ai/free/en/papers/liu-2016-leveraging-framenet-improve-automatic-event/) - [Exploiting Argument Information to Improve Event Detection via Supervised Attention Mechanisms](https://md.research.elamur.ai/free/en/papers/liu-2017-exploiting-argument-information-improve-event/) - [Jointly Multiple Events Extraction via Attention-based Graph Information Aggregation](https://md.research.elamur.ai/free/en/papers/liu-2018-jointly-multiple-events-extraction-attention/) - [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://md.research.elamur.ai/free/en/papers/lowe-2015-ubuntu-dialogue-corpus-large-dataset/) - [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://md.research.elamur.ai/free/en/papers/ma-2016-end-end-sequence-labeling-bi/) - [Paraphrasing Revisited with Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/mallinson-2017-paraphrasing-revisited-neural-machine-translation/) - [Learning Cognitive Features from Gaze Data for Sentiment and Sarcasm Classification using Convolutional Neural Network](https://md.research.elamur.ai/free/en/papers/mishra-2017-learning-cognitive-features-gaze-data/) - [Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding](https://md.research.elamur.ai/free/en/papers/monroe-2017-colors-context-pragmatic-neural-model/) - [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://md.research.elamur.ai/free/en/papers/nallapati-2016-summarunner-recurrent-neural-network-based/) - [Event Detection and Domain Adaptation with Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/nguyen-2015-event-detection-domain-adaptation-convolutional/) - [Relation Extraction: Perspective from Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/nguyen-2015-relation-extraction-perspective-convolutional-neural/) - [Joint Event Extraction via Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/nguyen-2016-joint-event-extraction-recurrent-neural/) - [Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/nguyen-2017-transfer-learning-across-low-resource/) - [Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation](https://md.research.elamur.ai/free/en/papers/peled-2017-sarcasm-sign-interpreting-sarcasm-sentiment/) - [Dissecting Contextual Word Embeddings: Architecture and Representation](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/) - [Using the Output Embedding to Improve Language Models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/) - [Artificial Error Generation with Machine Translation and Syntactic Patterns](https://md.research.elamur.ai/free/en/papers/rei-2017-artificial-error-generation-machine-translation/) - [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://md.research.elamur.ai/free/en/papers/reimers-2017-reporting-score-distributions-makes-difference/) - [Edinburgh Neural Machine Translation Systems for WMT 16](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/) - [Nematus: a Toolkit for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/sennrich-2017-nematus-toolkit-neural-machine-translation/) - [The University of Edinburgh's Neural MT Systems for WMT17](https://md.research.elamur.ai/free/en/papers/sennrich-2017-university-edinburgh-s-neural-mt/) - [Bidirectional Attention Flow for Machine Comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/) - [A Dataset for Multi-Target Stance Detection](https://md.research.elamur.ai/free/en/papers/sobhani-2017-dataset-multi-target-stance-detection/) - [Deep Semantic Role Labeling with Self-Attention](https://md.research.elamur.ai/free/en/papers/tan-2017-deep-semantic-role-labeling-self/) - [Gated Self-Matching Networks for Reading Comprehension and Question Answering](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/) - [One Model to Learn Both: Zero Pronoun Prediction and Translation](https://md.research.elamur.ai/free/en/papers/wang-2019-one-model-learn-both-zero/) - [pkudblab at SemEval-2016 Task 6 : A Specific Convolutional Neural Network System for Effective Stance Detection](https://md.research.elamur.ai/free/en/papers/wei-2016-pkudblab-semeval-2016-task-6/) - [Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework](https://md.research.elamur.ai/free/en/papers/white-2017-inference-is-everything-recasting-semantic/) - [Style Transfer as Unsupervised Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2018-style-transfer-unsupervised-machine-translation/) - [Bridging the Gap between Training and Inference for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2019-bridging-gap-between-training-inference/) - [Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification](https://md.research.elamur.ai/free/en/papers/zhou-2016-attention-based-bidirectional-long-short/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{zeiler2012adadelta, title = {{ADADELTA: An Adaptive Learning Rate Method}}, author = {Zeiler, Matthew D.}, year = {2012}, doi = {10.48550/arxiv.1212.5701}, eprint = {1212.5701}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1212.5701} } ``` - [← previous Active Learning for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/laws-2012-active-learning-coreference-resolution/) - [next → Alternative Lexicalizations of Discourse Connectives in Czech](https://md.research.elamur.ai/free/en/papers/rysova-2012-alternative-lexicalizations-discourse-connectives-czech/) - [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/) ## References 7 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. H. Robinds and S. Monro, “A stochastic approximation method,” Annals of Mathematical Statistics, vol. 22, pp. 400–407, 1951. [find](https://scholar.google.com/scholar?q=H%2E+Robinds+and+S%2E+Monro%2C+%E2%80%9CA+stochastic+approximation+method%2C%E2%80%9D+Annals+of+Mathematical+Statistics%2C+vol%2E+22%2C+pp%2E+400%E2%80%93407%2C+1951%2E) 2. D.E. Rumelhart, G.E. Hinton, and R.J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, pp. 533–536, 1986. [find](https://scholar.google.com/scholar?q=D%2EE%2E+Rumelhart%2C+G%2EE%2E+Hinton%2C+and+R%2EJ%2E+Williams%2C+%E2%80%9CLearning+representations+by+back%2Dpropagating+errors%2C%E2%80%9D+Nature%2C+vol%2E+323%2C+pp%2E+533%E2%80%93536%2C+1986%2E) 3. J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online leaning and stochastic optimization,” in COLT, 2010. [find](https://scholar.google.com/scholar?q=J%2E+Duchi%2C+E%2E+Hazan%2C+and+Y%2E+Singer%2C+%E2%80%9CAdaptive+subgradient+methods+for+online+leaning+and+stochastic+optimization%2C%E2%80%9D+in+COLT%2C+2010%2E) 4. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng, “Large scale distributed deep networks,” in NIPS, 2012. [find](https://scholar.google.com/scholar?q=J%2E+Dean%2C+G%2E+Corrado%2C+R%2E+Monga%2C+K%2E+Chen%2C+M%2E+Devin%2C+Q%2E+Le%2C+M%2E+Mao%2C+M%2E+Ranzato%2C+A%2E+Senior%2C+P%2E+Tucker%2C+K%2E+Yang%2C+and+A%2E+Ng%2C+%E2%80%9CLarge+scale+distributed+deep+networks%2C%E2%80%9D+in+NIPS%2C+2012%2E) 5. S. Becker and Y. LeCun, “Improving the convergence of back-propagation learning with second order methods,” Tech. Rep., Department of Computer Science, University of Toronto, Toronto, ON, Canada, 1988. [find](https://scholar.google.com/scholar?q=S%2E+Becker+and+Y%2E+LeCun%2C+%E2%80%9CImproving+the+convergence+of+back%2Dpropagation+learning+with+second+order+methods%2C%E2%80%9D+Tech%2E+Rep%2E%2C+Department+of+Computer+Science%2C+University+of+Toronto%2C+Toronto%2C+ON%2C+Canada%2C+1988) 6. T. Schaul, S. Zhang, and Y. LeCun, “No more pesky learning rates,” arXiv:1206.1106, 2012. — [arXiv:1206.1106](https://arxiv.org/abs/1206.1106) [find](https://scholar.google.com/scholar?q=T%2E+Schaul%2C+S%2E+Zhang%2C+and+Y%2E+LeCun%2C+%E2%80%9CNo+more+pesky+learning+rates%2C%E2%80%9D+arXiv%3A1206%2E1106%2C+2012%2E+%E2%80%94+%5BarXiv%3A1206%2E1106%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1206%2E1106%29) 7. N. Jaitly, P. Nguyen, A. Senior, and V. Vanhoucke, “Application of pretrained deep neural networks to large vocabulary speech recognition,” in Interspeech, 2012. [find](https://scholar.google.com/scholar?q=N%2E+Jaitly%2C+P%2E+Nguyen%2C+A%2E+Senior%2C+and+V%2E+Vanhoucke%2C+%E2%80%9CApplication+of+pretrained+deep+neural+networks+to+large+vocabulary+speech+recognition%2C%E2%80%9D+in+Interspeech%2C+2012%2E)