---
title: "Adam: A Method for Stochastic Optimization (2014)"
description: "Kingma and Ba propose Adam (adaptive moment estimation). 22 references, cited by 10,909 papers here."
authors: ["Diederik P. Kingma", "Jimmy Ba"]
year: 2014
venue: "arXiv:1412.6980"
source: "https://arxiv.org/abs/1412.6980"
doi: "10.48550/arxiv.1412.6980"
arxiv: "1412.6980"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/) / Adam: A Method for Stochastic Optimization

# Adam: A Method for Stochastic Optimization

**Authors:** [Diederik P. Kingma](https://md.research.elamur.ai/free/en/authors/diederik-p-kingma/) and [Jimmy Ba](https://md.research.elamur.ai/free/en/authors/jimmy-lei-ba/) · **Year:** 2014 · **Venue:** arXiv:1412.6980 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1412.6980) · [PDF](https://arxiv.org/pdf/1412.6980) · [DOI 10.48550/arxiv.1412.6980](https://doi.org/10.48550/arxiv.1412.6980) · [arXiv 1412.6980](https://arxiv.org/abs/1412.6980) · [OpenAlex](https://openalex.org/W1522301498)

**Citations:** 79,368 citations (OpenAlex, 2026-10-14) · [10,909](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training large models on noisy, minibatch-based objectives needs a first-order optimizer that is cheap, light on memory, and robust to gradient scale, sparse gradients and objectives that drift over time. Kingma and Ba propose Adam (adaptive moment estimation). For each parameter it keeps two exponential moving averages: one of the gradient and one of its elementwise square. The update divides the first by the square root of the second, so every parameter gets its own effective learning rate. Both averages start at zero and are therefore biased toward zero early in training. The authors derive a simple fix: divide each average by one minus its decay rate raised to the step count. They show that the effective step is roughly capped by the stepsize hyperparameter. It also stays the same when gradients are rescaled and shrinks on its own when the gradient direction is uncertain. In the online convex setting they prove an O(√T) regret bound, comparable to the best known results. Empirically, Adam compares favorably with other stochastic optimizers. The method combines strengths of AdaGrad and RMSProp, and a variant called AdaMax uses the infinity norm.

## Key points
- Suggested defaults for the tested problems are α = 0.001, β1 = 0.9, β2 = 0.999 and ε = 10^-8. All operations are element-wise.
- Bias correction divides m_t by (1 − β1^t) and v_t by (1 − β2^t). Without it, early steps would be much too large, especially for sparse gradients where β2 must be close to 1.
- The effective step |Δt| is approximately bounded by α. A looser bound of α·(1 − β1)/√(1 − β2) applies only in the extreme sparse case.
- The convergence analysis follows Zinkevich's online learning framework. It assumes bounded gradients and bounded parameter distances, a learning rate α_t = α/√t, and β1,t = β1·λ^(t−1) with λ typically close to 1 (e.g. 1 − 10^-8). The resulting regret is O(√T).
- The design aims to merge AdaGrad (good with sparse gradients) and RMSProp (good for on-line, non-stationary settings). A cheaper ordering of the computation folds the bias corrections into a per-step stepsize.

## Interesting ideas
- Any running average that starts from zero is biased low at the beginning, and dividing by one minus the decay rate to the power of the step count removes this cheaply and exactly under stationarity.
- The ratio of mean gradient to its root-mean-square works like a signal-to-noise measure, so updates automatically become small where evidence is inconsistent, giving annealing for free.
- Normalizing by the second moment makes the step size independent of gradient scale, so the single knob α acts as a trust-region radius whose order of magnitude can be chosen in advance.
- Scoring an online learner by regret, meaning its cumulative loss minus that of the best fixed choice in hindsight, measures how fast adaptation pays off rather than only the final quality.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Running estimates of witness-parser agreement or LLM-teacher accuracy could get this correction, so early silver-data batches are not wrongly judged as low-agreement.
- **I2** (tree): The perceptron parser could scale each feature's update by how consistent its update sign has been, damping features that flip between correct and wrong attachments.
- **I3** (tree): Training gparse with this property means one stepsize setting can transfer across treebanks and loss mixtures without retuning for each UD dialect conversion.
- **I4** (learn): Self-induced repair rules learned online could be judged against the best fixed rule set chosen afterwards, exposing rules that only help after they are frozen.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Amsterdam](https://md.research.elamur.ai/free/en/universities/university-of-amsterdam/) — University, Amsterdam, Netherlands · 144 works · 173,906 citations
- [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 96 works · 142,007 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 177 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 209 works · [🎯 Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) · 108 works · [➗ Numerical Analysis](https://md.research.elamur.ai/free/en/shelves/mathematics/numerical-analysis/) · 3 works

💡 Concepts (1): [💡 Machine learning](https://md.research.elamur.ai/free/en/concepts/machine-learning/) · 51 works

## Cite

Cite as: Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:1412.6980.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kingma2014adam,
  title = {{Adam: A Method for Stochastic Optimization}},
  author = {Diederik P. Kingma and Jimmy Ba},
  year = {2014},
  doi = {10.48550/arxiv.1412.6980},
  eprint = {1412.6980},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1412.6980}
}
```

- [next → Very Deep Convolutional Networks for Large-Scale Image Recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/)

## References

22 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amari, Shun-Ichi. Natural gradient works efficiently in learning. Neural computation, 10(2):251–276, 1998. [find](https://scholar.google.com/scholar?q=Amari%2C+Shun%2DIchi%2E+Natural+gradient+works+efficiently+in+learning%2E+Neural+computation%2C+10%282%29%3A251%E2%80%93276%2C+1998%2E)
2. Deng, Li, Li, Jinyu, Huang, Jui-Ting, Yao, Kaisheng, Yu, Dong, Seide, Frank, Seltzer, Michael, Zweig, Geoff, He, Xiaodong, Williams, Jason, et al. Recent advances in deep learning for speech research at microsoft. ICASSP 2013, 2013. [find](https://scholar.google.com/scholar?q=Deng%2C+Li%2C+Li%2C+Jinyu%2C+Huang%2C+Jui%2DTing%2C+Yao%2C+Kaisheng%2C+Yu%2C+Dong%2C+Seide%2C+Frank%2C+Seltzer%2C+Michael%2C+Zweig%2C+Geoff%2C+He%2C+Xiaodong%2C+Williams%2C+Jason%2C+et+al%2E+Recent+advances+in+deep+learning+for+speech+research+)
3. Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011. [find](https://scholar.google.com/scholar?q=Duchi%2C+John%2C+Hazan%2C+Elad%2C+and+Singer%2C+Yoram%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A2121%E2%80%932159%2C+2011%2E)
4. Graves, Alex. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2E+Generating+sequences+with+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1308%2E0850%2C+2013%2E)
5. Graves, Alex, Mohamed, Abdel-rahman, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6645–6649. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2C+Mohamed%2C+Abdel%2Drahman%2C+and+Hinton%2C+Geoffrey%2E+Speech+recognition+with+deep+recurrent+neural+networks%2E+In+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+2013+IEEE+International+Conferen)
6. Hinton, G.E. and Salakhutdinov, R.R. Reducing the dimensionality of data with neural networks. Science, 313 (5786):504–507, 2006. [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2EE%2E+and+Salakhutdinov%2C+R%2ER%2E+Reducing+the+dimensionality+of+data+with+neural+networks%2E+Science%2C+313+%285786%29%3A504%E2%80%93507%2C+2006%2E)
7. Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George E, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara N, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Mag… [find](https://scholar.google.com/scholar?q=Hinton%2C+Geoffrey%2C+Deng%2C+Li%2C+Yu%2C+Dong%2C+Dahl%2C+George+E%2C+Mohamed%2C+Abdel%2Drahman%2C+Jaitly%2C+Navdeep%2C+Senior%2C+Andrew%2C+Vanhoucke%2C+Vincent%2C+Nguyen%2C+Patrick%2C+Sainath%2C+Tara+N%2C+et+al%2E+Deep+neural+networks+for+acou)
8. Hinton, Geoffrey E, Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012b. [find](https://scholar.google.com/scholar?q=Hinton%2C+Geoffrey+E%2C+Srivastava%2C+Nitish%2C+Krizhevsky%2C+Alex%2C+Sutskever%2C+Ilya%2C+and+Salakhutdinov%2C+Ruslan+R%2E+Improving+neural+networks+by+preventing+co%2Dadaptation+of+feature+detectors%2E+arXiv+preprint+arXiv)
9. Kingma, Diederik P and Welling, Max. [Auto-Encoding Variational Bayes](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/). In The 2nd International Conference on Learning Representations (ICLR), 2013.
10. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In Advances in neural information processing systems, pp. 1097–1105, 2012.
11. Maas, Andrew L, Daly, Raymond E, Pham, Peter T, Huang, Dan, Ng, Andrew Y, and Potts, Christopher. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pp. 142–150. Association f…
12. Moulines, Eric and Bach, Francis R. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In Advances in Neural Information Processing Systems, pp. 451–459, 2011. [find](https://scholar.google.com/scholar?q=Moulines%2C+Eric+and+Bach%2C+Francis+R%2E+Non%2Dasymptotic+analysis+of+stochastic+approximation+algorithms+for+machine+learning%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+451%E2%80%93459%2C+2011%2E)
13. Pascanu, Razvan and Bengio, Yoshua. arXiv:1301.3584, 2013. — [arXiv:1301.3584](https://arxiv.org/abs/1301.3584) [find](https://scholar.google.com/scholar?q=Pascanu%2C+Razvan+and+Bengio%2C+Yoshua%2E+arXiv%3A1301%2E3584%2C+2013%2E+%E2%80%94+%5BarXiv%3A1301%2E3584%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1301%2E3584%29)
14. Revisiting natural gradient for deep networks. Polyak, Boris T and Juditsky, Anatoli B. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992. arXiv preprint Roux, Nicolas L and Fitzgibbon, Andrew W. A fast natural newton method. In Proc… [find](https://scholar.google.com/scholar?q=Revisiting+natural+gradient+for+deep+networks%2E+Polyak%2C+Boris+T+and+Juditsky%2C+Anatoli+B%2E+Acceleration+of+stochastic+approximation+by+averaging%2E+SIAM+Journal+on+Control+and+Optimization%2C+30%284%29%3A838%E2%80%93855%2C+)
15. Ruppert, David. Efficient estimations from a slowly convergent robbins-monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988. [find](https://scholar.google.com/scholar?q=Ruppert%2C+David%2E+Efficient+estimations+from+a+slowly+convergent+robbins%2Dmonro+process%2E+Technical+report%2C+Cornell+University+Operations+Research+and+Industrial+Engineering%2C+1988%2E)
16. Schaul, Tom, Zhang, Sixin, and LeCun, Yann. No more pesky learning rates. arXiv preprint arXiv:1206.1106, 2012. — [arXiv:1206.1106](https://arxiv.org/abs/1206.1106) [find](https://scholar.google.com/scholar?q=Schaul%2C+Tom%2C+Zhang%2C+Sixin%2C+and+LeCun%2C+Yann%2E+No+more+pesky+learning+rates%2E+arXiv+preprint+arXiv%3A1206%2E1106%2C+2012%2E+%E2%80%94+%5BarXiv%3A1206%2E1106%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1206%2E1106%29)
17. Sohl-Dickstein, Jascha, Poole, Ben, and Ganguli, Surya. Fast large-scale optimization by unifying stochastic gradient and quasi-newton methods. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 604–612, 2014. [find](https://scholar.google.com/scholar?q=Sohl%2DDickstein%2C+Jascha%2C+Poole%2C+Ben%2C+and+Ganguli%2C+Surya%2E+Fast+large%2Dscale+optimization+by+unifying+stochastic+gradient+and+quasi%2Dnewton+methods%2E+In+Proceedings+of+the+31st+International+Conference+on+M)
18. Sutskever, Ilya, Martens, James, Dahl, George, and Hinton, Geoffrey. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 1139–1147, 2013. [find](https://scholar.google.com/scholar?q=Sutskever%2C+Ilya%2C+Martens%2C+James%2C+Dahl%2C+George%2C+and+Hinton%2C+Geoffrey%2E+On+the+importance+of+initialization+and+momentum+in+deep+learning%2E+In+Proceedings+of+the+30th+International+Conference+on+Machine+L)
19. Tieleman, T. and Hinton, G. Lecture 6.5 - RMSProp, COURSERA: Neural Networks for Machine Learning. Technical report, 2012. [find](https://scholar.google.com/scholar?q=Tieleman%2C+T%2E+and+Hinton%2C+G%2E+Lecture+6%2E5+%2D+RMSProp%2C+COURSERA%3A+Neural+Networks+for+Machine+Learning%2E+Technical+report%2C+2012%2E)
20. Wang, Sida and Manning, Christopher. Fast dropout training. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 118–126, 2013. [find](https://scholar.google.com/scholar?q=Wang%2C+Sida+and+Manning%2C+Christopher%2E+Fast+dropout+training%2E+In+Proceedings+of+the+30th+International+Conference+on+Machine+Learning+%28ICML%2D13%29%2C+pp%2E+118%E2%80%93126%2C+2013%2E)
21. Zeiler, Matthew D. [Adadelta: An adaptive learning rate method](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/). arXiv preprint arXiv:1212.5701, 2012.
22. Zinkevich, Martin. Online convex programming and generalized infinitesimal gradient ascent. 2003. [find](https://scholar.google.com/scholar?q=Zinkevich%2C+Martin%2E+Online+convex+programming+and+generalized+infinitesimal+gradient+ascent%2E+2003%2E)

*Updated 29 October 2026*