---
title: "On Using Monolingual Corpora in Neural — Gulcehre 2015"
description: "Neural machine translation (NMT) relies on large sentence-aligned corpora."
authors: ["Caglar Gulcehre", "Orhan Firat", "Kelvin Xu", "Kyunghyun Cho", "Loic Barrault", "Huei-Chi Lin", "Fethi Bougares", "Holger Schwenk", "Yoshua Bengio"]
year: 2015
venue: "arXiv:1503.03535"
source: "https://arxiv.org/abs/1503.03535"
doi: "10.48550/arxiv.1503.03535"
arxiv: "1503.03535"
licence: "CC BY-NC-SA 3.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / On Using Monolingual Corpora in Neural Machine Translation

# On Using Monolingual Corpora in Neural Machine Translation

**Authors:** [Caglar Gulcehre](https://md.research.elamur.ai/free/en/authors/caglar-gulcehre/), [Orhan Firat](https://md.research.elamur.ai/free/en/authors/orhan-firat/), [Kelvin Xu](https://md.research.elamur.ai/free/en/authors/kelvin-xu/), [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/), [Loic Barrault](https://md.research.elamur.ai/free/en/authors/loic-barrault/), Huei-Chi Lin, [Fethi Bougares](https://md.research.elamur.ai/free/en/authors/fethi-bougares/), [Holger Schwenk](https://md.research.elamur.ai/free/en/authors/holger-schwenk/), [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/) · **Year:** 2015 · **Venue:** arXiv:1503.03535 · **Licence:** [CC BY-NC-SA 3.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1503.03535) · [PDF](https://arxiv.org/pdf/1503.03535) · [DOI 10.48550/arxiv.1503.03535](https://doi.org/10.48550/arxiv.1503.03535) · [arXiv 1503.03535](https://arxiv.org/abs/1503.03535) · [OpenAlex](https://openalex.org/W1915251500)

**Citations:** 466 citations (openalex-snapshot · 2026-10-14) · [150](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Neural machine translation (NMT) relies on large sentence-aligned corpora. Such data is scarce for low-resource language pairs and narrow domains, while target-language monolingual text is plentiful. The authors add a separately pre-trained recurrent language model (RNNLM) to an attention-based NMT system (Bahdanau et al.) in two ways. **Shallow fusion** reranks beam-search candidates using a weighted sum of the translation-model and LM log-probabilities. **Deep fusion** concatenates the hidden states of the LM and the NMT decoder, then fine-tunes only the output layer. A learned sigmoid gate scales the LM contribution for each word.

The methods were tested on four tasks:
- Zh-En SMS/chat and conversational speech (NIST OpenMT'15)
- Tr-En (IWSLT'14)
- De-En and Cs-En (WMT'15)

Deep fusion consistently gave the best results. It beat earlier Tr-En systems by up to 1.96 BLEU and gave smaller gains on the high-resource pairs. The size of the gain depended on how closely the monolingual data matched the translation domain. This shows that cheap monolingual text can help NMT, especially when parallel data is limited.

## Key points
- On Tr-En, deep fusion improved on plain NMT by up to +1.19 BLEU and surpassed the strongest earlier published system (Yılmaz et al., 2013) by as much as +1.96 BLEU.
- On the WMT'15 test sets, deep fusion improved on plain NMT by +0.39 BLEU for De-En (23.61 → 24.00) and +0.47 BLEU for Cs-En (21.89 → 22.36). Shallow fusion gained only 0.09 and 0.29 BLEU. The abstract assigns these two gains to the opposite language pairs, but the table supports the figures given here.
- Zh-En training used about 430K sentence pairs from DARPA BOLT and GALE data. The abstract reports a gain of up to 1.59 BLEU over phrase-based and hierarchical SMT baselines. On the CTS subset, deep fusion did not help.
- Domain match drove the gains. LM perplexity on the development sets was 223.68 for Zh-En (news-heavy English Gigaword vs. chat), 163.73 for Tr-En and 78.20 for De-En and Cs-En. The gate was also most active for De-En and Cs-En.
- The gate bias was initialised at b_g = −1, so the LM is used only when helpful. Separately, copying source words chosen by the attention weights in place of unknown tokens added roughly 1.0 BLEU on De-En and Cs-En.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

150 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 131 not yet.

- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
- [Neural versus Phrase-Based Machine Translation Quality: a Case Study](https://md.research.elamur.ai/free/en/papers/bentivogli-2016-neural-versus-phrase-based-machine/)
- [Neural Summarization by Extracting Sentences and Words](https://md.research.elamur.ai/free/en/papers/cheng-2016-neural-summarization-extracting-sentences-words/)
- [Attention-Based Models for Speech Recognition](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/)
- [A Survey of Domain Adaptation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chu-2018-survey-domain-adaptation-neural-machine/)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/)
- [APE at Scale and Its Implications on MT Evaluation Biases](https://md.research.elamur.ai/free/en/papers/freitag-2019-ape-scale-its-implications-mt/)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gu-2018-meta-learning-low-resource-neural/)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://md.research.elamur.ai/free/en/papers/ha-2016-multilingual-neural-machine-translation-universal/)
- [Survey of Low-Resource Machine Translation](https://md.research.elamur.ai/free/en/papers/haddow-2022-survey-low-resource-machine-translation/)
- [Unsupervised Translation of Programming Languages](https://md.research.elamur.ai/free/en/papers/lachaux-2020-unsupervised-translation-programming-languages/)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/)
- [Multi-task Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/park-2019-specaugment-simple-data-augmentation-method/)
- [Investigating Backtranslation in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/poncelas-2018-investigating-backtranslation-neural-machine-translation/)
- [Masked Language Model Scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/)
- [Minimum Risk Training for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/shen-2015-minimum-risk-training-neural-machine/)
- [Unsupervised Text Style Transfer using Language Models as Discriminators](https://md.research.elamur.ai/free/en/papers/yang-2018-unsupervised-text-style-transfer-using/)
- [Exploiting Source-side Monolingual Data in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2016-exploiting-source-side-monolingual-data/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gulcehre2015using,
  title = {{On Using Monolingual Corpora in Neural Machine Translation}},
  author = {Gulcehre, Caglar and Firat, Orhan and Xu, Kelvin and Cho, Kyunghyun and Barrault, Loic and Lin, Huei-Chi and Bougares, Fethi and Schwenk, Holger and Bengio, Yoshua},
  year = {2015},
  doi = {10.48550/arxiv.1503.03535},
  eprint = {1503.03535},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1503.03535}
}
```

- [← previous On Grammaticality in the Syntactic Annotation of Learner Language](https://md.research.elamur.ai/free/en/papers/dickinson-2015-grammaticality-syntactic-annotation-learner-language/)
- [next → On Using Very Large Target Vocabulary for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+arXiv+preprint+arXiv%3A1409%2E0473%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit3: Web inventory of transcribed and translated talks. Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), pages 261–268. [find](https://scholar.google.com/scholar?q=Mauro+Cettolo%2C+Christian+Girardi%2C+and+Marcello+Federico%2E+2012%2E+Wit3%3A+Web+inventory+of+transcribed+and+translated+talks%2E+Proceedings+of+the+16th+Conference+of+the+European+Association+for+Machine+Trans) [our card](https://md.research.elamur.ai/free/en/papers/cettolo-2012-wit3-web-inventory-transcribed-translated/)
3. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 263–270. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Chiang%2E+2005%2E+A+hierarchical+phrase%2Dbased+model+for+statistical+machine+translation%2E+In+Proceedings+of+the+43rd+Annual+Meeting+on+Association+for+Computational+Linguistics%2C+pages+263%E2%80%93270%2E+Associ) [our card](https://md.research.elamur.ai/free/en/papers/chiang-2005-hierarchical-phrase-based-model-statistical/)
4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), October.… [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Caglar+Gulcehre%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+RNN+encoder%2Ddecoder+for+statistical+machine+transla)
5. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Rabih+Zbib%2C+Zhongqiang+Huang%2C+Thomas+Lamar%2C+Richard+Schwartz%2C+and+John+Makhoul%2E+2014%2E+Fast+and+robust+neural+network+joint+models+for+statistical+machine+translation%2E+In+Association+for+)
6. Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2013. Maxout networks. In Proceedings of The 30th International Conference on Machine Learning, pages 1319–1327. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+David+Warde%2DFarley%2C+Mehdi+Mirza%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2013%2E+Maxout+networks%2E+In+Proceedings+of+The+30th+International+Conference+on+Machine+Learning%2C+pages+1319%E2%80%931327%2E)
7. Alex Graves. 2011. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pages 2348– 2356. [find](https://scholar.google.com/scholar?q=Alex+Graves%2E+2011%2E+Practical+variational+inference+for+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+2348%E2%80%93+2356%2E)
8. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. [find](https://scholar.google.com/scholar?q=Geoffrey+E+Hinton%2C+Nitish+Srivastava%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+R+Salakhutdinov%2E+2012%2E+Improving+neural+networks+by+preventing+co%2Dadaptation+of+feature+detectors%2E+arXiv+preprint+arXi)
9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
10. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2014. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2007. [find](https://scholar.google.com/scholar?q=Se%CC%81bastien+Jean%2C+Kyunghyun+Cho%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+2014%2E+On+using+very+large+target+vocabulary+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1412%2E2007%2E) [our card](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/)
11. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1700–1709. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner+and+Phil+Blunsom%2E+2013%2E+Recurrent+continuous+translation+models%2E+In+Proceedings+of+the+ACL+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+1700%E2%80%931709%2E+Ass)
12. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs.LG], December. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+Method+for+Stochastic+Optimization%2E+arXiv%3A1412%2E6980+%5Bcs%2ELG%5D%2C+December%2E)
13. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 48–54. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Franz+Josef+Och%2C+and+Daniel+Marcu%2E+2003%2E+Statistical+phrase%2Dbased+translation%2E+In+Proceedings+of+the+2003+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+L)
14. Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York, NY, USA. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2010%2E+Statistical+Machine+Translation%2E+Cambridge+University+Press%2C+New+York%2C+NY%2C+USA%2E)
15. Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2014. Addressing the rare word problem in neural machine translation. arXiv preprint arXiv:1410.8206. [find](https://scholar.google.com/scholar?q=Thang+Luong%2C+Ilya+Sutskever%2C+Quoc+V+Le%2C+Oriol+Vinyals%2C+and+Wojciech+Zaremba%2E+2014%2E+Addressing+the+rare+word+problem+in+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1410%2E8206%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/)
16. Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Cernocky. 2011. Rnnlm-recurrent neural network language modeling toolkit. In Proc. of the 2011 ASRU Workshop, pages 196–201. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Stefan+Kombrink%2C+Anoop+Deoras%2C+Lukar+Burget%2C+and+Jan+Cernocky%2E+2011%2E+Rnnlm%2Drecurrent+neural+network+language+modeling+toolkit%2E+In+Proc%2E+of+the+2011+ASRU+Workshop%2C+pages+196%E2%80%93201%2E)
17. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning (ICML 2013). [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+2013%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+In+Proceedings+of+the+30th+International+Conference+on+Machine+Learning+%28ICML+2013%29%2E)
18. R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio. 2014. How to construct deep recurrent neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014), April. [find](https://scholar.google.com/scholar?q=R%2E+Pascanu%2C+C%2E+Gulcehre%2C+K%2E+Cho%2C+and+Y%2E+Bengio%2E+2014%2E+How+to+construct+deep+recurrent+neural+networks%2E+In+Proceedings+of+the+Second+International+Conference+on+Learning+Representations+%28ICLR+2014%29%2C+Ap)
19. Haşim Sak, Tunga Güngör, and Murat Saraçlar. 2007. Morphological disambiguation of turkish text with perceptron algorithm. In Computational Linguistics and Intelligent Text Processing, pages 107–118. Springer. [find](https://scholar.google.com/scholar?q=Has%CC%A7im+Sak%2C+Tunga+Gu%CC%88ngo%CC%88r%2C+and+Murat+Sarac%CC%A7lar%2E+2007%2E+Morphological+disambiguation+of+turkish+text+with+perceptron+algorithm%2E+In+Computational+Linguistics+and+Intelligent+Text+Processing%2C+pages+107%E2%80%931)
20. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681. [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kuldip+K+Paliwal%2E+1997%2E+Bidirectional+recurrent+neural+networks%2E+Signal+Processing%2C+IEEE+Transactions+on%2C+45%2811%29%3A2673%E2%80%932681%2E)
21. Holger Schwenk. 2007a. Continuous space language models. Comput. Speech Lang., 21(3):492–518, July. [find](https://scholar.google.com/scholar?q=Holger+Schwenk%2E+2007a%2E+Continuous+space+language+models%2E+Comput%2E+Speech+Lang%2E%2C+21%283%29%3A492%E2%80%93518%2C+July%2E)
22. Holger Schwenk. 2007b. Continuous space language models. Computer Speech & Language, 21(3):492–518. [find](https://scholar.google.com/scholar?q=Holger+Schwenk%2E+2007b%2E+Continuous+space+language+models%2E+Computer+Speech+%26+Language%2C+21%283%29%3A492%E2%80%93518%2E)
23. Holger Schwenk. 2012. Continuous space translation models for phrase-based statistical machine translation. In Martin Kay and Christian Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics (COLIN), pages 1071–1080. Indian Institute of Technology Bombay. [find](https://scholar.google.com/scholar?q=Holger+Schwenk%2E+2012%2E+Continuous+space+translation+models+for+phrase%2Dbased+statistical+machine+translation%2E+In+Martin+Kay+and+Christian+Boitet%2C+editors%2C+Proceedings+of+the+24th+International+Conferenc)
24. Nakatani Shuyo. 2010. Language detection library for java. [find](https://scholar.google.com/scholar?q=Nakatani+Shuyo%2E+2010%2E+Language+detection+library+for+java%2E)
25. Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS 2014), December. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS+2014%29%2C+December%2E)
26. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4. [find](https://scholar.google.com/scholar?q=Tijmen+Tieleman+and+Geoffrey+Hinton%2E+2012%2E+Lecture+6%2E5%2Drmsprop%3A+Divide+the+gradient+by+a+running+average+of+its+recent+magnitude%2E+COURSERA%3A+Neural+Networks+for+Machine+Learning%2C+4%2E)
27. Ertugrul Yılmaz, Ilknur Durgar El-Kahlout, Burak Aydın, Zisan Sıla Ozil, and Coskun Mermer. 2013. Tubitak turkish-english submissions for iwslt 2013. Proceedings of the 10th International Workshop on Spoken Language Translation (IWSLT), pages 152–159. [find](https://scholar.google.com/scholar?q=Ertugrul+Y%C4%B1lmaz%2C+Ilknur+Durgar+El%2DKahlout%2C+Burak+Ayd%C4%B1n%2C+Zisan+S%C4%B1la+Ozil%2C+and+Coskun+Mermer%2E+2013%2E+Tubitak+turkish%2Denglish+submissions+for+iwslt+2013%2E+Proceedings+of+the+10th+International+Workshop+on+)
28. Matthew D Zeiler. 2012. ADADELTA: An adaptive learning rate method. arXiv:1212.5701 [cs.LG]. [find](https://scholar.google.com/scholar?q=Matthew+D+Zeiler%2E+2012%2E+ADADELTA%3A+An+adaptive+learning+rate+method%2E+arXiv%3A1212%2E5701+%5Bcs%2ELG%5D%2E) [our card](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/)