---
title: "Effective Approaches to Attention-based Neural Machine Translation — Luong 2015"
description: "The authors, from Stanford, proposed two simple variants of attention. 5.0 BLEU. 16 references, cited by 1,474 papers here."
authors: ["Thang Luong", "Hieu Pham", "Christopher D. Manning"]
year: 2015
venue: "Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D15-1166/"
doi: "10.18653/v1/d15-1166"
arxiv: "1508.04025"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Effective Approaches to Attention-based Neural Machine Translation

# Effective Approaches to Attention-based Neural Machine Translation

**Authors:** [Thang Luong](https://md.research.elamur.ai/free/en/authors/thang-luong/), [Hieu Pham](https://md.research.elamur.ai/free/en/authors/hieu-pham/) and [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/) · **Year:** 2015 · **Venue:** Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D15-1166/) · [PDF](https://aclanthology.org/D15-1166.pdf) · [DOI 10.18653/v1/d15-1166](https://doi.org/10.18653/v1/d15-1166) · [arXiv 1508.04025](https://arxiv.org/abs/1508.04025)

**Citations:** [1474](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary

The paper addresses neural machine translation (NMT) with an attention mechanism, which lets the decoder attend to the relevant parts of the source sentence when generating each word. The authors, from Stanford, proposed two simple variants of attention. Global attention weighs all encoder hidden states at every step. Local attention first predicts an alignment position in the source sentence and looks only at a window around it, so it is cheaper to compute while remaining differentiable. Both variants are built on a multilayer LSTM model. The authors also compared several scoring functions for the match between states. The models were evaluated on WMT tasks for English and German in both directions. Compared with models lacking attention but already trained with dropout, local attention improved results by as much as 5.0 BLEU. An ensemble of models set a new state of the art for English→German translation on WMT'14 and WMT'15. The work is significant because it simplified the attention architecture previously proposed by Bahdanau et al. and studied it systematically. Its terms global/local attention and dot/general scoring became standard in subsequent work.

## Key points

- The ensemble reached 25.9 BLEU on WMT'15 English→German. This is 1.0 BLEU higher than the previous best system, which combined NMT with an n-gram reranker.
- The scoring functions dot, general and concat are examined, along with an early location-based variant. According to the authors, the alternatives outperform concat, which was used by Bahdanau et al.
- Local attention blends ideas of soft and hard attention from Xu et al. (2015): it is cheaper than global attention and, unlike hard attention, is trained with standard backpropagation.
- In alignment quality (AER), the models matched the Berkeley aligner, with the local variants achieving a lower AER than the global one.
- Attentional models translate proper names and long sentences better. The code and models are published at nlp.stanford.edu/projects/nmt.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cite

Cite as: Thang Luong, Hieu Pham and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{luong2015effective,
  title = {{Effective Approaches to Attention-based Neural Machine Translation}},
  author = {Thang Luong, Hieu Pham and Christopher D. Manning},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1166},
  eprint = {1508.04025},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d15-1166}
}
```

- [← previous Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/)
- [next → SemEval-2015 Task 12: Aspect Based Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/pontiki-2015-semeval-2015-task-12-aspect/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

16 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014. N-gram counts and language models from the common crawl. In LREC. [our card](https://md.research.elamur.ai/free/en/papers/buck-2014-n-gram-counts-language-models/)
3. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Caglar+Gulcehre%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+RNN+encoder%2Ddecoder+for+statistical+machine+transla)
4. Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. End-to-end continuous speech recognition using attention-based recurrent NN: first results. CoRR, abs/1412.1602. — [arXiv:1412.1602](https://arxiv.org/abs/1412.1602) [find](https://scholar.google.com/scholar?q=Jan+Chorowski%2C+Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+End%2Dto%2Dend+continuous+speech+recognition+using+attention%2Dbased+recurrent+NN%3A+first+results%2E+CoRR%2C+abs%2F1412%2E1602%2E+%E2%80%94+%5BarXiv%3A1412%2E)
5. Alexander Fraser and Daniel Marcu. 2007. Measuring word alignment quality for statistical machine translation. Computational Linguistics, 33(3):293–303. [find](https://scholar.google.com/scholar?q=Alexander+Fraser+and+Daniel+Marcu%2E+2007%2E+Measuring+word+alignment+quality+for+statistical+machine+translation%2E+Computational+Linguistics%2C+33%283%29%3A293%E2%80%93303%2E)
6. Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. 2015. DRAW: A recurrent neural network for image generation. In ICML. [find](https://scholar.google.com/scholar?q=Karol+Gregor%2C+Ivo+Danihelka%2C+Alex+Graves%2C+Danilo+Jimenez+Rezende%2C+and+Daan+Wierstra%2E+2015%2E+DRAW%3A+A+recurrent+neural+network+for+image+generation%2E+In+ICML%2E)
7. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/)
8. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/)
9. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In NAACL. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Franz+Josef+Och%2C+and+Daniel+Marcu%2E+2003%2E+Statistical+phrase%2Dbased+translation%2E+In+NAACL%2E)
10. Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In NAACL. [our card](https://md.research.elamur.ai/free/en/papers/liang-2006-alignment-agreement/)
11. Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/)
12. Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. 2014. Recurrent models of visual attention. In NIPS. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Nicolas+Heess%2C+Alex+Graves%2C+and+Koray+Kavukcuoglu%2E+2014%2E+Recurrent+models+of+visual+attention%2E+In+NIPS%2E)
13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
14. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NIPS%2E)
15. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML. [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)
16. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2015. Recurrent neural network regularization. In ICLR. [our card](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/)