---
title: "Modeling Coverage for Neural Machine Translation — Tu 2016"
description: "Attention-based neural machine translation (NMT) does not record which source words it has already handled."
authors: ["Zhaopeng Tu", "Zhengdong Lu", "Yang Liu", "Xiaohua Liu", "Hang Li"]
year: 2016
venue: "Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P16-1008/"
doi: "10.18653/v1/p16-1008"
arxiv: "1601.04811"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Modeling Coverage for Neural Machine Translation

# Modeling Coverage for Neural Machine Translation

**Authors:** [Zhaopeng Tu](https://md.research.elamur.ai/free/en/authors/zhaopeng-tu/), [Zhengdong Lu](https://md.research.elamur.ai/free/en/authors/zhengdong-lu/), [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/), Xiaohua Liu, [Hang Li](https://md.research.elamur.ai/free/en/authors/hang-li/) · **Year:** 2016 · **Venue:** Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P16-1008/) · [PDF](https://aclanthology.org/P16-1008.pdf) · [DOI 10.18653/v1/p16-1008](https://doi.org/10.18653/v1/p16-1008) · [arXiv 1601.04811](https://arxiv.org/abs/1601.04811) · [OpenAlex](https://openalex.org/W2963260202)

**Citations:** 692 citations (openalex-snapshot · 2026-10-14) · [235](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Attention-based neural machine translation (NMT) does not record which source words it has already handled. This leads to two common errors: some source words are rendered repeatedly (over-translation) and others are dropped (under-translation). Phrase-based SMT avoids this with an explicit coverage vector, but standard NMT has no equivalent. Tu et al. propose NMT-COVERAGE, which attaches a coverage vector to each source annotation. The vector is updated after every attention step and fed back into the attention model, so future attention moves away from words that have already been heavily attended. They study two families. The first is a linguistically motivated scalar coverage that accumulates attention weights, optionally normalized by a predicted fertility. The second is a learned neural coverage, with or without gating. On Chinese–English translation with a Groundhog baseline, both families improve translation and alignment quality, and the gains are largest on long sentences. The work matters because it shows that a classic SMT idea can be transplanted into neural attention and trained end-to-end from data.

## Key points
- **Coverage model.** The general update is C_i,j = g_update(C_{i−1,j}, α_i,j, Φ(h_j), Ψ).
  - The linguistic variant sums past attention weights for each word.
  - The fertility variant divides that sum by a predicted fertility Φ_j = N·σ(U_f h_j), the expected number of target words per source word.
- **Alignment quality** (lower scores are better):
  - Groundhog baseline: AER 54.67, SAER 67.00.
  - Linguistic coverage with fertility: AER 52.13, SAER 64.85.
  - NN coverage with gating (d = 10): AER 50.50, SAER 64.25.
- **Long sentences (> 40 words).** Groundhog's translations become too short. The coverage models produce longer, more complete output, and the linguistic (with fertility) and NN (gated, d = 10) variants perform similarly on these inputs.
- **Translation quality.** The paper reports significant translation improvements, but the exact BLEU figures are in a section missing from the supplied text.
- **Code.** The implementation is released at github.com/tuzhaopeng/NMT-Coverage.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

235 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 217 not yet.

- [Neural Decipherment via Minimum-Cost Flow: From Ugaritic to Linear B](https://md.research.elamur.ai/free/en/papers/acl-2019-luo-neural-decipherment-minimum-cost-flow/)
- [Massive Exploration of Neural Machine Translation Architectures](https://md.research.elamur.ai/free/en/papers/britz-2017-massive-exploration-neural-machine-translation/)
- [Evaluating prose style transfer with the Bible](https://md.research.elamur.ai/free/en/papers/carlson-2017-evaluating-prose-style-transfer-bible/)
- [Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder](https://md.research.elamur.ai/free/en/papers/chen-2017-improved-neural-machine-translation-syntax/)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/)
- [Jointly Learning to Align and Translate with Transformer Models](https://md.research.elamur.ai/free/en/papers/garg-2019-jointly-learning-align-translate-transformer/)
- [Bottom-Up Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/)
- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [Text Generation from Knowledge Graphs with Graph Transformers](https://md.research.elamur.ai/free/en/papers/kedziorski-2019-text-generation-knowledge-graphs-graph/)
- [Globally Coherent Text Generation with Neural Checklist Models](https://md.research.elamur.ai/free/en/papers/kiddon-2016-globally-coherent-text-generation-neural/)
- [Masked Language Model Scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/)
- [A Graph-to-Sequence Model for AMR-to-Text Generation](https://md.research.elamur.ai/free/en/papers/song-2018-graph-sequence-model-amr-text/)
- [Exploiting Cross-Sentence Context for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/wang-2017-exploiting-cross-sentence-context-neural/)
- [Multi-Domain Dialogue Acts and Response Co-Generation](https://md.research.elamur.ai/free/en/papers/wang-2020-multi-domain-dialogue-acts-response/)
- [Bridging the Gap between Training and Inference for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2019-bridging-gap-between-training-inference/)
- [Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks](https://md.research.elamur.ai/free/en/papers/zhao-2018-paragraph-level-neural-question-generation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tu2016modeling,
  title = {{Modeling Coverage for Neural Machine Translation}},
  author = {Tu, Zhaopeng and Lu, Zhengdong and Liu, Yang and Liu, Xiaohua and Li, Hang},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1008},
  eprint = {1601.04811},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1008}
}
```

- [← previous Modeling Context in Referring Expressions](https://md.research.elamur.ai/free/en/papers/yu-2016-modeling-context-referring-expressions/)
- [next → Modelling Context with User Embeddings for Sarcasm Detection in Social Media](https://md.research.elamur.ai/free/en/papers/amir-2016-modelling-context-user-embeddings-sarcasm/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. ICLR 2015. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+ICLR+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. JMLR. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Re%CC%81jean+Ducharme%2C+Pascal+Vincent%2C+and+Christian+Janvin%2E+2003%2E+A+neural+probabilistic+language+model%2E+JMLR%2E)
3. Peter E. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. [find](https://scholar.google.com/scholar?q=Peter+E%2E+Brown%2C+Stephen+A%2E+Della+Pietra%2C+Vincent+J%2E+Della+Pietra%2C+and+Robert+L%2E+Mercer%2E+1993%2E+The+mathematics+of+statistical+machine+translation%3A+Parameter+estimation%2E+Computational+Linguistics%2C+19%282%29)
4. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2E)
5. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In ACL 2015. [find](https://scholar.google.com/scholar?q=Se%CC%81bastien+Jean%2C+Kyunghyun+Cho%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+2015%2E+On+using+very+large+target+vocabulary+for+neural+machine+translation%2E+In+ACL+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/)
6. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In NAACL 2003. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Franz+Josef+Och%2C+and+Daniel+Marcu%2E+2003%2E+Statistical+phrase%2Dbased+translation%2E+In+NAACL+2003%2E)
7. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: open source toolkit for statistical machine translation. In ACL… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondrej+Bojar%2C+Alexandra+Consta)
8. Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In NAACL 2006. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Ben+Taskar%2C+and+Dan+Klein%2E+2006%2E+Alignment+by+agreement%2E+In+NAACL+2006%2E) [our card](https://md.research.elamur.ai/free/en/papers/liang-2006-alignment-agreement/)
9. Yang Liu and Maosong Sun. 2015. Contrastive unsupervised word alignment with non-local features. In AAAI 2015. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Maosong+Sun%2E+2015%2E+Contrastive+unsupervised+word+alignment+with+non%2Dlocal+features%2E+In+AAAI+2015%2E)
10. Yong Cheng, Shiqi Shen, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Agreement-based Joint Training for Bidirectional Attention-based Neural Machine Translation. In IJCAI 2016. [find](https://scholar.google.com/scholar?q=Yong+Cheng%2C+Shiqi+Shen%2C+Zhongjun+He%2C+Wei+He%2C+Hua+Wu%2C+Maosong+Sun%2C+and+Yang+Liu%2E+2016%2E+Agreement%2Dbased+Joint+Training+for+Bidirectional+Attention%2Dbased+Neural+Machine+Translation%2E+In+IJCAI+2016%2E)
11. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP 2015. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D+Manning%2E+2015%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+EMNLP+2015%2E)
12. David Chiang. 2007. phrase-based translation. CL. [find](https://scholar.google.com/scholar?q=David+Chiang%2E+2007%2E+phrase%2Dbased+translation%2E+CL%2E)
13. Franz J. Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. Hierarchical [find](https://scholar.google.com/scholar?q=Franz+J%2E+Och+and+Hermann+Ney%2E+2003%2E+A+systematic+comparison+of+various+statistical+alignment+models%2E+Computational+Linguistics%2C+29%281%29%3A19%E2%80%9351%2E+Hierarchical)
14. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: encoder–decoder approaches. In SSST 2014. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Dzmitry+Bahdanau%2C+and+Yoshua+Bengio%2E+2014a%2E+On+the+properties+of+neural+machine+translation%3A+encoder%E2%80%93decoder+approaches%2E+In+SSST+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/)
15. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP 2014. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Caglar+Gulcehre%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014b%2E+Learning+phrase+representations+using+rnn+encoder%2Ddecoder+for+statistical+machine+transl)
16. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv. [find](https://scholar.google.com/scholar?q=Junyoung+Chung%2C+Caglar+Gulcehre%2C+KyungHyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Empirical+evaluation+of+gated+recurrent+neural+networks+on+sequence+modeling%2E+arXiv%2E)
17. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vylomova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating Structural Alignment Biases into an Attentional Neural Translation Model. In NAACL 2016. [find](https://scholar.google.com/scholar?q=Trevor+Cohn%2C+Cong+Duy+Vu+Hoang%2C+Ekaterina+Vylomova%2C+Kaisheng+Yao%2C+Chris+Dyer%2C+and+Gholamreza+Haffari%2E+2016%2E+Incorporating+Structural+Alignment+Biases+into+an+Attentional+Neural+Translation+Model%2E+In+N)
18. Michael Collins, Philipp Koehn, and Ivona Kučerová. 2005. Clause restructuring for statistical machine translation. In ACL 2005. [find](https://scholar.google.com/scholar?q=Michael+Collins%2C+Philipp+Koehn%2C+and+Ivona+Kuc%CC%8Cerova%CC%81%2E+2005%2E+Clause+restructuring+for+statistical+machine+translation%2E+In+ACL+2005%2E) [our card](https://md.research.elamur.ai/free/en/papers/collins-2005-clause-restructuring-statistical-machine-translation/)
19. Shi Feng, Shujie Liu, Mu Li, and Ming Zhou. 2016. Implicit distortion and fertility models for attention-based encoder-decoder nmt model. arXiv. [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Shujie+Liu%2C+Mu+Li%2C+and+Ming+Zhou%2E+2016%2E+Implicit+distortion+and+fertility+models+for+attention%2Dbased+encoder%2Ddecoder+nmt+model%2E+arXiv%2E)
20. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In ACL 2003. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och%2E+2003%2E+Minimum+error+rate+training+in+statistical+machine+translation%2E+In+ACL+2003%2E)
21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL 2002. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+ACL+2002%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
22. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681. [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kuldip+K+Paliwal%2E+1997%2E+Bidirectional+recurrent+neural+networks%2E+IEEE+Transactions+on+Signal+Processing%2C+45%2811%29%3A2673%E2%80%932681%2E)
23. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum Risk Training for Neural Machine Translation. In ACL 2016. [find](https://scholar.google.com/scholar?q=Shiqi+Shen%2C+Yong+Cheng%2C+Zhongjun+He%2C+Wei+He%2C+Hua+Wu%2C+Maosong+Sun%2C+and+Yang+Liu%2E+2016%2E+Minimum+Risk+Training+for+Neural+Machine+Translation%2E+In+ACL+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/shen-2015-minimum-risk-training-neural-machine/)
24. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In NIPS 2014. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+VV+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NIPS+2014%2E)
25. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In ICML 2015. [find](https://scholar.google.com/scholar?q=Kelvin+Xu%2C+Jimmy+Ba%2C+Ryan+Kiros%2C+Kyunghyun+Cho%2C+Aaron+Courville%2C+Ruslan+Salakhutdinov%2C+Richard+Zemel%2C+and+Yoshua+Bengio%2E+2015%2E+Show%2C+Attend+and+Tell%3A+Neural+Image+Caption+Generation+with+Visual+Attent) [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)