---
title: "Neural Machine Translation with Supervised Attention — Liu 2016"
description: "Attention in neural machine translation (Bahdanau et al., 2015) yields soft word alignments… 2.5 BLEU. 32 references, cited by 62 papers here."
authors: ["Lemao Liu", "Masao Utiyama", "Andrew Finch", "Eiichiro Sumita"]
year: 2016
venue: "Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers"
source: "https://aclanthology.org/C16-1291/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2016-neural-machine-translation-supervised-attention/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2016-neural-machine-translation-supervised-attention/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Neural Machine Translation with Supervised Attention

# Neural Machine Translation with Supervised Attention

**Authors:** [Lemao Liu](https://md.research.elamur.ai/free/en/authors/lemao-liu/), [Masao Utiyama](https://md.research.elamur.ai/free/en/authors/masao-utiyama/), [Andrew Finch](https://md.research.elamur.ai/free/en/authors/andrew-finch/) and [Eiichiro Sumita](https://md.research.elamur.ai/free/en/authors/eiichiro-sumita/) · **Year:** 2016 · **Venue:** [Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2016](https://md.research.elamur.ai/free/en/venues/coling/2016/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/C16-1291/) · [PDF](https://aclanthology.org/C16-1291.pdf) · [OpenAlex](https://openalex.org/W2520028199)

**Citations:** 22 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/liu-2016-neural-machine-translation-supervised-attention/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Attention in neural machine translation (Bahdanau et al., 2015) yields soft word alignments that are much worse than those from classical aligners. For Chinese-English, Cheng et al. (2016) report an AER of 54 for attention versus 30 for conventional models. The authors explain the gap this way: attention must choose the next source words before the target word is known, so it acts as a reordering model. Aligners like GIZA++, by contrast, can see the full target sentence. SMT usually trains reordering models on pre-computed alignments, and the authors carry this practice over in Supervised Attention NMT (SA-NMT). They first align the training corpus with an off-the-shelf tool and convert the hard links into per-word distributions. Training then adds a weighted penalty for disagreement between the model's attention and these targets to the usual translation likelihood. Decoding stays exactly as in standard NMT. On two Chinese-to-English tasks, the method gives better alignments and higher BLEU. This shows that cheap external alignment tools can usefully guide an internal component of a neural model.

## Key points
- The supervision signal comes from conventional aligners, either GIZA++ or fast_align, run on the bilingual training data before NMT training. The model is built on top of the Bahdanau et al. (2015) architecture, which uses a GRU decoder and a bidirectional RNN encoder.
- Hard alignments are converted into attention targets with simple rules:
  - An unaligned target word copies the alignment of the nearest aligned word, preferring the right side (following Devlin et al., 2014).
  - A word linked to several source words spreads its weight evenly across them.
  - The end-of-sentence tokens on both sides are aligned to each other.
- Three disagreement losses are compared, each weighted by a hyper-parameter λ: mean squared error (MSE), a multiplicative agreement loss (MUL), and cross entropy (CE).
- On the large-scale task, SA-NMT beats three baselines, including Moses, and gains up to 2.5 BLEU over the strongest one.
- On the low-resource BTEC task, the setup was as follows:
  - Training data: 30k sentence pairs (0.27M Chinese words, 0.33M English words).
  - Dev/test sets: CSTAR03 and IWSLT04.
  - Settings: vocabulary 6000, batch size 16, λ = 1.
  - BLEU results on IWSLT04: SA-NMT 40.7, NMT2 35.9, NMT1 33.0, Moses 45.1. SA-NMT is significantly better than both NMT baselines (p \< 0.01) but still trails Moses.

## Interesting ideas
- **Supervising an intermediate variable.** Treating a latent component as an observed variable and supervising it with output from an older pipeline tool is a cheap way to add prior knowledge. The same trick could guide attention toward parser arcs or toward gazetteer and toponym matches in geography data.
- **Supervision as a weighted regularizer.** The supervision is a weighted term in a joint objective, not a hard constraint, so the model can still drift from the noisy teacher when that helps translation. λ also gives a knob for trading task accuracy against faithfulness to the alignments.
- **Gradient benefit from mid-network supervision.** Because the attention sits closer to the encoder than the output does, supervising it sends a training signal into the middle of the network. The authors argue this eases vanishing gradients and overfitting, which may matter most when training data is scarce.
- **Explaining a weakness through information access.** The authors account for attention's poor alignments by pointing out that it cannot condition on the current target word. Asking what information a component actually has at prediction time is a useful diagnostic for other architectures.
- **Turning discrete annotations into soft targets.** Simple heuristics (inheriting from the nearest neighbour, splitting weight evenly) convert hard, incomplete labels into proper probability distributions. This recipe transfers to any task where symbolic annotations must supervise soft model outputs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Institute of Information and Communications Technology](https://md.research.elamur.ai/free/en/labs/national-institute-of-information-and-communications-technology/) — Research institute, Tokyo, Japan · 13 works · 1,970 citations
- [University of Malta](https://md.research.elamur.ai/free/en/universities/university-of-malta/) — University, Msida, Malta · 16 works · 775 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Lemao Liu, Masao Utiyama, Andrew Finch and Eiichiro Sumita. 2016. Neural Machine Translation with Supervised Attention. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2016neural,
  title = {{Neural Machine Translation with Supervised Attention}},
  author = {Lemao Liu, Masao Utiyama, Andrew Finch and Eiichiro Sumita},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  url = {https://aclanthology.org/C16-1291/}
}
```

- [← previous Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations](https://md.research.elamur.ai/free/en/papers/abercrombie-2016-putting-sarcasm-detection-into-context/)
- [next → CAMR at SemEval-2016 Task 8: An Extended Transition-based AMR Parser](https://md.research.elamur.ai/free/en/papers/wang-2016-camr-semeval-2016-task-8/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. [Incorporating discrete translation lexicons into neural machine translation](https://md.research.elamur.ai/free/en/papers/arthur-2016-incorporating-discrete-translation-lexicons-into/). CoRR, abs/1606.02006.
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). CoRR, abs/1409.0473.
3. Arianna Bisazza and Marcello Federico. 2016. A survey of word reordering in statistical machine translation: Computational models and language phenomena. Computational Linguistics, 42. [find](https://scholar.google.com/scholar?q=Arianna+Bisazza+and+Marcello+Federico%2E+2016%2E+A+survey+of+word+reordering+in+statistical+machine+translation%3A+Computational+models+and+language+phenomena%2E+Computational+Linguistics%2C+42%2E)
4. Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Comput. Linguist., 19(2):263–311. [find](https://scholar.google.com/scholar?q=Peter+F%2E+Brown%2C+Vincent+J%2E+Della+Pietra%2C+Stephen+A%2E+Della+Pietra%2C+and+Robert+L%2E+Mercer%2E+1993%2E+The+mathematics+of+statistical+machine+translation%3A+Parameter+estimation%2E+Comput%2E+Linguist%2E%2C+19%282%29%3A263%E2%80%93311)
5. Wenhu Chen, Evgeny Matusov, Shahram Khadivi, and Jan-Thorsten Peter. 2016. Guided alignment training for topic-aware neural machine translation. In Proceedings of AMTA. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Evgeny+Matusov%2C+Shahram+Khadivi%2C+and+Jan%2DThorsten+Peter%2E+2016%2E+Guided+alignment+training+for+topic%2Daware+neural+machine+translation%2E+In+Proceedings+of+AMTA%2E)
6. Yong Cheng, Shiqi Shen, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Agreement-based joint training for bidirectional attention-based neural machine translation. In Proceedings of IJCAI. [find](https://scholar.google.com/scholar?q=Yong+Cheng%2C+Shiqi+Shen%2C+Zhongjun+He%2C+Wei+He%2C+Hua+Wu%2C+Maosong+Sun%2C+and+Yang+Liu%2E+2016%2E+Agreement%2Dbased+joint+training+for+bidirectional+attention%2Dbased+neural+machine+translation%2E+In+Proceedings+of+IJC)
7. Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555. [find](https://scholar.google.com/scholar?q=Junyoung+Chung%2C+C%CC%A7aglar+Gu%CC%88lc%CC%A7ehre%2C+KyungHyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Empirical+evaluation+of+gated+recurrent+neural+networks+on+sequence+modeling%2E+CoRR%2C+abs%2F1412%2E3555%2E)
8. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. [Incorporating structural alignment biases into an attentional neural translation model](https://md.research.elamur.ai/free/en/papers/cohn-2016-incorporating-structural-alignment-biases-into/). In Proceedings of NAACL-HLT.
9. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. [Fast and robust neural network joint models for statistical machine translation](https://md.research.elamur.ai/free/en/papers/devlin-2014-fast-robust-neural-network-joint/). In Proceedings of ACL.
10. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In In Proc. NAACL. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Victor+Chahuneau%2C+and+Noah+A%2E+Smith%2E+2013%2E+A+simple%2C+fast%2C+and+effective+reparameterization+of+ibm+model+2%2E+In+In+Proc%2E+NAACL%2E)
11. Theodoros Evgeniou and Massimiliano Pontil. 2004. Regularized multi–task learning. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04. [find](https://scholar.google.com/scholar?q=Theodoros+Evgeniou+and+Massimiliano+Pontil%2E+2004%2E+Regularized+multi%E2%80%93task+learning%2E+In+Proceedings+of+the+Tenth+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+KDD+%E2%80%9904%2E)
12. Shi Feng, Shujie Liu, Mu Li, and Ming Zhou. 2016. Implicit distortion and fertility models for attention-based encoder-decoder NMT model. CoRR, abs/1601.03317. — [arXiv:1601.03317](https://arxiv.org/abs/1601.03317) [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Shujie+Liu%2C+Mu+Li%2C+and+Ming+Zhou%2E+2016%2E+Implicit+distortion+and+fertility+models+for+attention%2Dbased+encoder%2Ddecoder+NMT+model%2E+CoRR%2C+abs%2F1601%2E03317%2E+%E2%80%94+%5BarXiv%3A1601%2E03317%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
13. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst. 2007. Moses: open source toolkit for statistical machine translation. In Proceedings of ACL: Demonstrations. [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2C+H%2E+Hoang%2C+A%2E+Birch%2C+C%2E+Callison%2DBurch%2C+M%2E+Federico%2C+N%2E+Bertoldi%2C+B%2E+Cowan%2C+W%2E+Shen%2C+C%2E+Moran%2C+R%2E+Zens%2C+C%2E+Dyer%2C+O%2E+Bojar%2C+A%2E+Constantin%2C+and+E%2E+Herbst%2E+2007%2E+Moses%3A+open+source+toolkit+for+s)
14. E.L. Lehmann and G. Casella. 1998. Theory of Point Estimation. Springer Verlag. [find](https://scholar.google.com/scholar?q=E%2EL%2E+Lehmann+and+G%2E+Casella%2E+1998%2E+Theory+of+Point+Estimation%2E+Springer+Verlag%2E)
15. Peng Li, Yang Liu, Maosong Sun, Tatsuya Izuha, and Dakun Zhang. 2014. A neural reordering model for phrase-based translation. In Proceedings of COLING. [find](https://scholar.google.com/scholar?q=Peng+Li%2C+Yang+Liu%2C+Maosong+Sun%2C+Tatsuya+Izuha%2C+and+Dakun+Zhang%2E+2014%2E+A+neural+reordering+model+for+phrase%2Dbased+translation%2E+In+Proceedings+of+COLING%2E)
16. Percy Liang, Ben Taskar, and Dan Klein. 2006. [Alignment by agreement](https://md.research.elamur.ai/free/en/papers/liang-2006-alignment-agreement/). In Proceedings of HLT-NAACL.
17. Yang Liu and Maosong Sun. 2015. Contrastive unsupervised word alignment with non-local features. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Maosong+Sun%2E+2015%2E+Contrastive+unsupervised+word+alignment+with+non%2Dlocal+features%2E)
18. Chenxi Liu, Junhua Mao, Fei Sha, and Alan L. Yuille. 2016. Attention correctness in neural image captioning. CoRR, abs/1605.09553. — [arXiv:1605.09553](https://arxiv.org/abs/1605.09553) [find](https://scholar.google.com/scholar?q=Chenxi+Liu%2C+Junhua+Mao%2C+Fei+Sha%2C+and+Alan+L%2E+Yuille%2E+2016%2E+Attention+correctness+in+neural+image+captioning%2E+CoRR%2C+abs%2F1605%2E09553%2E+%E2%80%94+%5BarXiv%3A1605%2E09553%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1605%2E09553%29)
19. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proceedings of IWSLT. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong+and+Christopher+D%2E+Manning%2E+2015%2E+Stanford+neural+machine+translation+systems+for+spoken+language+domains%2E+In+Proceedings+of+IWSLT%2E)
20. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). In Proceedings of EMNLP.
21. Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. [Supervised attentions for neural machine translation](https://md.research.elamur.ai/free/en/papers/mi-2016-supervised-attentions-neural-machine-translation/). In Proceedings of EMNLP.
22. Graham Neubig, Makoto Morishita, and Satoshi Nakamura. 2015. Neural reranking improves subjective quality of machine translation: NAIST at WAT2015. In Proceedings of the 2nd Workshop on Asian Translation (WAT2015). [find](https://scholar.google.com/scholar?q=Graham+Neubig%2C+Makoto+Morishita%2C+and+Satoshi+Nakamura%2E+2015%2E+Neural+reranking+improves+subjective+quality+of+machine+translation%3A+NAIST+at+WAT2015%2E+In+Proceedings+of+the+2nd+Workshop+on+Asian+Translat)
23. Franz Josef Och and Hermann Ney. 2000. [Improved statistical alignment models](https://md.research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/). In Proceedings of ACL, pages 440–447.
24. Reuven Y. Rubinstein and Dirk P. Kroese. 2004. The Cross Entropy Method: A Unified Approach To Combinatorial Optimization, Monte-carlo Simulation (Information Science and Statistics). Springer-Verlag New York, Inc., Secaucus, NJ, USA. [find](https://scholar.google.com/scholar?q=Reuven+Y%2E+Rubinstein+and+Dirk+P%2E+Kroese%2E+2004%2E+The+Cross+Entropy+Method%3A+A+Unified+Approach+To+Combinatorial+Optimization%2C+Monte%2Dcarlo+Simulation+%28Information+Science+and+Statistics%29%2E+Springer%2DVerlag+)
25. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2015. Sequence to sequence learning with neural networks. In Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2015%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Proceedings+of+NIPS%2E)
26. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wei+Liu%2C+Yangqing+Jia%2C+Pierre+Sermanet%2C+Scott+Reed%2C+Dragomir+Anguelov%2C+Dumitru+Erhan%2C+Vincent+Vanhoucke%2C+and+Andrew+Rabinovich%2E+2015%2E+Going+deeper+with+convolutions%2E+In+Proceedings+)
27. Akihiro Tamura, Taro Watanabe, and Eiichiro Sumita. 2014. Recurrent neural networks for word alignment model. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Akihiro+Tamura%2C+Taro+Watanabe%2C+and+Eiichiro+Sumita%2E+2014%2E+Recurrent+neural+networks+for+word+alignment+model%2E+In+Proceedings+of+ACL%2E)
28. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. [Modeling coverage for neural machine translation](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/). In Proceedings of ACL.
29. Deyi Xiong, Qun Liu, and Shouxun Lin. 2006. [Maximum entropy based phrase reordering model for statistical machine translation](https://md.research.elamur.ai/free/en/papers/xiong-2006-maximum-entropy-based-phrase-reordering/). In Proceedings of ACL.
30. Nan Yang, Shujie Liu, Mu Li, Ming Zhou, and Nenghai Yu. 2013. Word alignment modeling with context dependent deep neural network. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), August. [find](https://scholar.google.com/scholar?q=Nan+Yang%2C+Shujie+Liu%2C+Mu+Li%2C+Ming+Zhou%2C+and+Nenghai+Yu%2E+2013%2E+Word+alignment+modeling+with+context+dependent+deep+neural+network%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Compu)
31. Matthew D. Zeiler. 2012. [ADADELTA: an adaptive learning rate method](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/). CoRR.
32. Jingyi Zhang, Masao Utiyama, Eiichiro Sumita, Hai Zhao, Graham Neubig, and Satoshi Nakamura. 2016. Learning local word reorderings for hierarchical phrase-based statistical machine translation. Machine Translation. [find](https://scholar.google.com/scholar?q=Jingyi+Zhang%2C+Masao+Utiyama%2C+Eiichiro+Sumita%2C+Hai+Zhao%2C+Graham+Neubig%2C+and+Satoshi+Nakamura%2E+2016%2E+Learning+local+word+reorderings+for+hierarchical+phrase%2Dbased+statistical+machine+translation%2E+Machin)

*Updated 24 October 2026*