---
title: "Supervised Attentions for Neural Machine Translation — Mi 2016"
description: "The method offers a simple, inexpensive way to inject alignment knowledge into neural translation. 20 references, cited by 52 papers here."
authors: ["Haitao Mi", "Zhiguo Wang", "Abe Ittycheriah"]
year: 2016
venue: "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D16-1249/"
doi: "10.18653/v1/d16-1249"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mi-2016-supervised-attentions-neural-machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mi-2016-supervised-attentions-neural-machine-translation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Supervised Attentions for Neural Machine Translation

# Supervised Attentions for Neural Machine Translation

**Authors:** [Haitao Mi](https://md.research.elamur.ai/free/en/authors/haitao-mi/), [Zhiguo Wang](https://md.research.elamur.ai/free/en/authors/zhiguo-wang/) and [Abe Ittycheriah](https://md.research.elamur.ai/free/en/authors/abe-ittycheriah/) · **Year:** 2016 · **Venue:** [Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2016](https://md.research.elamur.ai/free/en/venues/emnlp/2016/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D16-1249/) · [PDF](https://aclanthology.org/D16-1249.pdf) · [DOI 10.18653/v1/d16-1249](https://doi.org/10.18653/v1/d16-1249) · [OpenAlex](https://openalex.org/W2963598809)

**Citations:** 131 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/mi-2016-supervised-attentions-neural-machine-translation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
In attention-based NMT, the attention weights often fail to match real word correspondences, so the model's alignment quality is poor. Mi, Wang and Ittycheriah use alignments of the training pairs as supervision for the attention model. Each reference alignment matrix comes from one of three sources: GIZA++ in one direction, the GDFA merge, or a MaxEnt aligner. It is converted into row-wise probability distributions, and unaligned target words are attached to the source end-of-sentence token. An optional variant first spreads each link with a Gaussian window and then normalizes. The Euclidean distance between this matrix and the model's attention matrix is subtracted from the usual log-likelihood objective.

The authors trained on roughly 5 million Chinese-English sentence pairs from DARPA BOLT, building on a strong coverage-based large-vocabulary baseline (Cov. LVNMT). Joint training with smoothed MaxEnt supervision gave the best single-system translation quality and even outperformed a tree-to-string syntax-based SMT system. It also raised attention alignment F1 by about 5 points. The method offers a simple, inexpensive way to inject alignment knowledge into neural translation.

## Key points
- **Test sets and metric:** evaluation used NIST MT06, MT08 news and MT08 web, scored with (TER-BLEU)/2, where lower is better.
- **Translation results:** the best system (MaxEnt alignments, joint training, Gaussian smoothing) averaged 13.04. This compares with 14.24 for Cov. LVNMT and 13.36 for tree-to-string. BLEU gains over the baseline were significant at p \< 0.01 on all three test sets.
- **Optimization strategy:** training the alignment and translation parts separately did not help. The A → T setting was much worse, averaging 20.40, so joint optimization (J) is needed.
- **Alignment results:** on 447 hand-aligned test sentences, attention F1 rose from 45.76 to 51.94. The MaxEnt aligner itself, trained on 67k hand-aligned sentences, reaches 75.96.
- **Other observations:**
  - The choice of alignment source (Zh → En, GDFA or MaxEnt) changed translation quality only slightly.
  - The brevity penalty consistently moved closer to the reference length once the alignment cost was added.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 85 works · 66,118 citations
- [IBM Research - Thomas J. Watson Research Center](https://md.research.elamur.ai/free/en/labs/ibm-research-thomas-j-watson-research-center/) — Research institute, Yorktown Heights, United States · 41 works · 46,197 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Haitao Mi, Zhiguo Wang and Abe Ittycheriah. 2016. Supervised Attentions for Neural Machine Translation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mi2016supervised,
  title = {{Supervised Attentions for Neural Machine Translation}},
  author = {Haitao Mi, Zhiguo Wang and Abe Ittycheriah},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1249},
  url = {https://doi.org/10.18653/v1/d16-1249}
}
```

- [← previous CoNLL 2016 Shared Task on Multilingual Shallow Discourse Parsing](https://md.research.elamur.ai/free/en/papers/xue-2016-conll-2016-shared-task-multilingual/)
- [next → End-to-End Argumentation Mining in Student Essays](https://md.research.elamur.ai/free/en/papers/persing-2016-end-end-argumentation-mining-student/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sasa Hasan Arne Mauser and Hermann Ney. 2008. Automatic evaluation measures for statistical machine translation system optimization. In Proceedings of LREC 2008, Marrakech, Morocco, may. [find](https://scholar.google.com/scholar?q=Sasa+Hasan+Arne+Mauser+and+Hermann+Ney%2E+2008%2E+Automatic+evaluation+measures+for+statistical+machine+translation+system+optimization%2E+In+Proceedings+of+LREC+2008%2C+Marrakech%2C+Morocco%2C+may%2E)
2. D. Bahdanau, K. Cho, and Y. Bengio. 2014. [Neural Machine Translation by Jointly Learning to Align and Translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). ArXiv e-prints, September.
3. Yong Cheng, Shiqi Shen, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Agreement-based joint training for bidirectional attention-based neural machine translation. In Proceedings of IJCAI, New York, USA, July. [find](https://scholar.google.com/scholar?q=Yong+Cheng%2C+Shiqi+Shen%2C+Zhongjun+He%2C+Wei+He%2C+Hua+Wu%2C+Maosong+Sun%2C+and+Yang+Liu%2E+2016%2E+Agreement%2Dbased+joint+training+for+bidirectional+attention%2Dbased+neural+machine+translation%2E+In+Proceedings+of+IJC)
4. Martin Cmejrek, Haitao Mi, and Bowen Zhou. 2013. Flexible and efficient hypergraph interactions for joint hierarchical and forest-to-string decoding. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 545–555, Seattle, Washington, USA, October. Associati… [find](https://scholar.google.com/scholar?q=Martin+Cmejrek%2C+Haitao+Mi%2C+and+Bowen+Zhou%2E+2013%2E+Flexible+and+efficient+hypergraph+interactions+for+joint+hierarchical+and+forest%2Dto%2Dstring+decoding%2E+In+Proceedings+of+the+2013+Conference+on+Empirical)
5. Michael Collins, Philipp Koehn, and Ivona Kucerova. 2005. [Clause restructuring for statistical machine translation](https://md.research.elamur.ai/free/en/papers/collins-2005-clause-restructuring-statistical-machine-translation/). In Proceedings of ACL, pages 531–540, Ann Arbor, Michigan, June.
6. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia,… [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Victor+Chahuneau%2C+and+Noah+A%2E+Smith%2E+2013%2E+A+simple%2C+fast%2C+and+effective+reparameterization+of+ibm+model+2%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Assoc)
7. Mark Hopkins and Jonathan May. 2011. [Tuning as ranking](https://md.research.elamur.ai/free/en/papers/hopkins-2011-tuning-as-ranking/). In Proceedings of EMNLP.
8. Abraham Ittycheriah and Salim Roukos. 2005. A maximum entropy word aligner for arabic-english machine translation. In HLT ’05: Proceedings of the HLT and EMNLP, pages 89–96. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural ma… [find](https://scholar.google.com/scholar?q=Abraham+Ittycheriah+and+Salim+Roukos%2E+2005%2E+A+maximum+entropy+word+aligner+for+arabic%2Denglish+machine+translation%2E+In+HLT+%E2%80%9905%3A+Proceedings+of+the+HLT+and+EMNLP%2C+pages+89%E2%80%9396%2E+Se%CC%81bastien+Jean%2C+Kyunghyun)
9. Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. 2016. Towards zero unknown word in neural machine translation. In Proceedings of IJCAI 2016, pages 2852– 2858, New York, NY, USA, July. [find](https://scholar.google.com/scholar?q=Xiaoqing+Li%2C+Jiajun+Zhang%2C+and+Chengqing+Zong%2E+2016%2E+Towards+zero+unknown+word+in+neural+machine+translation%2E+In+Proceedings+of+IJCAI+2016%2C+pages+2852%E2%80%93+2858%2C+New+York%2C+NY%2C+USA%2C+July%2E)
10. P. Liang, B. Taskar, and D. Klein. 2006. [Alignment by agreement](https://md.research.elamur.ai/free/en/papers/liang-2006-alignment-agreement/). In North American Association for Computational Linguistics (NAACL), pages 104–111.
11. P. Liang, D. Klein, and M. I. Jordan. 2008. Agreement-based learning. In Advances in Neural Information Processing Systems (NIPS). [find](https://scholar.google.com/scholar?q=P%2E+Liang%2C+D%2E+Klein%2C+and+M%2E+I%2E+Jordan%2E+2008%2E+Agreement%2Dbased+learning%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2E)
12. Yang Liu, Haitao Mi, Yang Feng, and Qun Liu. 2009. Joint decoding with multiple translation models. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2, ACL ’09, pages… [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Haitao+Mi%2C+Yang+Feng%2C+and+Qun+Liu%2E+2009%2E+Joint+decoding+with+multiple+translation+models%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+and+the+4th+Internation)
13. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal, September. Association for Computational Linguis…
14. Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016a. A coverage embedding model for neural machine translation. ArXiv e-prints. [find](https://scholar.google.com/scholar?q=Haitao+Mi%2C+Baskaran+Sankaran%2C+Zhiguo+Wang%2C+and+Abe+Ittycheriah%2E+2016a%2E+A+coverage+embedding+model+for+neural+machine+translation%2E+ArXiv+e%2Dprints%2E)
15. Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016b. Vocabulary manipulation for neural machine translation. In Proceedings of ACL, Berlin, Germany, August. [find](https://scholar.google.com/scholar?q=Haitao+Mi%2C+Zhiguo+Wang%2C+and+Abe+Ittycheriah%2E+2016b%2E+Vocabulary+manipulation+for+neural+machine+translation%2E+In+Proceedings+of+ACL%2C+Berlin%2C+Germany%2C+August%2E)
16. Franz Josef Och and Hermann Ney. 2000. [Improved statistical alignment models](https://md.research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/). In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, ACL ’00, pages 440–447, Stroudsburg, PA, USA. Association for Computational Linguistics.
17. Z. Tu, Z. Lu, Y. Liu, X. Liu, and H. Li. 2016. Coverage-based Neural Machine Translation. ArXiv e-prints, January. [find](https://scholar.google.com/scholar?q=Z%2E+Tu%2C+Z%2E+Lu%2C+Y%2E+Liu%2C+X%2E+Liu%2C+and+H%2E+Li%2E+2016%2E+Coverage%2Dbased+Neural+Machine+Translation%2E+ArXiv+e%2Dprints%2C+January%2E)
18. Matthew D. Zeiler. 2012. [ADADELTA: an adaptive learning rate method](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/).
19. CoRR. Jiajun Zhang. 2016. [Exploiting source-side monolingual data in neural machine translation](https://md.research.elamur.ai/free/en/papers/zhang-2016-exploiting-source-side-monolingual-data/). In Proceedings of EMNLP 2016, Austin, Texas, USA, November.
20. Bing Zhao and Yaser Al-onaizan. 2008. Generalizing local and non-local word-reordering patterns for syntax-based machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 572–581, Stroudsburg, PA, USA. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Bing+Zhao+and+Yaser+Al%2Donaizan%2E+2008%2E+Generalizing+local+and+non%2Dlocal+word%2Dreordering+patterns+for+syntax%2Dbased+machine+translation%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+L)

*Updated 24 October 2026*