---
title: "Toward Multilingual Neural Machine Translation — Ha 2016"
description: "Attention-based NMT is usually trained for one language pair."
authors: ["Thanh-Le Ha", "Jan Niehues", "Alex Waibel"]
year: 2016
venue: "Proceedings of the 13th International Conference on Spoken Language Translation"
source: "https://aclanthology.org/2016.iwslt-1.6/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ha-2016-multilingual-neural-machine-translation-universal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ha-2016-multilingual-neural-machine-translation-universal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

# Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

**Authors:** Thanh-Le Ha, [Jan Niehues](https://md.research.elamur.ai/free/en/authors/jan-niehues/), [Alex Waibel](https://md.research.elamur.ai/free/en/authors/alex-waibel/) · **Year:** 2016 · **Venue:** Proceedings of the 13th International Conference on Spoken Language Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2016.iwslt-1.6/) · [PDF](https://aclanthology.org/2016.iwslt-1.6.pdf) · [OpenAlex](https://openalex.org/W6950195680)

**Citations:** 236 citations (openalex-snapshot · 2026-10-14) · [162](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Attention-based NMT is usually trained for one language pair. Earlier multilingual extensions used several encoders or decoders and custom ways of sharing or merging attention, which adds parameters and complicates training. Ha, Niehues and Waibel propose a simpler design: a single shared encoder and a single shared decoder handle every language, and the standard attention model of Bahdanau et al. is kept unchanged. Their technique, Language-specific Coding, tags each word with its language, so the same surface form in two languages becomes two separate entries in a mixed vocabulary. The network can still place words with similar meanings close together in a common embedding space. Because the architecture is not modified, the model stays about the size of one bilingual system and trains in the usual way. In a simulated low-resource setting, adding data from other languages raised quality by up to 2.6 BLEU. The same framework also gave a new way to use monolingual text, adding 3.15 BLEU on IWSLT'16 English→German. This is useful for languages with little parallel data.

## Key points
- The approach uses one universal encoder and one universal decoder for all languages, supports many-to-many translation, and needs no network changes to use attention.
- Language-specific Coding keeps words from different languages apart in the shared vocabulary. The free-parameter count stays at the scale of a single NMT system.
- Results: up to +2.6 BLEU in a simulated under-resourced scenario, and +3.15 BLEU on the IWSLT'16 English→German task when monolingual data is used through the same framework.
- The paper compares its method with multi-encoder/decoder systems by Luong et al., Dong et al., Firat et al. (shared attention) and Zoph & Knight (multi-source). Its setup draws on subword units (Byte-Pair Encoding), WIT3 data and ADADELTA.
- Venue: IWSLT 2016, the 13th edition of the international *workshop* series devoted to translating speech. Future work proposed by the authors includes real (not simulated) low-resource tests and zero-resource translation without direct parallel data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

162 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 154 not yet.

- [Massively Multilingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
- [Unsupervised Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gu-2018-meta-learning-low-resource-neural/)
- [A neural interlingua for multilingual machine translation](https://md.research.elamur.ai/free/en/papers/lu-2018-neural-interlingua-multilingual-machine-translation/)
- [Multilingual Translation from Denoising Pre-Training](https://md.research.elamur.ai/free/en/papers/tang-2021-multilingual-translation-denoising-pre-training/)
- [The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT](https://md.research.elamur.ai/free/en/papers/tiedemann-2020-tatoeba-translation-challenge-realistic-data/)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://md.research.elamur.ai/free/en/papers/zhang-2020-improving-massively-multilingual-neural-machine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ha2016toward,
  title = {{Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder}},
  author = {Ha, Thanh-Le and Niehues, Jan and Waibel, Alex},
  booktitle = {Proceedings of the 13th International Conference on Spoken Language Translation},
  year = {2016},
  url = {https://aclanthology.org/2016.iwslt-1.6/}
}
```

- [← previous Top Notch Participant Tracking in the Hebrew Bible](https://md.research.elamur.ai/free/en/papers/hbo-2016-winther-nielsen-top-notch-participant-tracking-hebrew/)
- [next → Transfer of Corpus-Specific Dialogue Act Annotation to ISO Standard: Is it worth it?](https://md.research.elamur.ai/free/en/papers/chowdhury-2016-transfer-corpus-specific-dialogue-act/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, R. Cattoni, and M. Federico, “The IWSLT 2015 Evaluation Campaign,” in Proceedings of the 12th International Workshop on Spoken Language Translation (IWSLT 2015), Danang, Vietnam, 2015. [find](https://scholar.google.com/scholar?q=M%2E+Cettolo%2C+J%2E+Niehues%2C+S%2E+Stu%CC%88ker%2C+L%2E+Bentivogli%2C+R%2E+Cattoni%2C+and+M%2E+Federico%2C+%E2%80%9CThe+IWSLT+2015+Evaluation+Campaign%2C%E2%80%9D+in+Proceedings+of+the+12th+International+Workshop+on+Spoken+Language+Translation+%28)
2. O. Bojar, R. Chatterjee, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, et al., “Findings of the 2016 Conference on Machine Translation (WMT16),” in Proceedings of the First Conference on Machine Translation (WMT16). Berlin, Germany: Association for Comp… [find](https://scholar.google.com/scholar?q=O%2E+Bojar%2C+R%2E+Chatterjee%2C+C%2E+Federmann%2C+B%2E+Haddow%2C+P%2E+Koehn%2C+J%2E+Leveling%2C+C%2E+Monz%2C+P%2E+Pecina%2C+M%2E+Post%2C+H%2E+Saint%2DAmand%2C+et+al%2E%2C+%E2%80%9CFindings+of+the+2016+Conference+on+Machine+Translation+%28WMT16%29%2C%E2%80%9D+in+Proce) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/)
3. D. Bahdanau, K. Cho, and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” CoRR, vol. abs/1409.0473, 2014. [Online]. Available: http://arxiv.org/abs/1409.0473 [↗](http://arxiv.org/abs/1409.0473) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
4. S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997. [Online]. Available: http://dx.doi.org/10.1162/neco.1997.9.8.1735 [↗](http://dx.doi.org/10.1162/neco.1997.9.8.1735)
5. K. Cho, B. van Merrienboer, Ç. Gülçehre, F. Bougares, H. Schwenk, and Y. Bengio, “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation,” in Proceedings of Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8. Baltimore, ML,… [find](https://scholar.google.com/scholar?q=K%2E+Cho%2C+B%2E+van+Merrienboer%2C+C%CC%A7%2E+Gu%CC%88lc%CC%A7ehre%2C+F%2E+Bougares%2C+H%2E+Schwenk%2C+and+Y%2E+Bengio%2C+%E2%80%9CLearning+Phrase+Representations+using+RNN+Encoder%2DDecoder+for+Statistical+Machine+Translation%2C%E2%80%9D+in+Proceedings+of+E)
6. M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 15. Lisbon, Portugal: Association for Computational Linguistics, September 2015, pp. 1412–14… [find](https://scholar.google.com/scholar?q=M%2E%2DT%2E+Luong%2C+H%2E+Pham%2C+and+C%2E+D%2E+Manning%2C+%E2%80%9CEffective+approaches+to+attention%2Dbased+neural+machine+translation%2C%E2%80%9D+in+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing)
7. M.-T. Luong, I. Sutskever, Q. V. Le, O. Vinyals, and W. Zaremba, “Addressing the rare word problem in neural machine translation,” in Proceedings of ACL-IJNLP 2015. Beijing, China: Association for Computational Linguistics, July 2015, pp. 11–19. [Online]. Available: [http://www.aclweb.org/anthology/…](http://www.aclweb.org/anthology/…) [↗](http://www.aclweb.org/anthology/…) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/)
8. R. Sennrich, B. Haddow, and A. Birch, “Neural Machine Translation of Rare Words with Subword Units,” in Association for Computational Linguistics (ACL 2016), Berlin, Germany, August 2016. [find](https://scholar.google.com/scholar?q=R%2E+Sennrich%2C+B%2E+Haddow%2C+and+A%2E+Birch%2C+%E2%80%9CNeural+Machine+Translation+of+Rare+Words+with+Subword+Units%2C%E2%80%9D+in+Association+for+Computational+Linguistics+%28ACL+2016%29%2C+Berlin%2C+Germany%2C+August+2016%2E)
9. M.-T. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and L. Kaiser, “Multi-task sequence to sequence learning,” in International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, May 2016. [find](https://scholar.google.com/scholar?q=M%2E%2DT%2E+Luong%2C+Q%2E+V%2E+Le%2C+I%2E+Sutskever%2C+O%2E+Vinyals%2C+and+L%2E+Kaiser%2C+%E2%80%9CMulti%2Dtask+sequence+to+sequence+learning%2C%E2%80%9D+in+International+Conference+on+Learning+Representations+%28ICLR%29%2C+San+Juan%2C+Puerto+Rico%2C+May+2) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
10. D. Dong, H. Wu, W. He, D. Yu, and H. Wang, “Multi-Task Learning for Multiple Language Translation,” in Proceedings of ACL-IJNLP 2015. Beijing, China: Association for Computational Linguistics, July 2015, pp. 1723–1732. [Online]. Available: [http://www.aclweb.org/anthology/P15-1166](http://www.aclweb.org/anthology/P15-1166) [↗](http://www.aclweb.org/anthology/P15-1166) [our card](https://md.research.elamur.ai/free/en/papers/dong-2015-multi-task-learning-multiple-language/)
11. O. Firat, K. Cho, and Y. Bengio, “Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism,” CoRR, vol. abs/1601.01073, 2016. [Online]. Available: http://arxiv.org/abs/1601. 01073 [↗](http://arxiv.org/abs/1601) [our card](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/)
12. B. Zoph and K. Knight, “Multi-Source Neural Translation,” in The North American Chapter of the Association for Computational Linguistics (NAACL 2016), San Diego, CA, USA, June 2016. [find](https://scholar.google.com/scholar?q=B%2E+Zoph+and+K%2E+Knight%2C+%E2%80%9CMulti%2DSource+Neural+Translation%2C%E2%80%9D+in+The+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NAACL+2016%29%2C+San+Diego%2C+CA%2C+USA%2C+June+2016%2E)
13. M. Cettolo, C. Girardi, and M. Federico, “Wit 3 : Web inventory of transcribed and translated talks,” in Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT), Trento, Italy, May 2012, pp. 261–268. [find](https://scholar.google.com/scholar?q=M%2E+Cettolo%2C+C%2E+Girardi%2C+and+M%2E+Federico%2C+%E2%80%9CWit+3+%3A+Web+inventory+of+transcribed+and+translated+talks%2C%E2%80%9D+in+Proceedings+of+the+16+th+Conference+of+the+European+Association+for+Machine+Translation+%28EAMT%29%2C)
14. M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,” CoRR, vol. abs/1212.5701, 2012. [Online]. Available: http://arxiv.org/abs/1212.5701 [↗](http://arxiv.org/abs/1212.5701) [our card](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/)
15. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL 2002). Association for Computational Linguistics, 2002, pp. 311–318. [find](https://scholar.google.com/scholar?q=K%2E+Papineni%2C+S%2E+Roukos%2C+T%2E+Ward%2C+and+W%2E%2DJ%2E+Zhu%2C+%E2%80%9CBleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2C%E2%80%9D+in+Proceedings+of+the+40th+Annual+Meeting+on+Association+for+Computational+Linguistics) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
16. R. Sennrich, B. Haddow, and A. Birch, “Improving Neural Machine Translation Models with Monolingual Data,” in Association for Computational Linguistics (ACL 2016), Berlin, Germany, August 2016. [find](https://scholar.google.com/scholar?q=R%2E+Sennrich%2C+B%2E+Haddow%2C+and+A%2E+Birch%2C+%E2%80%9CImproving+Neural+Machine+Translation+Models+with+Monolingual+Data%2C%E2%80%9D+in+Association+for+Computational+Linguistics+%28ACL+2016%29%2C+Berlin%2C+Germany%2C+August+2016%2E)
17. Ç. Gülçehre, O. Firat, K. Xu, K. Cho, L. Barrault, H. Lin, F. Bougares, H. Schwenk, and Y. Bengio, “On Using Monolingual Corpora in Neural Machine Translation,” CoRR, vol. abs/1503.03535, 2015. [Online]. Available: http://arxiv.org/abs/1503.03535 [↗](http://arxiv.org/abs/1503.03535) [our card](https://md.research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/)
18. E. Cho, J. Niehues, T.-L. Ha, M. Sperber, M. Mediani, and A. Waibel, “Adaptation and Combination of NMT Systems: The KIT Translation Systems for IWSLT 2016,” in Proceedings of the 13th International Workshop on Spoken Language Translation (IWSLT 2016) -To be appeared. [find](https://scholar.google.com/scholar?q=E%2E+Cho%2C+J%2E+Niehues%2C+T%2E%2DL%2E+Ha%2C+M%2E+Sperber%2C+M%2E+Mediani%2C+and+A%2E+Waibel%2C+%E2%80%9CAdaptation+and+Combination+of+NMT+Systems%3A+The+KIT+Translation+Systems+for+IWSLT+2016%2C%E2%80%9D+in+Proceedings+of+the+13th+International+W)