--- title: "Multi-Way, Multilingual Neural Machine — Firat 2016" description: "Attention-based neural MT systems are usually built for a single language pair." authors: ["Orhan Firat", "Kyunghyun Cho", "Yoshua Bengio"] year: 2016 venue: "Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies" source: "https://aclanthology.org/N16-1101/" doi: "10.18653/v1/n16-1101" arxiv: "1601.01073" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism # Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism **Authors:** [Orhan Firat](https://md.research.elamur.ai/free/en/authors/orhan-firat/), [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/), [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/N16-1101/) · [PDF](https://aclanthology.org/N16-1101.pdf) · [DOI 10.18653/v1/n16-1101](https://doi.org/10.18653/v1/n16-1101) · [arXiv 1601.01073](https://arxiv.org/abs/1601.01073) · [OpenAlex](https://openalex.org/W2229833550) **Citations:** 562 citations (openalex-snapshot · 2026-10-14) · [259](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Attention-based neural MT systems are usually built for a single language pair. Their attention (alignment) component was assumed to be pair-specific, so earlier multilingual NMT work either dropped attention or supported only one-to-many translation. Firat, Cho and Bengio design one network with a separate encoder for each source language, a separate decoder for each target language, and a single attention module reused by every pair. Since only encoders and decoders are added per language, model size scales linearly with the number of languages, rather than quadratically as with a collection of bilingual systems. The model is trained jointly on WMT'15 data for English, French, Czech, German, Russian and Finnish (ten translation directions). It is also tested in simulated low-resource settings where the target parallel corpus is artificially reduced. The multilingual model outperforms single-pair baselines in all low-resource cases and matches or beats them on most large-scale directions while using roughly half the parameters. This shows that attention can be shared across languages and that multilinguality especially benefits data-poor pairs. ## Key points - Architecture: N encoders and M decoders connected through one shared attention mechanism. Parameter count grows as O(L), versus O(L²) for separate single-pair models. - Data: all ten WMT'15 pairs/directions (En↔Fr, Cs, De, Ru, Fi). Low-resource conditions were created by artificially limiting the size of the target parallel corpus. - In low-resource tests, the multilingual model ("Multi") beats both "Single" and "Single+DF", which adds target-side monolingual data. The advantage widens as parallel data shrinks, so the gain is not just a result of seeing more target text. - At large scale, the multilingual model is better than or comparable to single-pair models in both BLEU and log-probability, with the clearest gains when translating into English. The authors attribute this to English appearing in every training corpus. Adding monolingual data via autoencoder-style reconstruction led to token copying and worse results. - Training splits language pairs across GPUs (NVidia Titan X, 12GB) and synchronises only the attention parameters, which gives near-linear speed-up. Code is released at github.com/nyu-dl/dl4mt-multi. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 259 papers of our library cite it (of the 140673 whose bibliographies we read); 26 of them have a page here, 233 not yet. - [Morphological Word Embeddings for Arabic Neural Machine Translation in Low-Resource Settings](https://md.research.elamur.ai/free/en/papers/aclbulk-2018-shapiro-morphological-word-embeddings-arabic-neural/) - [Few-Shot and Zero-Shot Learning for Historical Text Normalization](https://md.research.elamur.ai/free/en/papers/aclbulk-2019-bollmann-few-shot-zero-shot-learning/) - [Massively Multilingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/) - [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/) - [Unsupervised Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/) - [Simple, Scalable Adaptation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/) - [Evaluating prose style transfer with the Bible](https://md.research.elamur.ai/free/en/papers/carlson-2017-evaluating-prose-style-transfer-bible/) - [A Survey of Domain Adaptation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chu-2018-survey-domain-adaptation-neural-machine/) - [Meta-Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gu-2018-meta-learning-low-resource-neural/) - [Universal Neural Machine Translation for Extremely Low Resource Languages](https://md.research.elamur.ai/free/en/papers/gu-2018-universal-neural-machine-translation-extremely/) - [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://md.research.elamur.ai/free/en/papers/ha-2016-multilingual-neural-machine-translation-universal/) - [Survey of Low-Resource Machine Translation](https://md.research.elamur.ai/free/en/papers/haddow-2022-survey-low-resource-machine-translation/) - [Trivial Transfer Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/kocmi-2018-trivial-transfer-learning-low-resource/) - [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://md.research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/) - [A neural interlingua for multilingual machine translation](https://md.research.elamur.ai/free/en/papers/lu-2018-neural-interlingua-multilingual-machine-translation/) - [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://md.research.elamur.ai/free/en/papers/nekoto-2020-participatory-research-low-resourced-machine/) - [Exploring Neural Text Simplification Models](https://md.research.elamur.ai/free/en/papers/nisioi-2017-exploring-neural-text-simplification-models/) - [When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?](https://md.research.elamur.ai/free/en/papers/qi-2018-when-why-are-pre-trained/) - [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://md.research.elamur.ai/free/en/papers/ramesh-2022-samanantar-largest-publicly-available-parallel/) - [Multilingual Translation from Denoising Pre-Training](https://md.research.elamur.ai/free/en/papers/tang-2021-multilingual-translation-denoising-pre-training/) - [The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT](https://md.research.elamur.ai/free/en/papers/tiedemann-2020-tatoeba-translation-challenge-realistic-data/) - [Facebook AI's WMT21 News Translation Task Submission](https://md.research.elamur.ai/free/en/papers/tran-2021-facebook-ai-s-wmt21-news/) - [Paraphrasing as Zero-shot Translation with Feature-guided Diversity Enhancement](https://md.research.elamur.ai/free/en/papers/yan-2026-paraphrasing-as-zero-shot-translation/) - [Exploiting Source-side Monolingual Data in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2016-exploiting-source-side-monolingual-data/) - [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://md.research.elamur.ai/free/en/papers/zhang-2020-improving-massively-multilingual-neural-machine/) - [Legal Judgment Prediction via Topological Learning](https://md.research.elamur.ai/free/en/papers/zhong-2018-legal-judgment-prediction-topological-learning/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{firat2016multi, title = {{Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism}}, author = {Firat, Orhan and Cho, Kyunghyun and Bengio, Yoshua}, booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, year = {2016}, doi = {10.18653/v1/n16-1101}, eprint = {1601.01073}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/n16-1101} } ``` - [← previous Multi-view and multi-task training of RST discourse parsers](https://md.research.elamur.ai/free/en/papers/braud-2016-multi-view-multi-task-training/) - [next → Multi30K: Multilingual English-German Image Descriptions](https://md.research.elamur.ai/free/en/papers/elliott-2016-multi30k-multilingual-english-german-image/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) ## References 14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In ICLR 2015. Léon Bottou, Yoshua Bengio, and Yann Le Cun. 1997. Global training of document processing systems using graph transformer networks. In Computer Vision and P… [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+ICLR+2015%2E+Le%CC%81on+Bottou%2C+Yoshua+Bengio%2C+and+Yann+Le+Cun%2E+1997%2E+Globa) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) 2. IEEE Computer Society Conference on, pages 489– 494. IEEE. Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41–75. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: Encoder–Decoder approaches. In Eighth W… [find](https://scholar.google.com/scholar?q=IEEE+Computer+Society+Conference+on%2C+pages+489%E2%80%93+494%2E+IEEE%2E+Rich+Caruana%2E+1997%2E+Multitask+learning%2E+Machine+learning%2C+28%281%29%3A41%E2%80%9375%2E+Kyunghyun+Cho%2C+Bart+van+Merrie%CC%88nboer%2C+Dzmitry+Bahdanau%2C+and+Yoshua+Ben) 3. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems, pages 577–585. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Ku… [find](https://scholar.google.com/scholar?q=Jan+K+Chorowski%2C+Dzmitry+Bahdanau%2C+Dmitriy+Serdyuk%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Attention%2Dbased+models+for+speech+recognition%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+5) [our card](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/) 4. Weiguang Ding, Ruoyan Wang, Fei Mao, and Graham W. Taylor. 2014. Theano-based large-scale visual recognition with multiple gpus. arXiv:1412.2302. — [arXiv:1412.2302](https://arxiv.org/abs/1412.2302) [find](https://scholar.google.com/scholar?q=Weiguang+Ding%2C+Ruoyan+Wang%2C+Fei+Mao%2C+and+Graham+W%2E+Taylor%2E+2014%2E+Theano%2Dbased+large%2Dscale+visual+recognition+with+multiple+gpus%2E+arXiv%3A1412%2E2302%2E+%E2%80%94+%5BarXiv%3A1412%2E2302%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1412%2E2302%29) 5. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. ACL. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neu… [find](https://scholar.google.com/scholar?q=Daxiang+Dong%2C+Hua+Wu%2C+Wei+He%2C+Dianhai+Yu%2C+and+Haifeng+Wang%2E+2015%2E+Multi%2Dtask+learning+for+multiple+language+translation%2E+ACL%2E+Alex+Graves%2C+Santiago+Ferna%CC%81ndez%2C+Faustino+Gomez%2C+and+Ju%CC%88rgen+Schmidhuber%2E) [our card](https://md.research.elamur.ai/free/en/papers/dong-2015-multi-task-learning-multiple-language/) 6. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term me… [find](https://scholar.google.com/scholar?q=Caglar+Gulcehre%2C+Orhan+Firat%2C+Kelvin+Xu%2C+Kyunghyun+Cho%2C+Loic+Barrault%2C+Huei%2DChi+Lin%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2015%2E+On+using+monolingual+corpora+in+neural+machine+translation) [our card](https://md.research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/) 7. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. Montreal neural machine translation systems for wmt’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134–140, Lisbon, Portugal, September. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Se%CC%81bastien+Jean%2C+Orhan+Firat%2C+Kyunghyun+Cho%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+2015%2E+Montreal+neural+machine+translation+systems+for+wmt%E2%80%9915%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Mach) 8. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. The International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+The+International+Conference+on+Learning+Representations+%28ICLR%29%2E) 9. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015a. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Quoc+V+Le%2C+Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Lukasz+Kaiser%2E+2015a%2E+Multi%2Dtask+sequence+to+sequence+learning%2E+arXiv+preprint+arXiv%3A1511%2E06114%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/) 10. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015b. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D+Manning%2E+2015b%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1508%2E04025%2E) 11. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+2012%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1211%2E5063%2E) 12. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015a%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+arXiv+preprint+arXiv%3A1511%2E06709%2E) 13. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015b%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+arXiv+preprint+arXiv%3A1508%2E07909%2E) 14. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+VV+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3104%E2%80%933112%2E)