---
title: "Improved Zero-shot Neural Machine Translation via Ignoring"
description: "Multilingual NMT systems can translate between language pairs never seen together in training… 25 references, cited by 72 papers here."
authors: ["Jiatao Gu", "Yong Wang", "Kyunghyun Cho", "Victor O.K. Li"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1121/"
doi: "10.18653/v1/p19-1121"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gu-2019-improved-zero-shot-neural-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gu-2019-improved-zero-shot-neural-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations

# Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations

**Authors:** [Jiatao Gu](https://md.research.elamur.ai/free/en/authors/jiatao-gu/), [Yong Wang](https://md.research.elamur.ai/free/en/authors/yong-wang/), [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/) and [Victor O.K. Li](https://md.research.elamur.ai/free/en/authors/victor-o-k-li/) · **Year:** 2019 · **Venue:** [Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2019](https://md.research.elamur.ai/free/en/venues/acl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1121/) · [PDF](https://aclanthology.org/P19-1121.pdf) · [DOI 10.18653/v1/p19-1121](https://doi.org/10.18653/v1/p19-1121) · [OpenAlex](https://openalex.org/W2952153923)

**Citations:** 90 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/gu-2019-improved-zero-shot-neural-machine/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Multilingual NMT systems can translate between language pairs never seen together in training (zero-shot translation), but this ability is fragile. It depends heavily on hyper-parameters and usually falls well short of pivoting through a third language such as English. The authors trace the failure to a spurious dependency: with English-centric training data, maximum-likelihood training lets the decoder link the source language to the output language. The decoder then disregards the requested target-language ID and often produces English. They quantify this with an estimate of how much information the source-language ID and the language of the decoded output share, i.e. their mutual information. To weaken that dependency, they adapt two known techniques, decoder (language model) pre-training and back-translation, to the zero-shot setting. They test Transformer models on Europarl, IWSLT and MultiUN. Both techniques lift vanilla zero-shot quality substantially and stabilise training, and they match or beat pivot-based translation. This makes direct zero-shot translation a practical option for language pairs without parallel data.

## Key points
- Vanilla zero-shot NMT is highly sensitive to training settings. In preliminary Europarl experiments (En, Fr, Es, De), a layer-wise attention variant failed completely on zero-shot pairs, while parallel directions were barely affected.
- The degeneracy is measured through the estimated mutual information I(i; z) between source language ID and output language. As this value grows, the model drifts toward outputting the wrong language, typically English.
- Decoder (LM) pre-training and back-translation (BTZS) improve vanilla zero-shot results by roughly 4–22 BLEU. On some pairs they beat pivoting by about 2–3 BLEU.
- Evaluation covers 4 languages from Europarl, 5 from IWSLT and 4 from MultiUN. The base model is a Transformer (d_model 512, 6 layers, 8 heads) with 40,000 shared BPE operations.
- In an IWSLT setup with chained pairs (De-En, En-It, It-Ro, Ro-Nl), pivot translation must pass through several languages and degrades sharply. Zero-shot translation avoids this chain and gains clearly over pivoting.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 136 works · 86,390 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 44 works · 10,936 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Jiatao Gu, Yong Wang, Kyunghyun Cho and Victor O.K. Li. 2019. Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gu2019improved,
  title = {{Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations}},
  author = {Jiatao Gu, Yong Wang, Kyunghyun Cho and Victor O.K. Li},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1121},
  url = {https://doi.org/10.18653/v1/p19-1121}
}
```

- [← previous EASSE: Easier Automatic Sentence Simplification Evaluation](https://md.research.elamur.ai/free/en/papers/manchego-2019-easse-easier-automatic-sentence-simplification/)
- [next → On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models](https://md.research.elamur.ai/free/en/papers/michel-2019-evaluation-adversarial-perturbations-sequence-sequence/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2018. The missing ingredient in zero-shot neural machine translation. [find](https://scholar.google.com/scholar?q=Naveen+Arivazhagan%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Roee+Aharoni%2C+Melvin+Johnson%2C+and+Wolfgang+Macherey%2E+2018%2E+The+missing+ingredient+in+zero%2Dshot+neural+machine+translation%2E)
2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. [Unsupervised neural machine translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/). In Proceedings of International Conference on Learning Representations (ICLR), Vancouver, Canada.
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473.
4. Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. 2017. A teacher-student framework for zero-resource neural machine translation. arXiv preprint arXiv:1705.00753. [find](https://scholar.google.com/scholar?q=Yun+Chen%2C+Yang+Liu%2C+Yong+Cheng%2C+and+Victor+OK+Li%2E+2017%2E+A+teacher%2Dstudent+framework+for+zero%2Dresource+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1705%2E00753%2E)
5. Yun Chen, Yang Liu, and Victor OK Li. 2018. Zero-resource neural machine translation with multi-agent communication game. arXiv preprint arXiv:1802.03116. — [arXiv:1802.03116](https://arxiv.org/abs/1802.03116) [find](https://scholar.google.com/scholar?q=Yun+Chen%2C+Yang+Liu%2C+and+Victor+OK+Li%2E+2018%2E+Zero%2Dresource+neural+machine+translation+with+multi%2Dagent+communication+game%2E+arXiv+preprint+arXiv%3A1802%2E03116%2E+%E2%80%94+%5BarXiv%3A1802%2E03116%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F18)
6. Yong Cheng, Yang Liu, Qian Yang, Maosong Sun, and Wei Xu. 2016. Neural machine translation with pivot languages. arXiv preprint arXiv:1611.04928. — [arXiv:1611.04928](https://arxiv.org/abs/1611.04928) [find](https://scholar.google.com/scholar?q=Yong+Cheng%2C+Yang+Liu%2C+Qian+Yang%2C+Maosong+Sun%2C+and+Wei+Xu%2E+2016%2E+Neural+machine+translation+with+pivot+languages%2E+arXiv+preprint+arXiv%3A1611%2E04928%2E+%E2%80%94+%5BarXiv%3A1611%2E04928%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1611%2E04928%29)
7. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. [On the properties of neural machine translation: Encoder–Decoder approaches](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/). In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation.
8. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. [Multi-way, multilingual neural machine translation with a shared attention mechanism](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/). In NAACL.
9. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T Yarman Vural, and Kyunghyun Cho. 2016b. [Zero-resource translation with multi-lingual neural machine translation](https://md.research.elamur.ai/free/en/papers/firat-2016-zero-resource-translation-multi-lingual/). In EMNLP.
10. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017. [Convolutional sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). In Proceedings of International Conference on Machine Learning (ICML).
11. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. [find](https://scholar.google.com/scholar?q=Xavier+Glorot+and+Yoshua+Bengio%2E+2010%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+Proceedings+of+the+thirteenth+international+conference+on+artificial+intelligence+a)
12. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018a. [Non-autoregressive neural machine translation](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/). ICLR.
13. Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li. 2018b. [Universal neural machine translation for extremely low resource languages](https://md.research.elamur.ai/free/en/papers/gu-2018-universal-neural-machine-translation-extremely/). arXiv preprint arXiv:1802.05368.
14. Jiatao Gu, Yong Wang, Yun Chen, Kyunghyun Cho, and Victor OK Li. 2018c. [Meta-learning for low-resource neural machine translation](https://md.research.elamur.ai/free/en/papers/gu-2018-meta-learning-low-resource-neural/). arXiv preprint arXiv:1808.08437.
15. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the As… [find](https://scholar.google.com/scholar?q=Melvin+Johnson%2C+Mike+Schuster%2C+Quoc+V%2E+Le%2C+Maxim+Krikun%2C+Yonghui+Wu%2C+Zhifeng+Chen%2C+Nikhil+Thorat%2C+Fernanda+Vie%CC%81gas%2C+Martin+Wattenberg%2C+Greg+Corrado%2C+Macduff+Hughes%2C+and+Jeffrey+Dean%2E+2017%2E+Google%E2%80%99s+mu)
16. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 48–54. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Franz+Josef+Och%2C+and+Daniel+Marcu%2E+2003%2E+Statistical+phrase%2Dbased+translation%2E+In+Proceedings+of+the+2003+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+L)
17. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+arXiv+preprint+arXiv%3A1901%2E07291%2E)
18. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a. [Unsupervised machine translation using monolingual corpora only](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/). In Proceedings of International Conference on Learning Representations (ICLR), Vancouver, Canada.
19. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039–5049. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Myle+Ott%2C+Alexis+Conneau%2C+Ludovic+Denoyer%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2018b%2E+Phrase%2Dbased+%26+neural+unsupervised+machine+translation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical)
20. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2016. [Fully character-level neural machine translation without explicit segmentation](https://md.research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/). arXiv preprint arXiv:1610.03017.
21. Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. 2017. Emergent translation in multi-agent communication. arXiv preprint arXiv:1710.06922. — [arXiv:1710.06922](https://arxiv.org/abs/1710.06922) [find](https://scholar.google.com/scholar?q=Jason+Lee%2C+Kyunghyun+Cho%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2017%2E+Emergent+translation+in+multi%2Dagent+communication%2E+arXiv+preprint+arXiv%3A1710%2E06922%2E+%E2%80%94+%5BarXiv%3A1710%2E06922%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1710%2E06922)
22. [Marco Lui and Timothy Baldwin. 2012. langid. py: An off-the-shelf language identification tool. In Proceedings of the ACL 2012 system demonstrations, pages 25–30. Association for Computational Linguistics. machine translation. In Human Language Technology Conference of the North American Chapter of…](https://md.research.elamur.ai/free/en/papers/lui-2012-langid-py-off-shelf-language/)
23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS).
24. Hao Zheng, Yong Cheng, and Yang Liu. 2017. Maximum expected likelihood estimation for zero-resource neural machine translation. IJCAI. [find](https://scholar.google.com/scholar?q=Hao+Zheng%2C+Yong+Cheng%2C+and+Yang+Liu%2E+2017%2E+Maximum+expected+likelihood+estimation+for+zero%2Dresource+neural+machine+translation%2E+IJCAI%2E)
25. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Barret+Zoph%2C+Deniz+Yuret%2C+Jonathan+May%2C+and+Kevin+Knight%2E+2016%2E+Transfer+learning+for+low%2Dresource+neural+machine+translation%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Lan)

*Updated 24 October 2026*