---
title: "Zero-Resource Translation with Multi-Lingual — Firat 2016"
description: "The paper asks whether a multi-way, multilingual NMT model (Firat et al., 2016) can translate between two languages that have no direct parallel data…"
authors: ["Orhan Firat", "Baskaran Sankaran", "Yaser Al-onaizan", "Fatos T. Yarman Vural", "Kyunghyun Cho"]
year: 2016
venue: "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D16-1026/"
doi: "10.18653/v1/d16-1026"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/firat-2016-zero-resource-translation-multi-lingual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/firat-2016-zero-resource-translation-multi-lingual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / Zero-Resource Translation with Multi-Lingual Neural Machine Translation

# Zero-Resource Translation with Multi-Lingual Neural Machine Translation

**Authors:** [Orhan Firat](https://md.research.elamur.ai/free/en/authors/orhan-firat/), Baskaran Sankaran, [Yaser Al-onaizan](https://md.research.elamur.ai/free/en/authors/yaser-al-onaizan/), Fatos T. Yarman Vural, [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D16-1026/) · [PDF](https://aclanthology.org/D16-1026.pdf) · [DOI 10.18653/v1/d16-1026](https://doi.org/10.18653/v1/d16-1026) · [OpenAlex](https://openalex.org/W2964007535)

**Citations:** 261 citations (openalex-snapshot · 2026-10-14) · [89](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks whether a multi-way, multilingual NMT model (Firat et al., 2016) can translate between two languages that have no direct parallel data, such as Spanish→French when only En-Es and En-Fr corpora exist. The authors first add many-to-one decoding strategies to the model. These merge several source-language "views" of a sentence, either by averaging attention context vectors (early), by averaging output distributions (late), or both. They then demonstrate that the unmodified model fails at translating between language pairs it has no direct data for. To fix this, they finetune only a duplicated copy of the shared attention parameters on a pseudo-parallel corpus built with the model's own translations. Experiments on large English, Spanish and French corpora found that this approach matches or beats a single-pair model trained on 1M real Es-Fr sentence pairs. It also outperforms pivot translation through English. This matters because a usable direct system can be built from bilingual data that pass through a hub language, with only a small parameter overhead per new pair.

## Key points
- **Training data:** 34.71m En-Es pairs and 65.77m En-Fr pairs, drawn from UN, Europarl-v7, news-commentary-v7, LDC, internal technical-domain data and, for En-Fr, ReutersUN and Gigaword. A three-way En-Es-Fr subset contains 11.32m sentences.
- **Evaluation and preprocessing:**
  - newstest-2012 is the dev set and newstest-2013 is the test set.
  - Text is tokenized with Moses, and numbers, dates and URLs are replaced by placeholders.
  - BPE vocabularies are 37k (En), 43k (Es) and 45k (Fr) tokens.
  - Training keeps only sentences of at most 50 symbols.
- **Zero-resource Es→Fr results (test scores, Table 5):**
  - Pivot through English: 20.4.
  - Single-pair model trained on 1M true pairs: 20.35.
  - Early-average finetuning with 1M pseudo pairs: 21.46.
  - The same setup with 1M true pairs: 21.7.
- **Effect of pseudo vs. true pairs:** With few examples, pseudo pairs worked better than true ones. Early averaging scored 19.43 with 1k pseudo pairs versus 9.77 with 1k true pairs. Real pairs only paid off at around 1M or more.
- **Early+late averaging:** Combining both strategies helped in multi-source translation but brought no gain over early averaging alone in the zero-resource setting.

## Interesting ideas
- **Finetune only a copy of the attention module.** Freezing encoders and decoders while duplicating and retraining only the attention parameters adapts a model to a new language pair cheaply. This pattern could suit any modular multilingual system where adding a full model per pair is too costly. The authors note one drawback: the number of parameters still grows linearly with the number of target pairs.
- **Use the model's own outputs to realign components.** Pseudo-parallel data generated by the model repaired the mismatch between an encoder and a decoder that were never trained together. It even beat small amounts of real data, which the authors attribute to a better fit with the model's inductive bias and to a possible regularizing effect of noise. This is a useful trick whenever gold data for a direction is tiny.
- **Ensemble-like decoding from bilingual-only training.** Early averaging of context vectors and late averaging of output probabilities make multi-source translation possible without multi-way parallel corpora. Late averaging even works across separately trained models. This makes it easy to combine sources or systems at inference time.
- **Create a pseudo source on the fly.** At test time, the system can first generate a hub-language version of the input (e.g., English from Spanish) and then decode from both views at once. This turns a single-source problem into a multi-source one and could help low-resource pairs connected through a high-resource language like English.
- **Benchmark against real data budgets.** Comparing the zero-resource system with single-pair models trained on 1k–1M true pairs shows how much real data the method is "worth." This evaluation design transfers well to any low-resource method.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

89 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 74 not yet.

- [Massively Multilingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
- [Unsupervised Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/)
- [Rethinking embedding coupling in pre-trained language models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://md.research.elamur.ai/free/en/papers/gala-2023-indictrans2-towards-high-quality-accessible/)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gu-2018-meta-learning-low-resource-neural/)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://md.research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/)
- [A neural interlingua for multilingual machine translation](https://md.research.elamur.ai/free/en/papers/lu-2018-neural-interlingua-multilingual-machine-translation/)
- [Paraphrasing Revisited with Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/mallinson-2017-paraphrasing-revisited-neural-machine-translation/)
- [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://md.research.elamur.ai/free/en/papers/nekoto-2020-participatory-research-low-resourced-machine/)
- [Learning Joint Multilingual Sentence Representations with Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/schwenk-2017-learning-joint-multilingual-sentence-representations/)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/)
- [The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT](https://md.research.elamur.ai/free/en/papers/tiedemann-2020-tatoeba-translation-challenge-realistic-data/)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://md.research.elamur.ai/free/en/papers/zhang-2020-improving-massively-multilingual-neural-machine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{firat2016zero,
  title = {{Zero-Resource Translation with Multi-Lingual Neural Machine Translation}},
  author = {Firat, Orhan and Sankaran, Baskaran and Al-onaizan, Yaser and Vural, Fatos T. Yarman and Cho, Kyunghyun},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1026},
  url = {https://doi.org/10.18653/v1/d16-1026}
}
```

- [← previous Word Segmentation for Akkadian Cuneiform](https://md.research.elamur.ai/free/en/papers/lingbulk-2016-homburg-word-segmentation-akkadian-cuneiform/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015. Ozan Caglayan, Walid Aransa, Yaxing Wang, Marc Masana, Mercedes Garcı́a-Martı́nez, Fethi Bougares, Loı̈c Barrault, and Joost van de Weijer. 2016. Does multim… [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+ICLR+2015%2E+Ozan+Caglayan%2C+Walid+Aransa%2C+Yaxing+Wang%2C+Marc+Masana%2C+Me) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Junyoung+Chung%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2016%2E+A+character%2Dlevel+decoder+without+explicit+segmentation+for+neural+machine+translation%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/chung-2016-character-level-decoder-without-explicit/)
3. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. [find](https://scholar.google.com/scholar?q=Daxiang+Dong%2C+Hua+Wu%2C+Wei+He%2C+Dianhai+Yu%2C+and+Haifeng+Wang%2E+2015%2E+Multi%2Dtask+learning+for+multiple+language+translation%2E) [our card](https://md.research.elamur.ai/free/en/papers/dong-2015-multi-task-learning-multiple-language/)
4. ACL. Orhan Firat and Kyunghyun Cho. 2016. DL4MT-Tutorial: Conditional gated recurrent unit with attention mechanism. [find](https://scholar.google.com/scholar?q=ACL%2E+Orhan+Firat+and+Kyunghyun+Cho%2E+2016%2E+DL4MT%2DTutorial%3A+Conditional+gated+recurrent+unit+with+attention+mechanism%2E)
5. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In NAACL. Mikel L Forcada and Ramón P Ñeco. 1997. Recursive hetero-associative memories for translation. In Biological and Artificial Computation: From Neurosc… [find](https://scholar.google.com/scholar?q=Orhan+Firat%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2016%2E+Multi%2Dway%2C+multilingual+neural+machine+translation+with+a+shared+attention+mechanism%2E+In+NAACL%2E+Mikel+L+Forcada+and+Ramo%CC%81n+P+N%CC%83eco%2E+1997%2E+Recursive) [our card](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/)
6. Springer. Nizar Habash and Jun Hu. 2009. Improving arabic-chinese statistical machine translation using english as pivot language. In Proceedings of the Fourth Workshop on Statistical Machine Translation, StatMT ’09, pages 173–181. Association for Computational Linguistics. Sébastien Jean, Orhan Fi… [find](https://scholar.google.com/scholar?q=Springer%2E+Nizar+Habash+and+Jun+Hu%2E+2009%2E+Improving+arabic%2Dchinese+statistical+machine+translation+using+english+as+pivot+language%2E+In+Proceedings+of+the+Fourth+Workshop+on+Statistical+Machine+Translat)
7. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, pages 1700–1709. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner+and+Phil+Blunsom%2E+2013%2E+Recurrent+continuous+translation+models%2E+In+EMNLP%2C+pages+1700%E2%80%931709%2E) [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/)
8. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. The International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+The+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
9. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interacti… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+et+al%2E+2007%2E+Moses%3A+Open+source+toolkit+fo)
10. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. 2015. Character-based neural machine translation. arXiv:1511.04586. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Isabel+Trancoso%2C+Chris+Dyer%2C+and+Alan+W+Black%2E+2015%2E+Character%2Dbased+neural+machine+translation%2E+arXiv%3A1511%2E04586%2E) [our card](https://md.research.elamur.ai/free/en/papers/jussa-2016-character-based-neural-machine-translation/)
11. Minh-Thang Luong and Christopher D Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. arXiv:1604.00788. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong+and+Christopher+D+Manning%2E+2016%2E+Achieving+open+vocabulary+neural+machine+translation+with+hybrid+word%2Dcharacter+models%2E+arXiv%3A1604%2E00788%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2016-achieving-open-vocabulary-neural-machine/)
12. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015a. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Quoc+V+Le%2C+Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Lukasz+Kaiser%2E+2015a%2E+Multi%2Dtask+sequence+to+sequence+learning%2E+arXiv+preprint+arXiv%3A1511%2E06114%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
13. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015b. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model… [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D+Manning%2E+2015b%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1508%2E04025%2E+Tomas+Mikolov%2C+Martin+Karafia%CC%81t%2C+Luk)
14. Language Transfer. Cambridge University Press. Cambridge Books Online. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063. Håkan Ringbom. 2007. Cross-linguistic similarity in foreign language learning, volu… [find](https://scholar.google.com/scholar?q=Language+Transfer%2E+Cambridge+University+Press%2E+Cambridge+Books+Online%2E+Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+2012%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+arXiv+preprint+a)
15. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681. [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kuldip+K+Paliwal%2E+1997%2E+Bidirectional+recurrent+neural+networks%2E+Signal+Processing%2C+IEEE+Transactions+on%2C+45%2811%29%3A2673%E2%80%932681%2E)
16. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015a%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+arXiv+preprint+arXiv%3A1511%2E06709%2E)
17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015b%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+arXiv+preprint+arXiv%3A1508%2E07909%2E)
18. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+VV+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NIPS%2C+pages+3104%E2%80%933112%2E)
19. Masao Utiyama and Hitoshi Isahara. 2007. A comparison of pivot methods for phrase-based statistical machine translation. In HLT-NAACL, pages 484–491. [find](https://scholar.google.com/scholar?q=Masao+Utiyama+and+Hitoshi+Isahara%2E+2007%2E+A+comparison+of+pivot+methods+for+phrase%2Dbased+statistical+machine+translation%2E+In+HLT%2DNAACL%2C+pages+484%E2%80%93491%2E) [our card](https://md.research.elamur.ai/free/en/papers/utiyama-2007-comparison-pivot-methods-phrase-based/)
20. Hua Wu and Haifeng Wang. 2007. Pivot language approach for phrase-based statistical machine translation. Machine Translation, 21(3):165–181. [find](https://scholar.google.com/scholar?q=Hua+Wu+and+Haifeng+Wang%2E+2007%2E+Pivot+language+approach+for+phrase%2Dbased+statistical+machine+translation%2E+Machine+Translation%2C+21%283%29%3A165%E2%80%93181%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-2007-pivot-language-approach-phrase-based/)
21. Bing Zhao and Shengyuan Chen. 2009. A simplex armijo downhill algorithm for optimizing statistical machine translation decoding parameters. In HLT-NAACL, pages 21–24. [find](https://scholar.google.com/scholar?q=Bing+Zhao+and+Shengyuan+Chen%2E+2009%2E+A+simplex+armijo+downhill+algorithm+for+optimizing+statistical+machine+translation+decoding+parameters%2E+In+HLT%2DNAACL%2C+pages+21%E2%80%9324%2E)
22. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In NAACL. [find](https://scholar.google.com/scholar?q=Barret+Zoph+and+Kevin+Knight%2E+2016%2E+Multi%2Dsource+neural+translation%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/zoph-2016-multi-source-neural-translation/)