---
title: "A neural interlingua for multilingual machine — Lu 2018"
description: "Multilingual NMT usually takes one of two routes. One trains a separate model for every language pair, so the number of models grows quadratically. The…"
authors: ["Yichao Lu", "Phillip Keung", "Faisal Ladhak", "Vikas Bhardwaj", "Shaonan Zhang", "Jason Sun"]
year: 2018
venue: "Proceedings of the Third Conference on Machine Translation: Research Papers"
source: "https://aclanthology.org/W18-6309/"
doi: "10.18653/v1/w18-6309"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lu-2018-neural-interlingua-multilingual-machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lu-2018-neural-interlingua-multilingual-machine-translation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / A neural interlingua for multilingual machine translation

# A neural interlingua for multilingual machine translation

**Authors:** [Yichao Lu](https://md.research.elamur.ai/free/en/authors/yichao-lu/), [Phillip Keung](https://md.research.elamur.ai/free/en/authors/phillip-keung/), [Faisal Ladhak](https://md.research.elamur.ai/free/en/authors/faisal-ladhak/), Vikas Bhardwaj, Shaonan Zhang, Jason Sun · **Year:** 2018 · **Venue:** Proceedings of the Third Conference on Machine Translation: Research Papers · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-6309/) · [PDF](https://aclanthology.org/W18-6309.pdf) · [DOI 10.18653/v1/w18-6309](https://doi.org/10.18653/v1/w18-6309) · [OpenAlex](https://openalex.org/W2962778428)

**Citations:** 128 citations (openalex-snapshot · 2026-10-14) · [50](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Multilingual NMT usually takes one of two routes. One trains a separate model for every language pair, so the number of models grows quadratically. The other shares a single encoder-decoder and a joint vocabulary, which forces many languages into one fixed-size output layer. Lu et al. (Amazon) propose a middle path. Each language keeps its own BiLSTM encoder and attentional LSTM decoder. Between them sits a shared attentional LSTM layer, the neural interlingua, which converts any encoder's output into a fixed-length representation that all decoders consume. No language indicator tokens are used.

The model was trained on WMT15 data pairing English with French, German, Czech, Finnish and Russian, using a rotating schedule of language pairs. Despite having fewer parameters, it reaches BLEU comparable to separate bilingual baselines. It also performs direct zero-shot translation on the UN Parallel Corpus, outperforming a universal encoder-decoder baseline. Its sentence embeddings let a Yelp classifier trained only on English also handle French and German reviews. The work matters because components scale linearly with the number of languages, and a language-neutral representation supports crosslingual transfer.

## Key points
- **Architecture:** language-specific BiLSTM encoders and attentional LSTM decoders are connected through a shared attentional LSTM interlingua with a fixed output length of 50. Encoders, decoders and attention mechanisms all scale linearly with the number of languages.
- **Hyperparameters:** vocabulary of 30,000 per language, embeddings of 256, hidden size 512, Adam with learning rate 0.0002, batch size 400. Importance sampling was used to speed up training.
- **Training schedule:** each minibatch contains a single source-target pair. A naive schedule of only X→En and En→X failed to yield a shared representation. The final schedule (Algorithm 1) adds same-language pairs (S, S) for every language.
- **Zero-shot translation:** direct translation without pivoting was tested on Fr↔Ru, Zh↔Es and Es↔Fr from the updated UN Parallel Corpus. Results improve on the universal encoder-decoder approach (Johnson et al., 2017) but still lag behind pivot translation.
- **Crosslingual transfer:** an English Yelp review classifier built on interlingua embeddings also classifies French and German reviews. In a PCA projection to 2D, parallel sentences in different languages cluster together.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

50 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 42 not yet.

- [Massively Multilingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
- [The Multilingual Amazon Reviews Corpus](https://md.research.elamur.ai/free/en/papers/keung-2020-multilingual-amazon-reviews-corpus/)
- [Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information](https://md.research.elamur.ai/free/en/papers/lin-2020-pre-training-multilingual-neural-machine/)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/pan-2021-contrastive-learning-many-many-multilingual/)
- [Abstract Syntax as Interlingua: Scaling Up the Grammatical Framework from Controlled Languages to Robust Pipelines](https://md.research.elamur.ai/free/en/papers/ranta-2020-abstract-syntax-as-interlingua-scaling/)
- [Multilingual Translation from Denoising Pre-Training](https://md.research.elamur.ai/free/en/papers/tang-2021-multilingual-translation-denoising-pre-training/)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://md.research.elamur.ai/free/en/papers/zhang-2020-improving-massively-multilingual-neural-machine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lu2018neural,
  title = {{A neural interlingua for multilingual machine translation}},
  author = {Lu, Yichao and Keung, Phillip and Ladhak, Faisal and Bhardwaj, Vikas and Zhang, Shaonan and Sun, Jason},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Research Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6309},
  url = {https://doi.org/10.18653/v1/w18-6309}
}
```

- [← previous A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/muller-2018-large-scale-test-set-evaluation/)
- [next → A Neural Layered Model for Nested Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/ju-2018-neural-layered-model-nested-named/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/)

## References

20 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In ICLR. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Quoc+V+Le%2C+Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Lukasz+Kaiser%2E+2016%2E+Multi%2Dtask+sequence+to+sequence+learning%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Akiva Miura, Graham Neubig, Katsuhito Sudoh, and Satoshi Nakamura. 2017. Tree as a pivot: Syntactic matching methods in pivot translation. In WMT. [find](https://scholar.google.com/scholar?q=Akiva+Miura%2C+Graham+Neubig%2C+Katsuhito+Sudoh%2C+and+Satoshi+Nakamura%2E+2017%2E+Tree+as+a+pivot%3A+Syntactic+matching+methods+in+pivot+translation%2E+In+WMT%2E)
4. Christian Boitet. 1988. Pros and cons of the pivot and transfer approaches in multilingual machine translation. In Readings in machine translation, pages 273–279. [find](https://scholar.google.com/scholar?q=Christian+Boitet%2E+1988%2E+Pros+and+cons+of+the+pivot+and+transfer+approaches+in+multilingual+machine+translation%2E+In+Readings+in+machine+translation%2C+pages+273%E2%80%93279%2E)
5. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+ACL%2E)
6. Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. 2017. A teacher-student framework for zero-resource neural machine translation. ACL. [find](https://scholar.google.com/scholar?q=Yun+Chen%2C+Yang+Liu%2C+Yong+Cheng%2C+and+Victor+OK+Li%2E+2017%2E+A+teacher%2Dstudent+framework+for+zero%2Dresource+neural+machine+translation%2E+ACL%2E)
7. Masao Utiyama and Hitoshi Isahara. 2007. A comparison of pivot methods for phrase-based statistical machine translation. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Masao+Utiyama+and+Hitoshi+Isahara%2E+2007%2E+A+comparison+of+pivot+methods+for+phrase%2Dbased+statistical+machine+translation%2E+In+NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/utiyama-2007-comparison-pivot-methods-phrase-based/)
8. Trevor Cohn and Mirella Lapata. 2007. Machine translation by triangulation: Making effective use of multi-parallel corpora. In ACL. [find](https://scholar.google.com/scholar?q=Trevor+Cohn+and+Mirella+Lapata%2E+2007%2E+Machine+translation+by+triangulation%3A+Making+effective+use+of+multi%2Dparallel+corpora%2E+In+ACL%2E)
9. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NIPS%2E)
10. Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. 2017. Copied monolingual data improves low-resource neural machine translation. In WMT. [find](https://scholar.google.com/scholar?q=Anna+Currey%2C+Antonio+Valerio+Miceli+Barone%2C+and+Kenneth+Heafield%2E+2017%2E+Copied+monolingual+data+improves+low%2Dresource+neural+machine+translation%2E+In+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/currey-2017-copied-monolingual-data-improves-low/)
11. Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1. 0. In LREC. [find](https://scholar.google.com/scholar?q=Michal+Ziemski%2C+Marcin+Junczys%2DDowmunt%2C+and+Bruno+Pouliquen%2E+2016%2E+The+united+nations+parallel+corpus+v1%2E+0%2E+In+LREC%2E) [our card](https://md.research.elamur.ai/free/en/papers/ziemski-2016-united-nations-parallel-corpus-v1/)
12. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In ACL. [find](https://scholar.google.com/scholar?q=Daxiang+Dong%2C+Hua+Wu%2C+Wei+He%2C+Dianhai+Yu%2C+and+Haifeng+Wang%2E+2015%2E+Multi%2Dtask+learning+for+multiple+language+translation%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/dong-2015-multi-task-learning-multiple-language/)
13. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Barret+Zoph+and+Kevin+Knight%2E+2016%2E+Multi%2Dsource+neural+translation%2E+In+NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/zoph-2016-multi-source-neural-translation/)
14. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Orhan+Firat%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2016a%2E+Multi%2Dway%2C+multilingual+neural+machine+translation+with+a+shared+attention+mechanism%2E+In+NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/)
15. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Orhan+Firat%2C+Baskaran+Sankaran%2C+Yaser+Al%2DOnaizan%2C+Fatos+T+Yarman+Vural%2C+and+Kyunghyun+Cho%2E+2016b%2E+Zero%2Dresource+translation+with+multi%2Dlingual+neural+machine+translation%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/firat-2016-zero-resource-translation-multi-lingual/)
16. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In ICML. [find](https://scholar.google.com/scholar?q=Jonas+Gehring%2C+Michael+Auli%2C+David+Grangier%2C+Denis+Yarats%2C+and+Yann+N+Dauphin%2E+2017%2E+Convolutional+sequence+to+sequence+learning%2E+In+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/)
17. Thanh-Le Ha, Jan Niehues, and Alexander Waibel. 2016. Toward multilingual neural machine translation with universal encoder and decoder. In IWSLT. [find](https://scholar.google.com/scholar?q=Thanh%2DLe+Ha%2C+Jan+Niehues%2C+and+Alexander+Waibel%2E+2016%2E+Toward+multilingual+neural+machine+translation+with+universal+encoder+and+decoder%2E+In+IWSLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/ha-2016-multilingual-neural-machine-translation-universal/)
18. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Se%CC%81bastien+Jean%2C+Kyunghyun+Cho%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+2015%2E+On+using+very+large+target+vocabulary+for+neural+machine+translation%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/)
19. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: enabling zero-shot translation. In TACL. [find](https://scholar.google.com/scholar?q=Melvin+Johnson%2C+Mike+Schuster%2C+Quoc+V+Le%2C+Maxim+Krikun%2C+Yonghui+Wu%2C+Zhifeng+Chen%2C+Nikhil+Thorat%2C+Fernanda+Vie%CC%81gas%2C+Martin+Wattenberg%2C+Greg+Corrado%2C+et+al%2E+2017%2E+Google%E2%80%99s+multilingual+neural+machine+tr)
20. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR%2E)