--- title: "Multi-Source Neural Translation — Zoph 2016" description: "The paper tackles translation when the same sentence is already available in two source languages, for example when a document has been translated once…" authors: ["Barret Zoph", "Kevin Knight"] year: 2016 venue: "Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies" source: "https://aclanthology.org/N16-1004/" doi: "10.18653/v1/n16-1004" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zoph-2016-multi-source-neural-translation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zoph-2016-multi-source-neural-translation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Multi-Source Neural Translation # Multi-Source Neural Translation **Authors:** [Barret Zoph](https://md.research.elamur.ai/free/en/authors/barret-zoph/), [Kevin Knight](https://md.research.elamur.ai/free/en/authors/kevin-knight/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/N16-1004/) · [PDF](https://aclanthology.org/N16-1004.pdf) · [DOI 10.18653/v1/n16-1004](https://doi.org/10.18653/v1/n16-1004) · [OpenAlex](https://openalex.org/W2222949842) **Citations:** 300 citations (openalex-snapshot · 2026-10-14) · [132](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper tackles translation when the same sentence is already available in two source languages, for example when a document has been translated once by hand and must then be rendered into further languages. Earlier multi-source MT combined outputs of separately trained bilingual systems. Zoph and Knight instead train one neural model, P(e|f, g), directly on trilingual data. Each source language gets its own four-layer LSTM encoder, and the final encoder states are merged before decoding. They test two merging schemes: a Basic concatenate-and-project combiner and a Child-Sum combiner inspired by Tree-LSTMs. They also extend local attention so the decoder attends to both encoders at once. The data is a trilingual subset of WMT 2014. For French+German→English, the best multi-source attention model gains +4.8 BLEU over a strong single-source attention baseline. Gains are much smaller when the sources are English and French and the target is German. This suggests that more dissimilar source languages resolve each other's ambiguities better. The work shows that "triangulation" can be learned end-to-end inside a single network. ## Key points - Training data comes from WMT 2014 French/English/German: about 2.38M aligned segment triples. Development uses 3000 WMT sentences, and evaluation is done on 1503 lines drawn from the WMT test set that are available in all three languages. BLEU is case-sensitive, single-reference, computed with Moses multi-bleu.perl. - The architecture uses four-layer LSTM encoder-decoders with 1000-dimensional hidden states. The attention baseline follows Luong et al. (2015b) local-p attention with input feeding. - For French+German→English, Basic attention reaches 30.0 BLEU versus 25.2 for French-only attention (+4.8). Child-Sum attention reaches 29.6 (+4.4). Without attention, Basic improves 21.0→23.2 and Child-Sum improves 21.0→22.5. - Feeding two copies of the same French sentence gives no gain (20.7 BLEU). This shows the improvement comes from the second language, not from extra model capacity. - For French+English→German, gains are at most +1.1 BLEU (e.g., 17.6→18.6 with attention). The code is released as Zoph_RNN (github.com/isi-nlp). ## Interesting ideas - **Merge cell states by summing them instead of projecting them.** The authors report that concatenating and linearly transforming cell states made training diverge because of large values, while simple addition worked. This is a practical hint for anyone fusing several LSTM encoders. - **Give each input its own forget gate (Child-Sum style).** A learned gate per encoder lets the model decide how much to trust each source. This could transfer to fusing parallel inputs such as dialect variants, OCR versions, or multimodal signals. - **Use separate attention per source, concatenated before output.** Each encoder keeps its own attention parameters and context vector, and both vectors are joined with the decoder state. This is a simple way to add any number of aligned sources to an attention model. - **Control for extra capacity by duplicating the input.** Running the multi-source model on two copies of the same source separates the true benefit of a second language from the effect of simply having more parameters. This control is worth reusing in any multi-input experiment. - **Choose the second source language for its distance from the first.** The results suggest that a more distant second language adds more disambiguating information. For pipelines with existing human translations (e.g., Ukrainian alongside a typologically different language), the choice of auxiliary language may matter. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 132 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 120 not yet. - [Evaluating Discourse Phenomena in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/bawden-2017-evaluating-discourse-phenomena-neural-machine/) - [Zero-Resource Translation with Multi-Lingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/firat-2016-zero-resource-translation-multi-lingual/) - [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://md.research.elamur.ai/free/en/papers/ha-2016-multilingual-neural-machine-translation-universal/) - [Marian: Fast Neural Machine Translation in C++](https://md.research.elamur.ai/free/en/papers/junczys-2018-dowmunt-marian-fast-neural-machine-translation/) - [Scheduled Multi-Task Learning: From Syntax to Translation](https://md.research.elamur.ai/free/en/papers/kiperwasser-2018-scheduled-multi-task-learning-syntax/) - [A neural interlingua for multilingual machine translation](https://md.research.elamur.ai/free/en/papers/lu-2018-neural-interlingua-multilingual-machine-translation/) - [Paraphrasing Revisited with Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/mallinson-2017-paraphrasing-revisited-neural-machine-translation/) - [Data-Driven Sentence Simplification: Survey and Benchmark](https://md.research.elamur.ai/free/en/papers/manchego-2020-data-driven-sentence-simplification-survey/) - [Learning Joint Multilingual Sentence Representations with Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/schwenk-2017-learning-joint-multilingual-sentence-representations/) - [Leveraging Rule-Based Machine Translation Knowledge for Under-Resourced Neural Machine Translation Models](https://md.research.elamur.ai/free/en/papers/torregrosa-2019-leveraging-rule-based-machine-translation/) - [Exploiting Cross-Sentence Context for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/wang-2017-exploiting-cross-sentence-context-neural/) - [Exploiting Source-side Monolingual Data in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2016-exploiting-source-side-monolingual-data/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{zoph2016multi, title = {{Multi-Source Neural Translation}}, author = {Zoph, Barret and Knight, Kevin}, booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, year = {2016}, doi = {10.18653/v1/n16-1004}, url = {https://doi.org/10.18653/v1/n16-1004} } ``` - [← previous Multi-level Gated Recurrent Neural Network for dialog act classification](https://md.research.elamur.ai/free/en/papers/li-2016-multi-level-gated-recurrent-neural/) - [next → Multi-view and multi-task training of RST discourse parsers](https://md.research.elamur.ai/free/en/papers/braud-2016-multi-view-multi-task-training/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) ## References 21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. D. Bahdanau, K. Cho, and Y. Bengio. 2014. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, C. Monz, M. Post, and L. Specia, editors. 2014. Proc. of the Ninth Workshop on Statistical Machine Translation. Associ… [find](https://scholar.google.com/scholar?q=D%2E+Bahdanau%2C+K%2E+Cho%2C+and+Y%2E+Bengio%2E+2014%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proc%2E+ICLR%2E+O%2E+Bojar%2C+C%2E+Buck%2C+C%2E+Federmann%2C+B%2E+Haddow%2C+P%2E+Koehn%2C+C%2E+Monz%2C+M%2E+Post%2C+) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) 2. C. Callison-Burch. 2002. Co-training for statistical machine translation. Master’s thesis, School of Informatics, University of Edinburgh. M. A. Castaño and F. Casacuberta. 1997. A connectionist approach to machine translation. In EU-ROSPEECH. [find](https://scholar.google.com/scholar?q=C%2E+Callison%2DBurch%2E+2002%2E+Co%2Dtraining+for+statistical+machine+translation%2E+Master%E2%80%99s+thesis%2C+School+of+Informatics%2C+University+of+Edinburgh%2E+M%2E+A%2E+Castan%CC%83o+and+F%2E+Casacuberta%2E+1997%2E+A+connectionist+appr) 3. A. Deri and K. Knight. 2015. How to make a Frenemy: Multitape FSTs for portmanteau generation. In Proc. NAACL. D. Dong, H. Wu, W. he, D. Yu, and H. Wang. 2015. Multi-task learning for multiple language translation. In Proc. [find](https://scholar.google.com/scholar?q=A%2E+Deri+and+K%2E+Knight%2E+2015%2E+How+to+make+a+Frenemy%3A+Multitape+FSTs+for+portmanteau+generation%2E+In+Proc%2E+NAACL%2E+D%2E+Dong%2C+H%2E+Wu%2C+W%2E+he%2C+D%2E+Yu%2C+and+H%2E+Wang%2E+2015%2E+Multi%2Dtask+learning+for+multiple+languag) 4. ACL. C. Elgot and J. Mezei. 1965. On relations defined by generalized finite automata. IBM Journal of Research and Development, 9(1):47–68. [find](https://scholar.google.com/scholar?q=ACL%2E+C%2E+Elgot+and+J%2E+Mezei%2E+1965%2E+On+relations+defined+by+generalized+finite+automata%2E+IBM+Journal+of+Research+and+Development%2C+9%281%29%3A47%E2%80%9368%2E) 5. W. A Gale and K. W Church. 1993. A program for aligning sentences in bilingual corpora. Computational linguistics, 19(1):75–102. J. González-Rubio and F. Casacuberta. 2010. On the use of median string for multi-source translation. In Proc. [find](https://scholar.google.com/scholar?q=W%2E+A+Gale+and+K%2E+W+Church%2E+1993%2E+A+program+for+aligning+sentences+in+bilingual+corpora%2E+Computational+linguistics%2C+19%281%29%3A75%E2%80%93102%2E+J%2E+Gonza%CC%81lez%2DRubio+and+F%2E+Casacuberta%2E+2010%2E+On+the+use+of+median+strin) [our card](https://md.research.elamur.ai/free/en/papers/gale-1993-program-aligning-sentences-bilingual-corpora/) 6. ICPR. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8). [find](https://scholar.google.com/scholar?q=ICPR%2E+S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%2E) 7. R. Kaplan and M. Kay. 1994. Regular models of phonological rule systems. Computational Linguistics, 20(3):331–378. [find](https://scholar.google.com/scholar?q=R%2E+Kaplan+and+M%2E+Kay%2E+1994%2E+Regular+models+of+phonological+rule+systems%2E+Computational+Linguistics%2C+20%283%29%3A331%E2%80%93378%2E) 8. M. Kay. 2000. Triangulation in translation. Keynote at MT 2000 Conference, University of Exeter. [find](https://scholar.google.com/scholar?q=M%2E+Kay%2E+2000%2E+Triangulation+in+translation%2E+Keynote+at+MT+2000+Conference%2C+University+of+Exeter%2E) 9. M. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and L. Kaiser. 2015a. Multi-task sequence to sequence learning. In arXiv. http://arxiv.org/abs/1511.06114. [↗](http://arxiv.org/abs/1511.06114) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/) 10. M. Luong, H. Pham, and C. Manning. 2015b. Effective approaches to attention-based neural machine translation. In Proc. [find](https://scholar.google.com/scholar?q=M%2E+Luong%2C+H%2E+Pham%2C+and+C%2E+Manning%2E+2015b%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+Proc%2E) 11. EMNLP. E. Matusov, N. Ueffing, and H. Ney. 2006. Computing consensus translation from multiple machine translation systems using enhanced hypotheses alignment. In Proc. EACL. [find](https://scholar.google.com/scholar?q=EMNLP%2E+E%2E+Matusov%2C+N%2E+Ueffing%2C+and+H%2E+Ney%2E+2006%2E+Computing+consensus+translation+from+multiple+machine+translation+systems+using+enhanced+hypotheses+alignment%2E+In+Proc%2E+EACL%2E) 12. A. Max, J. Crego, and F. Yvon. 2010. Contrastive lexical evaluation of machine translation. In Proc. [find](https://scholar.google.com/scholar?q=A%2E+Max%2C+J%2E+Crego%2C+and+F%2E+Yvon%2E+2010%2E+Contrastive+lexical+evaluation+of+machine+translation%2E+In+Proc%2E) 13. LREC. R. Neco and M. Forcada. 1997. Asynchronous translations with recurrent neural nets. In International Conf. on Neural Networks, volume 4, pages 2535–2540. [find](https://scholar.google.com/scholar?q=LREC%2E+R%2E+Neco+and+M%2E+Forcada%2E+1997%2E+Asynchronous+translations+with+recurrent+neural+nets%2E+In+International+Conf%2E+on+Neural+Networks%2C+volume+4%2C+pages+2535%E2%80%932540%2E) 14. F. J. Och and H. Ney. 2001. Statistical multi-source translation. In Proc. [find](https://scholar.google.com/scholar?q=F%2E+J%2E+Och+and+H%2E+Ney%2E+2001%2E+Statistical+multi%2Dsource+translation%2E+In+Proc%2E) 15. MT Summit. J. Schroeder, T. Cohn, and P. Koehn. 2009. Word lattices for multi-source translation. In Proc. [find](https://scholar.google.com/scholar?q=MT+Summit%2E+J%2E+Schroeder%2C+T%2E+Cohn%2C+and+P%2E+Koehn%2E+2009%2E+Word+lattices+for+multi%2Dsource+translation%2E+In+Proc%2E) 16. EACL. L. Schwartz. 2008. Multi-source translation methods. [find](https://scholar.google.com/scholar?q=EACL%2E+L%2E+Schwartz%2E+2008%2E+Multi%2Dsource+translation+methods%2E) 17. Proc. AMTA. M. Simard. 1999. Text-translation alignment: Three languages are better than two. In Proc. EMNLP/VLC. [find](https://scholar.google.com/scholar?q=Proc%2E+AMTA%2E+M%2E+Simard%2E+1999%2E+Text%2Dtranslation+alignment%3A+Three+languages+are+better+than+two%2E+In+Proc%2E+EMNLP%2FVLC%2E) 18. I. Sutskever, O. Vinyals, and Q. V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. [find](https://scholar.google.com/scholar?q=I%2E+Sutskever%2C+O%2E+Vinyals%2C+and+Q%2E+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Proc%2E) 19. NIPS. K. S. Tai, R. Socher, and C. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proc. [find](https://scholar.google.com/scholar?q=NIPS%2E+K%2E+S%2E+Tai%2C+R%2E+Socher%2C+and+C%2E+Manning%2E+2015%2E+Improved+semantic+representations+from+tree%2Dstructured+long+short%2Dterm+memory+networks%2E+In+Proc%2E) 20. ACL. P. J. Werbos. 1990. Backpropagation through time: what it does and how to do it. Proc. IEEE, 78(10):1550– 1560. [find](https://scholar.google.com/scholar?q=ACL%2E+P%2E+J%2E+Werbos%2E+1990%2E+Backpropagation+through+time%3A+what+it+does+and+how+to+do+it%2E+Proc%2E+IEEE%2C+78%2810%29%3A1550%E2%80%93+1560%2E) 21. W. Zaremba, I. Sutskever, and O. Vinyals. 2014. Recurrent neural network regularization. CoRR, abs/1409.2329. [find](https://scholar.google.com/scholar?q=W%2E+Zaremba%2C+I%2E+Sutskever%2C+and+O%2E+Vinyals%2E+2014%2E+Recurrent+neural+network+regularization%2E+CoRR%2C+abs%2F1409%2E2329%2E) [our card](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/)