--- title: "Is Neural Machine Translation Ready — Junczys-Dowmunt 2016" description: "The paper asks whether neural machine translation can replace phrase-based statistical systems in a large, purely in-domain production setting such as…" authors: ["Marcin Junczys-Dowmunt", "Tomasz Dwojak", "Hieu Hoang"] year: 2016 venue: "Proceedings of the 13th International Conference on Spoken Language Translation" source: "https://aclanthology.org/2016.iwslt-1.5/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/dowmunt-2016-is-neural-machine-translation-ready/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/dowmunt-2016-is-neural-machine-translation-ready/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2016](https://md.research.elamur.ai/free/papers/2016/page/2/) / Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions # Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions **Authors:** [Marcin Junczys-Dowmunt](https://md.research.elamur.ai/free/authors/marcin-junczys-dowmunt/), [Tomasz Dwojak](https://md.research.elamur.ai/free/authors/tomasz-dwojak/), [Hieu Hoang](https://md.research.elamur.ai/free/authors/hieu-hoang/) · **Year:** 2016 · **Venue:** Proceedings of the 13th International Conference on Spoken Language Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2016.iwslt-1.5/) · [PDF](https://aclanthology.org/2016.iwslt-1.5.pdf) **Citations:** [67](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper asks whether neural machine translation can replace phrase-based statistical systems in a large, purely in-domain production setting such as the United Nations. The authors use the fully aligned six-language subcorpus of the United Nations Parallel Corpus v1.0, which covers Arabic, Chinese, English, French, Russian and Spanish. They train attentional encoder-decoder models with Nematus on BPE subwords and compare them, across all 30 translation directions, against a strong Moses configuration modelled on the UN's in-house systems. For the ten English-centred directions they also add a Hiero baseline. NMT equals or outperforms phrase-based SMT in every direction except French→Spanish. The largest gains are for pairs involving Chinese, and Arabic pairs also improve clearly. The second part presents AmuNMT, a C++ beam-search decoder for Nematus models, and examines speed trade-offs: checkpoint ensembling versus averaging, beam size and output-vocabulary selection. The authors find that one consumer GPU already delivers throughput competitive with the deployed Moses setup. This makes NMT a practical option for organizations with large in-house parallel data. ## Key points - Data: the 6-way UN subcorpus has 86,307 documents, about 11.4M lines and about 335M English tokens. Official dev and test sets of 4,000 sentences each come from 2015 documents. Text was lowercased and tokenized with Moses, Chinese was segmented with Jieba, and BPE used 30,000 units. - For directions involving Chinese, NMT beats PB-SMT by roughly 7–9 BLEU. Hiero improves over PB-SMT for zh–en and en–zh but recovers less than half of the NMT gain. - Doubling NMT training from 1.2M to 2.4M iterations (about 8 to 16 days on a GTX 1080) added 0.3–1.3 BLEU. No further benefit appeared beyond about 2M iterations. - On en–fr, averaging the last 8 checkpoints (51.35 BLEU, 860 words/s) nearly matches a 4-checkpoint ensemble (51.38 BLEU, 200 words/s). A beam of 5 scores only 0.2 BLEU below the best beam size. - AmuNMT's vocabulary selection (K = 75, K′ = 75, about 1,250 items per sentence) leaves BLEU unchanged. A single GPU outperforms 16 threads on server-grade Xeon CPUs. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/) ## Cited by Cited by 67 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 64 not yet. - [Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/) - [A Challenge Set Approach to Evaluating Machine Translation](https://md.research.elamur.ai/free/papers/isabelle-2017-challenge-set-approach-evaluating-machine/) - [Investigating Backtranslation in Neural Machine Translation](https://md.research.elamur.ai/free/papers/poncelas-2018-investigating-backtranslation-neural-machine-translation/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{junczysdowmunt2016is, title = {{Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions}}, author = {Junczys-Dowmunt, Marcin and Dwojak, Tomasz and Hoang, Hieu}, booktitle = {Proceedings of the 13th International Conference on Spoken Language Translation}, year = {2016}, url = {https://aclanthology.org/2016.iwslt-1.5/} } ``` - [← previous Intra-Sentential Subject Zero Anaphora Resolution using Multi-Column Convolutional Neural Network](https://md.research.elamur.ai/free/papers/iida-2016-intra-sentential-subject-zero-anaphora/) - [next → Is the adjective distinct from the noun as a grammatical category in biblical Hebrew?](https://md.research.elamur.ai/free/papers/hbo-2016-miller-naude-adjective-distinct-noun-grammatical-category/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/papers/2016/page/2/) ## References 20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. M. Ziemski, M. Junczys-Dowmunt, and B. Pouliquen, “The United Nations Parallel Corpus v1.0,” in LREC 2016. ELRA, 2016. [find](https://scholar.google.com/scholar?q=M%2E+Ziemski%2C+M%2E+Junczys%2DDowmunt%2C+and+B%2E+Pouliquen%2C+%E2%80%9CThe+United+Nations+Parallel+Corpus+v1%2E0%2C%E2%80%9D+in+LREC+2016%2E+ELRA%2C+2016%2E) [our card](https://md.research.elamur.ai/free/papers/ziemski-2016-united-nations-parallel-corpus-v1/) 2. T. Luong, I. Sutskever, Q. V. Le, O. Vinyals, and W. Zaremba, “Addressing the rare word problem in neural machine translation,” in ACL, 2015. [find](https://scholar.google.com/scholar?q=T%2E+Luong%2C+I%2E+Sutskever%2C+Q%2E+V%2E+Le%2C+O%2E+Vinyals%2C+and+W%2E+Zaremba%2C+%E2%80%9CAddressing+the+rare+word+problem+in+neural+machine+translation%2C%E2%80%9D+in+ACL%2C+2015%2E) [our card](https://md.research.elamur.ai/free/papers/luong-2015-addressing-rare-word-problem-neural/) 3. R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” CoRR, vol. abs/1508.07909, 2015. [find](https://scholar.google.com/scholar?q=R%2E+Sennrich%2C+B%2E+Haddow%2C+and+A%2E+Birch%2C+%E2%80%9CNeural+machine+translation+of+rare+words+with+subword+units%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1508%2E07909%2C+2015%2E) 4. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst, “Moses: Open source toolkit for statistical machine translation,” in ACL, 2007, pp. 177–180. [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2C+H%2E+Hoang%2C+A%2E+Birch%2C+C%2E+Callison%2DBurch%2C+M%2E+Federico%2C+N%2E+Bertoldi%2C+B%2E+Cowan%2C+W%2E+Shen%2C+C%2E+Moran%2C+R%2E+Zens%2C+C%2E+Dyer%2C+O%2E+Bojar%2C+A%2E+Constantin%2C+and+E%2E+Herbst%2C+%E2%80%9CMoses%3A+Open+source+toolkit+for+statis) 5. Q. Gao and S. Vogel, “Parallel implementations of word alignment tool,” in Software Engineering, Testing, and Quality Assurance for Natural Language Processing, 2008, pp. 49–57. [find](https://scholar.google.com/scholar?q=Q%2E+Gao+and+S%2E+Vogel%2C+%E2%80%9CParallel+implementations+of+word+alignment+tool%2C%E2%80%9D+in+Software+Engineering%2C+Testing%2C+and+Quality+Assurance+for+Natural+Language+Processing%2C+2008%2C+pp%2E+49%E2%80%9357%2E) 6. N. Durrani, A. Fraser, H. Schmid, H. Hoang, and P. Koehn, “Can Markov models over minimal translation units help phrase-based SMT?” in ACL, 2013, pp. 399–405. [find](https://scholar.google.com/scholar?q=N%2E+Durrani%2C+A%2E+Fraser%2C+H%2E+Schmid%2C+H%2E+Hoang%2C+and+P%2E+Koehn%2C+%E2%80%9CCan+Markov+models+over+minimal+translation+units+help+phrase%2Dbased+SMT%3F%E2%80%9D+in+ACL%2C+2013%2C+pp%2E+399%E2%80%93405%2E) 7. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” CoRR, vol. abs/1301.3781, 2013. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2C+K%2E+Chen%2C+G%2E+Corrado%2C+and+J%2E+Dean%2C+%E2%80%9CEfficient+estimation+of+word+representations+in+vector+space%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1301%2E3781%2C+2013%2E) 8. K. Heafield, I. Pouzyrevsky, J. H. Clark, and P. Koehn, “Scalable modified Kneser-Ney language model estimation,” in ACL, 2013, pp. 690– 696. [find](https://scholar.google.com/scholar?q=K%2E+Heafield%2C+I%2E+Pouzyrevsky%2C+J%2E+H%2E+Clark%2C+and+P%2E+Koehn%2C+%E2%80%9CScalable+modified+Kneser%2DNey+language+model+estimation%2C%E2%80%9D+in+ACL%2C+2013%2C+pp%2E+690%E2%80%93+696%2E) [our card](https://md.research.elamur.ai/free/papers/heafield-2013-scalable-modified-kneser-ney-language/) 9. H. Johnson, J. D. Martin, G. F. Foster, and R. Kuhn, “Improving translation quality by discarding most of the phrasetable,” in EMNLP-CoNLL, 2007, pp. 967–975. [find](https://scholar.google.com/scholar?q=H%2E+Johnson%2C+J%2E+D%2E+Martin%2C+G%2E+F%2E+Foster%2C+and+R%2E+Kuhn%2C+%E2%80%9CImproving+translation+quality+by+discarding+most+of+the+phrasetable%2C%E2%80%9D+in+EMNLP%2DCoNLL%2C+2007%2C+pp%2E+967%E2%80%93975%2E) [our card](https://md.research.elamur.ai/free/papers/johnson-2007-improving-translation-quality-discarding-most/) 10. M. Junczys-Dowmunt, “Phrasal Rank-Encoding: Exploiting phrase redundancy and translational relations for phrase table compression,” Prague Bull. Math. Linguistics, vol. 98, pp. 63–74, 2012. [find](https://scholar.google.com/scholar?q=M%2E+Junczys%2DDowmunt%2C+%E2%80%9CPhrasal+Rank%2DEncoding%3A+Exploiting+phrase+redundancy+and+translational+relations+for+phrase+table+compression%2C%E2%80%9D+Prague+Bull%2E+Math%2E+Linguistics%2C+vol%2E+98%2C+pp%2E+63%E2%80%9374%2C+2012%2E) 11. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” CoRR, vol. abs/1409.0473, 2014. [find](https://scholar.google.com/scholar?q=D%2E+Bahdanau%2C+K%2E+Cho%2C+and+Y%2E+Bengio%2C+%E2%80%9CNeural+machine+translation+by+jointly+learning+to+align+and+translate%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1409%2E0473%2C+2014%2E) [our card](https://md.research.elamur.ai/free/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) 12. R. Sennrich, B. Haddow, and A. Birch, “Edinburgh neural machine translation systems for WMT 16,” in WMT, 2016, pp. 371–376. [find](https://scholar.google.com/scholar?q=R%2E+Sennrich%2C+B%2E+Haddow%2C+and+A%2E+Birch%2C+%E2%80%9CEdinburgh+neural+machine+translation+systems+for+WMT+16%2C%E2%80%9D+in+WMT%2C+2016%2C+pp%2E+371%E2%80%93376%2E) [our card](https://md.research.elamur.ai/free/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/) 13. R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training recurrent neural networks,” in ICML, 2013, pp. 1310–1318. [find](https://scholar.google.com/scholar?q=R%2E+Pascanu%2C+T%2E+Mikolov%2C+and+Y%2E+Bengio%2C+%E2%80%9COn+the+difficulty+of+training+recurrent+neural+networks%2C%E2%80%9D+in+ICML%2C+2013%2C+pp%2E+1310%E2%80%931318%2E) 14. M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,” CoRR, vol. abs/1212.5701, 2012. [find](https://scholar.google.com/scholar?q=M%2E+D%2E+Zeiler%2C+%E2%80%9CADADELTA%3A+An+Adaptive+Learning+Rate+Method%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1212%2E5701%2C+2012%2E) [our card](https://md.research.elamur.ai/free/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/) 15. S. Jean, K. Cho, R. Memisevic, and Y. Bengio, “On using very large target vocabulary for neural machine translation,” in ACL, 2015, pp. 1–10. [find](https://scholar.google.com/scholar?q=S%2E+Jean%2C+K%2E+Cho%2C+R%2E+Memisevic%2C+and+Y%2E+Bengio%2C+%E2%80%9COn+using+very+large+target+vocabulary+for+neural+machine+translation%2C%E2%80%9D+in+ACL%2C+2015%2C+pp%2E+1%E2%80%9310%2E) [our card](https://md.research.elamur.ai/free/papers/jean-2015-very-large-target-vocabulary-neural/) 16. H. Mi, Z. Wang, and A. Ittycheriah, “Vocabulary manipulation for neural machine translation,” in ACL, 2016. [find](https://scholar.google.com/scholar?q=H%2E+Mi%2C+Z%2E+Wang%2C+and+A%2E+Ittycheriah%2C+%E2%80%9CVocabulary+manipulation+for+neural+machine+translation%2C%E2%80%9D+in+ACL%2C+2016%2E) 17. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corra… [find](https://scholar.google.com/scholar?q=Y%2E+Wu%2C+M%2E+Schuster%2C+Z%2E+Chen%2C+Q%2E+V%2E+Le%2C+M%2E+Norouzi%2C+W%2E+Macherey%2C+M%2E+Krikun%2C+Y%2E+Cao%2C+Q%2E+Gao%2C+K%2E+Macherey%2C+J%2E+Klingner%2C+A%2E+Shah%2C+M%2E+Johnson%2C+X%2E+Liu%2C+L%2E+Kaiser%2C+S%2E+Gouws%2C+Y%2E+Kato%2C+T%2E+Kudo%2C+H%2E+Kazawa%2C+K%2E+S) 18. M. Junczys-Dowmunt, T. Dwojak, and R. Sennrich, “The AMU-UEDIN submission to the WMT16 news translation task: Attention-based NMT models as feature functions in phrase-based SMT,” in WMT, 2016, pp. 319–325. [find](https://scholar.google.com/scholar?q=M%2E+Junczys%2DDowmunt%2C+T%2E+Dwojak%2C+and+R%2E+Sennrich%2C+%E2%80%9CThe+AMU%2DUEDIN+submission+to+the+WMT16+news+translation+task%3A+Attention%2Dbased+NMT+models+as+feature+functions+in+phrase%2Dbased+SMT%2C%E2%80%9D+in+WMT%2C+2016%2C+pp%2E+31) 19. B. Pouliquen, C. Elizalde, M. Junczys-Dowmunt, C. Mazenc, and J. Garcı́a-Verdugo, “Large-scale multiple language translation accelerator at the United Nations,” in MT-Summit XIV, 2013, pp. 345–352. [find](https://scholar.google.com/scholar?q=B%2E+Pouliquen%2C+C%2E+Elizalde%2C+M%2E+Junczys%2DDowmunt%2C+C%2E+Mazenc%2C+and+J%2E+Garc%C4%B1%CC%81a%2DVerdugo%2C+%E2%80%9CLarge%2Dscale+multiple+language+translation+accelerator+at+the+United+Nations%2C%E2%80%9D+in+MT%2DSummit+XIV%2C+2013%2C+pp%2E+345%E2%80%93352%2E) 20. R. Knowles and P. Koehn, “Neural interactive translation prediction,” in AMTA, 2016. [find](https://scholar.google.com/scholar?q=R%2E+Knowles+and+P%2E+Koehn%2C+%E2%80%9CNeural+interactive+translation+prediction%2C%E2%80%9D+in+AMTA%2C+2016%2E)