--- title: "The NiuTrans Machine Translation Systems for WMT19 — Li 2019" description: "This paper documents the NiuTrans team's (Northeastern University and NiuTrans Co., Ltd.) entries to the WMT19 news translation shared task." authors: ["Bei Li", "Yinqiao Li", "Chen Xu", "Ye Lin", "Jiqiang Liu", "Hui Liu", "Ziyang Wang", "Yuhao Zhang", "Nuo Xu", "Zeyang Wang", "Kai Feng", "Hexuan Chen", "Tengbo Liu", "Yanyang Li", "Qiang Wang", "Tong Xiao", "Jingbo Zhu"] year: 2019 venue: "Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)" source: "https://aclanthology.org/W19-5325/" doi: "10.18653/v1/w19-5325" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/li-2019-niutrans-machine-translation-systems-wmt19/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/li-2019-niutrans-machine-translation-systems-wmt19/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2019](https://md.research.elamur.ai/free/papers/2019/page/4/) / The NiuTrans Machine Translation Systems for WMT19 # The NiuTrans Machine Translation Systems for WMT19 **Authors:** [Bei Li](https://md.research.elamur.ai/free/authors/bei-li/), Yinqiao Li, [Chen Xu](https://md.research.elamur.ai/free/authors/chen-xu/), Ye Lin, Jiqiang Liu, [Hui Liu](https://md.research.elamur.ai/free/authors/hui-liu/), Ziyang Wang, [Yuhao Zhang](https://md.research.elamur.ai/free/authors/yuhao-zhang/), Nuo Xu, Zeyang Wang, Kai Feng, Hexuan Chen et al. (17 authors) · **Year:** 2019 · **Venue:** Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-5325/) · [PDF](https://aclanthology.org/W19-5325.pdf) · [DOI 10.18653/v1/w19-5325](https://doi.org/10.18653/v1/w19-5325) · [OpenAlex](https://openalex.org/W2970731908) **Citations:** 43 citations (openalex-snapshot · 2026-10-14) · [49](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This paper documents the NiuTrans team's (Northeastern University and NiuTrans Co., Ltd.) entries to the WMT19 news translation shared task. They covered 13 directions: 11 supervised (EN↔{ZH, DE, RU, KK, LT} and GU→EN) and the unsupervised DE↔CS sub-track. All supervised pairs used one shared pipeline: - **Architecture:** deep pre-norm Transformer variants, including Transformer-RPR and Transformer-DLCL, which learns a weighted combination of all earlier layers. - **Data:** strict corpus filtering, plus back-translation by beam search, unrestricted sampling or top-K sampling. - **Model improvement:** iterative ensemble knowledge distillation that uses multiple teachers to keep the student models diverse. - **Combination:** a greedy search for the best ensemble, followed by hypothesis combination and feature-based reranking (e.g., right-to-left models). The unsupervised systems were NMT models trained on synthetic data from an unsupervised SMT system, with joint or iterative training on top. Among constrained submissions, the systems placed first on KK↔EN and GU→EN. They were second on RU→EN and DE↔CS, and third on ZH→EN, LT→EN, EN→RU and EN↔DE. The work shows that one general recipe can compete across both high- and low-resource pairs. ## Key points - **Data filtering:** sentences over 100 words, HTML content, duplicates, fast-align scores below -6 and length ratios beyond 3:1 were removed. - **Back-translation choice:** sampling-based back-translation worked best for low-resource pairs (Kazakh, Gujarati, Lithuanian). ZH↔EN gained little from it, apart from a small improvement with top-10 sampling. - **Ensembling and reranking:** the greedy ensemble search gave about 1–1.5 BLEU over the best single model on development sets. Hypothesis combination raised the sentence-level oracle by roughly 5 BLEU. - **GU→EN:** about 0.5M parallel pairs were supplemented with HindEnCorp, converted via Hindi–Gujarati alphabet mapping, and with roughly 3M back-translated sentences. newstest2019 BLEU rose from 3.0 (baseline) to 24.9 (DLCL25RPR with distillation, ensemble and reranking). - **Unsupervised DE↔CS:** the systems used about 24.4M monolingual sentences per language from News Crawl 2007–2018 and 60,000 BPE merges. They reached roughly 18–19 BLEU on newstest2019. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/) ## Cited by Cited by 49 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 47 not yet. - [Findings of the 2019 Conference on Machine Translation (WMT19)](https://md.research.elamur.ai/free/papers/barrault-2019-findings-2019-conference-machine-translation/) - [Survey of Low-Resource Machine Translation](https://md.research.elamur.ai/free/papers/haddow-2022-survey-low-resource-machine-translation/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{li2019niutrans, title = {{The NiuTrans Machine Translation Systems for WMT19}}, author = {Li, Bei and Li, Yinqiao and Xu, Chen and Lin, Ye and Liu, Jiqiang and Liu, Hui and Wang, Ziyang and Zhang, Yuhao and Xu, Nuo and Wang, Zeyang and Feng, Kai and Chen, Hexuan and Liu, Tengbo and Li, Yanyang and Wang, Qiang and Xiao, Tong and Zhu, Jingbo}, booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)}, year = {2019}, doi = {10.18653/v1/w19-5325}, url = {https://doi.org/10.18653/v1/w19-5325} } ``` - [← previous The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution](https://md.research.elamur.ai/free/papers/emami-2019-knowref-coreference-corpus-removing-gender/) - [next → The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue](https://md.research.elamur.ai/free/papers/haber-2019-photobook-dataset-building-common-ground/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/papers/2019/page/4/) ## References 18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine translation. arXiv preprint arXiv:1902.01313. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Gorka+Labaka%2C+and+Eneko+Agirre%2E+2019%2E+An+effective+approach+to+unsupervised+machine+translation%2E+arXiv+preprint+arXiv%3A1902%2E01313%2E) 2. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv: Computation and Language. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+arXiv%3A+Computation+and+Language%2E) 3. Ankur Bapna, Mia Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine translation models with transparent attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3028–3033, Brussels, Belgium. Association for Computation… [find](https://scholar.google.com/scholar?q=Ankur+Bapna%2C+Mia+Chen%2C+Orhan+Firat%2C+Yuan+Cao%2C+and+Yonghui+Wu%2E+2018%2E+Training+deeper+neural+machine+translation+models+with+transparent+attention%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Met) 4. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. arXiv preprint arXiv:1804.07755. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Myle+Ott%2C+Alexis+Conneau%2C+Ludovic+Denoyer%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2018%2E+Phrase%2Dbased+%26+neural+unsupervised+machine+translation%2E+arXiv+preprint+arXiv%3A1804%2E07755%2E) 5. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. [find](https://scholar.google.com/scholar?q=Jimmy+Lei+Ba%2C+Jamie+Ryan+Kiros%2C+and+Geoffrey+E+Hinton%2E+2016%2E+Layer+normalization%2E+arXiv+preprint+arXiv%3A1607%2E06450%2E) [our card](https://md.research.elamur.ai/free/papers/ba-2016-layer-normalization/) 6. Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 427–436. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Colin+Cherry+and+George+Foster%2E+2012%2E+Batch+tuning+strategies+for+statistical+machine+translation%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapter+of+the+Association+for+Computatio) 7. Anthony Rousseau. 2013. Xenc: An open-source tool for data selection in natural language processing. The Prague Bulletin of Mathematical Linguistics, (100):73–82. [find](https://scholar.google.com/scholar?q=Anthony+Rousseau%2E+2013%2E+Xenc%3A+An+open%2Dsource+tool+for+data+selection+in+natural+language+processing%2E+The+Prague+Bulletin+of+Mathematical+Linguistics%2C+%28100%29%3A73%E2%80%9382%2E) 8. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016a%2E+Edinburgh+neural+machine+translation+systems+for+wmt+16%2E+arXiv+preprint+arXiv%3A1606%2E02891%2E) [our card](https://md.research.elamur.ai/free/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/) 9. Yongchao Deng, Shanbo Cheng, Jun Lu, Kai Song, Jingang Wang, Shenglan Wu, Liang Yao, Guchun Zhang, Haibo Zhang, Pei Zhang, Changfeng Zhu, and Boxing Chen. 2018. Alibaba’s neural machine translation systems for wmt18. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tas… [find](https://scholar.google.com/scholar?q=Yongchao+Deng%2C+Shanbo+Cheng%2C+Jun+Lu%2C+Kai+Song%2C+Jingang+Wang%2C+Shenglan+Wu%2C+Liang+Yao%2C+Guchun+Zhang%2C+Haibo+Zhang%2C+Pei+Zhang%2C+Changfeng+Zhu%2C+and+Boxing+Chen%2E+2018%2E+Alibaba%E2%80%99s+neural+machine+translation+sy) 10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational… [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016b%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computation) 11. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016c. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational L… [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016c%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Lin) 12. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155. [find](https://scholar.google.com/scholar?q=Peter+Shaw%2C+Jakob+Uszkoreit%2C+and+Ashish+Vaswani%2E+2018%2E+Self%2Dattention+with+relative+position+representations%2E+arXiv+preprint+arXiv%3A1803%2E02155%2E) [our card](https://md.research.elamur.ai/free/papers/shaw-2018-self-attention-relative-position-representations/) 13. Felix Stahlberg, Adria de Gispert, and Bill Byrne. 2018. The university of cambridge’s machine translation systems for wmt18. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 508–516, Belgium, Brussels. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Felix+Stahlberg%2C+Adria+de+Gispert%2C+and+Bill+Byrne%2E+2018%2E+The+university+of+cambridge%E2%80%99s+machine+translation+systems+for+wmt18%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%2C+Volume+2%3A+S) 14. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, et al. 2018. Tensor2tensor for neural machine translation. arXiv preprint arXiv:1803.07416. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Samy+Bengio%2C+Eugene+Brevdo%2C+Francois+Chollet%2C+Aidan+N+Gomez%2C+Stephan+Gouws%2C+Llion+Jones%2C+%C5%81ukasz+Kaiser%2C+Nal+Kalchbrenner%2C+Niki+Parmar%2C+et+al%2E+2018%2E+Tensor2tensor+for+neural+machine+tra) [our card](https://md.research.elamur.ai/free/papers/vaswani-2018-tensor2tensor-neural-machine-translation/) 15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processi) 16. Qiang Wang, Bei Li, Jiqiang Liu, Bojian Jiang, Zheyang Zhang, Yinqiao Li, Ye Lin, Tong Xiao, and Jingbo Zhu. 2018. The niutrans machine translation system for wmt18. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 532–538, Belgium, Brussels. Associa… [find](https://scholar.google.com/scholar?q=Qiang+Wang%2C+Bei+Li%2C+Jiqiang+Liu%2C+Bojian+Jiang%2C+Zheyang+Zhang%2C+Yinqiao+Li%2C+Ye+Lin%2C+Tong+Xiao%2C+and+Jingbo+Zhu%2E+2018%2E+The+niutrans+machine+translation+system+for+wmt18%2E+In+Proceedings+of+the+Third+Confer) 17. Qiang Wang, Bei Li, Tong Xiao, and Jingbo Zhu. 2019. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Italy, Florence. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qiang+Wang%2C+Bei+Li%2C+Tong+Xiao%2C+and+Jingbo+Zhu%2E+2019%2E+Learning+deep+transformer+models+for+machine+translation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistic) [our card](https://md.research.elamur.ai/free/papers/wang-2019-learning-deep-transformer-models-machine/) 18. Tong Xiao, Jingbo Zhu, Hao Zhang, and Qiang Li. 2012. Niutrans: An open source toolkit for phrase-based and syntax-based machine translation. In Proceedings of the ACL 2012 System Demonstrations, ACL ’12, pages 19–24, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tong+Xiao%2C+Jingbo+Zhu%2C+Hao+Zhang%2C+and+Qiang+Li%2E+2012%2E+Niutrans%3A+An+open+source+toolkit+for+phrase%2Dbased+and+syntax%2Dbased+machine+translation%2E+In+Proceedings+of+the+ACL+2012+System+Demonstrations%2C+ACL+)