--- title: "Facebook FAIR's WMT19 News Translation Task — Ng 2019" description: "This paper presents Facebook AI Research's systems for the WMT19 news translation shared task in four directions: English↔German and English↔Russian." authors: ["Nathan Ng", "Kyra Yee", "Alexei Baevski", "Myle Ott", "Michael Auli", "Sergey Edunov"] year: 2019 venue: "Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)" source: "https://aclanthology.org/W19-5333/" doi: "10.18653/v1/w19-5333" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ng-2019-facebook-fair-s-wmt19-news/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ng-2019-facebook-fair-s-wmt19-news/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / Facebook FAIR's WMT19 News Translation Task Submission # Facebook FAIR's WMT19 News Translation Task Submission **Authors:** [Nathan Ng](https://md.research.elamur.ai/free/en/authors/nathan-ng/), Kyra Yee, [Alexei Baevski](https://md.research.elamur.ai/free/en/authors/alexei-baevski/), [Myle Ott](https://md.research.elamur.ai/free/en/authors/myle-ott/), [Michael Auli](https://md.research.elamur.ai/free/en/authors/michael-auli/), [Sergey Edunov](https://md.research.elamur.ai/free/en/authors/sergey-edunov/) · **Year:** 2019 · **Venue:** Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-5333/) · [PDF](https://aclanthology.org/W19-5333.pdf) · [DOI 10.18653/v1/w19-5333](https://doi.org/10.18653/v1/w19-5333) · [OpenAlex](https://openalex.org/W2970295111) **Citations:** 296 citations (openalex-snapshot · 2026-10-14) · [247](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This paper presents Facebook AI Research's systems for the WMT19 news translation shared task in four directions: English↔German and English↔Russian. All systems work at the sentence level. They are built with the FAIRSEQ toolkit on big Transformer models, with a wider feed-forward layer (FFN 8192) and BPE subword units. The pipeline has five main steps: - **Bitext cleaning:** langid language identification plus length and length-ratio rules. - **Back-translation:** large-scale back-translation of Newscrawl, done by sampling from an ensemble of reverse-direction models. - **Extra Russian data:** because Russian Newscrawl is small, an in-domain slice of Russian Commoncrawl is added, chosen with Moore–Lewis cross-entropy difference scoring using KenLM n-gram models. - **Adaptation:** models are fine-tuned on earlier newstest sets and News-Commentary, then ensembled. - **Decoding:** n-best lists are reranked with a noisy channel model that combines forward, channel (reverse-direction) and Transformer language model scores. Each step adds measurable gains. Reranking still helps even on top of strong ensembles. The En→De system beats the team's WMT18 entry by about 4.5 BLEU, and the En→Ru submission achieved the top case-sensitive BLEU in the competition. The work shows that monolingual data and reranking remain valuable even for high-resource language pairs. ## Key points - **Bitext preprocessing:** filtering removed roughly 30% of the bitext. It dropped sentences over 250 tokens and pairs whose source/target length ratio exceeded 1.5. En–De used joint BPE with 32K merges. En–Ru used separate 24K BPE per language, which worked clearly better than joint BPE. - **Back-translation setup:** synthetic data was sampled from an ensemble of three target-to-source models and mixed 1:1 with bitext. Moore–Lewis selection kept about 5% of Russian Commoncrawl (1.2B → 60M sentences). - **Noisy channel reranking:** the forward model generates a beam-50 n-best list. Weights and length penalty were tuned by random search on newstest2014+2016. Reranking added 1.0 BLEU (En→Ru) to 1.5 BLEU (De→En). - **WMT19 test BLEU:** En→De 42.7, De→En 40.8, En→Ru 36.3 (best case-sensitive score), Ru→En 40.0. - **Training scale:** models were trained on 128 Volta GPUs. The weaker Russian language model, trained on less data, likely limited reranking gains for En→Ru. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 247 papers of our library cite it (of the 140673 whose bibliographies we read); 24 of them have a page here, 223 not yet. - [In-context Examples Selection for Machine Translation](https://md.research.elamur.ai/free/en/papers/agrawal-2023-context-examples-selection-machine-translation/) - [Unsupervised Domain Clusters in Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/aharoni-2020-unsupervised-domain-clusters-pretrained-language/) - [TICO-19: the Translation Initiative for COvid-19](https://md.research.elamur.ai/free/en/papers/anastasopoulos-2020-tico-19-translation-initiative-covid/) - [Findings of the 2019 Conference on Machine Translation (WMT19)](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/) - [AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://md.research.elamur.ai/free/en/papers/dai-2023-auggpt-leveraging-chatgpt-text-data/) - [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://md.research.elamur.ai/free/en/papers/dathathri-2019-plug-play-language-models-simple/) - [On The Evaluation of Machine Translation Systems Trained With Back-Translation](https://md.research.elamur.ai/free/en/papers/edunov-2020-evaluation-machine-translation-systems-trained/) - [Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution](https://md.research.elamur.ai/free/en/papers/emelin-2021-wino-x-multilingual-winograd-schemas/) - [Findings of the WMT 2020 Shared Task on Chat Translation](https://md.research.elamur.ai/free/en/papers/farajian-2020-findings-wmt-2020-shared-task/) - [Quality-Aware Decoding for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/fernandes-2022-quality-aware-decoding-neural-machine/) - [Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/) - [Automatic Detection of Machine Generated Text: A Critical Survey](https://md.research.elamur.ai/free/en/papers/jawahar-2020-automatic-detection-machine-generated-text/) - [How Can We Know What Language Models Know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) - [Self-Guided Contrastive Learning for BERT Sentence Representations](https://md.research.elamur.ai/free/en/papers/kim-2021-self-guided-contrastive-learning-bert/) - [Data Augmentation using Pre-trained Transformer Models](https://md.research.elamur.ai/free/en/papers/kumar-2020-data-augmentation-using-pre-trained/) - [Document-level Neural MT: A Systematic Comparison](https://md.research.elamur.ai/free/en/papers/lopes-2020-document-level-neural-mt-systematic/) - [Script Correction and Synthetic Pivoting: Adapting Tencent HY-MT for Low-Resource Turkic Translation](https://md.research.elamur.ai/free/en/papers/maxim-2026-script-correction-synthetic-pivoting-adapting/) - [Locally Typical Sampling](https://md.research.elamur.ai/free/en/papers/meister-2023-locally-typical-sampling/) - [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://md.research.elamur.ai/free/en/papers/schwenk-2019-ccmatrix-mining-billions-high-quality/) - [Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://md.research.elamur.ai/free/en/papers/smit-2020-combining-automatic-labelers-expert-annotations/) - [Facebook AI's WMT21 News Translation Task Submission](https://md.research.elamur.ai/free/en/papers/tran-2021-facebook-ai-s-wmt21-news/) - [Authorship Attribution for Neural Text Generation](https://md.research.elamur.ai/free/en/papers/uchendu-2020-authorship-attribution-neural-text-generation/) - [Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq](https://md.research.elamur.ai/free/en/papers/wang-2020-fairseq-s2t-fast-speech-text/) - [FUDGE: Controlled Text Generation With Future Discriminators](https://md.research.elamur.ai/free/en/papers/yang-2021-fudge-controlled-text-generation-future/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{ng2019facebook, title = {{Facebook FAIR's WMT19 News Translation Task Submission}}, author = {Ng, Nathan and Yee, Kyra and Baevski, Alexei and Ott, Myle and Auli, Michael and Edunov, Sergey}, booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)}, year = {2019}, doi = {10.18653/v1/w19-5333}, url = {https://doi.org/10.18653/v1/w19-5333} } ``` - [← previous Exploring Pre-trained Language Models for Event Extraction and Generation](https://md.research.elamur.ai/free/en/papers/yang-2019-exploring-pre-trained-language-models/) - [next → Factorising AMR generation through syntax](https://md.research.elamur.ai/free/en/papers/cao-2019-factorising-amr-generation-through-syntax/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) ## References 7 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+2018%2E+Understanding+back%2Dtranslation+at+scale%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pag) 2. Kenneth Heafield. 2011. Kenlm: faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197. [find](https://scholar.google.com/scholar?q=Kenneth+Heafield%2E+2011%2E+Kenlm%3A+faster+and+smaller+language+model+queries%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2C+pages+187%E2%80%93197%2E) [our card](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/) 3. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2016%2E+Bag+of+tricks+for+efficient+text+classification%2E+arXiv+preprint+arXiv%3A1607%2E01759%2E) 4. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondrej+Bojar%2C+Alexandra+Consta) 5. Marco Lui and Timothy Baldwin. 2012. langid. py: An off-the-shelf language identification tool. In Proceedings of the ACL 2012 system demonstrations, pages 25–30. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marco+Lui+and+Timothy+Baldwin%2E+2012%2E+langid%2E+py%3A+An+off%2Dthe%2Dshelf+language+identification+tool%2E+In+Proceedings+of+the+ACL+2012+system+demonstrations%2C+pages+25%E2%80%9330%2E+Association+for+Computational+Linguis) [our card](https://md.research.elamur.ai/free/en/papers/lui-2012-langid-py-off-shelf-language/) 6. Robert Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers, pages 220–224. [find](https://scholar.google.com/scholar?q=Robert+Moore+and+William+Lewis%2E+2010%2E+Intelligent+selection+of+language+model+training+data%2E+In+Proceedings+of+the+ACL+2010+Conference+Short+Papers%2C+pages+220%E2%80%93224%2E) [our card](https://md.research.elamur.ai/free/en/papers/moore-2010-intelligent-selection-language-model-training/) 7. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Sergey+Edunov%2C+Alexei+Baevski%2C+Angela+Fan%2C+Sam+Gross%2C+Nathan+Ng%2C+David+Grangier%2C+and+Michael+Auli%2E+2019%2E+fairseq%3A+A+fast%2C+extensible+toolkit+for+sequence+modeling%2E+In+Proceedings+of+NAACL%2DHL) [our card](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)