--- title: "Denoising Neural Machine Translation Training — Wang 2018" description: "Web-crawled and automatically mined parallel corpora often contain misaligned, partially translated or otherwise flawed sentence pairs, and NMT is more…" authors: ["Wei Wang", "Taro Watanabe", "Macduff Hughes", "Tetsuji Nakagawa", "Ciprian Chelba"] year: 2018 venue: "Proceedings of the Third Conference on Machine Translation: Research Papers" source: "https://aclanthology.org/W18-6314/" doi: "10.18653/v1/w18-6314" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/wang-2018-denoising-neural-machine-translation-training/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/wang-2018-denoising-neural-machine-translation-training/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/) / Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection # Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection **Authors:** [Wei Wang](https://md.research.elamur.ai/free/authors/wei-wang/), [Taro Watanabe](https://md.research.elamur.ai/free/authors/taro-watanabe/), Macduff Hughes, Tetsuji Nakagawa, [Ciprian Chelba](https://md.research.elamur.ai/free/authors/ciprian-chelba/) · **Year:** 2018 · **Venue:** Proceedings of the Third Conference on Machine Translation: Research Papers · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W18-6314/) · [PDF](https://aclanthology.org/W18-6314.pdf) · [DOI 10.18653/v1/w18-6314](https://doi.org/10.18653/v1/w18-6314) · [OpenAlex](https://openalex.org/W2962890089) **Citations:** 67 citations (openalex-snapshot · 2026-10-14) · [48](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Web-crawled and automatically mined parallel corpora often contain misaligned, partially translated or otherwise flawed sentence pairs, and NMT is more sensitive to such noise than SMT. Domain data selection is well studied, but its techniques had not been adapted for denoising. The authors (Google Research) assume a huge noisy corpus and only a small trusted set of a few thousand sentence pairs. They first fine-tune the noisy NMT model on the trusted data to get a slightly cleaner model. A sentence pair's noise is then scored by how differently the two seq2seq models treat it, which generalizes the bilingual cross-entropy difference (CED) of Axelrod et al. During online training, batches are chosen so the data the model sees becomes progressively cleaner, a curriculum-like schedule. On Paracrawl data, this gives large BLEU gains over random ordering, while a neural LM selector built the same way does not help. The work shows that a small amount of trusted data can make NMT training robust to heavy corpus noise. ## Key points - **Trusted data:** WMT newstest 2010–2011 served as the trusted set, and newstest 2012–2013 was used as the dev set. The approach needs only a tiny trusted fraction of the full corpus. - **Online denoising (P3) vs random order (P1) on Paracrawl:** | Test set | P1 (random order) | P3 (online denoising) | |---|---|---| | n2014 | 31.6 | 35.2 | | d2015 | 30.7 | 35.6 | | patent | 37.2 | 46.9 | - **Data order matters:** - Reversing the schedule so the model sees clean data first (P4) yields only 32.6 / 31.1 / 40.9 BLEU on n2014 / d2015 / patent. - Successive fine-tuning on nested subsets (S 80% ⊃ S 40% ⊃ S 20%) improves the score at each step, ending at 34.4 / 34.6 / 45.6. - **NNLM selection fails at denoising:** - Its scores do not track human quality ratings. - Fine-tuning on the top 20% chosen by NNLM (P9) gives 31.8 / 30.5 / 35.4, with patent dropping below the P1 baseline. - Fine-tuning on the top 20% chosen by the NMT-based method (P8) gives 34.3 / 34.7 / 45.8. - **Strong baseline:** the WMT baseline W1, a standard LSTM RNN, reaches 36.2 detokenized BLEU on n2014 (40.2 tokenized). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/) ## Cited by Cited by 48 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 45 not yet. - [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/) - [Language-agnostic BERT Sentence Embedding](https://md.research.elamur.ai/free/papers/feng-2020-language-agnostic-bert-sentence-embedding/) - [MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods](https://md.research.elamur.ai/free/papers/finkelstein-2023-mbr-qe-finetuning-training-time/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{wang2018denoising, title = {{Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection}}, author = {Wang, Wei and Watanabe, Taro and Hughes, Macduff and Nakagawa, Tetsuji and Chelba, Ciprian}, booktitle = {Proceedings of the Third Conference on Machine Translation: Research Papers}, year = {2018}, doi = {10.18653/v1/w18-6314}, url = {https://doi.org/10.18653/v1/w18-6314} } ``` - [← previous Deletion phenomena in comparative constructions: English comparatives in a cross-linguistic perspective](https://md.research.elamur.ai/free/papers/atkari-2018-deletion-phenomena-comparative-constructions/) - [next → Derivational Relations in Arabic WordNet](https://md.research.elamur.ai/free/papers/aclbulk-2018-batita-derivational-relations-arabic-wordnet/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355–362. \#\# Discussion [find](https://scholar.google.com/scholar?q=Amittai+Axelrod%2C+Xiaodong+He%2C+and+Jianfeng+Gao%2E+2011%2E+Domain+adaptation+via+pseudo+in%2Ddomain+data+selection%2E+In+Proceedings+of+the+2011+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+) [our card](https://md.research.elamur.ai/free/papers/axelrod-2011-domain-adaptation-pseudo-domain-data/)