---
title: "On the Impact of Various Types of Noise on — Khayrallah 2018"
description: "The paper asks which kinds of flaws in parallel training data damage translation quality, and whether neural and statistical systems react differently."
authors: ["Huda Khayrallah", "Philipp Koehn"]
year: 2018
venue: "Proceedings of the 2nd Workshop on Neural Machine Translation and Generation"
source: "https://aclanthology.org/W18-2709/"
doi: "10.18653/v1/w18-2709"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/khayrallah-2018-impact-various-types-noise-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/khayrallah-2018-impact-various-types-noise-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / On the Impact of Various Types of Noise on Neural Machine Translation

# On the Impact of Various Types of Noise on Neural Machine Translation

**Authors:** [Huda Khayrallah](https://md.research.elamur.ai/free/en/authors/huda-khayrallah/), [Philipp Koehn](https://md.research.elamur.ai/free/en/authors/philipp-koehn/) · **Year:** 2018 · **Venue:** Proceedings of the 2nd Workshop on Neural Machine Translation and Generation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-2709/) · [PDF](https://aclanthology.org/W18-2709.pdf) · [DOI 10.18653/v1/w18-2709](https://doi.org/10.18653/v1/w18-2709) · [OpenAlex](https://openalex.org/W2805411799)

**Citations:** 42 citations (openalex-snapshot · 2026-10-14) · [146](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper asks which kinds of flaws in parallel training data damage translation quality, and whether neural and statistical systems react differently. The motivation is an experiment on German–English: adding raw web-crawled Paracrawl data to clean WMT 2017 training data slightly improved a statistical system but sharply degraded a neural one. The authors manually labelled 200 crawled sentence pairs to see which errors occur in practice. Based on that study, they defined five synthetic noise types: misaligned sentences, misordered words, wrong language, untranslated sentences, and short segments. They mixed each type into a clean corpus in increasing proportions and trained RNN attention models in Marian and phrase-based systems in Moses. For nearly every noise type, the neural models suffered more than the statistical ones. The worst case was source text copied untranslated into the target side, which taught the neural system to copy its input. The results show that corpus cleaning matters much more for NMT, and they point to which types of noise filtering should target first.

## Key points
- Adding an equal amount of raw Paracrawl data changed BLEU from 27.2 to 17.3 (−9.9) for NMT and from 24.0 to 25.2 (+1.2) for SMT.
- In the manual Paracrawl sample, misalignment was the most common problem (41%). Wrong-language content made up 23%, untranslated pairs 4%, and only 23% of pairs were judged okay.
- The clean training data (Europarl, News Commentary, Rapid; about 83M tokens per language) was mixed with noise at 5–100% of its size. Evaluation used newstest2017, and the NMT models used BPE with a 50,000 vocabulary.
- Adding just 10% untranslated-target data made 57% of NMT outputs closer to the source than to the reference, by TER. Learning curves showed the models overfitting to the copied portion of the data.
- Both NMT and SMT were surprisingly robust to German–French data in the training set. Even at a 100% noise ratio, only 6.7% of NMT outputs were in French, which the authors attribute to the domain mismatch.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

146 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 133 not yet.

- [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/artetxe-2019-margin-based-parallel-corpus-mining/)
- [ParaCrawl: Web-Scale Acquisition of Parallel Corpora](https://md.research.elamur.ai/free/en/papers/banon-2020-paracrawl-web-scale-acquisition-parallel/)
- [Low-Resource Corpus Filtering Using Multilingual Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/chaudhary-2019-low-resource-corpus-filtering-multilingual/)
- [Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora](https://md.research.elamur.ai/free/en/papers/dowmunt-2018-dual-conditional-cross-entropy-filtering/)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://md.research.elamur.ai/free/en/papers/gala-2023-indictrans2-towards-high-quality-accessible/)
- [Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering](https://md.research.elamur.ai/free/en/papers/koehn-2018-findings-wmt-2018-shared-task/)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://md.research.elamur.ai/free/en/papers/kreutzer-2021-quality-glance-audit-web-crawled/)
- [ParaCLEAN: Improving Translation Quality through Systematic Parallel Data Cleaning](https://md.research.elamur.ai/free/en/papers/mash-2026-paraclean-improving-translation-quality-through/)
- [MTNT: A Testbed for Machine Translation of Noisy Text](https://md.research.elamur.ai/free/en/papers/michel-2018-mtnt-testbed-machine-translation-noisy/)
- [Bifixer and Bicleaner: two open-source tools to clean your parallel data](https://md.research.elamur.ai/free/en/papers/sanchez-2020-bifixer-bicleaner-two-open-source/)
- [Vecalign: Improved Sentence Alignment in Linear Time and Space](https://md.research.elamur.ai/free/en/papers/thompson-2019-vecalign-improved-sentence-alignment-linear/)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/)
- [Improving Arabic Diacritization by Learning to Diacritize and Translate](https://md.research.elamur.ai/free/en/papers/thompson-2022-improving-arabic-diacritization-learning-diacritize/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{khayrallah2018impact,
  title = {{On the Impact of Various Types of Noise on Neural Machine Translation}},
  author = {Khayrallah, Huda and Koehn, Philipp},
  booktitle = {Proceedings of the 2nd Workshop on Neural Machine Translation and Generation},
  year = {2018},
  doi = {10.18653/v1/w18-2709},
  url = {https://doi.org/10.18653/v1/w18-2709}
}
```

- [← previous Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words](https://md.research.elamur.ai/free/en/papers/mohammad-2018-obtaining-reliable-human-ratings-valence/)
- [next → On the Limitations of Unsupervised Bilingual Dictionary Induction](https://md.research.elamur.ai/free/en/papers/sogaard-2018-limitations-unsupervised-bilingual-dictionary-induction/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

43 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexandra Antonova and Alexey Misyurev. 2011. Building a web-based parallel corpus and filtering out machine-translated text. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web. Association for Computational Linguistics, Portland, Oregon, page… [find](https://scholar.google.com/scholar?q=Alexandra+Antonova+and+Alexey+Misyurev%2E+2011%2E+Building+a+web%2Dbased+parallel+corpus+and+filtering+out+machine%2Dtranslated+text%2E+In+Proceedings+of+the+4th+Workshop+on+Building+and+Using+Comparable+Corpor)
2. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics 33(2). [http://www.aclweb.org/anthology-new/J/J07/J07-2003.pdf](http://www.aclweb.org/anthology-new/J/J07/J07-2003.pdf). [↗](http://www.aclweb.org/anthology-new/J/J07/J07-2003.pdf) [our card](https://md.research.elamur.ai/free/en/papers/chiang-2007-hierarchical-phrase-based-translation/)
3. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Edinburgh, Scotland, UK., pages 355–362. [http://www.aclwe…](http://www.aclwe…) [↗](http://www.aclwe…) [our card](https://md.research.elamur.ai/free/en/papers/axelrod-2011-domain-adaptation-pseudo-domain-data/)
4. David Chiang, Kevin Knight, and Wei Wang. 2009. 11,001 new features for statistical machine translation. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=David+Chiang%2C+Kevin+Knight%2C+and+Wei+Wang%2E+2009%2E+11%2C001+new+features+for+statistical+machine+translation%2E+In+Proceedings+of+Human+Language+Technologies%3A+The+2009+Annual+Conference+of+the+North+American)
5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. [http://arxiv.org/pdf/1409.0473v6.pdf](http://arxiv.org/pdf/1409.0473v6.pdf). [↗](http://arxiv.org/pdf/1409.0473v6.pdf) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
6. Lei Cui, Dongdong Zhang, Shujie Liu, Mu Li, and Ming Zhou. 2013. Bilingual data cleaning for smt using graph-based random walk. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Sofia, Bulga… [find](https://scholar.google.com/scholar?q=Lei+Cui%2C+Dongdong+Zhang%2C+Shujie+Liu%2C+Mu+Li%2C+and+Ming+Zhou%2E+2013%2E+Bilingual+data+cleaning+for+smt+using+graph%2Dbased+random+walk%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computa)
7. Eduard Barbu, Carla Parra Escartı́n, Luisa Bentivogli, Matteo Negri, Marco Turchi, Constantin Orasan, and Marcello Federico. 2016. The first automatic translation memory cleaning shared task. Machine Translation 30(3):145–166. https://doi.org/10.1007/s10590-016-9183-x. — [doi:10.1007/s10590-016-9183-x](https://doi.org/10.1007/s10590-016-9183-x) [↗](https://doi.org/10.1007/s10590-016-9183-x)
8. Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. 2017. Copied monolingual data improves low-resource neural machine translation. In Proceedings of the Second Conference on Machine Translation, Volume 1: Research Paper. Association for Computational Linguistics, Copenhagen, Denmark,… [find](https://scholar.google.com/scholar?q=Anna+Currey%2C+Antonio+Valerio+Miceli+Barone%2C+and+Kenneth+Heafield%2E+2017%2E+Copied+monolingual+data+improves+low%2Dresource+neural+machine+translation%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Tra) [our card](https://md.research.elamur.ai/free/en/papers/currey-2017-copied-monolingual-data-improves-low/)
9. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. CoRR abs/1711.02173. http://arxiv.org/abs/1711.02173. [↗](http://arxiv.org/abs/1711.02173)
10. Phil Blunsom and Miles Osborne. 2008. Probabilistic inference for machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii, pages 215– 223. [http://www.aclweb.org/anthology/D08-1023](http://www.aclweb.org/anthology/D08-1023). [↗](http://www.aclweb.org/anthology/D08-1023)
11. Shuoyang Ding, Kevin Duh, Huda Khayrallah, Philipp Koehn, and Matt Post. 2016. The jhu machine translation systems for wmt 2016. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 272–280. [http://www.aclweb.org/anthology/W…](http://www.aclweb.org/anthology/W…) [↗](http://www.aclweb.org/anthology/W…)
12. Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translat… [find](https://scholar.google.com/scholar?q=Ondrej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Yvette+Graham%2C+Barry+Haddow%2C+Shujian+Huang%2C+Matthias+Huck%2C+Philipp+Koehn%2C+Qun+Liu%2C+Varvara+Logacheva%2C+Christof+Monz%2C+Matteo+Negri%2C+Matt+Post%2C+Rapha) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/)
13. Shuoyang Ding, Huda Khayrallah, Philipp Koehn, Matt Post, Gaurav Kumar, and Kevin Duh. 2017. The jhu machine translation systems for wmt 2017. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In… [find](https://scholar.google.com/scholar?q=Shuoyang+Ding%2C+Huda+Khayrallah%2C+Philipp+Koehn%2C+Matt+Post%2C+Gaurav+Kumar%2C+and+Kevin+Duh%2E+2017%2E+The+jhu+machine+translation+systems+for+wmt+2017%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Transl) [our card](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/)
14. Task Papers. Association for Computational Linguistics, Copenhagen, Denmark, pages 276–282. [http://www.aclweb.org/anthology/W17-4724](http://www.aclweb.org/anthology/W17-4724). [↗](http://www.aclweb.org/anthology/W17-4724)
15. Nadir Durrani, Alexander Fraser, Helmut Schmid, Hieu Hoang, and Philipp Koehn. 2013. Can Markov Models Over Minimal Translation Units Help Phrase-Based SMT? In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Sofia, B… [find](https://scholar.google.com/scholar?q=Nadir+Durrani%2C+Alexander+Fraser%2C+Helmut+Schmid%2C+Hieu+Hoang%2C+and+Philipp+Koehn%2E+2013%2E+Can+Markov+Models+Over+Minimal+Translation+Units+Help+Phrase%2DBased+SMT%3F+In+Proceedings+of+the+51st+Annual+Meeting+o)
16. Liang Huang and David Chiang. 2007. Forest rescoring: Faster decoding with integrated language models. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. Association for Computational Linguistics, Prague, Czech Republic, pages 144–151. [http://www.aclweb.org/an…](http://www.aclweb.org/an…) [↗](http://www.aclweb.org/an…) [our card](https://md.research.elamur.ai/free/en/papers/huang-2007-forest-rescoring-faster-decoding-integrated/)
17. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Vancouver, Canada,… [find](https://scholar.google.com/scholar?q=Marzieh+Fadaee%2C+Arianna+Bisazza%2C+and+Christof+Monz%2E+2017%2E+Data+augmentation+for+low%2Dresource+neural+machine+translation%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+) [our card](https://md.research.elamur.ai/free/en/papers/fadaee-2017-data-augmentation-low-resource-neural/)
18. M. Junczys-Dowmunt. 2012. A phrase table without phrases: Rank encoding for better phrase table compression. In Mauro Cettolo, Marcello Federico, Lucia Specia, and Andy Way, editors, Proceedings of th 16th International Conference of the European Association for Machine Translation (EAMT). pages 245… [find](https://scholar.google.com/scholar?q=M%2E+Junczys%2DDowmunt%2E+2012%2E+A+phrase+table+without+phrases%3A+Rank+encoding+for+better+phrase+table+compression%2E+In+Mauro+Cettolo%2C+Marcello+Federico%2C+Lucia+Specia%2C+and+Andy+Way%2C+editors%2C+Proceedings+of+th)
19. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inference and training of context-rich syntactic translation models. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the… [find](https://scholar.google.com/scholar?q=Michel+Galley%2C+Jonathan+Graehl%2C+Kevin+Knight%2C+Daniel+Marcu%2C+Steve+DeNeefe%2C+Wei+Wang%2C+and+Ignacio+Thayer%2E+2006%2E+Scalable+inference+and+training+of+context%2Drich+syntactic+translation+models%2E+In+Proceedi) [our card](https://md.research.elamur.ai/free/en/papers/galley-2006-scalable-inference-training-context-rich/)
20. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. arXiv preprint arXiv:1804.00344 h… [find](https://scholar.google.com/scholar?q=Marcin+Junczys%2DDowmunt%2C+Roman+Grundkiewicz%2C+Tomasz+Dwojak%2C+Hieu+Hoang%2C+Kenneth+Heafield%2C+Tom+Neckermann%2C+Frank+Seide%2C+Ulrich+Germann%2C+Alham+Fikri+Aji%2C+Nikolay+Bogoychev%2C+Andre%CC%81+F%2E+T%2E+Martins%2C+and+Alex) [our card](https://md.research.elamur.ai/free/en/papers/junczys-2018-dowmunt-marian-fast-neural-machine-translation/)
21. Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule? In Proceedings of the Joint Conference on Human Language Technologies and the Annual Meeting of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL). [http://www.aclweb.o…](http://www.aclweb.o…) [↗](http://www.aclweb.o…) [our card](https://md.research.elamur.ai/free/en/papers/galley-2004-what-s-translation-rule/)
22. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 1700–1709. [http://www.aclweb.org/anthology/D13-1176](http://www.aclweb.org/anthology/D13-1176). [↗](http://www.aclweb.org/anthology/D13-1176) [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/)
23. Michel Galley and Christopher D. Manning. 2008. A simple and effective hierarchical phrase reordering model. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii, pages 848–856. [http://www.aclweb.org/an…](http://www.aclweb.org/an…) [↗](http://www.aclweb.org/an…) [our card](https://md.research.elamur.ai/free/en/papers/galley-2008-simple-effective-hierarchical-phrase-reordering/)
24. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the Tenth Machine Translation Summit (MT Summit X). Phuket, Thailand. [http://mt-archive.info/MTS-2005-Koehn.pdf](http://mt-archive.info/MTS-2005-Koehn.pdf). [↗](http://mt-archive.info/MTS-2005-Koehn.pdf)
25. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Christopher J. Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translatio… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Christopher+J%2E+Dyer%2C+Ondr%CC%8Cej+Bojar%2C+Alexan)
26. Jonas Gehring, Michael Auli, David Grangier, and Yann Dauphin. 2017. A convolutional encoder model for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver,… [find](https://scholar.google.com/scholar?q=Jonas+Gehring%2C+Michael+Auli%2C+David+Grangier%2C+and+Yann+Dauphin%2E+2017%2E+A+convolutional+encoder+model+for+neural+machine+translation%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-encoder-model-neural-machine/)
27. Barry Haddow, Matthias Huck, Alexandra Birch, Nikolay Bogoychev, and Philipp Koehn. 2015. The edinburgh/jhu phrase-based machine translation systems for wmt 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Lisbon, Portugal, pag… [find](https://scholar.google.com/scholar?q=Barry+Haddow%2C+Matthias+Huck%2C+Alexandra+Birch%2C+Nikolay+Bogoychev%2C+and+Philipp+Koehn%2E+2015%2E+The+edinburgh%2Fjhu+phrase%2Dbased+machine+translation+systems+for+wmt+2015%2E+In+Proceedings+of+the+Tenth+Workshop+)
28. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation. Association for Computational Linguistics, Vancouver, pages 28–39. [http://www.aclweb.org/anthology/W17-3204](http://www.aclweb.org/anthology/W17-3204). [↗](http://www.aclweb.org/anthology/W17-3204)
29. Jörg Tiedemann. 2009. News from OPUS - A collection of multilingual parallel corpora with tools and interfaces. In N. Nicolov, K. Bontcheva, G. Angelova, and R. Mitkov, editors, Recent Advances in Natural Language Processing, John Benjamins, Amsterdam/Philadelphia, Borovets, Bulgaria, volume V. [find](https://scholar.google.com/scholar?q=Jo%CC%88rg+Tiedemann%2E+2009%2E+News+from+OPUS+%2D+A+collection+of+multilingual+parallel+corpora+with+tools+and+interfaces%2E+In+N%2E+Nicolov%2C+K%2E+Bontcheva%2C+G%2E+Angelova%2C+and+R%2E+Mitkov%2C+editors%2C+Recent+Advances+in+Na)
30. Shankar Kumar and William J. Byrne. 2004. Minimum Bayes-Risk Decoding for Statistical Machine Translation. In HLT-NAACL. pages 169–176. [find](https://scholar.google.com/scholar?q=Shankar+Kumar+and+William+J%2E+Byrne%2E+2004%2E+Minimum+Bayes%2DRisk+Decoding+for+Statistical+Machine+Translation%2E+In+HLT%2DNAACL%2E+pages+169%E2%80%93176%2E) [our card](https://md.research.elamur.ai/free/en/papers/kumar-2004-minimum-bayes-risk-decoding-statistical/)
31. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016). [find](https://scholar.google.com/scholar?q=Pierre+Lison+and+Jo%CC%88rg+Tiedemann%2E+2016%2E+Opensubtitles2016%3A+Extracting+large+parallel+corpora+from+movie+and+tv+subtitles%2E+In+Proceedings+of+the+10th+International+Conference+on+Language+Resources+and+) [our card](https://md.research.elamur.ai/free/en/papers/lison-2016-opensubtitles2016-extracting-large-parallel-corpora/)
32. Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the 8th International Conference on Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=Jo%CC%88rg+Tiedemann%2E+2012%2E+Parallel+Data%2C+Tools+and+Interfaces+in+OPUS%2E+In+Proceedings+of+the+8th+International+Conference+on+Language+Resources+and+Evaluation%2E)
33. Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. CoRR abs/1803.00047. http://arxiv.org/abs/1803.00047. [↗](http://arxiv.org/abs/1803.00047) [our card](https://md.research.elamur.ai/free/en/papers/ott-2018-analyzing-uncertainty-neural-machine-translation/)
34. Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017. Dynamic data selection for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1411–1421. [http://aclweb.org/anthology/D1…](http://aclweb.org/anthology/D1…) [↗](http://aclweb.org/anthology/D1…)
35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pages… [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+40th+Annual+Meeting+of+the+Association+for+Computat) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR abs/1706.03762. http://arxiv.org/abs/1706.03762. [↗](http://arxiv.org/abs/1706.03762)
37. Spencer Rarrick, Chris Quirk, and Will Lewis. 2011. MT detection in web-scraped parallel corpora. In Proceedings of the 13th Machine Translation Summit (MT Summit XIII). International Association for Machine Translation, pages 422–430. [http://www.mt-archive.info/MTS-2011-Rarrick.pdf](http://www.mt-archive.info/MTS-2011-Rarrick.pdf). [↗](http://www.mt-archive.info/MTS-2011-Rarrick.pdf)
38. Ashish Venugopal, Jakob Uszkoreit, David Talbot, Franz Och, and Juri Ganitkevitch. 2011. Watermarking the outputs of structured prediction with an application in statistical machine translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Association fo… [find](https://scholar.google.com/scholar?q=Ashish+Venugopal%2C+Jakob+Uszkoreit%2C+David+Talbot%2C+Franz+Och%2C+and+Juri+Ganitkevitch%2E+2011%2E+Watermarking+the+outputs+of+structured+prediction+with+an+application+in+statistical+machine+translation%2E+In+Pr)
39. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany,… [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016a%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computation)
40. Hainan Xu and Philipp Koehn. 2017. Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 2935–2940. [http://aclweb.org/antho…](http://aclweb.org/antho…) [↗](http://aclweb.org/antho…)
41. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Berlin, Germany. [http://www.aclweb.org/antholog…](http://www.aclweb.org/antholog…) [↗](http://www.aclweb.org/antholog…)
42. Raivis Skadiņš, Jörg Tiedemann, Roberts Rozis, and Daiga Deksne. 2014. Billions of parallel words for free: Building and using the eu bookshop corpus. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA… [find](https://scholar.google.com/scholar?q=Raivis+Skadin%CC%A7s%CC%8C%2C+Jo%CC%88rg+Tiedemann%2C+Roberts+Rozis%2C+and+Daiga+Deksne%2E+2014%2E+Billions+of+parallel+words+for+free%3A+Building+and+using+the+eu+bookshop+corpus%2E+In+Proceedings+of+the+9th+International+Confer)
43. Kaveh Taghipour, Shahram Khadivi, and Jia Xu. 2011. Parallel corpus refinement as an outlier detection algorithm. In Proceedings of the 13th Machine Translation Summit (MT Summit XIII). International Association for Machine Translation, pages 414–421. [http://www.mt-archive.info/MTS-2011-Taghipour.pd…](http://www.mt-archive.info/MTS-2011-Taghipour.pd…) [↗](http://www.mt-archive.info/MTS-2011-Taghipour.pd…)