---
title: "Training on Synthetic Noise Improves Robustness to Natural"
description: "Translation systems trained on clean text tend to fail on inputs with typos. 8.38 BLEU. 19 references, cited by 64 papers here."
authors: ["Vladimir Karpukhin", "Omer Levy", "Jacob Eisenstein", "Marjan Ghazvininejad"]
year: 2019
venue: "Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)"
source: "https://aclanthology.org/D19-5506/"
doi: "10.18653/v1/d19-5506"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/karpukhin-2019-training-synthetic-noise-improves-robustness/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/karpukhin-2019-training-synthetic-noise-improves-robustness/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation

# Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation

**Authors:** [Vladimir Karpukhin](https://md.research.elamur.ai/free/en/authors/vladimir-karpukhin/), [Omer Levy](https://md.research.elamur.ai/free/en/authors/omer-levy/), [Jacob Eisenstein](https://md.research.elamur.ai/free/en/authors/jacob-eisenstein/) and [Marjan Ghazvininejad](https://md.research.elamur.ai/free/en/authors/marjan-ghazvininejad/) · **Year:** 2019 · **Venue:** [Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)](https://md.research.elamur.ai/free/en/venues/wnut/) ([Workshop on Noisy User-generated Text 2019](https://md.research.elamur.ai/free/en/venues/wnut/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-5506/) · [PDF](https://aclanthology.org/D19-5506.pdf) · [DOI 10.18653/v1/d19-5506](https://doi.org/10.18653/v1/d19-5506) · [OpenAlex](https://openalex.org/W2914056350)

**Citations:** 95 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/karpukhin-2019-training-synthetic-noise-improves-robustness/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Translation systems trained on clean text tend to fail on inputs with typos. This holds even for subword (BPE) and character-CNN encoders, because a single misspelled character changes the units these models depend on. Earlier attempts to fix this with synthetic noise only helped on the same artificial noise and hurt clean-text quality. The authors train a Transformer with a CNN character encoder on IWSLT 2016 German, French and Czech→English data. During training, each token is randomly corrupted on the fly by one of four character edits: deletion, insertion, substitution or swap. They then test on real misspellings, drawn from Wikipedia edit histories and corrected Czech essays, which were never used in training. This mixed synthetic noise recovers about 19–54% of the BLEU lost to natural typos and shifts clean-text BLEU by less than one point. Deletions and insertions turn out to matter most. The approach does not help on MTNT Reddit data. This suggests that social-media text differs from standard text mainly in style and vocabulary rather than in spelling errors.

## Key points
- **Data and model.** About 200K sentence pairs per language pair from IWSLT 2016 (de/fr/cs→en). The model is a base Transformer (6 layers, 512/2048 dimensions, 8 heads) with a Kim et al. (2016) character CNN, 256-dimensional character embeddings and dropout 0.2.
- **Noise schedule.** Each training token stays clean with probability 60%. Otherwise it receives one of the four edits, each with probability 10%. Noise is resampled every epoch.
- **Main result.** On de-en with maximal natural noise (about 39% of tokens corrupted), BLEU rises from 12.49 to 23.34. Clean-test changes range from −0.67 to +0.33 BLEU.
- **Ablation on de-en.**
  - Adding deletion alone gives +4.90 BLEU, and insertion alone gives +2.51.
  - Substitution alone slightly hurts (−0.50).
  - Removing deletion from the full mix costs 8.38 BLEU.
- **MTNT results.** Noise training gives no meaningful gain: en-fr goes from 21.1 to 20.6 and fr-en from 23.6 to 24.1. Training on half of the natural typo set also behaved about the same as using no noise.

## Interesting ideas
- **Keep training and test noise separate.** Train with cheap, generic synthetic noise and evaluate only on real, held-out errors. This gives an honest estimate of robustness without requiring a language-specific error corpus, which is useful for low-resource languages such as Ukrainian.
- **Include edits that change word length.** Insertions and deletions shift character n-gram positions and word length, which stresses character encoders more than swaps do. Any noise-augmentation scheme for character or subword models should probably include them.
- **Report a "% recovered" metric.** This measures the share of the noise-induced BLEU drop that a method wins back. It makes robustness gains comparable across languages and noise levels better than raw score differences do.
- **Check how well synthetic noise covers real errors.** Measure what fraction of real errors a single synthetic edit could produce, then compare that with the gains. This is a quick diagnostic for whether a noise model fits the target error distribution, and it also flagged that random substitutions poorly mimic real ones.
- **Find out what kind of "noise" a domain really contains before augmenting.** The MTNT result shows that social-media text mainly differs through slang, code-switching and stylistic shortenings rather than typos. The augmentation should therefore target those phenomena instead.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 310 works · 284,480 citations
- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 69 works · 22,788 citations
- [Meta (Israel)](https://md.research.elamur.ai/free/en/companies/meta-israel/) — Company, Tel Aviv, Israel · 36 works · 31,506 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Vladimir Karpukhin, Omer Levy, Jacob Eisenstein and Marjan Ghazvininejad. 2019. Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation. Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{karpukhin2019training,
  title = {{Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation}},
  author = {Vladimir Karpukhin, Omer Levy, Jacob Eisenstein and Marjan Ghazvininejad},
  booktitle = {Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)},
  year = {2019},
  doi = {10.18653/v1/d19-5506},
  url = {https://doi.org/10.18653/v1/d19-5506}
}
```

- [← previous Semantic Neural Machine Translation Using AMR](https://md.research.elamur.ai/free/en/papers/song-2019-semantic-neural-machine-translation-amr/)
- [next → Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces](https://md.research.elamur.ai/free/en/papers/patra-2019-bilingual-lexicon-induction-semi-supervision/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214–223. [find](https://scholar.google.com/scholar?q=Martin+Arjovsky%2C+Soumith+Chintala%2C+and+Le%CC%81on+Bottou%2E+2017%2E+Wasserstein+generative+adversarial+networks%2E+In+International+Conference+on+Machine+Learning%2C+pages+214%E2%80%93223%2E)
2. Luz Rello and Ricardo A Baeza-Yates. 2012. Social media is not that bad! the lexical quality of social media. In ICWSM. [find](https://scholar.google.com/scholar?q=Luz+Rello+and+Ricardo+A+Baeza%2DYates%2E+2012%2E+Social+media+is+not+that+bad%21+the+lexical+quality+of+social+media%2E+In+ICWSM%2E)
3. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In ICLR. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2018%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+In+ICLR%2E)
4. Keisuke Sakaguchi, Kevin Duh, Matt Post, and Benjamin Van Durme. 2017. Robsut wrod reocginiton via semi-character recurrent neural network. In AAAI, pages 3281–3287. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Kevin+Duh%2C+Matt+Post%2C+and+Benjamin+Van+Durme%2E+2017%2E+Robsut+wrod+reocginiton+via+semi%2Dcharacter+recurrent+neural+network%2E+In+AAAI%2C+pages+3281%E2%80%933287%2E)
5. Chris M Bishop. 1995. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108–116. [find](https://scholar.google.com/scholar?q=Chris+M+Bishop%2E+1995%2E+Training+with+noise+is+equivalent+to+tikhonov+regularization%2E+Neural+computation%2C+7%281%29%3A108%E2%80%93116%2E)
6. Karel Šebesta, Zuzanna Bedřichová, Kateřina Šormová, Barbora Štindlová, Milan Hrdlička, Tereza Hrdličková, Jiřı́ Hana, Vladimı́r Petkevič, Tomáš Jelı́nek, Svatava Škodová, Petr Janeš, Kateřina Lundáková, Hana Skoumalová, Šimon Sládek, Piotr Pierscieniak, Dagmar Toufarová, Mi… [find](https://scholar.google.com/scholar?q=Karel+S%CC%8Cebesta%2C+Zuzanna+Bedr%CC%8Cichova%CC%81%2C+Kater%CC%8Cina+S%CC%8Cormova%CC%81%2C+Barbora+S%CC%8Ctindlova%CC%81%2C+Milan+Hrdlic%CC%8Cka%2C+Tereza+Hrdlic%CC%8Ckova%CC%81%2C+Jir%CC%8C%C4%B1%CC%81+Hana%2C+Vladim%C4%B1%CC%81r+Petkevic%CC%8C%2C+Toma%CC%81s%CC%8C+Jel%C4%B1%CC%81nek%2C+Svatava+S%CC%8Ckodova%CC%81%2C+Petr+Janes%CC%8C)
7. Mauro Cettolo, Niehues Jan, Stüker Sebastian, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2016. [find](https://scholar.google.com/scholar?q=Mauro+Cettolo%2C+Niehues+Jan%2C+Stu%CC%88ker+Sebastian%2C+Luisa+Bentivogli%2C+Roldano+Cattoni%2C+and+Marcello+Federico%2E+2016%2E)
8. The IWSLT 2016 evaluation campaign. In International Workshop on Spoken Language Translation. [find](https://scholar.google.com/scholar?q=The+IWSLT+2016+evaluation+campaign%2E+In+International+Workshop+on+Spoken+Language+Translation%2E)
9. Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. [Towards robust neural machine translation](https://md.research.elamur.ai/free/en/papers/cheng-2018-robust-neural-machine-translation/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756– 1766, Melbourne, Australia. Association for Computational…
10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). arXiv preprint arXiv:1508.07909.
11. Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. In International Workshop on Spoken Language Translation (IWSLT), Tokyo, Japan. [find](https://scholar.google.com/scholar?q=Matthias+Sperber%2C+Jan+Niehues%2C+and+Alex+Waibel%2E+2017%2E+Toward+robust+neural+machine+translation+for+noisy+input+sequences%2E+In+International+Workshop+on+Spoken+Language+Translation+%28IWSLT%29%2C+Tokyo%2C+Japan)
12. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. [Hotflip: White-box adversarial examples for text classification](https://md.research.elamur.ai/free/en/papers/ebrahimi-2018-hotflip-white-box-adversarial-examples/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31–36, Melbourne, Australia. Association for Comput…
13. Vaibhav, Sumeet Singh, Craig Stewart, and Graham Neubig. 2019. Improving robustness of machine translation with synthetic noise. arXiv preprint arXiv:1902.09508. [find](https://scholar.google.com/scholar?q=Vaibhav%2C+Sumeet+Singh%2C+Craig+Stewart%2C+and+Graham+Neubig%2E+2019%2E+Improving+robustness+of+machine+translation+with+synthetic+noise%2E+arXiv+preprint+arXiv%3A1902%2E09508%2E)
14. Jacob Eisenstein. 2013. [What to do about bad language on the internet](https://md.research.elamur.ai/free/en/papers/eisenstein-2013-what-do-about-bad-language/). In Proceedings of the 2013 conference of the North American Chapter of the association for computational linguistics: Human language technologies, pages 359–369.
15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pages 5998–6008.
16. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. [Explaining and harnessing adversarial examples](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/). arXiv preprint arXiv:1412.6572.
17. Stefan Wager, Sida Wang, and Percy S Liang. 2013. Dropout training as adaptive regularization. In Advances in neural information processing systems, pages 351–359. [find](https://scholar.google.com/scholar?q=Stefan+Wager%2C+Sida+Wang%2C+and+Percy+S+Liang%2E+2013%2E+Dropout+training+as+adaptive+regularization%2E+In+Advances+in+neural+information+processing+systems%2C+pages+351%E2%80%93359%2E)
18. Georg Heigold, Günter Neumann, and Josef van Genabith. 2017. How robust are character-based word embeddings in tagging and mt against wrod scramlbing or randdm nouse? arXiv preprint arXiv:1704.04441. [find](https://scholar.google.com/scholar?q=Georg+Heigold%2C+Gu%CC%88nter+Neumann%2C+and+Josef+van+Genabith%2E+2017%2E+How+robust+are+character%2Dbased+word+embeddings+in+tagging+and+mt+against+wrod+scramlbing+or+randdm+nouse%3F+arXiv+preprint+arXiv%3A1704%2E04441%2E)
19. Torsten Zesch. 2012. Measuring contextual fitness using error contexts extracted from the wikipedia revision history. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 529–538. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Torsten+Zesch%2E+2012%2E+Measuring+contextual+fitness+using+error+contexts+extracted+from+the+wikipedia+revision+history%2E+In+Proceedings+of+the+13th+Conference+of+the+European+Chapter+of+the+Association+f)

*Updated 24 October 2026*