---
title: "SwitchOut: an Efficient Data Augmentation — Wang 2018"
description: "Data augmentation for neural machine translation (NMT) is difficult because small edits to a sentence can change its meaning or make it inconsistent with…"
authors: ["Xinyi Wang", "Hieu Pham", "Zihang Dai", "Graham Neubig"]
year: 2018
venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D18-1100/"
doi: "10.18653/v1/d18-1100"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2018-switchout-efficient-data-augmentation-algorithm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2018-switchout-efficient-data-augmentation-algorithm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation

# SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation

**Authors:** [Xinyi Wang](https://md.research.elamur.ai/free/en/authors/xinyi-wang/), [Hieu Pham](https://md.research.elamur.ai/free/en/authors/hieu-pham/), [Zihang Dai](https://md.research.elamur.ai/free/en/authors/zihang-dai/), [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D18-1100/) · [PDF](https://aclanthology.org/D18-1100.pdf) · [DOI 10.18653/v1/d18-1100](https://doi.org/10.18653/v1/d18-1100) · [OpenAlex](https://openalex.org/W2888519496)

**Citations:** 194 citations (openalex-snapshot · 2026-10-14) · [93](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Data augmentation for neural machine translation (NMT) is difficult because small edits to a sentence can change its meaning or make it inconsistent with its translation. The authors treat the choice of an augmentation policy as an optimization problem. The objective rewards two things: augmented pairs that stay close to observed training pairs (smoothness) and a high-entropy augmentation distribution (diversity). The resulting maximum-entropy problem has a closed-form solution. Word dropout and RAML both turn out to be special cases of it. The framework also motivates a new method, SwitchOut. SwitchOut independently swaps random tokens in both the source and the target sentence for words drawn uniformly from the matching vocabulary. The number of swaps is sampled from a temperature-controlled distribution tied to negative Hamming distance. On WMT 15 English-German, IWSLT 2016 German-English and IWSLT 2015 English-Vietnamese, using a Transformer, SwitchOut adds roughly 0.5 BLEU. It matches or beats word dropout and RAML, and it combines well with RAML and back-translation. The method needs only a few lines of code, which makes it a cheap and widely applicable regularizer for NMT.

## Key points
- **Generic solution:** the optimal augmentation distribution is proportional to exp(similarity / τ).
  - RAML corresponds to target-only reward-based similarity.
  - Word dropout corresponds to a particular similarity function.
  - SwitchOut factorizes similarity into separate source and target terms, with temperatures τ_x and τ_y.
- **Main results (Transformer, median BLEU):**
  - WMT 15 en-de: 21.73 → 22.78 with SwitchOut, and 23.13 with RAML+SwitchOut.
  - IWSLT 16 de-en: up to 30.98 with RAML+SwitchOut.
  - IWSLT 15 en-vi: up to 29.09 with RAML+SwitchOut.
  - Word dropout (p = 0.1) hurt en-de, scoring 20.63.
- **Evaluation protocol:** runs with SwitchOut were repeated 9 times and runs without it 4 times. Paired bootstrap tests showed significant gains over the best baseline in 4 of 6 settings.
- **Back-translation on en-de:**
  - Back-translation alone gave only 21.82.
  - Back-translation + RAML + SwitchOut reached 23.76, so the two approaches are complementary.
- **Where it helps:**
  - Gains increase for test sentences that are farther, by WER, from their nearest training neighbor.
  - The best dev scores always used a non-zero source-side temperature τ_x.
  - TensorFlow and PyTorch code is provided in the appendix.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

93 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 86 not yet.

- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://md.research.elamur.ai/free/en/papers/cheng-2019-robust-neural-machine-translation-doubly/)
- [An Analysis of Simple Data Augmentation for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/dai-2020-analysis-simple-data-augmentation-named/)
- [A Survey of Data Augmentation Approaches for NLP](https://md.research.elamur.ai/free/en/papers/feng-2021-survey-data-augmentation-approaches-nlp/)
- [Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting](https://md.research.elamur.ai/free/en/papers/hu-2019-improved-lexically-constrained-decoding-translation/)
- [Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better](https://md.research.elamur.ai/free/en/papers/liu-2024-does-detectgpt-fully-utilize-perturbation/)
- [Masked Language Model Scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/)
- [Unsupervised Data Augmentation for Consistency Training](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2018switchout,
  title = {{SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation}},
  author = {Wang, Xinyi and Pham, Hieu and Dai, Zihang and Neubig, Graham},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1100},
  url = {https://doi.org/10.18653/v1/d18-1100}
}
```

- [← previous Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised](https://md.research.elamur.ai/free/en/papers/angelidis-2018-summarizing-opinions-aspect-extraction-meets/)
- [next → SzegedKoref: A Hungarian Coreference Corpus](https://md.research.elamur.ai/free/en/papers/vincze-2018-szegedkoref-hungarian-coreference-corpus/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, and more authors. 2016. Deep speech 2: End-to-end speech recognition in english and mandarin. In ICML. [find](https://scholar.google.com/scholar?q=Dario+Amodei%2C+Sundaram+Ananthanarayanan%2C+Rishita+Anubhai%2C+and+more+authors%2E+2016%2E+Deep+speech+2%3A+End%2Dto%2Dend+speech+recognition+in+english+and+mandarin%2E+In+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/)
2. Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. 2016. Reward augmented maximum likelihood for neural structured prediction. In NIPS. [find](https://scholar.google.com/scholar?q=Mohammad+Norouzi%2C+Samy+Bengio%2C+Zhifeng+Chen%2C+Navdeep+Jaitly%2C+Mike+Schuster%2C+Yonghui+Wu%2C+and+Dale+Schuurmans%2E+2016%2E+Reward+augmented+maximum+likelihood+for+neural+structured+prediction%2E+In+NIPS%2E)
3. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In ICLR. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2018%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+In+ICLR%2E)
4. Alberto Poncelas, Dimitar Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018. Investigating backtranslation in neural machine translation. Arxiv, 1804.06189. [find](https://scholar.google.com/scholar?q=Alberto+Poncelas%2C+Dimitar+Shterionov%2C+Andy+Way%2C+Gideon+Maillette+de+Buy+Wenniger%2C+and+Peyman+Passban%2E+2018%2E+Investigating+backtranslation+in+neural+machine+translation%2E+Arxiv%2C+1804%2E06189%2E) [our card](https://md.research.elamur.ai/free/en/papers/poncelas-2018-investigating-backtranslation-neural-machine-translation/)
5. Adam L Berger, Vincent J Della Pietra, and Stephen A Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational linguistics, 22(1):39–71. [find](https://scholar.google.com/scholar?q=Adam+L+Berger%2C+Vincent+J+Della+Pietra%2C+and+Stephen+A+Della+Pietra%2E+1996%2E+A+maximum+entropy+approach+to+natural+language+processing%2E+Computational+linguistics%2C+22%281%29%3A39%E2%80%9371%2E) [our card](https://md.research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/)
6. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In ICLR. [find](https://scholar.google.com/scholar?q=Marc%E2%80%99Aurelio+Ranzato%2C+Sumit+Chopra%2C+Michael+Auli%2C+and+Wojciech+Zaremba%2E+2016%2E+Sequence+level+training+with+recurrent+neural+networks%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/)
7. Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. 2009. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542–542. [find](https://scholar.google.com/scholar?q=Olivier+Chapelle%2C+Bernhard+Scholkopf%2C+and+Alexander+Zien%2E+2009%2E+Semi%2Dsupervised+learning+%28chapelle%2C+o%2E+et+al%2E%2C+eds%2E%3B+2006%29%5Bbook+reviews%5D%2E+IEEE+Transactions+on+Neural+Networks%2C+20%283%29%3A542%E2%80%93542%2E)
8. Jonathan Clark, Chris Dyer, Alon Lavie, and Noah Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In ACL. [find](https://scholar.google.com/scholar?q=Jonathan+Clark%2C+Chris+Dyer%2C+Alon+Lavie%2C+and+Noah+Smith%2E+2011%2E+Better+hypothesis+testing+for+statistical+machine+translation%3A+Controlling+for+optimizer+instability%2E+In+ACL%2E)
9. Terrance DeVries and Graham W. Taylor. 2017. Improved regularization of convolutional neural networks with cutout. Arxiv, 1708.04552. [find](https://scholar.google.com/scholar?q=Terrance+DeVries+and+Graham+W%2E+Taylor%2E+2017%2E+Improved+regularization+of+convolutional+neural+networks+with+cutout%2E+Arxiv%2C+1708%2E04552%2E)
10. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Marzieh+Fadaee%2C+Arianna+Bisazza%2C+and+Christof+Monz%2E+2017%2E+Data+augmentation+for+low%2Dresource+neural+machine+translation%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/fadaee-2017-data-augmentation-low-resource-neural/)
11. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=Yarin+Gal+and+Zoubin+Ghahramani%2E+2016%2E+A+theoretically+grounded+application+of+dropout+in+recurrent+neural+networks%2E+In+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/)
12. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In ICLR. [find](https://scholar.google.com/scholar?q=Jiatao+Gu%2C+James+Bradbury%2C+Caiming+Xiong%2C+Victor+O%2EK%2E+Li%2C+and+Richard+Socher%2E+2018%2E+Non%2Dautoregressive+neural+machine+translation%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/)
13. Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2016. Densely connected convolutional networks. In CVPR. [find](https://scholar.google.com/scholar?q=Gao+Huang%2C+Zhuang+Liu%2C+Laurens+van+der+Maaten%2C+and+Kilian+Q%2E+Weinberger%2E+2016%2E+Densely+connected+convolutional+networks%2E+In+CVPR%2E)
14. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In ICLR. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Lei+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR%2E)
15. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Geoffrey+E%2E+Hinton%2E+2012%2E+Imagenet+classification+with+deep+convolutional+neural+networks%2E+In+NIPS%2E)
16. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford neural machine translation systems for spoken language domain. In IWLST. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong+and+Christopher+D%2E+Manning%2E+2015%2E+Stanford+neural+machine+translation+systems+for+spoken+language+domain%2E+In+IWLST%2E)
17. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D%2E+Manning%2E+2015%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+EMNLP%2E)
18. Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, Graham Neubig, and Eduard Hovy. 2017. Softmax q-distribution estimation for structured prediction: A theoretical interpretation for raml. Arxiv, 1705.07136. [find](https://scholar.google.com/scholar?q=Xuezhe+Ma%2C+Pengcheng+Yin%2C+Jingzhou+Liu%2C+Graham+Neubig%2C+and+Eduard+Hovy%2E+2017%2E+Softmax+q%2Ddistribution+estimation+for+structured+prediction%3A+A+theoretical+interpretation+for+raml%2E+Arxiv%2C+1705%2E07136%2E)
19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. In WMT. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016a%2E+Edinburgh+neural+machine+translation+systems+for+wmt+16%2E+In+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/)
20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving neural machine translation models with monolingual data. In ACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016b%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+ACL%2E)
21. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going deeper with convolutions. In CVPR. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wei+Liu%2C+Yangqing+Jia%2C+Pierre+Sermanet%2C+Scott+Reed%2C+Dragomir+Anguelov%2C+Dumitru+Erhan%2C+Vincent+Vanhoucke%2C+and+Andrew+Rabinovich%2E+2014%2E+Going+deeper+with+convolutions%2E+In+CVPR%2E)
22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NIPS%2E)
23. Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. In BMVC. [find](https://scholar.google.com/scholar?q=Sergey+Zagoruyko+and+Nikos+Komodakis%2E+2016%2E+Wide+residual+networks%2E+In+BMVC%2E)