---
title: "Soft Contextual Data Augmentation for Neural Machine Translation"
description: "Gao et al. propose an alternative. 21 references, cited by 57 papers here. Among them: 115."
authors: ["Fei Gao", "Jinhua Zhu", "Lijun Wu", "Yingce Xia", "Tao Qin", "Xueqi Cheng", "Wengang Zhou", "Tie-Yan Liu"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1555/"
doi: "10.18653/v1/p19-1555"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gao-2019-soft-contextual-data-augmentation-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gao-2019-soft-contextual-data-augmentation-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Soft Contextual Data Augmentation for Neural Machine Translation

# Soft Contextual Data Augmentation for Neural Machine Translation

**Authors:** Fei Gao, Jinhua Zhu, [Lijun Wu](https://md.research.elamur.ai/free/en/authors/lijun-wu/), [Yingce Xia](https://md.research.elamur.ai/free/en/authors/yingce-xia/), [Tao Qin](https://md.research.elamur.ai/free/en/authors/tao-qin/), [Xueqi Cheng](https://md.research.elamur.ai/free/en/authors/xueqi-cheng/), Wengang Zhou and [Tie-Yan Liu](https://md.research.elamur.ai/free/en/authors/tie-yan-liu/) · **Year:** 2019 · **Venue:** [Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2019](https://md.research.elamur.ai/free/en/venues/acl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1555/) · [PDF](https://aclanthology.org/P19-1555.pdf) · [DOI 10.18653/v1/p19-1555](https://doi.org/10.18653/v1/p19-1555) · [OpenAlex](https://openalex.org/W2946068894)

**Citations:** 115 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/gao-2019-soft-contextual-data-augmentation-neural/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Data augmentation is routine in computer vision but uncommon in neural machine translation. Simple text perturbations such as swapping, dropping or blanking words often distort meaning. Contextual replacement with language-model samples keeps meaning better, but it covers only a small part of the possible variations unless it is repeated many times. Gao et al. propose an alternative. Instead of substituting a discrete token, a randomly selected word, chosen with probability γ, gets a "soft" representation. This is the probability distribution over the vocabulary that a pretrained Transformer language model predicts from the preceding words. The word's input embedding then becomes the weighted average of all vocabulary embeddings under that distribution. Separate source and target language models are trained only on the bilingual training corpus and kept frozen. They feed both the encoder and decoder of a Transformer NMT system, and no extra monolingual data is needed. On three IWSLT2014 tasks and WMT2014 En→De, the method outperformed every baseline augmentation scheme. This suggests that blending many plausible, context-aware substitutes is a cheap and robust way to regularize NMT.

## Key points
- **Benchmarks:** IWSLT2014 De→En (160k pairs), Es→En (181k) and He→En (151k), each with a joint 10K BPE vocabulary, plus WMT2014 En→De (4.5M pairs, 40k BPE). Quality was measured with case-sensitive BLEU.
- **BLEU gains over the Transformer baseline:**
  - De→En: 34.79 → 35.78
  - Es→En: 41.58 → 42.61
  - He→En: 33.64 → 34.91
  - En→De: 28.40 → 29.70 (+1.3)
- **Baselines beaten:** Swap, Dropout, Blank, Smooth and LM sample (Kobayashi, 2018). The soft method was the best on every task.
- **Replacement probability:** On De→En the best γ was 0.15. The method kept its gains across a wide range of γ values, whereas competing methods could fall below the baseline when γ > 0.15.
- **Code:** Available at github.com/teslacool/SCA. The authors plan to extend the method to monolingual data and to other tasks such as summarization.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft Research Asia](https://md.research.elamur.ai/free/en/companies/microsoft-research-asia/) — Company, Beijing, China · 54 works · 59,921 citations
- [University of Science and Technology of China](https://md.research.elamur.ai/free/en/universities/university-of-science-and-technology-of-china/) — University, Hefei, China · 26 works · 111,012 citations
- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 68 works · 22,753 citations
- [Sun Yat-sen University](https://md.research.elamur.ai/free/en/universities/sun-yat-sen-university/) — University, Guangzhou, China · 23 works · 8,980 citations
- [Institute of Computing Technology](https://md.research.elamur.ai/free/en/labs/institute-of-computing-technology/) — Research institute, Beijing, China · 18 works · 3,078 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou and Tie-Yan Liu. 2019. Soft Contextual Data Augmentation for Neural Machine Translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gao2019soft,
  title = {{Soft Contextual Data Augmentation for Neural Machine Translation}},
  author = {Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou and Tie-Yan Liu},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1555},
  url = {https://doi.org/10.18653/v1/p19-1555}
}
```

- [← previous Seeing Things from a Different Angle:Discovering Diverse Perspectives about Claims](https://md.research.elamur.ai/free/en/papers/chen-2019-seeing-things-different-angle-discovering/)
- [next → Understanding Undesirable Word Embedding Associations](https://md.research.elamur.ai/free/en/papers/ethayarajh-2019-understanding-undesirable-word-embedding-associations/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

21 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. [Unsupervised neural machine translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/). arXiv preprint arXiv:1710.11041.
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473.
3. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. [Semi-supervised learning for neural machine translation](https://md.research.elamur.ai/free/en/papers/cheng-2016-semi-supervised-learning-neural-machine/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1965–1974.
4. Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. 2018. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501. — [arXiv:1805.09501](https://arxiv.org/abs/1805.09501) [find](https://scholar.google.com/scholar?q=Ekin+D+Cubuk%2C+Barret+Zoph%2C+Dandelion+Mane%2C+Vijay+Vasudevan%2C+and+Quoc+V+Le%2E+2018%2E+Autoaugment%3A+Learning+augmentation+policies+from+data%2E+arXiv+preprint+arXiv%3A1805%2E09501%2E+%E2%80%94+%5BarXiv%3A1805%2E09501%5D%28https%3A%2F%2Far)
5. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. [Data augmentation for low-resource neural machine translation](https://md.research.elamur.ai/free/en/papers/fadaee-2017-data-augmentation-low-resource-neural/). arXiv preprint arXiv:1705.00440.
6. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. [Convolutional Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). In Proc. of ICML.
7. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. [On using monolingual corpora in neural machine translation](https://md.research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/). arXiv preprint arXiv:1503.03535.
8. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820–828. [find](https://scholar.google.com/scholar?q=Di+He%2C+Yingce+Xia%2C+Tao+Qin%2C+Liwei+Wang%2C+Nenghai+Yu%2C+Tie%2DYan+Liu%2C+and+Wei%2DYing+Ma%2E+2016%2E+Dual+learning+for+machine+translation%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+820%E2%80%93828%2E)
9. Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. [Iterative back-translation for neural machine translation](https://md.research.elamur.ai/free/en/papers/hoang-2018-iterative-back-translation-neural-machine/). In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 18–24.
10. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. [Deep unordered composition rivals syntactic methods for text classification](https://md.research.elamur.ai/free/en/papers/iyyer-2015-deep-unordered-composition-rivals-syntactic/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural L…
11. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
12. Sosuke Kobayashi. 2018. [Contextual augmentation: Data augmentation by words with paradigmatic relations](https://md.research.elamur.ai/free/en/papers/kobayashi-2018-contextual-augmentation-data-augmentation-words/). arXiv preprint arXiv:1805.06201.
13. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Geoffrey+E+Hinton%2E+2012%2E+Imagenet+classification+with+deep+convolutional+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+1097%E2%80%931105%2E)
14. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. [Unsupervised machine translation using monolingual corpora only](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/). arXiv preprint arXiv:1711.00043.
15. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015a%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+arXiv+preprint+arXiv%3A1511%2E06709%2E)
16. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). arXiv preprint arXiv:1508.07909.
17. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3104%E2%80%933112%2E)
18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pages 5998–6008.
19. Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. [Switchout: an efficient data augmentation algorithm for neural machine translation](https://md.research.elamur.ai/free/en/papers/wang-2018-switchout-efficient-data-augmentation-algorithm/). arXiv preprint arXiv:1808.07512.
20. Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2018. Conditional bert contextual augmentation. arXiv preprint arXiv:1812.06705. — [arXiv:1812.06705](https://arxiv.org/abs/1812.06705) [find](https://scholar.google.com/scholar?q=Xing+Wu%2C+Shangwen+Lv%2C+Liangjun+Zang%2C+Jizhong+Han%2C+and+Songlin+Hu%2E+2018%2E+Conditional+bert+contextual+augmentation%2E+arXiv+preprint+arXiv%3A1812%2E06705%2E+%E2%80%94+%5BarXiv%3A1812%2E06705%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1812%2E06705)
21. Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. 2017. Data noising as smoothing in neural network language models. arXiv preprint arXiv:1703.02573. [find](https://scholar.google.com/scholar?q=Ziang+Xie%2C+Sida+I+Wang%2C+Jiwei+Li%2C+Daniel+Le%CC%81vy%2C+Aiming+Nie%2C+Dan+Jurafsky%2C+and+Andrew+Y+Ng%2E+2017%2E+Data+noising+as+smoothing+in+neural+network+language+models%2E+arXiv+preprint+arXiv%3A1703%2E02573%2E)

*Updated 24 October 2026*