---
title: "Contextual Augmentation — Kobayashi 2018"
description: "Text classifiers often overfit, and NLP has few general augmentation tools."
authors: ["Sosuke Kobayashi"]
year: 2018
venue: "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)"
source: "https://aclanthology.org/N18-2072/"
doi: "10.18653/v1/n18-2072"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kobayashi-2018-contextual-augmentation-data-augmentation-words/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kobayashi-2018-contextual-augmentation-data-augmentation-words/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations

# Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations

**Authors:** Sosuke Kobayashi · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N18-2072/) · [PDF](https://aclanthology.org/N18-2072.pdf) · [DOI 10.18653/v1/n18-2072](https://doi.org/10.18653/v1/n18-2072) · [OpenAlex](https://openalex.org/W2963545917)

**Citations:** 603 citations (openalex-snapshot · 2026-10-14) · [201](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Text classifiers often overfit, and NLP has few general augmentation tools. The usual approach swaps words for WordNet synonyms, but true synonyms are rare, so only a small share of the vocabulary can be varied. Kobayashi proposes **contextual augmentation**. A bidirectional LSTM language model looks at the words around a position and proposes substitutes that fit there, words in a paradigmatic relation with the original rather than strict synonyms. Substitutes are sampled fresh at every training update, and a temperature parameter controls how aggressive the replacement is.

Unrestricted context-based swaps can flip a sentence's meaning, for example replacing a positive adjective with a negative one. To prevent this, the LM is extended with a label embedding so its predictions depend on the class label. The LM is pretrained on WikiText-103 and then fine-tuned on each labeled dataset.

On six classification benchmarks, with both CNN and RNN classifiers, the method beats synonym replacement, and the label-conditional variant gives the best average accuracy. The gains are modest, but the approach needs no task-specific rules or hand-built resources.

## Key points
- Evaluated on six datasets: SST5, SST2, Subj, MPQA, RT and TREC. CNN (Kim, 2014) and single-layer LSTM classifiers were used, with results averaged over eight random seeds.
- Average CNN accuracy:
  - baseline: 77.53
  - synonym augmentation: 77.50
  - contextual augmentation: 78.02
  - contextual + label conditioning: 78.20
- Average RNN accuracy:
  - baseline: 77.43
  - synonym augmentation: 77.40
  - contextual augmentation: 77.62
  - contextual + label conditioning: 77.83
- In the label-conditional LM, a positive label steers predictions for "fantastic" toward words like *funny* and *good*. A negative label steers them toward words like *dull* and *bad*. Label-neutral positions, such as "actors", change little.
- The code is written in Chainer and released at github.com/pfnet-research/contextual_augmentation. The author notes that the improvements are sometimes marginal.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

201 papers of our library cite it (of the 140884 whose bibliographies we read); 18 of them have a page here, 183 not yet.

- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://md.research.elamur.ai/free/en/papers/allen-2023-zhu-physics-language-models-part-3/)
- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/cao-2021-cliff-contrastive-learning-improving-faithfulness/)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://md.research.elamur.ai/free/en/papers/chen-2023-empirical-survey-data-augmentation-limited/)
- [An Analysis of Simple Data Augmentation for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/dai-2020-analysis-simple-data-augmentation-named/)
- [AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://md.research.elamur.ai/free/en/papers/dai-2023-auggpt-leveraging-chatgpt-text-data/)
- [A Survey of Data Augmentation Approaches for NLP](https://md.research.elamur.ai/free/en/papers/feng-2021-survey-data-augmentation-approaches-nlp/)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://md.research.elamur.ai/free/en/papers/hedderich-2021-survey-recent-approaches-natural-language/)
- [Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting](https://md.research.elamur.ai/free/en/papers/hu-2019-improved-lexically-constrained-decoding-translation/)
- [An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/konno-2020-empirical-study-contextual-data-augmentation/)
- [Submodular Optimization-based Diverse Paraphrasing and its Effectiveness in Data Augmentation](https://md.research.elamur.ai/free/en/papers/kumar-2019-submodular-optimization-based-diverse-paraphrasing/)
- [Data Augmentation using Pre-trained Transformer Models](https://md.research.elamur.ai/free/en/papers/kumar-2020-data-augmentation-using-pre-trained/)
- [Target-Aware Data Augmentation for Stance Detection](https://md.research.elamur.ai/free/en/papers/li-2021-target-aware-data-augmentation-stance/)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://md.research.elamur.ai/free/en/papers/markov-2022-holistic-approach-undesired-content-detection/)
- [Text and Code Embeddings by Contrastive Pre-Training](https://md.research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://md.research.elamur.ai/free/en/papers/wei-2019-eda-easy-data-augmentation-techniques/)
- [CorefQA: Coreference Resolution as Query-based Span Prediction](https://md.research.elamur.ai/free/en/papers/wu-2020-corefqa-coreference-resolution-as-query/)
- [Generative Data Augmentation for Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/yang-2020-generative-data-augmentation-commonsense-reasoning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kobayashi2018contextual,
  title = {{Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations}},
  author = {Kobayashi, Sosuke},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2072},
  url = {https://doi.org/10.18653/v1/n18-2072}
}
```

- [← previous Context-Aware Neural Machine Translation Learns Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/voita-2018-context-aware-neural-machine-translation/)
- [next → Contextual String Embeddings for Sequence Labeling](https://md.research.elamur.ai/free/en/papers/akbik-2018-contextual-string-embeddings-sequence-labeling/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/)

## References

29 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS, pages 1171–1179. [find](https://scholar.google.com/scholar?q=Samy+Bengio%2C+Oriol+Vinyals%2C+Navdeep+Jaitly%2C+and+Noam+Shazeer%2E+2015%2E+Scheduled+sampling+for+sequence+prediction+with+recurrent+neural+networks%2E+In+NIPS%2C+pages+1171%E2%80%931179%2E)
2. Sosuke Kobayashi, Naoaki Okazaki, and Kentaro Inui. 2017. A neural language model for dynamically representing the meanings of unknown words and entities in a discourse. In IJCNLP, pages 473–483. [find](https://scholar.google.com/scholar?q=Sosuke+Kobayashi%2C+Naoaki+Okazaki%2C+and+Kentaro+Inui%2E+2017%2E+A+neural+language+model+for+dynamically+representing+the+meanings+of+unknown+words+and+entities+in+a+discourse%2E+In+IJCNLP%2C+pages+473%E2%80%93483%2E)
3. Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017. Training data augmentation for low-resource morphological inflection. In CoNLL SIGMORPHON, pages 31–39. [find](https://scholar.google.com/scholar?q=Toms+Bergmanis%2C+Katharina+Kann%2C+Hinrich+Schu%CC%88tze%2C+and+Sharon+Goldwater%2E+2017%2E+Training+data+augmentation+for+low%2Dresource+morphological+inflection%2E+In+CoNLL+SIGMORPHON%2C+pages+31%E2%80%9339%2E)
4. Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, and Kentaro Inui. 2016. Dynamic entity representation with max-pooling improves machine reading. In Proceedings of NAACL-HLT, pages 850–855. [find](https://scholar.google.com/scholar?q=Sosuke+Kobayashi%2C+Ran+Tian%2C+Naoaki+Okazaki%2C+and+Kentaro+Inui%2E+2016%2E+Dynamic+entity+representation+with+max%2Dpooling+improves+machine+reading%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+850%E2%80%93855%2E)
5. Koen Deschacht and Marie-Francine Moens. 2009. Semi-supervised semantic role labeling using the latent words language model. In EMNLP, pages 21– 29. [find](https://scholar.google.com/scholar?q=Koen+Deschacht+and+Marie%2DFrancine+Moens%2E+2009%2E+Semi%2Dsupervised+semantic+role+labeling+using+the+latent+words+language+model%2E+In+EMNLP%2C+pages+21%E2%80%93+29%2E)
6. Oleksandr Kolomiyets, Steven Bethard, and Marie-Francine Moens. 2011. Model-portability experiments for textual temporal analysis. In ACL, pages 271–276. [find](https://scholar.google.com/scholar?q=Oleksandr+Kolomiyets%2C+Steven+Bethard%2C+and+Marie%2DFrancine+Moens%2E+2011%2E+Model%2Dportability+experiments+for+textual+temporal+analysis%2E+In+ACL%2C+pages+271%E2%80%93276%2E)
7. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In ACL, pages 334–343. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Miguel+Ballesteros%2C+Wang+Ling%2C+Austin+Matthews%2C+and+Noah+A%2E+Smith%2E+2015%2E+Transition%2Dbased+dependency+parsing+with+stack+long+short%2Dterm+memory%2E+In+ACL%2C+pages+334%E2%80%93343%2E) [our card](https://md.research.elamur.ai/free/en/papers/missing-2015-dyer-transition-based-dependency-parsing-stack/)
8. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097– 1105. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Geoffrey+E+Hinton%2E+2012%2E+Imagenet+classification+with+deep+convolutional+neural+networks%2E+In+NIPS%2C+pages+1097%E2%80%93+1105%2E)
9. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In ACL, pages 567–573. [find](https://scholar.google.com/scholar?q=Marzieh+Fadaee%2C+Arianna+Bisazza%2C+and+Christof+Monz%2E+2017%2E+Data+augmentation+for+low%2Dresource+neural+machine+translation%2E+In+ACL%2C+pages+567%E2%80%93573%2E) [our card](https://md.research.elamur.ai/free/en/papers/fadaee-2017-data-augmentation-low-resource-neural/)
10. Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING, pages 1–7. [find](https://scholar.google.com/scholar?q=Xin+Li+and+Dan+Roth%2E+2002%2E+Learning+question+classifiers%2E+In+COLING%2C+pages+1%E2%80%937%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/)
11. William Fedus, Ian Goodfellow, and Andrew M. Dai. 2018. MaskGAN: Better text generation via filling in the . In ICLR. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Ian+Goodfellow%2C+and+Andrew+M%2E+Dai%2E+2018%2E+MaskGAN%3A+Better+text+generation+via+filling+in+the+%2E+In+ICLR%2E)
12. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In ICLR. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Caiming+Xiong%2C+James+Bradbury%2C+and+Richard+Socher%2E+2017%2E+Pointer+sentinel+mixture+models%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/)
13. Hagen Fürstenau and Mirella Lapata. 2009. Semi-supervised semantic role labeling. In EACL, pages 220–228. [find](https://scholar.google.com/scholar?q=Hagen+Fu%CC%88rstenau+and+Mirella+Lapata%2E+2009%2E+Semi%2Dsupervised+semantic+role+labeling%2E+In+EACL%2C+pages+220%E2%80%93228%2E)
14. George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41. [find](https://scholar.google.com/scholar?q=George+A%2E+Miller%2E+1995%2E+Wordnet%3A+A+lexical+database+for+english%2E+Commun%2E+ACM%2C+38%2811%29%3A39%E2%80%9341%2E) [our card](https://md.research.elamur.ai/free/en/papers/lex-1994-miller-wordnet/)
15. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS, pages 649–657. [find](https://scholar.google.com/scholar?q=Xiang+Zhang%2C+Junbo+Zhao%2C+and+Yann+LeCun%2E+2015%2E+Character%2Dlevel+convolutional+networks+for+text+classification%2E+In+NIPS%2C+pages+649%E2%80%93657%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)
16. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In ACL. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2004%2E+A+sentimental+education%3A+Sentiment+analysis+using+subjectivity+summarization+based+on+minimum+cuts%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/pang-2004-sentimental-education-sentiment-analysis-subjectivity/)
17. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pages 115– 124. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2005%2E+Seeing+stars%3A+Exploiting+class+relationships+for+sentiment+categorization+with+respect+to+rating+scales%2E+In+ACL%2C+pages+115%E2%80%93+124%2E) [our card](https://md.research.elamur.ai/free/en/papers/pang-2005-seeing-stars-exploiting-class-relationships/)
18. Charles Bally Albert Sechehaye Saussure, Ferdinand de and Albert Riedlinger. 1916. Cours de linguistique generale. Lausanne: Payot. [find](https://scholar.google.com/scholar?q=Charles+Bally+Albert+Sechehaye+Saussure%2C+Ferdinand+de+and+Albert+Riedlinger%2E+1916%2E+Cours+de+linguistique+generale%2E+Lausanne%3A+Payot%2E)
19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL, pages 86–96. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+ACL%2C+pages+86%E2%80%9396%2E)
20. Miikka Silfverberg, Adam Wiemerslage, Ling Liu, and Lingshuang Jack Mao. 2017. Data augmentation for morphological reinflection. In CoNLL SIGMOR-PHON, pages 90–99. [find](https://scholar.google.com/scholar?q=Miikka+Silfverberg%2C+Adam+Wiemerslage%2C+Ling+Liu%2C+and+Lingshuang+Jack+Mao%2E+2017%2E+Data+augmentation+for+morphological+reinflection%2E+In+CoNLL+SIGMOR%2DPHON%2C+pages+90%E2%80%9399%2E) [our card](https://md.research.elamur.ai/free/en/papers/silfverberg-2017-data-augmentation-morphological-reinflection/)
21. Patrice Y. Simard, Yann A. LeCun, John S. Denker, and Bernard Victorri. 1998. Transformation Invariance in Pattern Recognition — Tangent Distance and Tangent Propagation. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Patrice+Y%2E+Simard%2C+Yann+A%2E+LeCun%2C+John+S%2E+Denker%2C+and+Bernard+Victorri%2E+1998%2E+Transformation+Invariance+in+Pattern+Recognition+%E2%80%94+Tangent+Distance+and+Tangent+Propagation%2E+Springer+Berlin+Heidelberg%2E)
22. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In)
23. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NIPS%2C+pages+3104%E2%80%933112%2E)
24. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In CVPR. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wei+Liu%2C+Yangqing+Jia%2C+Pierre+Sermanet%2C+Scott+Reed%2C+Dragomir+Anguelov%2C+Dumitru+Erhan%2C+Vincent+Vanhoucke%2C+and+Andrew+Rabinovich%2E+2015%2E+Going+deeper+with+convolutions%2E+In+CVPR%2E)
25. Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. 2015. Chainer: a next-generation open source framework for deep learning. In Proceedings of Workshop on LearningSys in NIPS 28. [find](https://scholar.google.com/scholar?q=Seiya+Tokui%2C+Kenta+Oono%2C+Shohei+Hido%2C+and+Justin+Clayton%2E+2015%2E+Chainer%3A+a+next%2Dgeneration+open+source+framework+for+deep+learning%2E+In+Proceedings+of+Workshop+on+LearningSys+in+NIPS+28%2E)
26. William Yang Wang and Diyi Yang. 2015. That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using \#petpeeve tweets. In EMNLP, pages 2557–2563. [find](https://scholar.google.com/scholar?q=William+Yang+Wang+and+Diyi+Yang%2E+2015%2E+That%E2%80%99s+so+annoying%21%21%21%3A+A+lexical+and+frame%2Dsemantic+embedding+based+data+augmentation+approach+to+automatic+categorization+of+annoying+behaviors+using+%5C%23petpeeve) [our card](https://md.research.elamur.ai/free/en/papers/wang-2015-that-s-so-annoying-lexical/)
27. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2):165–210. [find](https://scholar.google.com/scholar?q=Janyce+Wiebe%2C+Theresa+Wilson%2C+and+Claire+Cardie%2E+2005%2E+Annotating+expressions+of+opinions+and+emotions+in+language%2E+Language+Resources+and+Evaluation%2C+39%282%29%3A165%E2%80%93210%2E)
28. Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tie-Yan Liu. 2017. Dual supervised learning. In ICML, pages 3789–3798. [find](https://scholar.google.com/scholar?q=Yingce+Xia%2C+Tao+Qin%2C+Wei+Chen%2C+Jiang+Bian%2C+Nenghai+Yu%2C+and+Tie%2DYan+Liu%2E+2017%2E+Dual+supervised+learning%2E+In+ICML%2C+pages+3789%E2%80%933798%2E)
29. Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan. 2017. Variational autoencoder for semi-supervised text classification. In AAAI, pages 3358–3364. [find](https://scholar.google.com/scholar?q=Weidi+Xu%2C+Haoze+Sun%2C+Chao+Deng%2C+and+Ying+Tan%2E+2017%2E+Variational+autoencoder+for+semi%2Dsupervised+text+classification%2E+In+AAAI%2C+pages+3358%E2%80%933364%2E)