---
title: "BAE: BERT-based Adversarial Examples for Text — Garg 2020"
description: "Text classifiers can be fooled by small edits that humans barely notice."
authors: ["Siddhant Garg", "Goutham Ramakrishnan"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.498/"
doi: "10.18653/v1/2020.emnlp-main.498"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/garg-2020-bae-bert-based-adversarial-examples/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/garg-2020-bae-bert-based-adversarial-examples/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / BAE: BERT-based Adversarial Examples for Text Classification

# BAE: BERT-based Adversarial Examples for Text Classification

**Authors:** Siddhant Garg, Goutham Ramakrishnan · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.498/) · [PDF](https://aclanthology.org/2020.emnlp-main.498.pdf) · [DOI 10.18653/v1/2020.emnlp-main.498](https://doi.org/10.18653/v1/2020.emnlp-main.498) · [OpenAlex](https://openalex.org/W3014153848)

**Citations:** 478 citations (openalex-snapshot · 2026-10-14) · [187](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Text classifiers can be fooled by small edits that humans barely notice. Earlier attacks, such as TextFooler, swap words for synonyms from a fixed embedding space. They ignore sentence context, so the substitutions are often awkward or out of place. Garg and Ramakrishnan propose BAE, a soft-label black-box attack that uses the BERT masked language model to produce context-aware perturbations.

BAE ranks tokens by how much deleting each one lowers the probability of the correct label. It then either masks a token (replace) or adds a mask next to it (insert), and lets BERT-MLM propose candidates. Candidates are filtered by Universal Sentence Encoder similarity, and replacements must also match the original token's part of speech. The paper defines four modes: BAE-R, BAE-I, BAE-R/I and BAE-R+I.

The authors attacked word-LSTM, word-CNN and fine-tuned BERT on seven datasets. BAE nearly always beat TextFooler, cutting accuracy by roughly 40–80% while keeping USE similarity higher. In human evaluation, BAE outputs were judged more natural and better at preserving sentiment. This makes BAE a stronger and more realistic tool for testing NLP robustness.

## Key points
- Datasets: Amazon, Yelp, IMDB, MR (sentiment), MPQA (opinion polarity), Subj (subjectivity) and TREC (question type). The victim models were word-LSTM, word-CNN and BERT base-uncased.
- Settings: the top K=50 BERT-MLM candidates are kept, with a USE cosine-similarity threshold of 0.8. TextFooler serves as the baseline.
- BAE-R+I was the strongest mode in all but one case. For example, it lowered word-LSTM accuracy on Amazon from 88.0% to 4.0% (TextFooler reached 31.0%). It also cut BERT accuracy on TREC from 97.6% to 20.2%. BERT was more robust overall than word-LSTM and word-CNN.
- Human evaluation used 3 annotators on Amazon, IMDB and MR against the BERT classifier. Naturalness (Likert 1–5) was about 3.84–3.91 for BAE-R versus 3.17–3.41 for TextFooler. Insertions in BAE-R+I strengthened the attack but slightly reduced naturalness, even though its USE scores were higher. This exposes the limits of USE as a measure of meaning preservation.
- An ablation showed that insert operations matter more than replace operations for successful attacks. BAE is available in the TextAttack library.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

187 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 178 not yet.

- [Evaluation of Adversarial Robustness in Arabic Language Models](https://md.research.elamur.ai/free/en/papers/alajmi-2026-evaluation-adversarial-robustness-arabic-language/)
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://md.research.elamur.ai/free/en/papers/chen-2023-empirical-survey-data-augmentation-limited/)
- [Gradient-based Adversarial Attacks against Text Transformers](https://md.research.elamur.ai/free/en/papers/guo-2021-gradient-based-adversarial-attacks-against/)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://md.research.elamur.ai/free/en/papers/hedderich-2021-survey-recent-approaches-natural-language/)
- [Contextualized Perturbation for Textual Adversarial Attack](https://md.research.elamur.ai/free/en/papers/li-2021-contextualized-perturbation-textual-adversarial-attack/)
- [TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP](https://md.research.elamur.ai/free/en/papers/morris-2020-textattack-framework-adversarial-attacks-data/)
- [中文对抗攻击下的ChatGPT鲁棒性评估](https://md.research.elamur.ai/free/en/papers/zhang-2025-robustness-evaluation-chatgpt/)
- [Explainability for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{garg2020bae,
  title = {{BAE: BERT-based Adversarial Examples for Text Classification}},
  author = {Garg, Siddhant and Ramakrishnan, Goutham},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.498},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.498}
}
```

- [← previous BabyFST - Towards a Finite-State Based Computational Model of Ancient Babylonian](https://md.research.elamur.ai/free/en/papers/lingbulk-2020-sahala-babyfst-finite-state-based-computational/)
- [next → BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

27 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Moustafa+Alzantot%2C+Yash+Sharma%2C+Ahmed+Elgohary%2C+Bo%2DJhang+Ho%2C+Mani+Srivastava%2C+and+Kai%2DWei+Chang%2E+2018%2E+Generating+natural+language+adversarial+examples%2E+In+Proceedings+of+the+2018+Conference+on+Empiri) [our card](https://md.research.elamur.ai/free/en/papers/alzantot-2018-generating-natural-language-adversarial-examples/)
2. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Yinfei+Yang%2C+Sheng%2Dyi+Kong%2C+Nan+Hua%2C+Nicole+Limtiaco%2C+Rhomni+St%2E+John%2C+Noah+Constant%2C+Mario+Guajardo%2DCespedes%2C+Steve+Yuan%2C+Chris+Tar%2C+Yun%2DHsuan+Sung%2C+Brian+Strope%2C+and+Ray+Kurzweil%2E+2018%2E+) [our card](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
4. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 31–36, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Javid+Ebrahimi%2C+Anyi+Rao%2C+Daniel+Lowd%2C+and+Dejing+Dou%2E+2018%2E+HotFlip%3A+White%2Dbox+adversarial+examples+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computati) [our card](https://md.research.elamur.ai/free/en/papers/ebrahimi-2018-hotflip-white-box-adversarial-examples/)
5. Shi Feng, Eric Wallace, Mohit Iyyer, Pedro Rodriguez, Alvin Grissom II, and Jordan L. Boyd-Graber. 2018. Right answer for the wrong reason: Discovery and mitigation. CoRR, abs/1804.07781. [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+Mohit+Iyyer%2C+Pedro+Rodriguez%2C+Alvin+Grissom+II%2C+and+Jordan+L%2E+Boyd%2DGraber%2E+2018%2E+Right+answer+for+the+wrong+reason%3A+Discovery+and+mitigation%2E+CoRR%2C+abs%2F1804%2E07781%2E)
6. Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. CoRR, abs/1801.04354. [find](https://scholar.google.com/scholar?q=Ji+Gao%2C+Jack+Lanchantin%2C+Mary+Lou+Soffa%2C+and+Yanjun+Qi%2E+2018%2E+Black%2Dbox+generation+of+adversarial+text+sequences+to+evade+deep+learning+classifiers%2E+CoRR%2C+abs%2F1801%2E04354%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2018-black-box-generation-adversarial-text/)
7. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Jonathon+Shlens%2C+and+Christian+Szegedy%2E+2015%2E+Explaining+and+harnessing+adversarial+examples%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/)
8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Com., 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Com%2E%2C+9%288%29%3A1735%E2%80%931780%2E)
9. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is bert really robust? natural language attack on text classification and entailment. arXiv preprint arXiv:1907.11932. [find](https://scholar.google.com/scholar?q=Di+Jin%2C+Zhijing+Jin%2C+Joey+Tianyi+Zhou%2C+and+Peter+Szolovits%2E+2019%2E+Is+bert+really+robust%3F+natural+language+attack+on+text+classification+and+entailment%2E+arXiv+preprint+arXiv%3A1907%2E11932%2E) [our card](https://md.research.elamur.ai/free/en/papers/jin-2019-bert-really-robust-strong-baseline/)
10. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1746–1751. [find](https://scholar.google.com/scholar?q=Yoon+Kim%2E+2014%2E+Convolutional+neural+networks+for+sentence+classification%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1746%E2%80%931751%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)
11. Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2017. Adversarial examples in the physical world. ICLR Workshop. [find](https://scholar.google.com/scholar?q=Alexey+Kurakin%2C+Ian+Goodfellow%2C+and+Samy+Bengio%2E+2017%2E+Adversarial+examples+in+the+physical+world%2E+ICLR+Workshop%2E)
12. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. CoRR, abs/1612.08220. — [arXiv:1612.08220](https://arxiv.org/abs/1612.08220) [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+2016%2E+Understanding+neural+networks+through+representation+erasure%2E+CoRR%2C+abs%2F1612%2E08220%2E+%E2%80%94+%5BarXiv%3A1612%2E08220%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1612%2E08220%29)
13. Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings of the 19th International Conference on Computational Linguistics - Volume 1, COLING ’02, pages 1–7, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xin+Li+and+Dan+Roth%2E+2002%2E+Learning+question+classifiers%2E+In+Proceedings+of+the+19th+International+Conference+on+Computational+Linguistics+%2D+Volume+1%2C+COLING+%E2%80%9902%2C+pages+1%E2%80%937%2C+Stroudsburg%2C+PA%2C+USA%2E+Asso) [our card](https://md.research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/)
14. Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2017. Deep text classification can be fooled. CoRR, abs/1704.08006. — [arXiv:1704.08006](https://arxiv.org/abs/1704.08006) [find](https://scholar.google.com/scholar?q=Bin+Liang%2C+Hongcheng+Li%2C+Miaoqiang+Su%2C+Pan+Bian%2C+Xirong+Li%2C+and+Wenchang+Shi%2E+2017%2E+Deep+text+classification+can+be+fooled%2E+CoRR%2C+abs%2F1704%2E08006%2E+%E2%80%94+%5BarXiv%3A1704%2E08006%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1704%2E08006%29)
15. John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. [find](https://scholar.google.com/scholar?q=John+X%2E+Morris%2C+Eli+Lifland%2C+Jin+Yong+Yoo%2C+Jake+Grigsby%2C+Di+Jin%2C+and+Yanjun+Qi%2E+2020%2E+Textattack%3A+A+framework+for+adversarial+attacks%2C+data+augmentation%2C+and+adversarial+training+in+nlp%2E) [our card](https://md.research.elamur.ai/free/en/papers/morris-2020-textattack-framework-adversarial-attacks-data/)
16. Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017. Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Nikola+Mrks%CC%8Cic%CC%81%2C+Ivan+Vulic%CC%81%2C+Diarmuid+O%CC%81+Se%CC%81aghdha%2C+Ira+Leviant%2C+Roi+Reichart%2C+Milica+Gas%CC%8Cic%CC%81%2C+Anna+Korhonen%2C+and+Steve+Young%2E+2017%2E+Semantic+specialization+of+distributional+word+vector+spaces+using)
17. Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of HLT-NAACL. [find](https://scholar.google.com/scholar?q=Nikola+Mrks%CC%8Cic%CC%81%2C+Diarmuid+O%CC%81+Se%CC%81aghdha%2C+Blaise+Thomson%2C+Milica+Gas%CC%8Cic%CC%81%2C+Lina+Rojas%2DBarahona%2C+Pei%2DHao+Su%2C+David+Vandyke%2C+Tsung%2DHsien+Wen%2C+and+Steve+Young%2E+2016%2E+Counter%2Dfitting+word+vectors+to+linguist)
18. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity. In Proceedings of ACL, pages 271–278. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2004%2E+A+sentimental+education%3A+Sentiment+analysis+using+subjectivity%2E+In+Proceedings+of+ACL%2C+pages+271%E2%80%93278%2E)
19. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2005%2E+Seeing+stars%3A+Exploiting+class+relationships+for+sentiment+categorization+with+respect+to+rating+scales%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/pang-2005-seeing-stars-exploiting-class-relationships/)
20. Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. 2017. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, ASIA CCS ’17, pages 506–519, New York, NY, USA.… [find](https://scholar.google.com/scholar?q=Nicolas+Papernot%2C+Patrick+McDaniel%2C+Ian+Goodfellow%2C+Somesh+Jha%2C+Z%2E+Berkay+Celik%2C+and+Ananthram+Swami%2E+2017%2E+Practical+black%2Dbox+attacks+against+machine+learning%2E+In+Proceedings+of+the+2017+ACM+on+Asia)
21. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
22. Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shuhuai+Ren%2C+Yihe+Deng%2C+Kun+He%2C+and+Wanxiang+Che%2E+2019%2E+Generating+natural+language+adversarial+examples+through+probability+weighted+word+saliency%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+As) [our card](https://md.research.elamur.ai/free/en/papers/ren-2019-generating-natural-language-adversarial-examples/)
23. Prathusha K Sarma, Yingyu Liang, and Bill Sethares. 2018. Domain adapted word embeddings for improved sentiment classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 37–42. [find](https://scholar.google.com/scholar?q=Prathusha+K+Sarma%2C+Yingyu+Liang%2C+and+Bill+Sethares%2E+2018%2E+Domain+adapted+word+embeddings+for+improved+sentiment+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computa)
24. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wojciech+Zaremba%2C+Ilya+Sutskever%2C+Joan+Bruna%2C+Dumitru+Erhan%2C+Ian+Goodfellow%2C+and+Rob+Fergus%2E+2014%2E+Intriguing+properties+of+neural+networks%2E+In+International+Conference+on+Learning+) [our card](https://md.research.elamur.ai/free/en/papers/szegedy-2013-intriguing-properties-neural-networks/)
25. Janyce Wiebe and Theresa Wilson. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=Janyce+Wiebe+and+Theresa+Wilson%2E+2005%2E+Annotating+expressions+of+opinions+and+emotions+in+language%2E+Language+Resources+and+Evaluation%2E)
26. Wei Emma Zhang, Quan Z. Sheng, and Ahoud Abdul-rahmn F. Alhazmi. 2019. Generating textual adversarial examples for deep learning models: A survey. CoRR, abs/1901.06796. — [arXiv:1901.06796](https://arxiv.org/abs/1901.06796) [find](https://scholar.google.com/scholar?q=Wei+Emma+Zhang%2C+Quan+Z%2E+Sheng%2C+and+Ahoud+Abdul%2Drahmn+F%2E+Alhazmi%2E+2019%2E+Generating+textual+adversarial+examples+for+deep+learning+models%3A+A+survey%2E+CoRR%2C+abs%2F1901%2E06796%2E+%E2%80%94+%5BarXiv%3A1901%2E06796%5D%28https%3A%2F%2Far)
27. Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhengli+Zhao%2C+Dheeru+Dua%2C+and+Sameer+Singh%2E+2018%2E+Generating+natural+adversarial+examples%2E+In+International+Conference+on+Learning+Representations%2E)