---
title: "HotFlip: White-Box Adversarial Examples for — Ebrahimi 2018"
description: "Character-level neural text classifiers can be fooled by tiny, nearly invisible edits, yet earlier NLP attacks mostly used black-box heuristics or random…"
authors: ["Javid Ebrahimi", "Anyi Rao", "Daniel Lowd", "Dejing Dou"]
year: 2018
venue: "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/P18-2006/"
doi: "10.18653/v1/p18-2006"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ebrahimi-2018-hotflip-white-box-adversarial-examples/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ebrahimi-2018-hotflip-white-box-adversarial-examples/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / HotFlip: White-Box Adversarial Examples for Text Classification

# HotFlip: White-Box Adversarial Examples for Text Classification

**Authors:** Javid Ebrahimi, Anyi Rao, Daniel Lowd, [Dejing Dou](https://md.research.elamur.ai/free/en/authors/dejing-dou/) · **Year:** 2018 · **Venue:** Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P18-2006/) · [PDF](https://aclanthology.org/P18-2006.pdf) · [DOI 10.18653/v1/p18-2006](https://doi.org/10.18653/v1/p18-2006) · [OpenAlex](https://openalex.org/W2777353073)

**Citations:** 31 citations (openalex-snapshot · 2026-10-14) · [326](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Character-level neural text classifiers can be fooled by tiny, nearly invisible edits, yet earlier NLP attacks mostly used black-box heuristics or random noise. This paper introduces HotFlip, a white-box attack. It treats each character edit as a direction in one-hot input space. Substitutions are handled directly, and insertions and deletions are expressed as chains of substitutions. One forward pass and one backward pass give first-order estimates of how much each possible edit would raise the loss. A beam search then combines several edits. On AG's news with a CharCNN-LSTM classifier, the attack succeeded on more than 90% of attacked instances even when it had to cause a misclassification at 0.9 confidence. At 0.5 confidence it changed only about 4% of characters, and crowd workers seldom saw any change in meaning. Because the examples are cheap to generate, the authors also use them for adversarial training. This lowers clean test error and sharply reduces attack success. A constrained word-level variant, tested on a sentiment CNN trained on SST, works but finds few valid examples. The paper offers an early, efficient gradient-based tool for probing and hardening NLP models.

## Key points
- **Data and model:** The main experiments use AG's news, with 120,000 training and 7,600 test texts in four classes. The target is a CharCNN-LSTM with two stacked LSTM layers of 500 units each. The attacker uses beam width 10 and may change at most 10% of characters.
- **Cost:** Beam search needs O(br) forward and backward passes, where b is the beam width and r is the edit budget. A naive approach that measures the loss of every possible change needs O(brL|V|) queries. Flips make up about 80% of the chosen operations, and deletions are used twice as often as insertions.
- **Adversarial training (Table 2):** Training with HotFlip examples lowers clean error from 8.27% to 7.65%. It also lowers attack success from 98.16% to 69.32%. This beats the embedding-noise method of Miyato et al. (2017) and training with black-box examples.
- **Human check:** 600 Mechanical Turk items were annotated, half of them adversarial. Median human accuracy fell only from 87.49% to 85.71%.
- **Word level:** Against Kim's CNN on SST, the attack is restricted to swaps with cosine similarity above 0.8 and the same part of speech, and it may not replace stop-words. Under these rules it produced only 41 adversarial examples, about 2% of the correctly classified test instances.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

326 papers of our library cite it (of the 140673 whose bibliographies we read); 29 of them have a page here, 297 not yet.

- [Generating Natural Language Adversarial Examples](https://md.research.elamur.ai/free/en/papers/alzantot-2018-generating-natural-language-adversarial-examples/)
- [Analysis Methods in Neural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://md.research.elamur.ai/free/en/papers/chen-2023-empirical-survey-data-augmentation-limited/)
- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://md.research.elamur.ai/free/en/papers/cheng-2019-robust-neural-machine-translation-doubly/)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://md.research.elamur.ai/free/en/papers/dathathri-2019-plug-play-language-models-simple/)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://md.research.elamur.ai/free/en/papers/dinan-2019-build-it-break-it-fix/)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://md.research.elamur.ai/free/en/papers/garg-2020-bae-bert-based-adversarial-examples/)
- [Gradient-based Adversarial Attacks against Text Transformers](https://md.research.elamur.ai/free/en/papers/guo-2021-gradient-based-adversarial-attacks-against/)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://md.research.elamur.ai/free/en/papers/han-2020-explaining-black-box-predictions-unveiling/)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://md.research.elamur.ai/free/en/papers/jain-2023-baseline-defenses-adversarial-attacks-against/)
- [Certified Robustness to Adversarial Word Substitutions](https://md.research.elamur.ai/free/en/papers/jia-2019-certified-robustness-adversarial-word-substitutions/)
- [How Can We Know What Language Models Know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
- [Weight Poisoning Attacks on Pretrained Models](https://md.research.elamur.ai/free/en/papers/kurita-2020-weight-poisoning-attacks-pretrained-models/)
- [Contextualized Perturbation for Textual Adversarial Attack](https://md.research.elamur.ai/free/en/papers/li-2021-contextualized-perturbation-textual-adversarial-attack/)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://md.research.elamur.ai/free/en/papers/li-2024-llms-judges-comprehensive-survey-llm/)
- [Diff4TST: Masked Diffusion Language Model for Text Style Transfer](https://md.research.elamur.ai/free/en/papers/ma-2026-diff4tst-masked-diffusion-language-model/)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://md.research.elamur.ai/free/en/papers/mehrotra-2023-tree-attacks-jailbreaking-black-box/)
- [TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP](https://md.research.elamur.ai/free/en/papers/morris-2020-textattack-framework-adversarial-attacks-data/)
- [Red Teaming Language Models with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/)
- [Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer](https://md.research.elamur.ai/free/en/papers/qi-2021-mind-style-text-adversarial-backdoor/)
- [Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency](https://md.research.elamur.ai/free/en/papers/ren-2019-generating-natural-language-adversarial-examples/)
- [Word-level Textual Adversarial Attacking as Combinatorial Optimization](https://md.research.elamur.ai/free/en/papers/zang-2020-word-level-textual-adversarial-attacking/)
- [中文对抗攻击下的ChatGPT鲁棒性评估](https://md.research.elamur.ai/free/en/papers/zhang-2025-robustness-evaluation-chatgpt/)
- [Evaluating and Enhancing the Robustness of Neural Network-based Dependency Parsing Models with Adversarial Examples](https://md.research.elamur.ai/free/en/papers/zheng-2020-evaluating-enhancing-robustness-neural-network/)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/zhu-2019-freelb-enhanced-adversarial-training-natural/)
- [VALUE: Understanding Dialect Disparity in NLU](https://md.research.elamur.ai/free/en/papers/ziems-2022-value-understanding-dialect-disparity-nlu/)
- [SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space](https://md.research.elamur.ai/free/en/papers/zinkovich-2026-sparta-evaluating-reasoning-segmentation-robustness/)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ebrahimi2018hotflip,
  title = {{HotFlip: White-Box Adversarial Examples for Text Classification}},
  author = {Ebrahimi, Javid and Rao, Anyi and Lowd, Daniel and Dou, Dejing},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2006},
  url = {https://doi.org/10.18653/v1/p18-2006}
}
```

- [← previous Higher-Order Coreference Resolution with Coarse-to-Fine Inference](https://md.research.elamur.ai/free/en/papers/lee-2018-higher-order-coreference-resolution-coarse/)
- [next → HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pages 1631–1642. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+)
2. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2018%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+In+Proceedings+of+ICLR%2E)
3. Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on, pages 39–57. IEEE. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini+and+David+Wagner%2E+2017%2E+Towards+evaluating+the+robustness+of+neural+networks%2E+In+Security+and+Privacy+%28SP%29%2C+2017+IEEE+Symposium+on%2C+pages+39%E2%80%9357%2E+IEEE%2E)
4. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wojciech+Zaremba%2C+Ilya+Sutskever%2C+Joan+Bruna%2C+Dumitru+Erhan%2C+Ian+Goodfellow%2C+and+Rob+Fergus%2E+2014%2E+Intriguing+properties+of+neural+networks%2E+In+Proceedings+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/szegedy-2013-intriguing-properties-neural-networks/)
5. Alexis Conneau, Holger Schwenk, Loı̈c Barrault, and Yann Lecun. 2017. Very deep convolutional networks for text classification. In Proceedings of EACL, volume 1, pages 1107–1116. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Holger+Schwenk%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Yann+Lecun%2E+2017%2E+Very+deep+convolutional+networks+for+text+classification%2E+In+Proceedings+of+EACL%2C+volume+1%2C+pages+1107%E2%80%931116%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-very-deep-convolutional-networks-text/)
6. Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Zhengli+Zhao%2C+Dheeru+Dua%2C+and+Sameer+Singh%2E+2018%2E+Generating+natural+adversarial+examples%2E+In+Proceedings+of+ICLR%2E)
7. Nilesh Dalvi, Pedro Domingos, Sumit Sanghai, Deepak Verma, et al. 2004. Adversarial classification. In Proceedings of KDD, pages 99–108. [find](https://scholar.google.com/scholar?q=Nilesh+Dalvi%2C+Pedro+Domingos%2C+Sumit+Sanghai%2C+Deepak+Verma%2C+et+al%2E+2004%2E+Adversarial+classification%2E+In+Proceedings+of+KDD%2C+pages+99%E2%80%93108%2E)
8. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Ian+J+Goodfellow%2C+Jonathon+Shlens%2C+and+Christian+Szegedy%2E+2015%2E+Explaining+and+harnessing+adversarial+examples%2E+In+Proceedings+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/)
9. Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google’s perspective api built for detecting toxic comments. arXiv preprint arXiv:1702.08138. — [arXiv:1702.08138](https://arxiv.org/abs/1702.08138) [find](https://scholar.google.com/scholar?q=Hossein+Hosseini%2C+Sreeram+Kannan%2C+Baosen+Zhang%2C+and+Radha+Poovendran%2E+2017%2E+Deceiving+google%E2%80%99s+perspective+api+built+for+detecting+toxic+comments%2E+arXiv+preprint+arXiv%3A1702%2E08138%2E+%E2%80%94+%5BarXiv%3A1702%2E08138%5D)
10. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. arXiv preprint arXiv:1804.06059. [find](https://scholar.google.com/scholar?q=Mohit+Iyyer%2C+John+Wieting%2C+Kevin+Gimpel%2C+and+Luke+Zettlemoyer%2E+2018%2E+Adversarial+example+generation+with+syntactically+controlled+paraphrase+networks%2E+arXiv+preprint+arXiv%3A1804%2E06059%2E) [our card](https://md.research.elamur.ai/free/en/papers/iyyer-2018-adversarial-example-generation-syntactically-controlled/)
11. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Robin+Jia+and+Percy+Liang%2E+2017%2E+Adversarial+examples+for+evaluating+reading+comprehension+systems%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Yoon+Kim%2E+2014%2E+Convolutional+neural+networks+for+sentence+classification%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)
13. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Yoon+Kim%2C+Yacine+Jernite%2C+David+Sontag%2C+and+Alexander+M+Rush%2E+2016%2E+Character%2Daware+neural+language+models%2E+Proceedings+of+AAAI%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/)
14. Daniel Lowd and Christopher Meek. 2005. Adversarial learning. In Proceedings of KDD, pages 641–647. [find](https://scholar.google.com/scholar?q=Daniel+Lowd+and+Christopher+Meek%2E+2005%2E+Adversarial+learning%2E+In+Proceedings+of+KDD%2C+pages+641%E2%80%93647%2E)
15. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Aleksander+Madry%2C+Aleksandar+Makelov%2C+Ludwig+Schmidt%2C+Dimitris+Tsipras%2C+and+Adrian+Vladu%2E+2018%2E+Towards+deep+learning+models+resistant+to+adversarial+attacks%2E+In+Proceedings+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/madry-2017-towards-deep-learning-models-resistant/)
16. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Takeru+Miyato%2C+Andrew+M+Dai%2C+and+Ian+Goodfellow%2E+2017%2E+Adversarial+training+methods+for+semi%2Dsupervised+text+classification%2E+In+Proceedings+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/miyato-2016-adversarial-training-methods-semi-supervised/)
17. Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. 2016. The limitations of deep learning in adversarial settings. In Security and Privacy (EuroS&P), 2016 IEEE European Symposium on, pages 372– 387. IEEE. [find](https://scholar.google.com/scholar?q=Nicolas+Papernot%2C+Patrick+McDaniel%2C+Somesh+Jha%2C+Matt+Fredrikson%2C+Z+Berkay+Celik%2C+and+Ananthram+Swami%2E+2016%2E+The+limitations+of+deep+learning+in+adversarial+settings%2E+In+Security+and+Privacy+%28EuroS%26P%29%2C)
18. Graham Ernest Rawlinson. 1976. The Significance of Letter Position in Word Recognition. Ph.D. thesis, University of Nottingham. [find](https://scholar.google.com/scholar?q=Graham+Ernest+Rawlinson%2E+1976%2E+The+Significance+of+Letter+Position+in+Word+Recognition%2E+Ph%2ED%2E+thesis%2C+University+of+Nottingham%2E)