---
title: "Achieving Verified Robustness to Symbol Substitutions via"
description: "Neural text classifiers can be fooled by small edits such as swapping a word for a synonym or flipping… 45 references, cited by 50 papers here."
authors: ["Po-Sen Huang", "Robert Stanforth", "Johannes Welbl", "Chris Dyer", "Dani Yogatama", "Sven Gowal", "Krishnamurthy Dvijotham", "Pushmeet Kohli"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1419/"
doi: "10.18653/v1/d19-1419"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/huang-2019-achieving-verified-robustness-symbol-substitutions/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/huang-2019-achieving-verified-robustness-symbol-substitutions/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation

# Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation

**Authors:** [Po-Sen Huang](https://md.research.elamur.ai/free/en/authors/po-sen-huang/), Robert Stanforth, [Johannes Welbl](https://md.research.elamur.ai/free/en/authors/johannes-welbl/), [Chris Dyer](https://md.research.elamur.ai/free/en/authors/chris-dyer/), [Dani Yogatama](https://md.research.elamur.ai/free/en/authors/dani-yogatama/), Sven Gowal, Krishnamurthy Dvijotham and [Pushmeet Kohli](https://md.research.elamur.ai/free/en/authors/pushmeet-kohli/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1419/) · [PDF](https://aclanthology.org/D19-1419.pdf) · [DOI 10.18653/v1/d19-1419](https://doi.org/10.18653/v1/d19-1419) · [OpenAlex](https://openalex.org/W2970449623)

**Citations:** 113 citations (OpenAlex, 2026-10-14) · [50](https://md.research.elamur.ai/free/en/papers/huang-2019-achieving-verified-robustness-symbol-substitutions/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Neural text classifiers can be fooled by small edits such as swapping a word for a synonym or flipping characters. Adversarial training and data augmentation reduce this weakness, but they cannot show that no stronger attack exists. The authors (DeepMind / UCL) instead aim to formally certify robustness. They work with word- and character-level HotFlip-style substitutions on Stanford Sentiment Treebank (SST) and AG News. Their key step is to over-approximate the set of perturbed inputs. They take the convex hull of single-substitution embeddings and scale it up to cover multiple substitutions, which gives a simplex in embedding space. Interval Bound Propagation (IBP) then pushes this simplex through the network. This yields cheap worst-case bounds that also serve as an extra training objective. Models trained this way stay close to normal models in nominal accuracy but are far more often verifiably robust. On both datasets they reach the best exhaustively verified accuracy. Exhaustive search, by contrast, shows that adversarial training gives a false sense of security. The paper is presented as the first application of verification and verifiable training to NLP.

## Key points
- The perturbation space is a dilated simplex built from single-symbol substitutions. This is much tighter than an axis-aligned box, which would allow every token to change at once.
- IBP verification costs only two forward passes per sample, one for upper and one for lower bounds. Exhaustive search grows exponentially with the perturbation radius. For SST character-level, the maximum space in the reported test sets exceeds 1.4 million sentences.
- With standard GloVe, bounds stay loose unless the model is trained to be IBP-verifiable.
- Counter-fitted embeddings (Mrkšić et al., 2016; built with PPDB and WordNet) shrink the simplex. They raise the share of IBP-verified examples by up to 33.2% (normal training, radius 1), with a mild drop in nominal accuracy.
- Limitations: only shallow CNN and fully connected models were studied. Recurrent and self-attention architectures are left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 59 works · 16,463 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 116 works

## Cite

Cite as: Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham and Pushmeet Kohli. 2019. Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{huang2019achieving,
  title = {{Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation}},
  author = {Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham and Pushmeet Kohli},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1419},
  url = {https://doi.org/10.18653/v1/d19-1419}
}
```

- [← previous Semantic Noise Matters for Neural Natural Language Generation](https://md.research.elamur.ai/free/en/papers/dusek-2019-semantic-noise-matters-neural-natural/)
- [next → Question Answering for Privacy Policies: Combining Computational and Legal Perspectives](https://md.research.elamur.ai/free/en/papers/ravichander-2019-question-answering-privacy-policies-combining/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

45 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anish Athalye, Nicholas Carlini, and David A. Wagner. 2018. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, pages 274–283. [find](https://scholar.google.com/scholar?q=Anish+Athalye%2C+Nicholas+Carlini%2C+and+David+A%2E+Wagner%2E+2018%2E+Obfuscated+gradients+give+a+false+sense+of+security%3A+Circumventing+defenses+to+adversarial+examples%2E+In+ICML%2C+pages+274%E2%80%93283%2E)
2. Christel Baier and Joost-Pieter Katoen. 2008. Principles of Model Checking. MIT press. [find](https://scholar.google.com/scholar?q=Christel+Baier+and+Joost%2DPieter+Katoen%2E+2008%2E+Principles+of+Model+Checking%2E+MIT+press%2E)
3. Clark Barrett and Cesare Tinelli. 2018. Satisfiability modulo theories. In Handbook of Model Checking, pages 305–343. Springer. [find](https://scholar.google.com/scholar?q=Clark+Barrett+and+Cesare+Tinelli%2E+2018%2E+Satisfiability+modulo+theories%2E+In+Handbook+of+Model+Checking%2C+pages+305%E2%80%93343%2E+Springer%2E)
4. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2018%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+In+International+Conference+on+Learning+Representations%2E)
5. Nicola Bertoldi, Richard Zens, and Marcello Federico. 2007. Speech translation by confusion network decoding. In Proc. ICASSP. [find](https://scholar.google.com/scholar?q=Nicola+Bertoldi%2C+Richard+Zens%2C+and+Marcello+Federico%2E+2007%2E+Speech+translation+by+confusion+network+decoding%2E+In+Proc%2E+ICASSP%2E)
6. Rudy Bunel, Ilker Turkaslan, Philip HS Torr, Pushmeet Kohli, and M Pawan Kumar. 2017. Piecewise linear neural network verification: a comparative study. arXiv preprint arXiv:1711.00455. — [arXiv:1711.00455](https://arxiv.org/abs/1711.00455) [find](https://scholar.google.com/scholar?q=Rudy+Bunel%2C+Ilker+Turkaslan%2C+Philip+HS+Torr%2C+Pushmeet+Kohli%2C+and+M+Pawan+Kumar%2E+2017%2E+Piecewise+linear+neural+network+verification%3A+a+comparative+study%2E+arXiv+preprint+arXiv%3A1711%2E00455%2E+%E2%80%94+%5BarXiv%3A1711%2E)
7. Nicholas Carlini, Guy Katz, Clark Barrett, and David L Dill. 2017. Ground-truth adversarial examples. arXiv preprint arXiv:1709.10207. — [arXiv:1709.10207](https://arxiv.org/abs/1709.10207) [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Guy+Katz%2C+Clark+Barrett%2C+and+David+L+Dill%2E+2017%2E+Ground%2Dtruth+adversarial+examples%2E+arXiv+preprint+arXiv%3A1709%2E10207%2E+%E2%80%94+%5BarXiv%3A1709%2E10207%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1709%2E10207%29)
8. Nicholas Carlini and David Wagner. 2017. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 3–14. ACM. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini+and+David+Wagner%2E+2017%2E+Adversarial+examples+are+not+easily+detected%3A+Bypassing+ten+detection+methods%2E+In+Proceedings+of+the+10th+ACM+Workshop+on+Artificial+Intelligence+and+Security%2C)
9. Chih-Hong Cheng, Georg Nührenberg, and Harald Ruess. 2017. Maximum resilience of artificial neural networks. In International Symposium on Automated Technology for Verification and Analysis, pages 251–268. Springer. [find](https://scholar.google.com/scholar?q=Chih%2DHong+Cheng%2C+Georg+Nu%CC%88hrenberg%2C+and+Harald+Ruess%2E+2017%2E+Maximum+resilience+of+artificial+neural+networks%2E+In+International+Symposium+on+Automated+Technology+for+Verification+and+Analysis%2C+pages+25)
10. Minhao Cheng, Jinfeng Yi, Huan Zhang, Pin-Yu Chen, and Cho-Jui Hsieh. 2018. Seq2Sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. CoRR, abs/1803.01128. — [arXiv:1803.01128](https://arxiv.org/abs/1803.01128) [find](https://scholar.google.com/scholar?q=Minhao+Cheng%2C+Jinfeng+Yi%2C+Huan+Zhang%2C+Pin%2DYu+Chen%2C+and+Cho%2DJui+Hsieh%2E+2018%2E+Seq2Sick%3A+Evaluating+the+robustness+of+sequence%2Dto%2Dsequence+models+with+adversarial+examples%2E+CoRR%2C+abs%2F1803%2E01128%2E+%E2%80%94+%5BarXiv)
11. Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O’Donoghue, Jonathan Uesato, and Pushmeet Kohli. 2018. Training verified learners with learned verifiers. arXiv preprint arXiv:1805.10265. — [arXiv:1805.10265](https://arxiv.org/abs/1805.10265) [find](https://scholar.google.com/scholar?q=Krishnamurthy+Dvijotham%2C+Sven+Gowal%2C+Robert+Stanforth%2C+Relja+Arandjelovic%2C+Brendan+O%E2%80%99Donoghue%2C+Jonathan+Uesato%2C+and+Pushmeet+Kohli%2E+2018%2E+Training+verified+learners+with+learned+verifiers%2E+arXiv+prepr)
12. Christopher Dyer, Smaranda Muresan, and Philip Resnik. 2008. [Generalizing word lattice translation](https://md.research.elamur.ai/free/en/papers/dyer-2008-generalizing-word-lattice-translation/). In Proceedings of ACL-08: HLT, pages 1012– 1020, Columbus, Ohio. Association for Computational Linguistics.
13. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. [HotFlip: White-box adversarial examples for text classification](https://md.research.elamur.ai/free/en/papers/ebrahimi-2018-hotflip-white-box-adversarial-examples/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31–36. Association for Computational Linguistics.
14. Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M. Bender. 2017. Towards linguistically generalizable NLP systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems. [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2C+Sudha+Rao%2C+Hal+Daume%CC%81+III%2C+and+Emily+M%2E+Bender%2E+2017%2E+Towards+linguistically+generalizable+NLP+systems%3A+A+workshop+and+shared+task%2E+In+Proceedings+of+the+First+Workshop+on+Building+L)
15. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. [PPDB: The paraphrase database](https://md.research.elamur.ai/free/en/papers/ganitkevitch-2013-ppdb-paraphrase-database/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758–764, Atlanta, Georgia. Association for Co…
16. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. [Breaking NLI systems with sentences that require simple lexical inferences](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655, Melbourne, Australia. Association for…
17. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. [Explaining and harnessing adversarial examples](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/). arXiv preprint arXiv:1412.6572.
18. Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy A. Mann, and Pushmeet Kohli. 2018. On the effectiveness of interval bound propagation for training verifiably robust models. CoRR, abs/1810.12715. — [arXiv:1810.12715](https://arxiv.org/abs/1810.12715) [find](https://scholar.google.com/scholar?q=Sven+Gowal%2C+Krishnamurthy+Dvijotham%2C+Robert+Stanforth%2C+Rudy+Bunel%2C+Chongli+Qin%2C+Jonathan+Uesato%2C+Relja+Arandjelovic%2C+Timothy+A%2E+Mann%2C+and+Pushmeet+Kohli%2E+2018%2E+On+the+effectiveness+of+interval+bound+p)
19. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal+and+Ines+Montani%2E+2017%2E+spaCy+2%3A+Natural+language+understanding+with+Bloom+embeddings%2C+convolutional+neural+networks+and+incremental+parsing%2E+To+appear%2E)
20. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. [Adversarial example generation with syntactically controlled paraphrase networks](https://md.research.elamur.ai/free/en/papers/iyyer-2018-adversarial-example-generation-syntactically-controlled/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
21. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Empirical Methods in Natural Language Processing (EMNLP).
22. Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Reluplex: An efficient SMT solver for verifying deep neural networks. In International Conference on Computer Aided Verification, pages 97–117. Springer. [find](https://scholar.google.com/scholar?q=Guy+Katz%2C+Clark+Barrett%2C+David+L+Dill%2C+Kyle+Julian%2C+and+Mykel+J+Kochenderfer%2E+2017%2E+Reluplex%3A+An+efficient+SMT+solver+for+verifying+deep+neural+networks%2E+In+International+Conference+on+Computer+Aided+)
23. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+International+Conference+on+Learning+Representations%2E)
24. Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533. — [arXiv:1607.02533](https://arxiv.org/abs/1607.02533) [find](https://scholar.google.com/scholar?q=Alexey+Kurakin%2C+Ian+Goodfellow%2C+and+Samy+Bengio%2E+2016%2E+Adversarial+examples+in+the+physical+world%2E+arXiv+preprint+arXiv%3A1607%2E02533%2E+%E2%80%94+%5BarXiv%3A1607%2E02533%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1607%2E02533%29)
25. Yitong Li, Trevor Cohn, and Timothy Baldwin. 2017. Robust training under linguistic adversity. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 21–27. [find](https://scholar.google.com/scholar?q=Yitong+Li%2C+Trevor+Cohn%2C+and+Timothy+Baldwin%2E+2017%2E+Robust+training+under+linguistic+adversity%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Lingui)
26. Xunying Liu, Xie Chen, Yongqiang Wang, Mark J. F. Gales, and Philip C. Woodland. 2016. Two efficient lattice rescoring methods using recurrent neural network language models. IEEE/ACM Trans. Audio, Speech & Language Processing, 24(8):1438–1449. [find](https://scholar.google.com/scholar?q=Xunying+Liu%2C+Xie+Chen%2C+Yongqiang+Wang%2C+Mark+J%2E+F%2E+Gales%2C+and+Philip+C%2E+Woodland%2E+2016%2E+Two+efficient+lattice+rescoring+methods+using+recurrent+neural+network+language+models%2E+IEEE%2FACM+Trans%2E+Audio%2C+Sp)
27. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. [Towards deep learning models resistant to adversarial attacks](https://md.research.elamur.ai/free/en/papers/madry-2017-towards-deep-learning-models-resistant/). In International Conference on Learning Representations.
28. George A. Miller. 1995. [WordNet: A lexical database for English](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Commun. ACM, 38(11):39–41.
29. Matthew Mirman, Timon Gehr, and Martin Vechev. 2018. Differentiable abstract interpretation for provably robust neural networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 3578–3586. [find](https://scholar.google.com/scholar?q=Matthew+Mirman%2C+Timon+Gehr%2C+and+Martin+Vechev%2E+2018%2E+Differentiable+abstract+interpretation+for+provably+robust+neural+networks%2E+In+Proceedings+of+the+35th+International+Conference+on+Machine+Learning)
30. Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Associ… [find](https://scholar.google.com/scholar?q=Nikola+Mrks%CC%8Cic%CC%81%2C+Diarmuid+O%CC%81+Se%CC%81aghdha%2C+Blaise+Thomson%2C+Milica+Gas%CC%8Cic%CC%81%2C+Lina+M%2E+Rojas%2DBarahona%2C+Pei%2DHao+Su%2C+David+Vandyke%2C+Tsung%2DHsien+Wen%2C+and+Steve+Young%2E+2016%2E+Counter%2Dfitting+word+vectors+to+lingu)
31. Takashi Onishi, Masao Utiyama, and Eiichiro Sumita. 2010. Paraphrase lattice for statistical machine translation. In Proceedings of the ACL 2010 Conference Short Papers, pages 1–5, Uppsala, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Takashi+Onishi%2C+Masao+Utiyama%2C+and+Eiichiro+Sumita%2E+2010%2E+Paraphrase+lattice+for+statistical+machine+translation%2E+In+Proceedings+of+the+ACL+2010+Conference+Short+Papers%2C+pages+1%E2%80%935%2C+Uppsala%2C+Sweden%2E+As)
32. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. Association for Computational Linguistics.
33. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
34. Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Certified defenses against adversarial examples. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Aditi+Raghunathan%2C+Jacob+Steinhardt%2C+and+Percy+Liang%2E+2018%2E+Certified+defenses+against+adversarial+examples%2E+In+International+Conference+on+Learning+Representations%2E)
35. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2018%2E+Semantically+equivalent+adversarial+rules+for+debugging+NLP+models%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Compu)
36. Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. 2018. [Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms](https://md.research.elamur.ai/free/en/papers/shen-2018-baseline-needs-more-love-simple/). In Proceedings of the 56th Annual Meeting of the Association…
37. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In)
38. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. [Intriguing properties of neural networks](https://md.research.elamur.ai/free/en/papers/szegedy-2013-intriguing-properties-neural-networks/). arXiv preprint arXiv:1312.6199.
39. Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness may be at odds with accuracy. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dimitris+Tsipras%2C+Shibani+Santurkar%2C+Logan+Engstrom%2C+Alexander+Turner%2C+and+Aleksander+Madry%2E+2019%2E+Robustness+may+be+at+odds+with+accuracy%2E+In+International+Conference+on+Learning+Representations%2E)
40. Jonathan Uesato, Brendan O’Donoghue, Pushmeet Kohli, and Aron van den Oord. 2018. Adversarial risk and the dangers of evaluating against weak attacks. In ICML, pages 5032–5041. [find](https://scholar.google.com/scholar?q=Jonathan+Uesato%2C+Brendan+O%E2%80%99Donoghue%2C+Pushmeet+Kohli%2C+and+Aron+van+den+Oord%2E+2018%2E+Adversarial+risk+and+the+dangers+of+evaluating+against+weak+attacks%2E+In+ICML%2C+pages+5032%E2%80%935041%2E)
41. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal trigger sequences for attacking and analyzing NLP. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Shi+Feng%2C+Nikhil+Kandpal%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2019%2E+Universal+trigger+sequences+for+attacking+and+analyzing+NLP%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
42. Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Formal security analysis of neural networks using symbolic intervals. In 27th USENIX Security Symposium (USENIX Security 18), pages 1599–1614, Baltimore, MD. USENIX Association. [find](https://scholar.google.com/scholar?q=Shiqi+Wang%2C+Kexin+Pei%2C+Justin+Whitehouse%2C+Junfeng+Yang%2C+and+Suman+Jana%2E+2018%2E+Formal+security+analysis+of+neural+networks+using+symbolic+intervals%2E+In+27th+USENIX+Security+Symposium+%28USENIX+Security+1)
43. Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. 2018. Towards fast computation of certified robustness for ReLU networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learni… [find](https://scholar.google.com/scholar?q=Lily+Weng%2C+Huan+Zhang%2C+Hongge+Chen%2C+Zhao+Song%2C+Cho%2DJui+Hsieh%2C+Luca+Daniel%2C+Duane+Boning%2C+and+Inderjit+Dhillon%2E+2018%2E+Towards+fast+computation+of+certified+robustness+for+ReLU+networks%2E+In+Proceedings+)
44. Eric Wong and Zico Kolter. 2018. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pages 5283–5292. [find](https://scholar.google.com/scholar?q=Eric+Wong+and+Zico+Kolter%2E+2018%2E+Provable+defenses+against+adversarial+examples+via+the+convex+outer+adversarial+polytope%2E+In+International+Conference+on+Machine+Learning%2C+pages+5283%E2%80%935292%2E)
45. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). In Advances in Neural Information Processing Systems, pages 649–657.

*Updated 24 October 2026*