---
title: "AutoDAN — paper by Zhu et al. (2023) — summary and ideas"
description: "This paper presents AutoDAN, an attack that combines the strengths of both. 61 references, cited by 85 papers here."
authors: ["Sicheng Zhu", "Ruiyi Zhang", "Bang An", "Gang Wu", "Joe Barrow", "Zichao Wang", "Furong Huang", "Ani Nenkova", "Tong Sun"]
year: 2023
venue: "arXiv:2310.15140"
source: "https://arxiv.org/abs/2310.15140"
doi: "10.48550/arxiv.2310.15140"
arxiv: "2310.15140"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2023-autodan-interpretable-gradient-based-adversarial/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2023-autodan-interpretable-gradient-based-adversarial/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models

# AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models

**Authors:** Sicheng Zhu, [Ruiyi Zhang](https://md.research.elamur.ai/free/en/authors/ruiyi-zhang/), [Bang An](https://md.research.elamur.ai/free/en/authors/bang-an/), [Gang Wu](https://md.research.elamur.ai/free/en/authors/gang-wu/), [Joe Barrow](https://md.research.elamur.ai/free/en/authors/joe-barrow/), [Zichao Wang](https://md.research.elamur.ai/free/en/authors/zichao-wang/), [Furong Huang](https://md.research.elamur.ai/free/en/authors/furong-huang/), [Ani Nenkova](https://md.research.elamur.ai/free/en/authors/ani-nenkova/) and [Tong Sun](https://md.research.elamur.ai/free/en/authors/tong-sun/) · **Year:** 2023 · **Venue:** arXiv:2310.15140 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.15140) · [PDF](https://arxiv.org/pdf/2310.15140) · [DOI 10.48550/arxiv.2310.15140](https://doi.org/10.48550/arxiv.2310.15140) · [arXiv 2310.15140](https://arxiv.org/abs/2310.15140) · [OpenAlex](https://openalex.org/W4387929776)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [85](https://md.research.elamur.ai/free/en/papers/zhu-2023-autodan-interpretable-gradient-based-adversarial/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models can be pushed past their safety alignment in two ways. Hand-written jailbreaks read naturally, but there are few of them and providers can blocklist them. Gradient-based attacks such as GCG can produce endless prompts, but the output is gibberish that perplexity filters catch easily. This paper presents AutoDAN, an attack that combines the strengths of both. It builds an adversarial suffix from scratch, one token at a time, from left to right, much as a model writes text. Each new token is picked by a two-stage search: a coarse shortlist from gradients, then a finer check. Both stages balance two goals, jailbreaking and readability, and the way the goals are combined adjusts to the entropy of the next-token distribution. The resulting prompts are fluent and get past perplexity filters while still succeeding often. They also show strategies familiar from manual jailbreaks and transfer to black-box models better than unreadable prompts. This suggests perplexity filtering alone is too optimistic a defense, and it gives red-teamers a new, interpretable tool.

## Key points
- The method has two nested loops. The inner loop, Single Token Optimization (STO), refines one token through preliminary and fine selection. The outer loop extends the prompt token by token.
- Generated suffixes reach lower perplexity than typical benign user prompts. The baseline GCG-reg, which adds perplexity regularization (e.g., weight 0.1), cannot achieve high attack success and readability together.
- With one white-box proxy model (e.g., Vicuna 7B), AutoDAN prompts transfer to GPT-3.5 and GPT-4 better than GCG prompts. They also generalize better to unseen harmful behaviors when training data is limited.
- Without any prior knowledge, the prompts show recognizable strategies: roleplay, switching to a foreign or coding language, formatting instructions, and quoting sources.
- The approach extends to other variants and tasks: AutoDAN-Prefix, Semi-AutoDAN (adds manual prefixes and suffixes), and system prompt leaking via a custom objective.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 141 works · 29,174 citations
- [Adobe Systems](https://md.research.elamur.ai/free/en/companies/adobe-systems/) — Company, San Jose, United States · 40 works · 4,248 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova and Tong Sun. 2023. AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models. arXiv:2310.15140.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhu2023autodan,
  title = {{AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models}},
  author = {Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova and Tong Sun},
  year = {2023},
  doi = {10.48550/arxiv.2310.15140},
  eprint = {2310.15140},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.15140}
}
```

- [← previous Alignment for Honesty](https://md.research.elamur.ai/free/en/papers/yang-2023-alignment-honesty/)
- [next → Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/qi-2023-cross-lingual-consistency-factual-knowledge/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

61 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gabriel Alon and Michael Kamfonas. [Detecting language model attacks with perplexity](https://md.research.elamur.ai/free/en/papers/alon-2023-detecting-language-model-attacks-perplexity/). ArXiv, abs/2308.14132, 2023.
2. Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. 274–283. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Anish+Athalye%2C+Nicholas+Carlini%2C+and+David+Wagner%2E+Obfuscated+gradients+give+a+false+sense+of+security%3A+Circumventing+defenses+to+adversarial+examples%2E+In+International+conference+on+machine+learning%2C)
3. Clark W. Barrett, Bradley L Boyd, Ellie Burzstein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, Kathleen Fisher, Tatsunori Hashimoto, Dan Hendrycks, Somesh Jha, Daniel Kang, Florian Kerschbaum, Eric Mitchell, John C. Mitchell, Zulfi… [find](https://scholar.google.com/scholar?q=Clark+W%2E+Barrett%2C+Bradley+L+Boyd%2C+Ellie+Burzstein%2C+Nicholas+Carlini%2C+Brad+Chen%2C+Jihye+Choi%2C+Amrita+Roy+Chowdhury%2C+Mihai+Christodorescu%2C+Anupam+Datta%2C+Soheil+Feizi%2C+Kathleen+Fisher%2C+Tatsunori+Hashimoto)
4. Dimitris Bertsimas and John N Tsitsiklis. Introduction to linear optimization, volume 6. Athena scientific Belmont, MA, 1997. [find](https://scholar.google.com/scholar?q=Dimitris+Bertsimas+and+John+N+Tsitsiklis%2E+Introduction+to+linear+optimization%2C+volume+6%2E+Athena+scientific+Belmont%2C+MA%2C+1997%2E)
5. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on M… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+Pythia%3A+A+sui)
6. Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. [Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM](https://md.research.elamur.ai/free/en/papers/cao-2024-defending-against-alignment-breaking-attacks/), September 2023.
7. Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. [Are aligned neural networks adversarially aligned?](https://md.research.elamur.ai/free/en/papers/carlini-2023-aligned-neural-networks-adversarially-aligned/) arXiv preprint arXiv:2306.15447, 2023.
8. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. [Jailbreaking Black Box Large Language Models in Twenty Queries](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/), October 2023.
9. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna…](https://lmsys.org/blog/2023-03-30-vicuna…) [↗](https://lmsys.org/blog/2023-03-30-vicuna…)
10. DAN. Chat gpt "dan" (and other "jailbreaks"), 2023. URL [https://gist.github.com/coolaj86/6f4f7b30129b0251f61fa7baaa881516](https://gist.github.com/coolaj86/6f4f7b30129b0251f61fa7baaa881516). GitHub repository. [↗](https://gist.github.com/coolaj86/6f4f7b30129b0251f61fa7baaa881516)
11. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots, October 2023. [find](https://scholar.google.com/scholar?q=Gelei+Deng%2C+Yi+Liu%2C+Yuekang+Li%2C+Kailong+Wang%2C+Ying+Zhang%2C+Zefeng+Li%2C+Haoyu+Wang%2C+Tianwei+Zhang%2C+and+Yang+Liu%2E+MasterKey%3A+Automated+Jailbreak+Across+Multiple+Large+Language+Model+Chatbots%2C+October+2023)
12. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). arXiv preprint arXiv:2305.14314, 2023.
13. Xiaohan Fu, Zihan Wang, Shuheng Li, Rajesh K. Gupta, Niloofar Mireshghallah, Taylor Berg-Kirkpatrick, and Earlence Fernandes. Misusing Tools in Large Language Models With Visual Adversarial Examples, October 2023. [find](https://scholar.google.com/scholar?q=Xiaohan+Fu%2C+Zihan+Wang%2C+Shuheng+Li%2C+Rajesh+K%2E+Gupta%2C+Niloofar+Mireshghallah%2C+Taylor+Berg%2DKirkpatrick%2C+and+Earlence+Fernandes%2E+Misusing+Tools+in+Large+Language+Models+With+Visual+Adversarial+Examples%2C+)
14. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, May 2023. — [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://md.research.elamur.ai/free/en/papers/greshake-2023-ve-signed-up-compromising-real/)
15. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. [Gradient-based adversarial attacks against text transformers](https://md.research.elamur.ai/free/en/papers/guo-2021-gradient-based-adversarial-attacks-against/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5747–5757, Online and Punta Cana, Dominican Republic, November 2021. Associati…
16. Thilo Hagendorff. Deception abilities emerged in large language models. ArXiv, abs/2307.16513, 2023. URL [https://api.semanticscholar.org/CorpusID:260334697](https://api.semanticscholar.org/CorpusID:260334697). Citation Key: Hagendorff2023DeceptionAE. — [arXiv:2307.16513](https://arxiv.org/abs/2307.16513) [↗](https://api.semanticscholar.org/CorpusID:260334697)
17. Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. [LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked](https://md.research.elamur.ai/free/en/papers/phute-2023-llm-self-defense-self-examination/). ArXiv, abs/2308.07308, 2023.
18. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. [Catastrophic jailbreak of open-source llms via exploiting generation](https://md.research.elamur.ai/free/en/papers/huang-2023-catastrophic-jailbreak-open-source-llms/). arXiv preprint arXiv:2310.06987, 2023a.
19. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://md.research.elamur.ai/free/en/papers/huang-2023-catastrophic-jailbreak-open-source-llms/), October 2023b.
20. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://md.research.elamur.ai/free/en/papers/jain-2023-baseline-defenses-adversarial-attacks-against/), September 2023.
21. Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. [Automatically Auditing Large Language Models via Discrete Optimization](https://md.research.elamur.ai/free/en/papers/jones-2023-automatically-auditing-large-language-models/), March 2023.
22. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. [Ctrl: A conditional transformer language model for controllable generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/). arXiv preprint arXiv:1909.05858, 2023.
23. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. [A watermark for large language models](https://md.research.elamur.ai/free/en/papers/kirchenbauer-2023-watermark-large-language-models/). arXiv preprint arXiv:2301.10226, 2023.
24. Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. [Certifying LLM Safety against Adversarial Prompting](https://md.research.elamur.ai/free/en/papers/kumar-2023-certifying-llm-safety-against-adversarial/), September 2023.
25. Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models. ArXiv, September 2023. doi: 10.48550/arXiv.2309.01446. URL http://arxiv.org/abs/2309.01446. arXiv:2309.01446 [cs]. — [doi:10.48550/arxiv.2309.01446](https://doi.org/10.48550/arxiv.2309.01446) [↗](http://arxiv.org/abs/2309.01446)
26. Yann LeCun and Corinna Cortes. MNIST handwritten digit database. Tech Report, 2010. [find](https://scholar.google.com/scholar?q=Yann+LeCun+and+Corinna+Cortes%2E+MNIST+handwritten+digit+database%2E+Tech+Report%2C+2010%2E)
27. Brian Lester, Rami Al-Rfou, and Noah Constant. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational L…
28. Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582–4597, On… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+Inte)
29. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D. Manning, Christopher Re, Diana Acosta-Navas, Drew Arad H… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
30. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://md.research.elamur.ai/free/en/papers/liu-2023-autodan-generating-stealthy-jailbreak-prompts/), October 2023a.
31. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. [Prompt Injection attack against LLM-integrated Applications](https://md.research.elamur.ai/free/en/papers/liu-2023-prompt-injection-attack-against-llm/), June 2023b.
32. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://md.research.elamur.ai/free/en/papers/liu-2023-jailbreaking-chatgpt-prompt-engineering-empirical/), May 2023c.
33. Natalie Maus, Patrick Chao, Eric Wong, and Jacob Gardner. Black Box Adversarial Prompting for Foundation Models. https://arxiv.org/abs/2302.04237v2, February 2023. — [arXiv:2302.04237](https://arxiv.org/abs/2302.04237) [↗](https://arxiv.org/abs/2302.04237v2)
34. John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. [Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp](https://md.research.elamur.ai/free/en/papers/morris-2020-textattack-framework-adversarial-attacks-data/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 119–…
35. R OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, pp. 2303–08774, 2023.
36. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35: 27730–27744, 2022.
37. Fábio Perez and Ian Ribeiro. [Ignore Previous Prompt: Attack Techniques For Language Models](https://md.research.elamur.ai/free/en/papers/perez-2022-ignore-previous-prompt-attack-techniques/), November 2022.
38. Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic Prompt Optimization with "Gradient Descent" and Beam Search. ArXiv, October 2023. [doi: 10.48550/arXiv.2305.03495](https://md.research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/).
39. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. [Visual Adversarial Examples Jailbreak Aligned Large Language Models](https://md.research.elamur.ai/free/en/papers/qi-2023-visual-adversarial-examples-jailbreak-aligned/), August 2023.
40. Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury. Tricking LLMs into Disobedience: Understanding, Analyzing, and Preventing Jailbreaks, May 2023. [find](https://scholar.google.com/scholar?q=Abhinav+Rao%2C+Sachin+Vashistha%2C+Atharva+Naik%2C+Somak+Aditya%2C+and+Monojit+Choudhury%2E+Tricking+LLMs+into+Disobedience%3A+Understanding%2C+Analyzing%2C+and+Preventing+Jailbreaks%2C+May+2023%2E)
41. Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://md.research.elamur.ai/free/en/papers/robey-2023-smoothllm-defending-large-language-models/), October 2023.
42. Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In International Conference on Learning Representations, 2019. [find](https://scholar.google.com/scholar?q=Lukas+Schott%2C+Jonas+Rauber%2C+Matthias+Bethge%2C+and+Wieland+Brendel%2E+Towards+the+first+adversarially+robust+neural+network+model+on+MNIST%2E+In+International+Conference+on+Learning+Representations%2C+2019%2E)
43. Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. [Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models](https://md.research.elamur.ai/free/en/papers/shayegani-2023-jailbreak-pieces-compositional-adversarial-attacks/), October 2023.
44. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.[03825, 2023](https://md.research.elamur.ai/free/en/papers/shen-2023-anything-now-characterizing-evaluating-wild/).
45. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto-Prompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 [Conference on Empirical Methods in Natural Language Processing (EMNLP), pp](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). 4222–4235, Online, Nove…
46. Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. On the Exploitability of Instruction Tuning. ArXiv, June 2023. doi: 10.48550/arXiv.2306.17194. — [doi:10.48550/arxiv.2306.17194](https://doi.org/10.48550/arxiv.2306.17194) [find](https://scholar.google.com/scholar?q=Manli+Shu%2C+Jiongxiao+Wang%2C+Chen+Zhu%2C+Jonas+Geiping%2C+Chaowei+Xiao%2C+and+Tom+Goldstein%2E+On+the+Exploitability+of+Instruction+Tuning%2E+ArXiv%2C+June+2023%2E+doi%3A+10%2E48550%2FarXiv%2E2306%2E17194%2E+%E2%80%94+%5Bdoi%3A10%2E48550%2Farxi)
47. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
48. Florian Tramèr. Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them, June 2022. [find](https://scholar.google.com/scholar?q=Florian+Tram%C3%A8r%2E+Detecting+Adversarial+Examples+Is+%28Nearly%29+As+Hard+As+Classifying+Them%2C+June+2022%2E)
49. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. [Jailbroken: How does llm safety training fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) arXiv preprint arXiv:2307.02483, 2023a.
50. Zeming Wei, Yifei Wang, and Yisen Wang. [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbreak-guard-aligned-language-models/), October 2023b.
51. Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://md.research.elamur.ai/free/en/papers/wen-2023-hard-prompts-made-easy-gradient/), June 2023.
52. Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, and Amnon Shashua. [Fundamental Limitations of Alignment in Large Language Models](https://md.research.elamur.ai/free/en/papers/wolf-2023-fundamental-limitations-alignment-large-language/), August 2023.
53. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://md.research.elamur.ai/free/en/papers/yu-2023-gptfuzzer-red-teaming-large-language/), October 2023.
54. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. [Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher](https://md.research.elamur.ai/free/en/papers/yuan-2023-gpt-4-too-smart-safe/). ArXiv, August 2023. doi: 10.48550/arXiv.2308.06463. URL http://arxiv.org/abs/2308.06463. arXiv:2308.06463 [cs]. [↗](http://arxiv.org/abs/2308.06463)
55. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating Large Language Models at Evaluating Instruction Following, October 2023. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Jiatong+Yu%2C+Tianyu+Gao%2C+Yu+Meng%2C+Tanya+Goyal%2C+and+Danqi+Chen%2E+Evaluating+Large+Language+Models+at+Evaluating+Instruction+Following%2C+October+2023%2E)
56. Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology, 11(3), April 2020. ISSN 2157-6904. doi: 10.1145/3374217. URL https://doi.org/10.1145/3374217. [↗](https://doi.org/10.1145/3374217)
57. Yiming Zhang and Daphne Ippolito. Prompts Should not be Seen as Secrets: Systematically Measuring Prompt Extraction Attack Success, July 2023. [find](https://scholar.google.com/scholar?q=Yiming+Zhang+and+Daphne+Ippolito%2E+Prompts+Should+not+be+Seen+as+Secrets%3A+Systematically+Measuring+Prompt+Extraction+Attack+Success%2C+July+2023%2E)
58. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/), 2023.
59. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.[04528, October 2023](https://md.research.elamur.ai/free/en/papers/zhu-2023-promptrobust-towards-evaluating-robustness-large/). doi: 10.48550/a…
60. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendry… — [Representation Engineering: A Top-Down Approach to AI Transparency](https://md.research.elamur.ai/free/en/papers/zou-2023-representation-engineering-top-down-approach/)
61. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/), July 2023b.

*Updated 2 November 2026*