---
title: "LLM Self Defense — paper by Phute et al. (2023)"
description: "The authors propose LLM Self Defense, a zero-shot output filter. 35 references, cited by 89 papers here."
authors: ["Mansi Phute", "Alec Helbling", "Matthew Hull", "ShengYun Peng", "Sebastian Szyller", "Cory Cornelius", "Duen Horng Chau"]
year: 2023
venue: "arXiv:2308.07308"
source: "https://arxiv.org/abs/2308.07308"
doi: "10.48550/arxiv.2308.07308"
arxiv: "2308.07308"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/phute-2023-llm-self-defense-self-examination/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/phute-2023-llm-self-defense-self-examination/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked

# LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked

**Authors:** Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius and Duen Horng Chau · **Year:** 2023 · **Venue:** arXiv:2308.07308 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.07308) · [PDF](https://arxiv.org/pdf/2308.07308) · [DOI 10.48550/arxiv.2308.07308](https://doi.org/10.48550/arxiv.2308.07308) · [arXiv 2308.07308](https://arxiv.org/abs/2308.07308) · [OpenAlex](https://openalex.org/W4385889721)

**Citations:** 30 citations (OpenAlex, 2026-10-14) · [89](https://md.research.elamur.ai/free/en/papers/phute-2023-llm-self-defense-self-examination/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Safety-aligned LLMs can still be manipulated into producing dangerous content, for example through role-play or fictional framing, or by forcing the model to begin with an affirmative reply. The authors propose LLM Self Defense, a zero-shot output filter. A second LLM instance, which can be the same model, receives the generated response inside a fixed instruction template and is asked to label it harmful or not. The approach requires no fine-tuning, no input preprocessing and no repeated generation, so it is cheaper than earlier defenses. The authors tested it on GPT 3.5 and Llama 2 7B, using harmful responses elicited with modified AdvBench prompts plus benign responses. Both models served as harm filters for their own outputs and for each other's. The filters caught nearly all harmful text, which reduced attack success to near zero. Placing the harm question after the text, rather than before it, sharply reduced false alarms, especially for Llama 2. The result matters because it offers a lightweight, model-agnostic safeguard for deployed chatbots.

## Key points

- **Evaluation data:** For each model, the authors collected 50 harmful and 50 benign responses. The harmful ones were elicited with adapted AdvBench prompts: completion-style "Sure, here is…" steering for Llama 2, and hypothetical or fictional framing for GPT 3.5.
- **GPT 3.5 as a self-filter:** Accuracy was 98% with the question as a prefix and 99% as a suffix, with zero false positives.
- **Llama 2 as a self-filter:** Accuracy rose from 77% to 94.6% when the question came after the text. The false positive rate fell from 0.42 to 0.09. In the prefix setting, Llama 2 even flagged harmless requests such as opening a spreadsheet.
- **Limitations:** The authors could not reproduce the gradient-based adversarial suffix attack of Zou et al. because training was unstable. Llama 2's filter outputs had to be mapped to yes/no by hand. The authors suggest logit biasing and in-context examples as future fixes.
- **Code:** Available at github.com/poloclub/llm-self-defense.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 152 works · 27,763 citations
- [Intel](https://md.research.elamur.ai/free/en/companies/intel/) — Company, Santa Clara, United States · 18 works · 43,058 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius and Duen Horng Chau. 2023. LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked. arXiv:2308.07308.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{phute2023llm,
  title = {{LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked}},
  author = {Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius and Duen Horng Chau},
  year = {2023},
  doi = {10.48550/arxiv.2308.07308},
  eprint = {2308.07308},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.07308}
}
```

- [← previous Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-models-are-not/)
- [next → RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://md.research.elamur.ai/free/en/papers/zhang-2023-repocoder-repository-level-code-completion/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

35 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. “Musiclm: Generating music from text”. In: arXiv preprint arXiv:2301.11325 (2023). — [arXiv:2301.11325](https://arxiv.org/abs/2301.11325) [find](https://scholar.google.com/scholar?q=Andrea+Agostinelli%2C+Timo+I+Denk%2C+Zal%C3%A1n+Borsos%2C+Jesse+Engel%2C+Mauro+Verzetti%2C+Antoine+Caillon%2C+Qingqing+Huang%2C+Aren+Jansen%2C+Adam+Roberts%2C+Marco+Tagliasacchi%2C+et+al%2E+%E2%80%9CMusiclm%3A+Generating+music+from+text%E2%80%9D)
2. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. “[Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)”. In: arXiv preprint arXiv:2212.08073 (2022).
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. “[Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)”. In: Advances in neural information processing systems 33 (2020), pp. 1877–1901.
4. Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. “[Are aligned neural networks adversarially aligned?](https://md.research.elamur.ai/free/en/papers/carlini-2023-aligned-neural-networks-adversarially-aligned/)” In: arXiv preprint arXiv:2306.15447 (2023).
5. Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. “Certified adversarial robustness via randomized smoothing”. In: international conference on machine learning. PMLR. 2019, pp. 1310– 1320. [find](https://scholar.google.com/scholar?q=Jeremy+Cohen%2C+Elan+Rosenfeld%2C+and+Zico+Kolter%2E+%E2%80%9CCertified+adversarial+robustness+via+randomized+smoothing%E2%80%9D%2E+In%3A+international+conference+on+machine+learning%2E+PMLR%2E+2019%2C+pp%2E+1310%E2%80%93+1320%2E)
6. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. “[Chain-of-Verification Reduces Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/dhuliawala-2023-chain-verification-reduces-hallucination-large/)”. In: arXiv preprint arXiv:2309.11495 (2023).
7. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. “A survey for in-context learning”. In: arXiv preprint arXiv:2301.[00234 (2022)](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/).
8. Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. “[Improving alignment of dialogue agents via targeted human judgements](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/)”. In: arXiv preprint arXiv:2209.14375 (2022).
9. K Greshake, S Abdelnabi, S Mishra, C Endres, T Holz, and M Fritz. “Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, May 2023”. In: URL http://arxiv.org/abs/2302.12173 (). — [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://md.research.elamur.ai/free/en/papers/greshake-2023-ve-signed-up-compromising-real/) [↗](http://arxiv.org/abs/2302.12173)
10. Maanak Gupta, CharanKumar Akiri, Kshitiz Aryal, Eli Parker, and Lopamudra Praharaj. “From ChatGPT to ThreatGPT: Impact of generative AI in cybersecurity and privacy”. In: IEEE Access (2023). [find](https://scholar.google.com/scholar?q=Maanak+Gupta%2C+CharanKumar+Akiri%2C+Kshitiz+Aryal%2C+Eli+Parker%2C+and+Lopamudra+Praharaj%2E+%E2%80%9CFrom+ChatGPT+to+ThreatGPT%3A+Impact+of+generative+AI+in+cybersecurity+and+privacy%E2%80%9D%2E+In%3A+IEEE+Access+%282023%29%2E)
11. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. “Audiogpt: Understanding and generating speech, music, sound, and talking head”. In: arXiv preprint arXiv:2304.12995 (2023). — [arXiv:2304.12995](https://arxiv.org/abs/2304.12995) [find](https://scholar.google.com/scholar?q=Rongjie+Huang%2C+Mingze+Li%2C+Dongchao+Yang%2C+Jiatong+Shi%2C+Xuankai+Chang%2C+Zhenhui+Ye%2C+Yuning+Wu%2C+Zhiqing+Hong%2C+Jiawei+Huang%2C+Jinglin+Liu%2C+et+al%2E+%E2%80%9CAudiogpt%3A+Understanding+and+generating+speech%2C+music%2C+sound)
12. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. “[Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://md.research.elamur.ai/free/en/papers/jain-2023-baseline-defenses-adversarial-attacks-against/)”. In: arXiv preprint arXiv:2309.00614 (2023).
13. Martin Josifoski, Lars Klein, Maxime Peyrard, Yifei Li, Saibo Geng, Julian Paul Schnitzler, Yuxing Yao, Jiheng Wei, Debjit Paul, and Robert West. “Flows: Building blocks of reasoning and collaborating ai”. In: arXiv preprint arXiv:2308.01285 (2023). — [arXiv:2308.01285](https://arxiv.org/abs/2308.01285) [find](https://scholar.google.com/scholar?q=Martin+Josifoski%2C+Lars+Klein%2C+Maxime+Peyrard%2C+Yifei+Li%2C+Saibo+Geng%2C+Julian+Paul+Schnitzler%2C+Yuxing+Yao%2C+Jiheng+Wei%2C+Debjit+Paul%2C+and+Robert+West%2E+%E2%80%9CFlows%3A+Building+blocks+of+reasoning+and+collaborating)
14. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. “[Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)”. In: Advances in neural information processing systems 35 (2022), pp. 22199–22213.
15. Klas Leino, Zifan Wang, and Matt Fredrikson. “Globally-robust neural networks”. In: International Conference on Machine Learning. PMLR. 2021, pp. 6212–6222. [find](https://scholar.google.com/scholar?q=Klas+Leino%2C+Zifan+Wang%2C+and+Matt+Fredrikson%2E+%E2%80%9CGlobally%2Drobust+neural+networks%E2%80%9D%2E+In%3A+International+Conference+on+Machine+Learning%2E+PMLR%2E+2021%2C+pp%2E+6212%E2%80%936222%2E)
16. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. “RAIN: Your Language Models Can Align Themselves without Finetuning”. In: arXiv preprint arXiv:2309.07124 (2023). — [arXiv:2309.07124](https://arxiv.org/abs/2309.07124) [find](https://scholar.google.com/scholar?q=Yuhui+Li%2C+Fangyun+Wei%2C+Jinjing+Zhao%2C+Chao+Zhang%2C+and+Hongyang+Zhang%2E+%E2%80%9CRAIN%3A+Your+Language+Models+Can+Align+Themselves+without+Finetuning%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2309%2E07124+%282023%29%2E+%E2%80%94+%5BarXiv%3A2309%2E071)
17. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. “[Jailbreaking chatgpt via prompt engineering: An empirical study](https://md.research.elamur.ai/free/en/papers/liu-2023-jailbreaking-chatgpt-prompt-engineering-empirical/)”. In: arXiv preprint arXiv:2305.13860 (2023).
18. Alexandra Luccioni and Joseph Viviano. “What’s in the box? an analysis of undesirable content in the Common Crawl corpus”. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2:… — [What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus](https://md.research.elamur.ai/free/en/papers/luccioni-2021-what-s-box-analysis-undesirable/)
19. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. “[Towards deep learning models resistant to adversarial attacks](https://md.research.elamur.ai/free/en/papers/madry-2017-towards-deep-learning-models-resistant/)”. In: arXiv preprint arXiv:1706.06083 (2017).
20. Ning Miao, Yee Whye Teh, and Tom Rainforth. “[Selfcheck: Using llms to zero-shot check their own step-by-step reasoning](https://md.research.elamur.ai/free/en/papers/miao-2023-selfcheck-using-llms-zero-shot/)”. In: arXiv preprint arXiv:2308.00436 (2023).
21. Yisroel Mirsky et al. “The Threat of Offensive AI to Organizations”. In: Computers and Security 124 (2023), p. 103006. ISSN : 0167-4048. DOI : https://doi.org/10.1016/j. cose.2022.103006. URL : https://www.sciencedirect.com/science/article/pii/S0167404822003984. — [doi:10.1016/j](https://doi.org/10.1016/j) [find](https://scholar.google.com/scholar?q=Yisroel+Mirsky+et+al%2E+%E2%80%9CThe+Threat+of+Offensive+AI+to+Organizations%E2%80%9D%2E+In%3A+Computers+and+Security+124+%282023%29%2C+p%2E+103006%2E+ISSN+%3A+0167%2D4048%2E+DOI+%3A+https%3A%2F%2Fdoi%2Eorg%2F10%2E1016%2Fj%2E+cose%2E2022%2E103006%2E+URL+%3A+https%3A)
22. OpenAI. ChatGPT. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). Accessed on: Aug 08, 2023. Nov. 2022. [↗](https://openai.com/blog/chatgpt)
23. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. “[Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 27730–2774…
24. Huachuan Qiu, Shuai Zhang, Anqi Li, Hongliang He, and Zhenzhong Lan. “Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models”. In: arXiv preprint arXiv:2307.08487 (2023). — [arXiv:2307.08487](https://arxiv.org/abs/2307.08487) [find](https://scholar.google.com/scholar?q=Huachuan+Qiu%2C+Shuai+Zhang%2C+Anqi+Li%2C+Hongliang+He%2C+and+Zhenzhong+Lan%2E+%E2%80%9CLatent+Jailbreak%3A+A+Benchmark+for+Evaluating+Text+Safety+and+Output+Robustness+of+Large+Language+Models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv)
25. Nino Scherrer, Claudia Shi, Amir Feder, and David M Blei. “[Evaluating the Moral Beliefs Encoded in LLMs](https://md.research.elamur.ai/free/en/papers/scherrer-2023-evaluating-moral-beliefs-encoded-llms/)”. In: arXiv preprint arXiv:2307.14324 (2023).
26. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. “[Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)”. In: arXiv preprint arXiv:2307.09288 (2023).
27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. “[Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)”. In: Advances in neural information processing systems 30 (2017).
28. Tongzhou Wang. Github issue "[reproduce] Optimization unstable / not working". [https://github.com/llm-attacks/llm-attacks/issues/44](https://github.com/llm-attacks/llm-attacks/issues/44). Accessed: 2023-09-29. 2023. [↗](https://github.com/llm-attacks/llm-attacks/issues/44)
29. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. “[Jailbroken: How does llm safety training fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/)” In: arXiv preprint arXiv:2307.02483 (2023).
30. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. “[Large Language Models Are Better Reasoners with Self-Verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/)”. In: CoRR, abs/2212.09561 (2023).
31. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. “[AutoGen: Enabling next-gen LLM applications via multi-agent conversation](https://md.research.elamur.ai/free/en/papers/wu-2023-autogen-enabling-next-gen-llm/) framework”. In: arXiv preprint arXiv:2308.08155 (2023).
32. Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. “Adversarial examples: Attacks and defenses for deep learning”. In: IEEE transactions on neural networks and learning systems 30.9 (2019), pp. 2805–2824. [find](https://scholar.google.com/scholar?q=Xiaoyong+Yuan%2C+Pan+He%2C+Qile+Zhu%2C+and+Xiaolin+Li%2E+%E2%80%9CAdversarial+examples%3A+Attacks+and+defenses+for+deep+learning%E2%80%9D%2E+In%3A+IEEE+transactions+on+neural+networks+and+learning+systems+30%2E9+%282019%29%2C+pp%2E+2805%E2%80%93282)
33. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. “[A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)”. In: arXiv preprint arXiv:2303.18223 (2023).
34. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. “[Can Large Language Models Transform Computational Social Science?](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/) arXiv 2023”. In: arXiv preprint arXiv:2305.03514 ().
35. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. “[Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/)”. In: arXiv preprint arXiv:2307.15043 (2023). The following pages contain raw LLM generated text induced by harmful prompts that readers may find offensive.

*Updated 2 November 2026*