---
title: "Jailbreaking Black Box Large Language Models in — Chao 2023"
description: "Safety-aligned LLMs can still be pushed into producing harmful content."
authors: ["Patrick Chao", "Alexander Robey", "Edgar Dobriban", "Hamed Hassani", "George J. Pappas", "Eric Wong"]
year: 2023
venue: "arXiv:2310.08419"
source: "https://arxiv.org/abs/2310.08419"
doi: "10.48550/arxiv.2310.08419"
arxiv: "2310.08419"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / Jailbreaking Black Box Large Language Models in Twenty Queries

# Jailbreaking Black Box Large Language Models in Twenty Queries

**Authors:** [Patrick Chao](https://md.research.elamur.ai/free/en/authors/patrick-chao/), [Alexander Robey](https://md.research.elamur.ai/free/en/authors/alexander-robey/), [Edgar Dobriban](https://md.research.elamur.ai/free/en/authors/edgar-dobriban/), [Hamed Hassani](https://md.research.elamur.ai/free/en/authors/hamed-hassani/), [George J. Pappas](https://md.research.elamur.ai/free/en/authors/george-j-pappas/), [Eric Wong](https://md.research.elamur.ai/free/en/authors/eric-wong/) · **Year:** 2023 · **Venue:** arXiv:2310.08419 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.08419) · [PDF](https://arxiv.org/pdf/2310.08419) · [DOI 10.48550/arxiv.2310.08419](https://doi.org/10.48550/arxiv.2310.08419) · [arXiv 2310.08419](https://arxiv.org/abs/2310.08419) · [OpenAlex](https://openalex.org/W4410609100)

**Citations:** 193 citations (openalex-snapshot · 2026-10-14) · [340](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Safety-aligned LLMs can still be pushed into producing harmful content. Until this paper, testing for that weakness meant one of two things. The first is hand-written persuasive prompts, which take a lot of human effort. The second is token-level optimisation such as GCG, which needs white-box access and hundreds of thousands of queries, and produces suffixes that people can't read. The authors propose PAIR (Prompt Automatic Iterative Refinement), a fully automated red-teaming method that needs only black-box query access. An attacker LLM writes a candidate prompt, following a system prompt built around one of three strategies: role-playing, logical appeal or authority endorsement. The target model answers, and a separate judge model decides whether the answer counts as a jailbreak. If it does not, the prompt, the answer and the attacker's own chain-of-thought notes on how to improve are sent back to the attacker for the next attempt. Many of these streams run in parallel. PAIR often succeeds within twenty queries and breaks GPT-3.5/4, Vicuna and Gemini-Pro. Its prompts are readable and transfer between models, which makes cheap and interpretable stress-testing of deployed LLMs practical.

## Key points
- PAIR uses over 250 times fewer queries than GCG. Against Vicuna, an average successful attack took about 34 seconds, used 366 MB of CPU memory and cost under $0.03, with no GPU required.
- Jailbreak success rates were about 50% on GPT-3.5/4, 88% on Vicuna-13B and 73% on Gemini-Pro. The authors say it is the first automated attack shown to work on Gemini-Pro.
- For the judge, six classifiers were compared against the majority vote of three expert annotators on 100 prompt–response pairs. GPT-4 agreed best (88%, with 16% false positives and 7% false negatives).
- Ablations on the 100 JailbreakBench behaviors used Vicuna-13B as the target. Mixtral was the best attacker (88%), ahead of Vicuna (78%) and GPT-3.5 (69%). The settings chosen were N = 30 streams with depth K = 3. Role-playing produced 82 of the 88 successes, and removing the in-context examples or the improvement notes lowered performance.
- Limitation: PAIR does poorly against heavily safety-tuned models such as Llama-2 and Claude-1/2.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

340 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 330 not yet.

- [Refusal in Language Models Is Mediated by a Single Direction](https://md.research.elamur.ai/free/en/papers/arditi-2024-refusal-language-models-mediated-single/)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://md.research.elamur.ai/free/en/papers/chao-2024-jailbreakbench-open-robustness-benchmark-jailbreaking/)
- [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/)
- [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://md.research.elamur.ai/free/en/papers/li-2023-deepinception-hypnotize-large-language-model/)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://md.research.elamur.ai/free/en/papers/mehrotra-2023-tree-attacks-jailbreaking-black-box/)
- [OpenAI o1 System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://md.research.elamur.ai/free/en/papers/robey-2023-smoothllm-defending-large-language-models/)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbreak-guard-aligned-language-models/)
- [SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space](https://md.research.elamur.ai/free/en/papers/zinkovich-2026-sparta-evaluating-reasoning-segmentation-robustness/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chao2023jailbreaking,
  title = {{Jailbreaking Black Box Large Language Models in Twenty Queries}},
  author = {Chao, Patrick and Robey, Alexander and Dobriban, Edgar and Hassani, Hamed and Pappas, George J. and Wong, Eric},
  year = {2023},
  doi = {10.48550/arxiv.2310.08419},
  eprint = {2310.08419},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.08419}
}
```

- [← previous Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbreak-guard-aligned-language-models/)
- [next → Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://md.research.elamur.ai/free/en/papers/liu-2023-jailbreaking-chatgpt-prompt-engineering-empirical/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

55 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. 1 [find](https://scholar.google.com/scholar?q=Baptiste+Rozie%CC%80re%2C+Jonas+Gehring%2C+Fabian+Gloeckle%2C+Sten+Sootla%2C+Itai+Gat%2C+Xiaoqing+Ellen+Tan%2C+Yossi+Adi%2C+Jingyu+Liu%2C+Tal+Remez%2C+Je%CC%81re%CC%81my+Rapin%2C+et+al%2E+Code+llama%3A+Open+foundation+models+for+code%2E+arXi) [our card](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
2. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023. 1 [find](https://scholar.google.com/scholar?q=Shijie+Wu%2C+Ozan+Irsoy%2C+Steven+Lu%2C+Vadim+Dabravolski%2C+Mark+Dredze%2C+Sebastian+Gehrmann%2C+Prabhanjan+Kambadur%2C+David+Rosenberg%2C+and+Gideon+Mann%2E+Bloomberggpt%3A+A+large+language+model+for+finance%2E+arXiv+pre) [our card](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/)
3. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, pages 1–11, 2023. 1 [find](https://scholar.google.com/scholar?q=Arun+James+Thirunavukarasu%2C+Darren+Shu+Jeng+Ting%2C+Kabilan+Elangovan%2C+Laura+Gutierrez%2C+Ting+Fang+Tan%2C+and+Daniel+Shu+Wei+Ting%2E+Large+language+models+in+medicine%2E+Nature+medicine%2C+pages+1%E2%80%9311%2C+2023%2E+1)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 1 [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
5. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 1, 6 [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
6. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335, 2023. 1 [find](https://scholar.google.com/scholar?q=Ameet+Deshpande%2C+Vishvak+Murahari%2C+Tanmay+Rajpurohit%2C+Ashwin+Kalyan%2C+and+Karthik+Narasimhan%2E+Toxicity+in+chatgpt%3A+Analyzing+persona%2Dassigned+language+models%2E+arXiv+preprint+arXiv%3A2304%2E05335%2C+2023%2E+1) [our card](https://md.research.elamur.ai/free/en/papers/deshpande-2023-toxicity-chatgpt-analyzing-persona-assigned/)
7. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. 1 [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+)
8. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506–17533. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Tomasz+Korbak%2C+Kejian+Shi%2C+Angelica+Chen%2C+Rasika+Vinayak+Bhalerao%2C+Christopher+Buckley%2C+Jason+Phang%2C+Samuel+R+Bowman%2C+and+Ethan+Perez%2E+Pretraining+language+models+with+human+preferences%2E+In+Internatio)
9. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback, 2022. URL https://arxiv.org/abs/2203.02155, 13, 2022. [↗](https://arxiv.org/abs/2203.02155)
10. Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. 1 [find](https://scholar.google.com/scholar?q=Amelia+Glaese%2C+Nat+McAleese%2C+Maja+Trebacz%2C+John+Aslanides%2C+Vlad+Firoiu%2C+Timo+Ewalds%2C+Maribeth+Rauh%2C+Laura+Weidinger%2C+Martin+Chadwick%2C+Phoebe+Thacker%2C+et+al%2E+Improving+alignment+of+dialogue+agents+via+) [our card](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/)
11. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. 1, 2, 3, 5, 6, 7, 15, 21 [find](https://scholar.google.com/scholar?q=Andy+Zou%2C+Zifan+Wang%2C+J%2E+Zico+Kolter%2C+and+Matt+Fredrikson%2E+Universal+and+transferable+adversarial+attacks+on+aligned+language+models%2C+2023%2E+1%2C+2%2C+3%2C+5%2C+6%2C+7%2C+15%2C+21) [our card](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/)
12. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483, 2023. 1 [find](https://scholar.google.com/scholar?q=Alexander+Wei%2C+Nika+Haghtalab%2C+and+Jacob+Steinhardt%2E+Jailbroken%3A+How+does+llm+safety+training+fail%3F+arXiv+preprint+arXiv%3A2307%2E02483%2C+2023%2E+1) [our card](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/)
13. Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned?, 2023. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Milad+Nasr%2C+Christopher+A%2E+Choquette%2DChoo%2C+Matthew+Jagielski%2C+Irena+Gao%2C+Anas+Awadalla%2C+Pang+Wei+Koh%2C+Daphne+Ippolito%2C+Katherine+Lee%2C+Florian+Tramer%2C+and+Ludwig+Schmidt%2E+Are+aligned+)
14. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In The Second Workshop on New Frontiers in Adversarial Machine Learning, 2023. [find](https://scholar.google.com/scholar?q=Xiangyu+Qi%2C+Kaixuan+Huang%2C+Ashwinee+Panda%2C+Mengdi+Wang%2C+and+Prateek+Mittal%2E+Visual+adversarial+examples+jailbreak+aligned+large+language+models%2E+In+The+Second+Workshop+on+New+Frontiers+in+Adversarial+)
15. Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03348, 2023. 1 — [arXiv:2311.03348](https://arxiv.org/abs/2311.03348) [find](https://scholar.google.com/scholar?q=Rusheb+Shah%2C+Soroush+Pour%2C+Arush+Tagade%2C+Stephen+Casper%2C+Javier+Rando%2C+et+al%2E+Scalable+and+transferable+black%2Dbox+jailbreaks+for+language+models+via+persona+modulation%2E+arXiv+preprint+arXiv%3A2311%2E03348)
16. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. arXiv preprint arXiv:1908.06083, 2019. 1, 3 [find](https://scholar.google.com/scholar?q=Emily+Dinan%2C+Samuel+Humeau%2C+Bharath+Chintagunta%2C+and+Jason+Weston%2E+Build+it+break+it+fix+it+for+dialogue+safety%3A+Robustness+from+adversarial+human+attack%2E+arXiv+preprint+arXiv%3A1908%2E06083%2C+2019%2E+1%2C+3) [our card](https://md.research.elamur.ai/free/en/papers/dinan-2019-build-it-break-it-fix/)
17. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118, 2020. 1, 3 [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Tongshuang+Wu%2C+Carlos+Guestrin%2C+and+Sameer+Singh%2E+Beyond+accuracy%3A+Behavioral+testing+of+nlp+models+with+checklist%2E+arXiv+preprint+arXiv%3A2005%2E04118%2C+2020%2E+1%2C+3)
18. Natalie Maus, Patrick Chao, Eric Wong, and Jacob Gardner. Black box adversarial prompting for foundation models, 2023. 1, 14 [find](https://scholar.google.com/scholar?q=Natalie+Maus%2C+Patrick+Chao%2C+Eric+Wong%2C+and+Jacob+Gardner%2E+Black+box+adversarial+prompting+for+foundation+models%2C+2023%2E+1%2C+14)
19. Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically auditing large language models via discrete optimization, 2023. 1 [find](https://scholar.google.com/scholar?q=Erik+Jones%2C+Anca+Dragan%2C+Aditi+Raghunathan%2C+and+Jacob+Steinhardt%2E+Automatically+auditing+large+language+models+via+discrete+optimization%2C+2023%2E+1)
20. Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023. 1, 7, 14, 15 [find](https://scholar.google.com/scholar?q=Alexander+Robey%2C+Eric+Wong%2C+Hamed+Hassani%2C+and+George+J%2E+Pappas%2E+Smoothllm%3A+Defending+large+language+models+against+jailbreaking+attacks%2E+arXiv+preprint+arXiv%3A2310%2E03684%2C+2023%2E+1%2C+7%2C+14%2C+15) [our card](https://md.research.elamur.ai/free/en/papers/robey-2023-smoothllm-defending-large-language-models/)
21. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023. 3, 5, 6 [find](https://scholar.google.com/scholar?q=Hakan+Inan%2C+Kartikeya+Upasani%2C+Jianfeng+Chi%2C+Rashi+Rungta%2C+Krithika+Iyer%2C+Yuning+Mao%2C+Michael+Tontchev%2C+Qing+Hu%2C+Brian+Fuller%2C+Davide+Testuggine%2C+and+Madian+Khabsa%2E+Llama+guard%3A+Llm%2Dbased+input%2Doutput) [our card](https://md.research.elamur.ai/free/en/papers/inan-2023-llama-guard-llm-based-input/)
22. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, et al. Constitutional ai: Harmlessness from ai feedback, 2022. 3 [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Saurav+Kadavath%2C+Sandipan+Kundu%2C+Amanda+Askell%2C+Jackson+Kernion%2C+Andy+Jones%2C+et+al%2E+Constitutional+ai%3A+Harmlessness+from+ai+feedback%2C+2022%2E+3) [our card](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
23. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022. 3 [find](https://scholar.google.com/scholar?q=Deep+Ganguli%2C+Liane+Lovitt%2C+Jackson+Kernion%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Saurav+Kadavath%2C+et+al%2E+Red+teaming+language+models+to+reduce+harms%3A+Methods%2C+scaling+behaviors%2C+and+lessons+learned%2C+2022%2E+3) [our card](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/)
24. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022. 3 [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Saffron+Huang%2C+Francis+Song%2C+Trevor+Cai%2C+Roman+Ring%2C+John+Aslanides%2C+Amelia+Glaese%2C+Nat+McAleese%2C+and+Geoffrey+Irving%2E+Red+teaming+language+models+with+language+models%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/)
25. Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. Improving question answering model robustness with synthetic adversarial data generation. arXiv preprint arXiv:2104.08678, 2021. 3 [find](https://scholar.google.com/scholar?q=Max+Bartolo%2C+Tristan+Thrush%2C+Robin+Jia%2C+Sebastian+Riedel%2C+Pontus+Stenetorp%2C+and+Douwe+Kiela%2E+Improving+question+answering+model+robustness+with+synthetic+adversarial+data+generation%2E+arXiv+preprint+ar)
26. Max Bartolo, Tristan Thrush, Sebastian Riedel, Pontus Stenetorp, Robin Jia, and Douwe Kiela. Models in the loop: Aiding crowdworkers with generative annotation assistants. arXiv preprint arXiv:2112.09062, 2021. 3 [find](https://scholar.google.com/scholar?q=Max+Bartolo%2C+Tristan+Thrush%2C+Sebastian+Riedel%2C+Pontus+Stenetorp%2C+Robin+Jia%2C+and+Douwe+Kiela%2E+Models+in+the+loop%3A+Aiding+crowdworkers+with+generative+annotation+assistants%2E+arXiv+preprint+arXiv%3A2112%2E09)
27. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020. 3, 14 [find](https://scholar.google.com/scholar?q=Taylor+Shin%2C+Yasaman+Razeghi%2C+Robert+L+Logan+IV%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+Autoprompt%3A+Eliciting+knowledge+from+language+models+with+automatically+generated+prompts%2E+arXiv+preprint+arXiv%3A2010%2E15) [our card](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/)
28. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms, 2024. 3, 24, 25 [find](https://scholar.google.com/scholar?q=Yi+Zeng%2C+Hongpeng+Lin%2C+Jingwen+Zhang%2C+Diyi+Yang%2C+Ruoxi+Jia%2C+and+Weiyan+Shi%2E+How+johnny+can+persuade+llms+to+jailbreak+them%3A+Rethinking+persuasion+to+challenge+ai+safety+by+humanizing+llms%2C+2024%2E+3%2C+24)
29. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Proce… [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+brian+ichter%2C+Fei+Xia%2C+Ed+Chi%2C+Quoc+V+Le%2C+and+Denny+Zhou%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+In+S%2E+Koyejo%2C+S) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
30. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation, 2023. 5 [find](https://scholar.google.com/scholar?q=Yangsibo+Huang%2C+Samyak+Gupta%2C+Mengzhou+Xia%2C+Kai+Li%2C+and+Danqi+Chen%2E+Catastrophic+jailbreak+of+open%2Dsource+llms+via+exploiting+generation%2C+2023%2E+5) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-catastrophic-jailbreak-open-source-llms/)
31. Mantas Mazeika, Andy Zou, Norman Mu, Long Phan, Zifan Wang, Chunru Yu, Adam Khoja, Fengqing Jiang, Aidan O’Gara, Ellie Sakhaee, Zhen Xiang, Arezoo Rajabi, Dan Hendrycks, Radha Poovendran, Bo Li, and David Forsyth. Tdc 2023 (llm edition): The trojan detection challenge. In NeurIPS Competition Track,… [find](https://scholar.google.com/scholar?q=Mantas+Mazeika%2C+Andy+Zou%2C+Norman+Mu%2C+Long+Phan%2C+Zifan+Wang%2C+Chunru+Yu%2C+Adam+Khoja%2C+Fengqing+Jiang%2C+Aidan+O%E2%80%99Gara%2C+Ellie+Sakhaee%2C+Zhen+Xiang%2C+Arezoo+Rajabi%2C+Dan+Hendrycks%2C+Radha+Poovendran%2C+Bo+Li%2C+and+D)
32. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv preprint arXiv:2404.01318,… [find](https://scholar.google.com/scholar?q=Patrick+Chao%2C+Edoardo+Debenedetti%2C+Alexander+Robey%2C+Maksym+Andriushchenko%2C+Francesco+Croce%2C+Vikash+Sehwag%2C+Edgar+Dobriban%2C+Nicolas+Flammarion%2C+George+J+Pappas%2C+Florian+Tramer%2C+et+al%2E+Jailbreakbench%3A+A) [our card](https://md.research.elamur.ai/free/en/papers/chao-2024-jailbreakbench-open-robustness-benchmark-jailbreaking/)
33. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
34. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 6 [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric%2E+P+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+Judging+llm%2Das%2Da%2Djud)
35. OpenAI. Gpt-4 technical report, 2023. 6 [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E+6) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
36. Gemini Team. Gemini: A family of highly capable multimodal models, 2023. 6 [find](https://scholar.google.com/scholar?q=Gemini+Team%2E+Gemini%3A+A+family+of+highly+capable+multimodal+models%2C+2023%2E+6) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/)
37. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614, 2023. 7, 14 [find](https://scholar.google.com/scholar?q=Neel+Jain%2C+Avi+Schwarzschild%2C+Yuxin+Wen%2C+Gowthami+Somepalli%2C+John+Kirchenbauer%2C+Ping%2Dyeh+Chiang%2C+Micah+Goldblum%2C+Aniruddha+Saha%2C+Jonas+Geiping%2C+and+Tom+Goldstein%2E+Baseline+defenses+for+adversarial+att) [our card](https://md.research.elamur.ai/free/en/papers/jain-2023-baseline-defenses-adversarial-attacks-against/)
38. Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132, 2023. 7, 14 [find](https://scholar.google.com/scholar?q=Gabriel+Alon+and+Michael+Kamfonas%2E+Detecting+language+model+attacks+with+perplexity%2E+arXiv+preprint+arXiv%3A2308%2E14132%2C+2023%2E+7%2C+14) [our card](https://md.research.elamur.ai/free/en/papers/alon-2023-detecting-language-model-attacks-perplexity/)
39. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. 14 [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wojciech+Zaremba%2C+Ilya+Sutskever%2C+Joan+Bruna%2C+Dumitru+Erhan%2C+Ian+Goodfellow%2C+and+Rob+Fergus%2E+Intriguing+properties+of+neural+networks%2E+arXiv+preprint+arXiv%3A1312%2E6199%2C+2013%2E+14) [our card](https://md.research.elamur.ai/free/en/papers/szegedy-2013-intriguing-properties-neural-networks/)
40. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. 14 [find](https://scholar.google.com/scholar?q=Ian+J+Goodfellow%2C+Jonathon+Shlens%2C+and+Christian+Szegedy%2E+Explaining+and+harnessing+adversarial+examples%2E+arXiv+preprint+arXiv%3A1412%2E6572%2C+2014%2E+14) [our card](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/)
41. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. 14 [find](https://scholar.google.com/scholar?q=Aleksander+Madry%2C+Aleksandar+Makelov%2C+Ludwig+Schmidt%2C+Dimitris+Tsipras%2C+and+Adrian+Vladu%2E+Towards+deep+learning+models+resistant+to+adversarial+attacks%2E+arXiv+preprint+arXiv%3A1706%2E06083%2C+2017%2E+14) [our card](https://md.research.elamur.ai/free/en/papers/madry-2017-towards-deep-learning-models-resistant/)
42. Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, et al. Adversarial attacks and defences competition. In The NIPS’17 Competition: Building Intelligent Systems, pages 195–231. Springer, 2018. [find](https://scholar.google.com/scholar?q=Alexey+Kurakin%2C+Ian+Goodfellow%2C+Samy+Bengio%2C+Yinpeng+Dong%2C+Fangzhou+Liao%2C+Ming+Liang%2C+Tianyu+Pang%2C+Jun+Zhu%2C+Xiaolin+Hu%2C+Cihang+Xie%2C+et+al%2E+Adversarial+attacks+and+defences+competition%2E+In+The+NIPS%E2%80%9917+)
43. Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International conference on learning representations, 2019. 14 [find](https://scholar.google.com/scholar?q=Yisen+Wang%2C+Difan+Zou%2C+Jinfeng+Yi%2C+James+Bailey%2C+Xingjun+Ma%2C+and+Quanquan+Gu%2E+Improving+adversarial+robustness+requires+revisiting+misclassified+examples%2E+In+International+conference+on+learning+repre)
44. Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE symposium on security and privacy (SP), pages 656–672. IEEE, 2019. 14 [find](https://scholar.google.com/scholar?q=Mathias+Lecuyer%2C+Vaggelis+Atlidakis%2C+Roxana+Geambasu%2C+Daniel+Hsu%2C+and+Suman+Jana%2E+Certified+robustness+to+adversarial+examples+with+differential+privacy%2E+In+2019+IEEE+symposium+on+security+and+privacy)
45. Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pages 1310–1320. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Jeremy+Cohen%2C+Elan+Rosenfeld%2C+and+Zico+Kolter%2E+Certified+adversarial+robustness+via+randomized+smoothing%2E+In+international+conference+on+machine+learning%2C+pages+1310%E2%80%931320%2E+PMLR%2C+2019%2E)
46. Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in Neural Information Processing Systems, 32, 2019. 14 [find](https://scholar.google.com/scholar?q=Hadi+Salman%2C+Jerry+Li%2C+Ilya+Razenshteyn%2C+Pengchuan+Zhang%2C+Huan+Zhang%2C+Sebastien+Bubeck%2C+and+Greg+Yang%2E+Provably+robust+deep+learning+via+adversarially+trained+smoothed+classifiers%2E+Advances+in+Neural+)
47. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 38… [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Adam+Fisch%2C+and+Danqi+Chen%2E+Making+pre%2Dtrained+language+models+better+few%2Dshot+learners%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/)
48. Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with ”gradient descent” and beam search, 2023. 14 [find](https://scholar.google.com/scholar?q=Reid+Pryzant%2C+Dan+Iter%2C+Jerry+Li%2C+Yin+Tat+Lee%2C+Chenguang+Zhu%2C+and+Michael+Zeng%2E+Automatic+prompt+optimization+with+%E2%80%9Dgradient+descent%E2%80%9D+and+beam+search%2C+2023%2E+14) [our card](https://md.research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/)
49. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers, 2023. 14 [find](https://scholar.google.com/scholar?q=Yongchao+Zhou%2C+Andrei+Ioan+Muresanu%2C+Ziwen+Han%2C+Keiran+Paster%2C+Silviu+Pitis%2C+Harris+Chan%2C+and+Jimmy+Ba%2E+Large+language+models+are+human%2Dlevel+prompt+engineers%2C+2023%2E+14) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/)
50. Siyuan Liang, Baoyuan Wu, Yanbo Fan, Xingxing Wei, and Xiaochun Cao. Parallel rectangle flip attack: A query-based black-box attack against object detection, 2022. 14 [find](https://scholar.google.com/scholar?q=Siyuan+Liang%2C+Baoyuan+Wu%2C+Yanbo+Fan%2C+Xingxing+Wei%2C+and+Xiaochun+Cao%2E+Parallel+rectangle+flip+attack%3A+A+query%2Dbased+black%2Dbox+attack+against+object+detection%2C+2022%2E+14)
51. Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, July 2018. 14 [find](https://scholar.google.com/scholar?q=Andrew+Ilyas%2C+Logan+Engstrom%2C+Anish+Athalye%2C+and+Jessy+Lin%2E+Black%2Dbox+adversarial+attacks+with+limited+queries+and+information%2E+In+Proceedings+of+the+35th+International+Conference+on+Machine+Learning%2C)
52. Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv:1611.02770, 2016. 14 — [arXiv:1611.02770](https://arxiv.org/abs/1611.02770) [find](https://scholar.google.com/scholar?q=Yanpei+Liu%2C+Xinyun+Chen%2C+Chang+Liu%2C+and+Dawn+Song%2E+Delving+into+transferable+adversarial+examples+and+black%2Dbox+attacks%2E+arXiv+preprint+arXiv%3A1611%2E02770%2C+2016%2E+14+%E2%80%94+%5BarXiv%3A1611%2E02770%5D%28https%3A%2F%2Farxiv%2Eor)
53. Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, CCS ’17. ACM, November 2017. 14 [find](https://scholar.google.com/scholar?q=Pin%2DYu+Chen%2C+Huan+Zhang%2C+Yash+Sharma%2C+Jinfeng+Yi%2C+and+Cho%2DJui+Hsieh%2E+Zoo%3A+Zeroth+order+optimization+based+black%2Dbox+attacks+to+deep+neural+networks+without+training+substitute+models%2E+In+Proceedings+o)
54. Jiabao Ji, Bairu Hou, Alexander Robey, George J Pappas, Hamed Hassani, Yang Zhang, Eric Wong, and Shiyu Chang. Defending large language models against jailbreak attacks via semantic smoothing. arXiv preprint arXiv:2402.16192, 2024. 14 [find](https://scholar.google.com/scholar?q=Jiabao+Ji%2C+Bairu+Hou%2C+Alexander+Robey%2C+George+J+Pappas%2C+Hamed+Hassani%2C+Yang+Zhang%2C+Eric+Wong%2C+and+Shiyu+Chang%2E+Defending+large+language+models+against+jailbreak+attacks+via+semantic+smoothing%2E+arXiv+p)
55. Andy Zhou, Bo Li, and Haohan Wang. Robust prompt optimization for defending language models against jailbreaking attacks. arXiv preprint arXiv:2401.17263, 2024. 14 — [arXiv:2401.17263](https://arxiv.org/abs/2401.17263) [find](https://scholar.google.com/scholar?q=Andy+Zhou%2C+Bo+Li%2C+and+Haohan+Wang%2E+Robust+prompt+optimization+for+defending+language+models+against+jailbreaking+attacks%2E+arXiv+preprint+arXiv%3A2401%2E17263%2C+2024%2E+14+%E2%80%94+%5BarXiv%3A2401%2E17263%5D%28https%3A%2F%2Farxiv%2Eo)