---
title: "Catastrophic Jailbreak of Open-source LLMs via — Huang 2023"
description: "Safety tuning of open-source LLMs is usually checked under one fixed decoding setup."
authors: ["Yangsibo Huang", "Samyak Gupta", "Mengzhou Xia", "Kai Li", "Danqi Chen"]
year: 2023
venue: "arXiv:2310.06987"
source: "https://arxiv.org/abs/2310.06987"
doi: "10.48550/arxiv.2310.06987"
arxiv: "2310.06987"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/huang-2023-catastrophic-jailbreak-open-source-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/huang-2023-catastrophic-jailbreak-open-source-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

# Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

**Authors:** Yangsibo Huang, Samyak Gupta, [Mengzhou Xia](https://md.research.elamur.ai/free/en/authors/mengzhou-xia/), Kai Li, [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) · **Year:** 2023 · **Venue:** arXiv:2310.06987 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.06987) · [PDF](https://arxiv.org/pdf/2310.06987) · [DOI 10.48550/arxiv.2310.06987](https://doi.org/10.48550/arxiv.2310.06987) · [arXiv 2310.06987](https://arxiv.org/abs/2310.06987) · [OpenAlex](https://openalex.org/W4387595881)

**Citations:** 11 citations (openalex-snapshot · 2026-10-14) · [151](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Safety tuning of open-source LLMs is usually checked under one fixed decoding setup. For example, LLaMA2 was assessed with top-p 0.9, temperature 0.1 and a system prompt. The authors argue that this narrow testing hides serious weaknesses. They introduce the generation exploitation attack, which changes no input text. It simply drops the system prompt and sweeps decoding choices: temperature, top-k, top-p, plus multiple sampling and constrained decoding. They tested 11 models from the LLaMA2, Vicuna, Falcon and MPT families on AdvBench and on MaliciousInstruct, a new benchmark they built. Outputs were scored with a trained classifier that matches human judgments better than string matching does. Misalignment rose from near zero to above 95% at roughly 30× less compute than the leading adversarial-prompt attack. As a countermeasure, the authors propose generation-aware alignment, which fine-tunes on responses sampled under many decoding configurations. The study shows that red teaming should cover a range of generation strategies, not just the defaults.

## Key points
- **Benchmarks:** AdvBench has 500 harmful instructions. MaliciousInstruct has 100 instructions spread over ten intent categories, including tax fraud, hacking, defamation and cyberbullying.
- **Attack strength:** Varying decoding alone pushed 9 of 11 models above 95% misalignment. Combining all techniques pushed every model past that level. Human review found that at least half of the misaligned answers contained genuinely harmful instructions.
- **LLaMA2-chat on MaliciousInstruct:** With the system prompt and greedy decoding, the attack success rate was 0%. Without the system prompt and with varied decoding, it reached 81% (7B) and 88% (13B).
- **Defense:** Generation-aware alignment lowered misalignment from 95% to 69%.
- **Proprietary model:** The same approach against gpt-3.5-turbo through the OpenAI API succeeded only about 7% of the time. Code is available at github.com/Princeton-SysML/Jailbreak_LLM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/)

## Cited by

Cited by

151 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 136 not yet.

- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://md.research.elamur.ai/free/en/papers/chao-2024-jailbreakbench-open-robustness-benchmark-jailbreaking/)
- [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/)
- [Refusal in Language Models Is Mediated by a Single Direction](https://md.research.elamur.ai/free/en/papers/arditi-2024-refusal-language-models-mediated-single/)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://md.research.elamur.ai/free/en/papers/yuan-2023-gpt-4-too-smart-safe/)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://md.research.elamur.ai/free/en/papers/robey-2023-smoothllm-defending-large-language-models/)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://md.research.elamur.ai/free/en/papers/lin-2023-unlocking-spell-base-llms-rethinking/)
- [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://md.research.elamur.ai/free/en/papers/li-2023-deepinception-hypnotize-large-language-model/)
- [SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding](https://md.research.elamur.ai/free/en/papers/xu-2024-safedecoding-defending-against-jailbreak-attacks/)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbreak-guard-aligned-language-models/)
- [SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-salad-bench-hierarchical-comprehensive-safety/)
- [In-Distribution Steering: Balancing Control and Coherence in Language Model Generation](https://md.research.elamur.ai/free/en/papers/vogels-2026-distribution-steering-balancing-control-coherence/)
- [Judging Instruction Responses in a Low-Resource Language: A Case Study on Basque](https://md.research.elamur.ai/free/en/papers/ponce-2026-judging-instruction-responses-low-resource/)
- [面向大语言模型的越狱攻击综述](https://md.research.elamur.ai/free/en/papers/li-2024-jailbreak-attack-large/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{huang2023catastrophic,
  title = {{Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation}},
  author = {Huang, Yangsibo and Gupta, Samyak and Xia, Mengzhou and Li, Kai and Chen, Danqi},
  year = {2023},
  doi = {10.48550/arxiv.2310.06987},
  eprint = {2310.06987},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.06987}
}
```

- [← previous Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/)
- [next → GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://md.research.elamur.ai/free/en/papers/yu-2023-gptfuzzer-red-teaming-large-language/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

47 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alex Albert. Jailbreak chat, 2023. URL [https://www.jailbreakchat.com/](https://www.jailbreakchat.com/). [↗](https://www.jailbreakchat.com/)
2. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language model with state-of-the-art… [find](https://scholar.google.com/scholar?q=Ebtesam+Almazrouei%2C+Hamza+Alobeidli%2C+Abdulaziz+Alshamsi%2C+Alessandro+Cappelli%2C+Ruxandra+Cojocaru%2C+Merouane+Debbah%2C+Etienne+Goffinet%2C+Daniel+Heslow%2C+Julien+Launay%2C+Quentin+Malartic%2C+Badreddine+Noune%2C+Ba)
3. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. [find](https://scholar.google.com/scholar?q=Rohan+Anil%2C+Andrew+M+Dai%2C+Orhan+Firat%2C+Melvin+Johnson%2C+Dmitry+Lepikhin%2C+Alexandre+Passos%2C+Siamak+Shakeri%2C+Emanuel+Taropa%2C+Paige+Bailey%2C+Zhifeng+Chen%2C+et+al%2E+Palm+2+technical+report%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
4. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+a)
5. Azure. Content filtering, 2023. URL [https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter). [↗](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter)
6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
7. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Saurav+Kadavath%2C+Sandipan+Kundu%2C+Amanda+Askell%2C+Jackson+Kernion%2C+Andy+Jones%2C+Anna+Chen%2C+Anna+Goldie%2C+Azalia+Mirhoseini%2C+Cameron+McKinnon%2C+et+al%2E+Constitutional+ai%3A+Harmlessness+from+ai+fee) [our card](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
8. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Tom+B+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
9. Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447, 2023. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Milad+Nasr%2C+Christopher+A+Choquette%2DChoo%2C+Matthew+Jagielski%2C+Irena+Gao%2C+Anas+Awadalla%2C+Pang+Wei+Koh%2C+Daphne+Ippolito%2C+Katherine+Lee%2C+Florian+Tramer%2C+et+al%2E+Are+aligned+neural+network)
10. Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, establish, exploit: Red teaming language models from scratch. arXiv preprint arXiv:2306.09442, 2023. [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Jason+Lin%2C+Joe+Kwon%2C+Gatlen+Culp%2C+and+Dylan+Hadfield%2DMenell%2E+Explore%2C+establish%2C+exploit%3A+Red+teaming+language+models+from+scratch%2E+arXiv+preprint+arXiv%3A2306%2E09442%2C+2023%2E)
11. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna…](https://lmsys.org/blog/2023-03-30-vicuna…) [↗](https://lmsys.org/blog/2023-03-30-vicuna…)
12. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurIPS, 2017. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+NeurIPS%2C+2017%2E)
13. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+prep)
14. Lavina Daryanani. How to jailbreak chatgpt., 2023. how-to-jailbreak-chatgpt. [find](https://scholar.google.com/scholar?q=Lavina+Daryanani%2E+How+to+jailbreak+chatgpt%2E%2C+2023%2E+how%2Dto%2Djailbreak%2Dchatgpt%2E)
15. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In ACL, 2018. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+Hierarchical+neural+story+generation%2E+In+ACL%2C+2018%2E)
16. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. [find](https://scholar.google.com/scholar?q=Deep+Ganguli%2C+Liane+Lovitt%2C+Jackson+Kernion%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Saurav+Kadavath%2C+Ben+Mann%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Kamal+Ndousse%2C+et+al%2E+Red+teaming+language+models+to+reduce+harms%3A+Me) [our card](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/)
17. Amelia Glaese, Nat McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. [find](https://scholar.google.com/scholar?q=Amelia+Glaese%2C+Nat+McAleese%2C+Maja+Trkebacz%2C+John+Aslanides%2C+Vlad+Firoiu%2C+Timo+Ewalds%2C+Maribeth+Rauh%2C+Laura+Weidinger%2C+Martin+Chadwick%2C+Phoebe+Thacker%2C+et+al%2E+Improving+alignment+of+dialogue+agents+via) [our card](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/)
18. URL [https://watcher.guru/news/Dongyoung](https://watcher.guru/news/Dongyoung) Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302.08215, 2023. [↗](https://watcher.guru/news/Dongyoung)
19. Google. An important next step on our ai journey, 2023. URL [https://blog.google/technology/ai/bard-google-ai-search-updates/](https://blog.google/technology/ai/bard-google-ai-search-updates/). [↗](https://blog.google/technology/ai/bard-google-ai-search-updates/)
20. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers. In ACL, 2021. [find](https://scholar.google.com/scholar?q=Chuan+Guo%2C+Alexandre+Sablayrolles%2C+Herve%CC%81+Je%CC%81gou%2C+and+Douwe+Kiela%2E+Gradient%2Dbased+adversarial+attacks+against+text+transformers%2E+In+ACL%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/guo-2021-gradient-based-adversarial-attacks-against/)
21. Julian Hazell. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972, 2023. — [arXiv:2305.06972](https://arxiv.org/abs/2305.06972) [find](https://scholar.google.com/scholar?q=Julian+Hazell%2E+Large+language+models+can+be+used+to+effectively+scale+spear+phishing+campaigns%2E+arXiv+preprint+arXiv%3A2305%2E06972%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E06972%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E06972%29)
22. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2020. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+The+curious+case+of+neural+text+degeneration%2E+In+ICLR%2C+2020%2E)
23. Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically auditing large language models via discrete optimization. arXiv preprint arXiv:2303.04381, 2023. [find](https://scholar.google.com/scholar?q=Erik+Jones%2C+Anca+Dragan%2C+Aditi+Raghunathan%2C+and+Jacob+Steinhardt%2E+Automatically+auditing+large+language+models+via+discrete+optimization%2E+arXiv+preprint+arXiv%3A2303%2E04381%2C+2023%2E)
24. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733, 2023. — [arXiv:2302.05733](https://arxiv.org/abs/2302.05733) [find](https://scholar.google.com/scholar?q=Daniel+Kang%2C+Xuechen+Li%2C+Ion+Stoica%2C+Carlos+Guestrin%2C+Matei+Zaharia%2C+and+Tatsunori+Hashimoto%2E+Exploiting+programmatic+behavior+of+llms%3A+Dual%2Duse+through+standard+security+attacks%2E+arXiv+preprint+arXiv)
25. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In ICML, 2023. [find](https://scholar.google.com/scholar?q=Tomasz+Korbak%2C+Kejian+Shi%2C+Angelica+Chen%2C+Rasika+Vinayak+Bhalerao%2C+Christopher+Buckley%2C+Jason+Phang%2C+Samuel+R+Bowman%2C+and+Ethan+Perez%2E+Pretraining+language+models+with+human+preferences%2E+In+ICML%2C+2023) [our card](https://md.research.elamur.ai/free/en/papers/korbak-2023-pretraining-language-models-human-preferences/)
26. Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models, 2023. [find](https://scholar.google.com/scholar?q=Raz+Lapid%2C+Ron+Langberg%2C+and+Moshe+Sipper%2E+Open+sesame%21+universal+black+box+jailbreaking+of+large+language+models%2C+2023%2E)
27. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197, 2023. [find](https://scholar.google.com/scholar?q=Haoran+Li%2C+Dadi+Guo%2C+Wei+Fan%2C+Mingshi+Xu%2C+and+Yangqiu+Song%2E+Multi%2Dstep+jailbreaking+privacy+attacks+on+chatgpt%2E+arXiv+preprint+arXiv%3A2304%2E05197%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-multi-step-jailbreaking-privacy-attacks/)
28. Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02676, 2023a. [find](https://scholar.google.com/scholar?q=Hao+Liu%2C+Carmelo+Sferrazza%2C+and+Pieter+Abbeel%2E+Languages+are+rewards%3A+Hindsight+finetuning+using+human+feedback%2E+arXiv+preprint+arXiv%3A2302%2E02676%2C+2023a%2E)
29. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860, 2023b. [find](https://scholar.google.com/scholar?q=Yi+Liu%2C+Gelei+Deng%2C+Zhengzi+Xu%2C+Yuekang+Li%2C+Yaowen+Zheng%2C+Ying+Zhang%2C+Lida+Zhao%2C+Tianwei+Zhang%2C+and+Yang+Liu%2E+Jailbreaking+chatgpt+via+prompt+engineering%3A+An+empirical+study%2E+arXiv+preprint+arXiv%3A2305) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-jailbreaking-chatgpt-prompt-engineering-empirical/)
30. MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www.mosaicml.com/blog/mpt-7b. Accessed: 2023-05-05. [find](https://scholar.google.com/scholar?q=MosaicML%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2C+2023%2E+URL+www%2Emosaicml%2Ecom%2Fblog%2Fmpt%2D7b%2E+Accessed%3A+2023%2D05%2D05%2E)
31. OpenAI. OpenAI: Introducing ChatGPT, 2022. URL [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
32. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+wi)
33. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In EMNLP, 2022. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Saffron+Huang%2C+Francis+Song%2C+Trevor+Cai%2C+Roman+Ring%2C+John+Aslanides%2C+Amelia+Glaese%2C+Nat+McAleese%2C+and+Geoffrey+Irving%2E+Red+teaming+language+models+with+language+models%2E+In+EMNLP%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/)
34. Huachuan Qiu, Shuai Zhang, Anqi Li, Hongliang He, and Zhenzhong Lan. Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models, 2023. [find](https://scholar.google.com/scholar?q=Huachuan+Qiu%2C+Shuai+Zhang%2C+Anqi+Li%2C+Hongliang+He%2C+and+Zhenzhong+Lan%2E+Latent+jailbreak%3A+A+benchmark+for+evaluating+text+safety+and+output+robustness+of+large+language+models%2C+2023%2E)
35. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+Technical+report%2C+OpenAI%2C+2018%2E)
36. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
37. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. ” do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825, 2023. [find](https://scholar.google.com/scholar?q=Xinyue+Shen%2C+Zeyuan+Chen%2C+Michael+Backes%2C+Yun+Shen%2C+and+Yang+Zhang%2E+%E2%80%9D+do+anything+now%E2%80%9D%3A+Characterizing+and+evaluating+in%2Dthe%2Dwild+jailbreak+prompts+on+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E) [our card](https://md.research.elamur.ai/free/en/papers/shen-2023-anything-now-characterizing-evaluating-wild/)
38. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020. [find](https://scholar.google.com/scholar?q=Taylor+Shin%2C+Yasaman+Razeghi%2C+Robert+L+Logan+IV%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+Autoprompt%3A+Eliciting+knowledge+from+language+models+with+automatically+generated+prompts%2E+arXiv+preprint+arXiv%3A2010%2E15) [our card](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/)
39. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+In+NeurIPS%2C+2020%2E)
40. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timothe%CC%81e+Lacroix%2C+Baptiste+Rozie%CC%80re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+Llama%3A+Open+and+efficient+found)
41. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
42. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023. [find](https://scholar.google.com/scholar?q=Yufei+Wang%2C+Wanjun+Zhong%2C+Liangyou+Li%2C+Fei+Mi%2C+Xingshan+Zeng%2C+Wenyong+Huang%2C+Lifeng+Shang%2C+Xin+Jiang%2C+and+Qun+Liu%2E+Aligning+large+language+models+with+human%3A+A+survey%2E+arXiv+preprint+arXiv%3A2307%2E12966%2C) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-aligning-large-language-models-human/)
43. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483, 2023. [find](https://scholar.google.com/scholar?q=Alexander+Wei%2C+Nika+Haghtalab%2C+and+Jacob+Steinhardt%2E+Jailbroken%3A+How+does+llm+safety+training+fail%3F+arXiv+preprint+arXiv%3A2307%2E02483%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/)
44. Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302.03668, 2023. [find](https://scholar.google.com/scholar?q=Yuxin+Wen%2C+Neel+Jain%2C+John+Kirchenbauer%2C+Micah+Goldblum%2C+Jonas+Geiping%2C+and+Tom+Goldstein%2E+Hard+prompts+made+easy%3A+Gradient%2Dbased+discrete+optimization+for+prompt+tuning+and+discovery%2E+arXiv+preprint+)
45. Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Kaidi+Xu%2C+Zhouxing+Shi%2C+Huan+Zhang%2C+Yihan+Wang%2C+Kai%2DWei+Chang%2C+Minlie+Huang%2C+Bhavya+Kailkhura%2C+Xue+Lin%2C+and+Cho%2DJui+Hsieh%2E+Automatic+perturbation+analysis+for+scalable+certified+robustness+and+beyond%2E)
46. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+LIMA%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2C+2023%2E)
47. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. [find](https://scholar.google.com/scholar?q=Andy+Zou%2C+Zifan+Wang%2C+J+Zico+Kolter%2C+and+Matt+Fredrikson%2E+Universal+and+transferable+adversarial+attacks+on+aligned+language+models%2E+arXiv+preprint+arXiv%3A2307%2E15043%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/)