--- title: "Jailbreaking ChatGPT via Prompt Engineering — Liu 2023" description: "The paper examines how prompt engineering alone can bypass the safety restrictions OpenAI places on ChatGPT." authors: ["Yi Liu", "Gelei Deng", "Zhengzi Xu", "Yuekang Li", "Yaowen Zheng", "Ying Zhang", "Lida Zhao", "Tianwei Zhang", "Kailong Wang", "Yang Liu"] year: 2023 venue: "arXiv:2305.13860" source: "https://arxiv.org/abs/2305.13860" doi: "10.48550/arxiv.2305.13860" arxiv: "2305.13860" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/liu-2023-jailbreaking-chatgpt-prompt-engineering-empirical/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2023-jailbreaking-chatgpt-prompt-engineering-empirical/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study # Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study **Authors:** [Yi Liu](https://md.research.elamur.ai/free/en/authors/yi-liu/), Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, [Tianwei Zhang](https://md.research.elamur.ai/free/en/authors/tianwei-zhang/), Kailong Wang, [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/) · **Year:** 2023 · **Venue:** arXiv:2305.13860 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2305.13860) · [PDF](https://arxiv.org/pdf/2305.13860) · [DOI 10.48550/arxiv.2305.13860](https://doi.org/10.48550/arxiv.2305.13860) · [arXiv 2305.13860](https://arxiv.org/abs/2305.13860) · [OpenAlex](https://openalex.org/W4378506863) **Citations:** 98 citations (openalex-snapshot · 2026-10-14) · [168](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper examines how prompt engineering alone can bypass the safety restrictions OpenAI places on ChatGPT. Because the model is closed-source, the authors treat jailbreaking as a black-box problem. They study which kinds of jailbreak prompts exist, how well they work, and how strongly GPT-3.5-Turbo and GPT-4 resist them. They gathered 78 verified prompts from the jailbreakchat website, collected between February and late April 2023, and used them to build a taxonomy of 3 types and 10 patterns. Each prompt was paired with questions covering 8 scenarios prohibited by OpenAI's usage policy, and the authors queried both model versions at scale. Jailbreaks succeeded widely. Effectiveness depended strongly on prompt structure, and prompts combining several techniques, especially privilege escalation, performed best. GPT-4 resisted more than GPT-3.5-Turbo, and protection strength varied by scenario. The work matters because it supplies an open dataset and a systematic baseline for studying LLM jailbreaks. It also argues that content policies should be better aligned with real-world law and ethics. ## Key points - **Dataset and taxonomy:** 78 real-world jailbreak prompts, released openly on the project site (llm-jailbreak-study), were classified into 10 patterns grouped into 3 types. The types are pretending (97.44% of prompts), privilege escalation (17.96%) and attention shifting (6.41%). A single prompt can use more than one technique. - **Evaluation scale:** The study used 3,120 jailbreak questions across 8 prohibited scenarios, which expanded into 40 use-case scenarios. In total, 31,200 queries were sent to GPT-3.5-Turbo and GPT-4. - **Success rate:** Jailbreaks worked in 86.3% of the tested real-world scenarios, and prompts consistently evaded restrictions across all 40 of them. - **Model differences:** GPT-4 provided stronger protection than GPT-3.5-Turbo, and resistance differed depending on the content-policy area. - **Prompt evolution:** Analysis of prompt histories suggests that jailbreak prompts became more sophisticated and more effective over time. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 168 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 150 not yet. - [Security and Privacy Challenges of Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/das-2024-security-privacy-challenges-large-language/) - [Multilingual Jailbreak Challenges in Large Language Models](https://md.research.elamur.ai/free/en/papers/deng-2023-multilingual-jailbreak-challenges-large-language/) - [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/) - [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://md.research.elamur.ai/free/en/papers/huang-2023-catastrophic-jailbreak-open-source-llms/) - [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/) - [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://md.research.elamur.ai/free/en/papers/li-2023-deepinception-hypnotize-large-language-model/) - [SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-salad-bench-hierarchical-comprehensive-safety/) - [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://md.research.elamur.ai/free/en/papers/liu-2023-autodan-generating-stealthy-jailbreak-prompts/) - [Prompt Injection attack against LLM-integrated Applications](https://md.research.elamur.ai/free/en/papers/liu-2023-prompt-injection-attack-against-llm/) - [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/) - [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://md.research.elamur.ai/free/en/papers/mehrotra-2023-tree-attacks-jailbreaking-black-box/) - [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://md.research.elamur.ai/free/en/papers/qi-2023-fine-tuning-aligned-language-models/) - ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://md.research.elamur.ai/free/en/papers/shen-2023-anything-now-characterizing-evaluating-wild/) - [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-rolellm-benchmarking-eliciting-enhancing-role/) - [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbreak-guard-aligned-language-models/) - [SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding](https://md.research.elamur.ai/free/en/papers/xu-2024-safedecoding-defending-against-jailbreak-attacks/) - [Low-Resource Languages Jailbreak GPT-4](https://md.research.elamur.ai/free/en/papers/yong-2023-low-resource-languages-jailbreak-gpt/) - [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://md.research.elamur.ai/free/en/papers/yu-2023-gptfuzzer-red-teaming-large-language/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{liu2023jailbreaking, title = {{Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study}}, author = {Liu, Yi and Deng, Gelei and Xu, Zhengzi and Li, Yuekang and Zheng, Yaowen and Zhang, Ying and Zhao, Lida and Zhang, Tianwei and Wang, Kailong and Liu, Yang}, year = {2023}, doi = {10.48550/arxiv.2305.13860}, eprint = {2305.13860}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2305.13860} } ``` - [← previous Jailbreaking Black Box Large Language Models in Twenty Queries](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/) - [next → Jailbroken: How Does LLM Safety Training Fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) ## References 13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. B. Zhang, B. Haddow, and A. Birch, “Prompting large language model for machine translation: A case study,” CoRR, vol. abs/2301.07069, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2301.07069 [↗](https://doi.org/10.48550/arXiv.2301.07069) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2023-prompting-large-language-model-machine/) 2. C. Zhang, C. Zhang, S. Zheng, Y. Qiao, C. Li, M. Zhang, S. K. Dam, C. M. Thwal, Y. L. Tun, L. L. Huy, D. U. Kim, S. Bae, L. Lee, Y. Yang, H. T. Shen, I. S. Kweon, and C. S. Hong, “A complete survey on generative AI (AIGC): is chatgpt from GPT-4 to GPT-5 all you need?” CoRR, vol. abs/2303.11717, 2023… — [doi:10.48550/arxiv.2303.11717](https://doi.org/10.48550/arxiv.2303.11717) [find](https://scholar.google.com/scholar?q=C%2E+Zhang%2C+C%2E+Zhang%2C+S%2E+Zheng%2C+Y%2E+Qiao%2C+C%2E+Li%2C+M%2E+Zhang%2C+S%2E+K%2E+Dam%2C+C%2E+M%2E+Thwal%2C+Y%2E+L%2E+Tun%2C+L%2E+L%2E+Huy%2C+D%2E+U%2E+Kim%2C+S%2E+Bae%2C+L%2E+Lee%2C+Y%2E+Yang%2C+H%2E+T%2E+Shen%2C+I%2E+S%2E+Kweon%2C+and+C%2E+S%2E+Hong%2C+%E2%80%9CA+complete+survey+on) 3. J. Ni, T. Young, V. Pandelea, F. Xue, and E. Cambria, “Recent advances in deep learning based dialogue systems: a systematic survey,” Artif. Intell. Rev., vol. 56, no. 4, pp. 3055–3155, 2023. [Online]. Available: https://doi.org/10.1007/s10462-022-10248-8 — [doi:10.1007/s10462-022-10248-8](https://doi.org/10.1007/s10462-022-10248-8) [↗](https://doi.org/10.1007/s10462-022-10248-8) 4. “New chat,” [https://chat.openai.com/](https://chat.openai.com/), (Accessed on 02/02/2023). [↗](https://chat.openai.com/) 5. “Models - openai api,” [https://platform.openai.com/docs/models/](https://platform.openai.com/docs/models/), (Accessed on 02/02/2023). [↗](https://platform.openai.com/docs/models/) 6. “Openai,” [https://openai.com/](https://openai.com/), (Accessed on 02/02/2023). [↗](https://openai.com/) 7. H. Li, D. Guo, W. Fan, M. Xu, and Y. Song, “Multi-step jailbreaking privacy attacks on chatgpt,” 2023. [find](https://scholar.google.com/scholar?q=H%2E+Li%2C+D%2E+Guo%2C+W%2E+Fan%2C+M%2E+Xu%2C+and+Y%2E+Song%2C+%E2%80%9CMulti%2Dstep+jailbreaking+privacy+attacks+on+chatgpt%2C%E2%80%9D+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-multi-step-jailbreaking-privacy-attacks/) 8. J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” 2023. [find](https://scholar.google.com/scholar?q=J%2E+White%2C+Q%2E+Fu%2C+S%2E+Hays%2C+M%2E+Sandborn%2C+C%2E+Olea%2C+H%2E+Gilbert%2C+A%2E+Elnashar%2C+J%2E+Spencer%2DSmith%2C+and+D%2E+C%2E+Schmidt%2C+%E2%80%9CA+prompt+pattern+catalog+to+enhance+prompt+engineering+with+chatgpt%2C%E2%80%9D+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/white-2023-prompt-pattern-catalog-enhance-prompt/) 9. “Meet dan âC” the â C˜jailbreakâC™ version of chatgpt and how to use it â C” ai unchained and unfiltered \| by michael king \| medium,” [https://medium.com/@neonforge/meet-dan-the-jailbreak-version-of-chatgpt-and-how-to-use-it-ai-unchained-and-unfiltered-f91bfa679024](https://medium.com/@neonforge/meet-dan-the-jailbreak-version-of-chatgpt-and-how-to-use-it-ai-unchained-and-unfiltered-f91bfa679024), (Accessed on 02/02/2023). [↗](https://medium.com/@neonforge/meet-dan-the-jailbreak-version-of-chatgpt-and-how-to-use-it-ai-unchained-and-unfiltered-f91bfa679024) 10. “Moderation - openai api,” [https://platform.openai.com/docs/guides/](https://platform.openai.com/docs/guides/) moderation, (Accessed on 02/02/2023). [↗](https://platform.openai.com/docs/guides/) 11. “Llm jailbreak study,” [https://sites.google.com/view/llm-jailbreak-study](https://sites.google.com/view/llm-jailbreak-study), (Accessed on 05/06/2023). [↗](https://sites.google.com/view/llm-jailbreak-study) 12. “Alex albert,” [https://alexalbert.me/](https://alexalbert.me/), (Accessed on 05/06/2023). [↗](https://alexalbert.me/) 13. K. Stol, P. Ralph, and B. Fitzgerald, “Grounded theory in software engineering research: a critical review and guidelines,” in Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016, L. K. Dillon, W. Visser, and [find](https://scholar.google.com/scholar?q=K%2E+Stol%2C+P%2E+Ralph%2C+and+B%2E+Fitzgerald%2C+%E2%80%9CGrounded+theory+in+software+engineering+research%3A+a+critical+review+and+guidelines%2C%E2%80%9D+in+Proceedings+of+the+38th+International+Conference+on+Software+Engineering%2C)