---
title: "Defending Large Language Models Against Jailbreaking Attacks"
description: "The authors propose explicit goal prioritization at two stages. 16 references, cited by 69 papers here."
authors: ["Zhexin Zhang", "Junxiao Yang", "Pei Ke", "Fei Mi", "Hongning Wang", "Minlie Huang"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.481/"
doi: "10.18653/v1/2024.acl-long.481"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-defending-large-language-models-against/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-defending-large-language-models-against/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization

# Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization

**Authors:** [Zhexin Zhang](https://md.research.elamur.ai/free/en/authors/zhexin-zhang/), Junxiao Yang, [Pei Ke](https://md.research.elamur.ai/free/en/authors/pei-ke/), [Fei Mi](https://md.research.elamur.ai/free/en/authors/fei-mi/), [Hongning Wang](https://md.research.elamur.ai/free/en/authors/hongning-wang/) and [Minlie Huang](https://md.research.elamur.ai/free/en/authors/minlie-huang/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.481/) · [PDF](https://aclanthology.org/2024.acl-long.481.pdf) · [DOI 10.18653/v1/2024.acl-long.481](https://doi.org/10.18653/v1/2024.acl-long.481) · [OpenAlex](https://openalex.org/W4402671828)

**Citations:** 43 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/zhang-2024-defending-large-language-models-against/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper argues that jailbreaking attacks on LLMs succeed because models are tuned for two competing objectives, helpfulness and safety, without being told which one wins when they clash. Standard SFT and RLHF teach refusals for plain harmful requests, but this does not generalize to out-of-distribution jailbreak prompts. The authors propose explicit goal prioritization at two stages. At inference, a plug-and-play prompt tells the model to put safety first and includes two in-context demonstrations: one benign query answered and one harmful query refused. For training, they build contrastive data in which the same queries are paired with opposite priority instructions (safety over helpfulness vs. helpfulness over safety). The target outputs contain internal thoughts that analyze the requirement, followed by a response that complies with it. At deployment, a short safety-first instruction is enough. Both variants sharply reduce attack success rates across prompt, gradient, perturbation and combination attacks while largely preserving general ability. This supports the goal-conflict explanation and offers a practical defense for both API-based and open models.

## Key points
- Prompt-only defense lowers ChatGPT's average Attack Success Rate (ASR) from 66.4% to 3.6%, compared with 28.1% for the Self-Reminder baseline. Across 7 prompt variants, ChatGPT's average ASR stays around 4.5% (±1.8).
- Training with goal prioritization brings Llama2-13B from 71.0% ASR down to 6.6%, compared with 20.3% for ordinary SFT. Even with no jailbreak prompts in the training data, ASR falls from 56.8% to 34.0%.
- Stronger models such as GPT-4 are open to a wider range of attacks, yet are also easier to steer toward safety. The authors attribute both effects to better instruction following.
- On newer OpenAI versions, the method still beats the baselines; for example, GPT-4-0125 drops from 11.9% to 0.5% ASR. Additional validation uses the Wild Jailbreak dataset.
- Limitations: the inference-only variant trades off some helpfulness, and the internal-thoughts output adds decoding cost. Code is available at github.com/thu-coai/JailbreakDefense_GoalPriority.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 137 works · 83,262 citations
- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 29 works · 9,991 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works

## Cite

Cite as: Zhexin Zhang, Junxiao Yang, Pei Ke, Fei Mi, Hongning Wang and Minlie Huang. 2024. Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2024defending,
  title = {{Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization}},
  author = {Zhexin Zhang, Junxiao Yang, Pei Ke, Fei Mi, Hongning Wang and Minlie Huang},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.481},
  url = {https://doi.org/10.18653/v1/2024.acl-long.481}
}
```

- [← previous Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://md.research.elamur.ai/free/en/papers/fu-2024-break-sequential-dependency-llm-inference/)
- [next → DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data](https://md.research.elamur.ai/free/en/papers/xin-2024-deepseek-prover-advancing-theorem-proving/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

16 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023. [Xstest: A test suite for identifying exaggerated safety behaviours in large language models](https://md.research.elamur.ai/free/en/papers/rottger-2023-xstest-test-suite-identifying-exaggerated/). CoRR, abs/2308.01263.
2. Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. 2023. [Defending against alignment-breaking attacks via robustly aligned llm](https://md.research.elamur.ai/free/en/papers/cao-2024-defending-against-alignment-breaking-attacks/). arXiv preprint arXiv:2309.14348.
3. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. 2023c. Rain: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.07124. — [arXiv:2309.07124](https://arxiv.org/abs/2309.07124) [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Yuhui+Li%2C+Fangyun+Wei%2C+Jinjing+Zhao%2C+Chao+Zhang%2C+and+Hongyang+Zhang%2E+2023c%2E+Rain%3A+Your+language+models+can)
4. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023. [Jailbreaking chatgpt via prompt engineering: An empirical study](https://md.research.elamur.ai/free/en/papers/liu-2023-jailbreaking-chatgpt-prompt-engineering-empirical/). CoRR, abs/2305.13860.
5. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. [BBQ: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/). In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, p…
6. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. [Smoothllm: Defending large language models against jailbreaking attacks](https://md.research.elamur.ai/free/en/papers/robey-2023-smoothllm-defending-large-language-models/). arXiv preprint arXiv:2310.03684.
7. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. ["do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models](https://md.research.elamur.ai/free/en/papers/shen-2023-anything-now-characterizing-evaluating-wild/). CoRR, abs/2308.03825.
8. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Anjana+Arunkumar%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)
9. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. [Jailbroken: How does LLM safety training fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) CoRR, abs/2307.02483.
10. Fangzhao Wu, Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, and Xing Xie. 2023. Defending chatgpt against jailbreak attack via self-reminder. [find](https://scholar.google.com/scholar?q=Fangzhao+Wu%2C+Yueqi+Xie%2C+Jingwei+Yi%2C+Jiawei+Shao%2C+Justin+Curl%2C+Lingjuan+Lyu%2C+Qifeng+Chen%2C+and+Xing+Xie%2E+2023%2E+Defending+chatgpt+against+jailbreak+attack+via+self%2Dreminder%2E)
11. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. [GPTFUZZER: red teaming large language models with auto-generated jailbreak prompts](https://md.research.elamur.ai/free/en/papers/yu-2023-gptfuzzer-red-teaming-large-language/). CoRR, abs/2309.10253.
12. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. [Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher](https://md.research.elamur.ai/free/en/papers/yuan-2023-gpt-4-too-smart-safe/). arXiv preprint arXiv:2308.06463.
13. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023a. [Safetybench: Evaluating the safety of large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-safetybench-evaluating-safety-large/) with multiple choice questions. CoRR, abs/2309.07045.
14. Zhexin Zhang, Jiaxin Wen, and Minlie Huang. 2023b. ETHICIST: targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toro… [find](https://scholar.google.com/scholar?q=Zhexin+Zhang%2C+Jiaxin+Wen%2C+and+Minlie+Huang%2E+2023b%2E+ETHICIST%3A+targeted+training+data+extraction+through+loss+smoothed+soft+prompting+and+calibrated+confidence+estimation%2E+In+Proceedings+of+the+61st+Ann)
15. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+2023%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+a)
16. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). CoRR, abs/2307.15043.

*Updated 24 October 2026*