---
title: "Reasoning-to-Defend — paper by Zhu et al. (2025)"
description: "The authors propose Reasoning-to-Defend (R2D), a training paradigm with two stages: SwaRD… 68 references, cited by 18 papers here."
authors: ["Junda Zhu", "Lingyong Yan", "Shuaiqiang Wang", "Dawei Yin", "Lei Sha"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.1493/"
doi: "10.18653/v1/2025.emnlp-main.1493"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2025-reasoning-defend-safety-aware-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2025-reasoning-defend-safety-aware-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking

# Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking

**Authors:** Junda Zhu, [Lingyong Yan](https://md.research.elamur.ai/free/en/authors/lingyong-yan/), [Shuaiqiang Wang](https://md.research.elamur.ai/free/en/authors/shuaiqiang-wang/), [Dawei Yin](https://md.research.elamur.ai/free/en/authors/dawei-yin/) and [Lei Sha](https://md.research.elamur.ai/free/en/authors/lei-sha/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.1493/) · [PDF](https://aclanthology.org/2025.emnlp-main.1493.pdf) · [DOI 10.18653/v1/2025.emnlp-main.1493](https://doi.org/10.18653/v1/2025.emnlp-main.1493) · [OpenAlex](https://openalex.org/W4416035332)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [18](https://md.research.elamur.ai/free/en/papers/zhu-2025-reasoning-defend-safety-aware-reasoning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Making LLMs safer usually depends on external filters or on supervised tuning that teaches blunt refusals. These approaches ignore the models' own reasoning abilities, and blunt refusals tend to generalize poorly. The authors propose Reasoning-to-Defend (R2D), a training paradigm with two stages:

- **SwaRD (Safety-aware Reasoning Distillation):** distills a staged, step-by-step thinking habit into models that do not reason by default.
- **CPO (Contrastive Pivot Optimization):** a contrastive objective that teaches the model to label each reasoning step with a safety PIVOT TOKEN ([SAFE], [UNSAFE] or [RETHINK]).

During generation, the model judges its own intermediate steps and changes course when needed, so no external guardrail is required. The authors test several open LLMs and LRMs against PAIR, GCG and JBC attacks on JailbreakBench and HarmBench. They check over-refusal on XSTest and general ability on broader datasets. R2D sharply lowers attack success rates without hurting helpfulness on benign prompts. This suggests that explicit safety reasoning can serve as a practical, built-in defense against jailbreaks.

## Key points
- Evaluated models include Llama v3-8B, Qwen v2-7B, Mistral v0.3-7B, Vicuna v1.5-7B, R1 7B, R1 32B, QwQ 32B and Qwen v2.5-14B. The metric is Attack Success Rate (ASR). On the JBC attack, Qwen v2-7B's ASR falls from 92% to 0%, and Llama v3-8B's PAIR ASR drops from 52% to 1%.
- Ablations show both components matter. Dropping CPO raises ASR by up to 23%, and removing pivot tokens from the training data raises it by up to 45%.
- R2D outperforms the prompt-based defenses Self-Reminder and IA. It is competitive with adversarially trained RPO, cutting ASR by up to 9% relative to it.
- On XSTest, R2D models refuse unsafe prompts more often while keeping compliance on safe ones. Qwen v2.5-14B's full compliance on safe prompts rises by 4.8%.
- Harmful queries trigger longer outputs because of self-evaluation and rethinking steps. Benign queries keep normal response lengths. Code is available at github.com/chuhac/Reasoning-to-Defend.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 84 works · 17,681 citations
- [Beihang University](https://md.research.elamur.ai/free/en/universities/beihang-university/) — University, Beijing, China · 86 works · 9,097 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Junda Zhu, Lingyong Yan, Shuaiqiang Wang, Dawei Yin and Lei Sha. 2025. Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhu2025reasoning,
  title = {{Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking}},
  author = {Junda Zhu, Lingyong Yan, Shuaiqiang Wang, Dawei Yin and Lei Sha},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1493},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1493}
}
```

- [← previous Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2025-multimodal-needle-haystack-benchmarking-long/)
- [next → RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios](https://md.research.elamur.ai/free/en/papers/zhou-2025-rulearena-benchmark-rule-guided-reasoning/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

68 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gabriel Alon and Michael Kamfonas. 2023. [Detecting language model attacks with perplexity](https://md.research.elamur.ai/free/en/papers/alon-2023-detecting-language-model-attacks-perplexity/). arXiv preprint arXiv:2308.14132.
2. Maksym Andriushchenko and Nicolas Flammarion. 2025. Does refusal training in LLMs generalize to the past tense? In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Maksym+Andriushchenko+and+Nicolas+Flammarion%2E+2025%2E+Does+refusal+training+in+LLMs+generalize+to+the+past+tense%3F+In+The+Thirteenth+International+Conference+on+Learning+Representations%2E)
3. Anthropic. 2024. Introducing the next generation of claude. [https://www.anthropic.com/news/](https://www.anthropic.com/news/) claude-3-family/. [↗](https://www.anthropic.com/news/)
4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073.
5. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv preprint arXiv:2108.07258.
6. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. 2024. [Jailbreakbench: An open robustness benchmark for jailbreaking large language models](https://md.research.elamur.ai/free/en/papers/chao-2024-jailbreakbench-open-robustness-benchmark-jailbreaking/). In…
7. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. [Jailbreaking black box large language models in twenty queries](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/). arXiv preprint arXiv:2310.08419.
8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
9. Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2024. [Or-bench: An over-refusal benchmark for large language models](https://md.research.elamur.ai/free/en/papers/cui-2024-bench-over-refusal-benchmark-large/). arXiv preprint arXiv:2405.20947.
10. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2024. [Safe RLHF: Safe reinforcement learning from human feedback](https://md.research.elamur.ai/free/en/papers/dai-2023-safe-rlhf-safe-reinforcement-learning/). In The Twelfth International Conference on Learning Representations.
11. Tri Dao. 2024. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In The Twelfth International Conference on Learning Representations.
12. Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. 2022. Flashattention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+Re%2E+2022%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Sys)
13. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
14. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
15. Rima Hazra, Sayan Layek, Somnath Banerjee, and Soujanya Poria. 2024. Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21759–21776,… [find](https://scholar.google.com/scholar?q=Rima+Hazra%2C+Sayan+Layek%2C+Somnath+Banerjee%2C+and+Soujanya+Poria%2E+2024%2E+Safety+arithmetic%3A+A+framework+for+test%2Dtime+safety+alignment+of+language+models+by+steering+parameters+and+activations%2E+In+Proceed)
16. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
17. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
18. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. [Llama guard: Llm-based input-output safeguard for human-ai conversations](https://md.research.elamur.ai/free/en/papers/inan-2023-llama-guard-llm-based-input/). arXiv preprint arXiv:2312.06674.
19. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). arXiv preprint arXiv:2412.16720.
20. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. [Baseline defenses for adversarial attacks against aligned language models](https://md.research.elamur.ai/free/en/papers/jain-2023-baseline-defenses-adversarial-attacks-against/). arXiv preprint arXiv:2309.00614.
21. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
22. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+2024%2E+Mixtral+o)
23. Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. 2025. Safechain: Safety of language models with long chain-of-thought reasoning capabilities. In ICLR 2025 Workshop on Bidirectional Human-AI Alignment. [find](https://scholar.google.com/scholar?q=Fengqing+Jiang%2C+Zhangchen+Xu%2C+Yuetai+Li%2C+Luyao+Niu%2C+Zhen+Xiang%2C+Bo+Li%2C+Bill+Yuchen+Lin%2C+and+Radha+Poovendran%2E+2025%2E+Safechain%3A+Safety+of+language+models+with+long+chain%2Dof%2Dthought+reasoning+capabiliti)
24. Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. 2019. A study of bfloat16 for deep learning training. arXiv preprint arXiv:1905.12322. — [arXiv:1905.12322](https://arxiv.org/abs/1905.12322) [find](https://scholar.google.com/scholar?q=Dhiraj+Kalamkar%2C+Dheevatsa+Mudigere%2C+Naveen+Mellempudi%2C+Dipankar+Das%2C+Kunal+Banerjee%2C+Sasikanth+Avancha%2C+Dharma+Teja+Vooturi%2C+Nataraj+Jammalamadaka%2C+Jianyu+Huang%2C+Hector+Yuen%2C+et+al%2E+2019%2E+A+study+of+)
25. Mintong Kang and Bo Li. 2025. R 2 -guard: Robust reasoning enabled LLM guardrail via knowledge-enhanced logical reasoning. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Mintong+Kang+and+Bo+Li%2E+2025%2E+R+2+%2Dguard%3A+Robust+reasoning+enabled+LLM+guardrail+via+knowledge%2Denhanced+logical+reasoning%2E+In+The+Thirteenth+International+Conference+on+Learning+Representations%2E)
26. Team Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. — [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/)
27. Doga Kisa, Guy Van den Broeck, Arthur Choi, and Adnan Darwiche. 2014. Probabilistic sentential decision diagrams. In Fourteenth International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Doga+Kisa%2C+Guy+Van+den+Broeck%2C+Arthur+Choi%2C+and+Adnan+Darwiche%2E+2014%2E+Probabilistic+sentential+decision+diagrams%2E+In+Fourteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+a)
28. Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. 2023. [Certifying llm safety against adversarial prompting](https://md.research.elamur.ai/free/en/papers/kumar-2023-certifying-llm-safety-against-adversarial/). arXiv preprint arXiv:2309.02705.
29. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
30. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback. In Proceedings of the 41st Internatio… [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Thomas+Mesnard%2C+Johan+Ferret%2C+Kellie+Ren+Lu%2C+Colton+Bishop%2C+Ethan+Hall%2C+Victor+Carbune%2C+Abhinav+Rastogi%2C+and+Sushant+Prakash%2E+2024%2E+RLAIF+vs%2E+RLHF%3A+Scalin)
31. Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. 2020. [Pytorch distributed: experiences on accelerating data parallel training](https://md.research.elamur.ai/free/en/papers/li-2020-pytorch-distributed-experiences-accelerating-data/). Proc. VLDB Endow., 13(12):3005–3018.
32. Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Conference+on+Neural+Information+Processing+Systems%2E)
33. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. [HarmBench: A standardized evaluation framework for automated red teaming and robust refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/). In Proceedings of the 41st International Con…
34. Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Bin Sun, Xinglin Wang, Heda Wang, and Kan Li. 2024. Turning dust into gold: Distilling complex reasoning capabilities from llms by leveraging negative data. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):18591–18599. [find](https://scholar.google.com/scholar?q=Yiwei+Li%2C+Peiwen+Yuan%2C+Shaoxiong+Feng%2C+Boyuan+Pan%2C+Bin+Sun%2C+Xinglin+Wang%2C+Heda+Wang%2C+and+Kan+Li%2E+2024%2E+Turning+dust+into+gold%3A+Distilling+complex+reasoning+capabilities+from+llms+by+leveraging+negativ)
35. Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian D Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. 2024. Rule based rewards for language model safety. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Tong+Mu%2C+Alec+Helyar%2C+Johannes+Heidecke%2C+Joshua+Achiam%2C+Andrea+Vallone%2C+Ian+D+Kivlichan%2C+Molly+Lin%2C+Alex+Beutel%2C+John+Schulman%2C+and+Lilian+Weng%2E+2024%2E+Rule+based+rewards+for+language+model+safety%2E+In+)
36. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). In The Twelfth International Conference on Learning Representations.
37. OpenAI. 2025. Openai o3-mini system card. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Openai+o3%2Dmini+system+card%2E)
38. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+et+al%2E+2024a%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437)
39. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Trai… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
40. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024b. [AutoDAN: Generating stealthy jailbreak prompts on aligned large language models](https://md.research.elamur.ai/free/en/papers/liu-2023-autodan-generating-stealthy-jailbreak-prompts/). In The Twelfth International Conference on Learning Representations.
41. Swetasudha Panda, Naveen Jafer Nizar, and Michael L Wick. 2024. LLM improvement for jailbreak defense: Analysis through the lens of over-refusal. In Neurips Safe Generative AI Workshop 2024. [find](https://scholar.google.com/scholar?q=Swetasudha+Panda%2C+Naveen+Jafer+Nizar%2C+and+Michael+L+Wick%2E+2024%2E+LLM+improvement+for+jailbreak+defense%3A+Analysis+through+the+lens+of+over%2Drefusal%2E+In+Neurips+Safe+Generative+AI+Workshop+2024%2E)
42. Yue Liu, Hongcheng Gao, Shengfang Zhai, Jun Xia, Tianyi Wu, Zhiwei Xue, Yulin Chen, Kenji Kawaguchi, Jiaheng Zhang, and Bryan Hooi. 2025. Guardreasoner: Towards reasoning-based llm safeguards. arXiv preprint arXiv:2501.18492. — [arXiv:2501.18492](https://arxiv.org/abs/2501.18492) [find](https://scholar.google.com/scholar?q=Yue+Liu%2C+Hongcheng+Gao%2C+Shengfang+Zhai%2C+Jun+Xia%2C+Tianyi+Wu%2C+Zhiwei+Xue%2C+Yulin+Chen%2C+Kenji+Kawaguchi%2C+Jiaheng+Zhang%2C+and+Bryan+Hooi%2E+2025%2E+Guardreasoner%3A+Towards+reasoning%2Dbased+llm+safeguards%2E+arXiv+p)
43. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. [Safety alignment should be made more than just a few tokens deep](https://md.research.elamur.ai/free/en/papers/qi-2024-safety-alignment-should-made-more/). In The Thirteenth International Conference on Learning Representations.
44. Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. 2024c. Enhancing llm safety via constrained direct preference optimization. arXiv preprint arXiv:2403.02475. — [arXiv:2403.02475](https://arxiv.org/abs/2403.02475) [find](https://scholar.google.com/scholar?q=Zixuan+Liu%2C+Xiaolin+Sun%2C+and+Zizhan+Zheng%2E+2024c%2E+Enhancing+llm+safety+via+constrained+direct+preference+optimization%2E+arXiv+preprint+arXiv%3A2403%2E02475%2E+%E2%80%94+%5BarXiv%3A2403%2E02475%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E)
45. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
46. Weikai Lu, Ziqian Zeng, Jianwei Wang, Zhengdong Lu, Zelin Chen, Huiping Zhuang, and Cen Chen. 2024. Eraser: Jailbreaking defense in large language models via unlearning harmful knowledge. arXiv preprint arXiv:2404.05880. — [arXiv:2404.05880](https://arxiv.org/abs/2404.05880) [find](https://scholar.google.com/scholar?q=Weikai+Lu%2C+Ziqian+Zeng%2C+Jianwei+Wang%2C+Zhengdong+Lu%2C+Zelin+Chen%2C+Huiping+Zhuang%2C+and+Cen+Chen%2E+2024%2E+Eraser%3A+Jailbreaking+defense+in+large+language+models+via+unlearning+harmful+knowledge%2E+arXiv+prepri)
47. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
48. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Thirty-seventh Conference on Neural Information Processing Systems.
49. Matthew Richardson and Pedro Domingos. 2006. Markov logic networks. Machine learning, 62:107– 136. [find](https://scholar.google.com/scholar?q=Matthew+Richardson+and+Pedro+Domingos%2E+2006%2E+Markov+logic+networks%2E+Machine+learning%2C+62%3A107%E2%80%93+136%2E)
50. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. [Smoothllm: Defending large language models against jailbreaking attacks](https://md.research.elamur.ai/free/en/papers/robey-2023-smoothllm-defending-large-language-models/). arXiv preprint arXiv:2310.03684.
51. Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. [XSTest: A test suite for identifying exaggerated safety behaviours in large language models](https://md.research.elamur.ai/free/en/papers/rottger-2023-xstest-test-suite-identifying-exaggerated/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational L…
52. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, et al. 2024a. Qwen2 technical report. arXiv preprint arXiv:2407.10671. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+et+al%2E+2024a%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2E)
53. Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2025. [Great, now write an article about that: the crescendo multi-turn llm jailbreak attack](https://md.research.elamur.ai/free/en/papers/russinovich-2024-great-now-write-article-about/). In Proceedings of the 34th USENIX Conference on Security Symposium, SEC ’25, USA. USENIX Association.
54. Yiming Zhang, Jianfeng Chi, Hailey Nguyen, Kartikeya Upasani, Daniel M. Bikel, Jason E Weston, and Eric Michael Smith. 2025a. Backtracking improves generation safety. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yiming+Zhang%2C+Jianfeng+Chi%2C+Hailey+Nguyen%2C+Kartikeya+Upasani%2C+Daniel+M%2E+Bikel%2C+Jason+E+Weston%2C+and+Eric+Michael+Smith%2E+2025a%2E+Backtracking+improves+generation+safety%2E+In+The+Thirteenth+International+C)
55. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. [Distilling reasoning capabilities into smaller language models](https://md.research.elamur.ai/free/en/papers/shridhar-2023-distilling-reasoning-capabilities-into-smaller/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073, Toronto, Canada. Association for Computational Linguistics.
56. Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao. 2025b. [Intention analysis makes LLMs a good jailbreak defender](https://md.research.elamur.ai/free/en/papers/zhang-2025-intention-analysis-makes-llms-good/). In Proceedings of the 31st International Conference on Computational Linguistics, pages 2947–2968, Abu Dhabi, UAE. Association for Computational Linguistics.
57. Hua Tang, Lingyong Yan, Yukun Zhao, Shuaiqiang Wang, Jizhou Huang, and Dawei Yin. 2025. Multi-turn jailbreaking via global refinement and active fabrication. arXiv preprint arXiv:2506.17881. — [arXiv:2506.17881](https://arxiv.org/abs/2506.17881) [find](https://scholar.google.com/scholar?q=Hua+Tang%2C+Lingyong+Yan%2C+Yukun+Zhao%2C+Shuaiqiang+Wang%2C+Jizhou+Huang%2C+and+Dawei+Yin%2E+2025%2E+Multi%2Dturn+jailbreaking+via+global+refinement+and+active+fabrication%2E+arXiv+preprint+arXiv%3A2506%2E17881%2E+%E2%80%94+%5BarXiv%3A)
58. Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang. 2024. Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks. arXiv preprint arXiv:2407.02855. — [arXiv:2407.02855](https://arxiv.org/abs/2407.02855) [find](https://scholar.google.com/scholar?q=Zhexin+Zhang%2C+Junxiao+Yang%2C+Pei+Ke%2C+Shiyao+Cui%2C+Chujie+Zheng%2C+Hongning+Wang%2C+and+Minlie+Huang%2E+2024%2E+Safe+unlearning%3A+A+surprisingly+effective+and+generalizable+solution+to+defend+against+jailbreak+at)
59. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models., 3(6):7. [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2023%2E+Alpaca%3A+A+strong%2C+replicable+instruction%2Dfollowing+model%2E+Stanford+)
60. Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. 2024. [ReFT: Reasoning with reinforced fine-tuning](https://md.research.elamur.ai/free/en/papers/trung-2024-reft-reasoning-reinforced-fine-tuning/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7601–7614, Bangkok, Thailand. Association for Compu…
61. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. [Jailbroken: How does LLM safety training fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) In Thirty-seventh Conference on Neural Information Processing Systems.
62. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems.
63. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024b. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+et+al%2E+2024b%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2E)
64. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Thirty-seventh Conference on Neural Information Processing…
65. Andy Zhou, Bo Li, and Haohan Wang. 2024. Robust prompt optimization for defending language models against jailbreaking attacks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Andy+Zhou%2C+Bo+Li%2C+and+Haohan+Wang%2E+2024%2E+Robust+prompt+optimization+for+defending+language+models+against+jailbreaking+attacks%2E+In+The+Thirty%2Deighth+Annual+Conference+on+Neural+Information+Processing+)
66. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). arXiv preprint arXiv:2307.15043.
67. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12):1486–1496. [find](https://scholar.google.com/scholar?q=Yueqi+Xie%2C+Jingwei+Yi%2C+Jiawei+Shao%2C+Justin+Curl%2C+Lingjuan+Lyu%2C+Qifeng+Chen%2C+Xing+Xie%2C+and+Fangzhao+Wu%2E+2023%2E+Defending+chatgpt+against+jailbreak+attack+via+self%2Dreminders%2E+Nature+Machine+Intelligence%2C)
68. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024. [SafeDecoding: Defending against jailbreak attacks via safety-aware decoding](https://md.research.elamur.ai/free/en/papers/xu-2024-safedecoding-defending-against-jailbreak-attacks/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 55…

*Updated 2 November 2026*