---
title: "SafeDecoding — Xu 2024"
description: "Aligned LLMs can still be jailbroken: adversarial prompts push them into producing harmful content."
authors: ["Zhangchen Xu", "Fengqing Jiang", "Luyao Niu", "Jinyuan Jia", "Bill Yuchen Lin", "Radha Poovendran"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.303/"
doi: "10.18653/v1/2024.acl-long.303"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2024-safedecoding-defending-against-jailbreak-attacks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2024-safedecoding-defending-against-jailbreak-attacks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding

# SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding

**Authors:** [Zhangchen Xu](https://md.research.elamur.ai/free/en/authors/zhangchen-xu/), [Fengqing Jiang](https://md.research.elamur.ai/free/en/authors/fengqing-jiang/), [Luyao Niu](https://md.research.elamur.ai/free/en/authors/luyao-niu/), Jinyuan Jia, [Bill Yuchen Lin](https://md.research.elamur.ai/free/en/authors/bill-yuchen-lin/), [Radha Poovendran](https://md.research.elamur.ai/free/en/authors/radha-poovendran/) · **Year:** 2024 · **Venue:** Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.303/) · [PDF](https://aclanthology.org/2024.acl-long.303.pdf) · [DOI 10.18653/v1/2024.acl-long.303](https://doi.org/10.18653/v1/2024.acl-long.303) · [OpenAlex](https://openalex.org/W4402671718)

**Citations:** 38 citations (openalex-snapshot · 2026-10-14) · [88](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Aligned LLMs can still be jailbroken: adversarial prompts push them into producing harmful content. Existing defenses often protect poorly, slow down inference, or make models less useful to ordinary users. The authors studied token distributions under attack and found that the compliant opening (e.g., "Sure") usually dominates, but refusal-style tokens such as "Sorry" or "I" still rank near the top. The model therefore appears to retain some awareness that the request is unsafe.

SafeDecoding exploits this observation in two stages:
- **Training:** an expert model is fine-tuned on a safety-oriented dataset generated by the original model itself.
- **Inference:** for the first few generated tokens, the method keeps only candidates ranked highly by both models. It then builds a new distribution from both models' probabilities, which boosts disclaimers and suppresses attacker-aligned continuations.

The approach was tested on five LLMs against six state-of-the-art jailbreak attacks, using two harmful and two utility benchmarks. It beat six baseline defenses, lowering attack success and harmfulness while keeping benign helpfulness and adding little latency. This makes it a practical, model-agnostic safety layer.

## Key points

- The evaluation covers models including Vicuna, Llama2 and Guanaco. The attacks include optimization-based ones such as GCG, and SafeDecoding was compared with defenses like Perplexity filtering, Self-Reminder, ICD, Retokenization, Self-Examination and Paraphrase.
- Measured by ATGR, SafeDecoding's time overhead relative to no defense is about 3% on Llama2 (1.03×) and 7% on Vicuna (1.07×). Paraphrase, by comparison, costs up to 2.15×.
- Some competing defenses hurt utility badly. For example, Self-Examination scores only 1.31 on MT-bench with Llama2, because it flags benign outputs as harmful.
- Ablations show the method is robust to its hyper-parameters α, m and c. Using top-p sampling raises ASR slightly in exchange for more diverse responses.
- Known limitations:
  - In 31 of 250 responses, the model first refuses and then complies.
  - Multimodal LLMs were not examined.
- Code is available at github.com/uw-nsl/SafeDecoding.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

88 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 85 not yet.

- [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/)
- [Toxic Subword Pruning for Dialogue Response Generation on Large Language Models](https://md.research.elamur.ai/free/en/papers/lu-2026-toxic-subword-pruning-dialogue-response/)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://md.research.elamur.ai/free/en/papers/yuan-2023-gpt-4-too-smart-safe/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2024safedecoding,
  title = {{SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding}},
  author = {Xu, Zhangchen and Jiang, Fengqing and Niu, Luyao and Jia, Jinyuan and Lin, Bill Yuchen and Poovendran, Radha},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.303},
  url = {https://doi.org/10.18653/v1/2024.acl-long.303}
}
```

- [← previous RULER: What's the Real Context Size of Your Long-Context Language Models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/)
- [next → SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-salad-bench-hierarchical-comprehensive-safety/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

52 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. Technical report. [find](https://scholar.google.com/scholar?q=Josh+Achiam%2C+Steven+Adler%2C+Sandhini+Agarwal%2C+Lama+Ahmad%2C+Ilge+Akkaya%2C+Florencia+Leoni+Aleman%2C+Diogo+Almeida%2C+Janko+Altenschmidt%2C+Sam+Altman%2C+Shyamal+Anadkat%2C+et+al%2E+2023%2E+GPT%2D4+technical+report%2E+Techn)
2. Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. [find](https://scholar.google.com/scholar?q=Gabriel+Alon+and+Michael+Kamfonas%2E+2023%2E+Detecting+language+model+attacks+with+perplexity%2E) [our card](https://md.research.elamur.ai/free/en/papers/alon-2023-detecting-language-model-attacks-perplexity/)
3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+2022%2E+Training+a+helpful+and+harmless+assistant+with+rein)
4. Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. 2023. Defending against alignment-breaking attacks via robustly aligned LLM. arXiv preprint arXiv:2309.14348. [find](https://scholar.google.com/scholar?q=Bochuan+Cao%2C+Yuanpu+Cao%2C+Lu+Lin%2C+and+Jinghui+Chen%2E+2023%2E+Defending+against+alignment%2Dbreaking+attacks+via+robustly+aligned+LLM%2E+arXiv+preprint+arXiv%3A2309%2E14348%2E)
5. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419. [find](https://scholar.google.com/scholar?q=Patrick+Chao%2C+Alexander+Robey%2C+Edgar+Dobriban%2C+Hamed+Hassani%2C+George+J+Pappas%2C+and+Eric+Wong%2E+2023%2E+Jailbreaking+black+box+large+language+models+in+twenty+queries%2E+arXiv+preprint+arXiv%3A2310%2E08419%2E) [our card](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/)
6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality. See [https://vicuna.lmsys](https://vicuna.lmsys). org (accessed 14 April 2023). [↗](https://vicuna.lmsys)
7. Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He. 2023a. Attack prompt generation for red teaming and defending large language models. arXiv preprint arXiv:2310.12505. [find](https://scholar.google.com/scholar?q=Boyi+Deng%2C+Wenjie+Wang%2C+Fuli+Feng%2C+Yang+Deng%2C+Qifan+Wang%2C+and+Xiangnan+He%2E+2023a%2E+Attack+prompt+generation+for+red+teaming+and+defending+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E12505%2E)
8. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023b. Masterkey: Automated jailbreak across multiple large language model chatbots. [find](https://scholar.google.com/scholar?q=Gelei+Deng%2C+Yi+Liu%2C+Yuekang+Li%2C+Kailong+Wang%2C+Ying+Zhang%2C+Zefeng+Li%2C+Haoyu+Wang%2C+Tianwei+Zhang%2C+and+Yang+Liu%2E+2023b%2E+Masterkey%3A+Automated+jailbreak+across+multiple+large+language+model+chatbots%2E)
9. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023c. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474. [find](https://scholar.google.com/scholar?q=Yue+Deng%2C+Wenxuan+Zhang%2C+Sinno+Jialin+Pan%2C+and+Lidong+Bing%2E+2023c%2E+Multilingual+jailbreak+challenges+in+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E06474%2E) [our card](https://md.research.elamur.ai/free/en/papers/deng-2023-multilingual-jailbreak-challenges-large-language/)
10. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized LLMs. arXiv preprint arXiv:2305.14314. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+2023%2E+Qlora%3A+Efficient+finetuning+of+quantized+LLMs%2E+arXiv+preprint+arXiv%3A2305%2E14314%2E)
11. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+arXiv+preprint+arXiv%3A1805%2E04833%2E)
12. Emilio Ferrara. 2023. Should ChatGPT be biased? Challenges and risks of bias in large language models. arXiv preprint arXiv:2304.03738. — [arXiv:2304.03738](https://arxiv.org/abs/2304.03738) [find](https://scholar.google.com/scholar?q=Emilio+Ferrara%2E+2023%2E+Should+ChatGPT+be+biased%3F+Challenges+and+risks+of+bias+in+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E03738%2E+%E2%80%94+%5BarXiv%3A2304%2E03738%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2304%2E03738%29)
13. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858. [find](https://scholar.google.com/scholar?q=Deep+Ganguli%2C+Liane+Lovitt%2C+Jackson+Kernion%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Saurav+Kadavath%2C+Ben+Mann%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Kamal+Ndousse%2C+et+al%2E+2022%2E+Red+teaming+language+models+to+reduce+har) [our card](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/)
14. Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375. [find](https://scholar.google.com/scholar?q=Amelia+Glaese%2C+Nat+McAleese%2C+Maja+Tr%CB%9Bebacz%2C+John+Aslanides%2C+Vlad+Firoiu%2C+Timo+Ewalds%2C+Maribeth+Rauh%2C+Laura+Weidinger%2C+Martin+Chadwick%2C+Phoebe+Thacker%2C+et+al%2E+2022%2E+Improving+alignment+of+dialogue+agen) [our card](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/)
15. Eric Hartford. 2023. Dolphin. [find](https://scholar.google.com/scholar?q=Eric+Hartford%2E+2023%2E+Dolphin%2E)
16. Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. 2023. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308. [find](https://scholar.google.com/scholar?q=Alec+Helbling%2C+Mansi+Phute%2C+Matthew+Hull%2C+and+Duen+Horng+Chau%2E+2023%2E+Llm+self+defense%3A+By+self+examination%2C+llms+know+they+are+being+tricked%2E+arXiv+preprint+arXiv%3A2308%2E07308%2E)
17. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2019%2E+The+curious+case+of+neural+text+degeneration%2E+arXiv+preprint+arXiv%3A1904%2E09751%2E)
18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. [find](https://scholar.google.com/scholar?q=Edward+J+Hu%2C+Yelong+Shen%2C+Phillip+Wallis%2C+Zeyuan+Allen%2DZhu%2C+Yuanzhi+Li%2C+Shean+Wang%2C+Lu+Wang%2C+and+Weizhu+Chen%2E+2021%2E+Lora%3A+Low%2Drank+adaptation+of+large+language+models%2E+arXiv+preprint+arXiv%3A2106%2E09685%2E)
19. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2023. Catastrophic jailbreak of open-source llms via exploiting generation. arXiv preprint arXiv:2310.06987. [find](https://scholar.google.com/scholar?q=Yangsibo+Huang%2C+Samyak+Gupta%2C+Mengzhou+Xia%2C+Kai+Li%2C+and+Danqi+Chen%2E+2023%2E+Catastrophic+jailbreak+of+open%2Dsource+llms+via+exploiting+generation%2E+arXiv+preprint+arXiv%3A2310%2E06987%2E) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-catastrophic-jailbreak-open-source-llms/)
20. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614. [find](https://scholar.google.com/scholar?q=Neel+Jain%2C+Avi+Schwarzschild%2C+Yuxin+Wen%2C+Gowthami+Somepalli%2C+John+Kirchenbauer%2C+Ping%2Dyeh+Chiang%2C+Micah+Goldblum%2C+Aniruddha+Saha%2C+Jonas+Geiping%2C+and+Tom+Goldstein%2E+2023%2E+Baseline+defenses+for+adversari) [our card](https://md.research.elamur.ai/free/en/papers/jain-2023-baseline-defenses-adversarial-attacks-against/)
21. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38. [find](https://scholar.google.com/scholar?q=Ziwei+Ji%2C+Nayeon+Lee%2C+Rita+Frieske%2C+Tiezheng+Yu%2C+Dan+Su%2C+Yan+Xu%2C+Etsuko+Ishii%2C+Ye+Jin+Bang%2C+Andrea+Madotto%2C+and+Pascale+Fung%2E+2023%2E+Survey+of+hallucination+in+natural+language+generation%2E+ACM+Computin) [our card](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
22. Fengqing Jiang, Zhangchen Xu, Luyao Niu, Boxin Wang, Jinyuan Jia, Bo Li, and Radha Poovendran. 2023. Identifying and mitigating vulnerabilities in llm-integrated applications. arXiv preprint arXiv:2311.16153. — [arXiv:2311.16153](https://arxiv.org/abs/2311.16153) [find](https://scholar.google.com/scholar?q=Fengqing+Jiang%2C+Zhangchen+Xu%2C+Luyao+Niu%2C+Boxin+Wang%2C+Jinyuan+Jia%2C+Bo+Li%2C+and+Radha+Poovendran%2E+2023%2E+Identifying+and+mitigating+vulnerabilities+in+llm%2Dintegrated+applications%2E+arXiv+preprint+arXiv%3A231)
23. Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. 2024. Artprompt: Ascii art-based jailbreak attacks against aligned llms. arXiv preprint arXiv:2402.11753. [find](https://scholar.google.com/scholar?q=Fengqing+Jiang%2C+Zhangchen+Xu%2C+Luyao+Niu%2C+Zhen+Xiang%2C+Bhaskar+Ramasubramanian%2C+Bo+Li%2C+and+Radha+Poovendran%2E+2024%2E+Artprompt%3A+Ascii+art%2Dbased+jailbreak+attacks+against+aligned+llms%2E+arXiv+preprint+arXiv)
24. Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. 2023. Automatically auditing large language models via discrete optimization. arXiv preprint arXiv:2303.04381. [find](https://scholar.google.com/scholar?q=Erik+Jones%2C+Anca+Dragan%2C+Aditi+Raghunathan%2C+and+Jacob+Steinhardt%2E+2023%2E+Automatically+auditing+large+language+models+via+discrete+optimization%2E+arXiv+preprint+arXiv%3A2303%2E04381%2E)
25. Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2023a. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191. [find](https://scholar.google.com/scholar?q=Xuan+Li%2C+Zhanke+Zhou%2C+Jianing+Zhu%2C+Jiangchao+Yao%2C+Tongliang+Liu%2C+and+Bo+Han%2E+2023a%2E+Deepinception%3A+Hypnotize+large+language+model+to+be+jailbreaker%2E+arXiv+preprint+arXiv%3A2311%2E03191%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-deepinception-hypnotize-large-language-model/)
26. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. 2023b. Rain: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.07124. — [arXiv:2309.07124](https://arxiv.org/abs/2309.07124) [find](https://scholar.google.com/scholar?q=Yuhui+Li%2C+Fangyun+Wei%2C+Jinjing+Zhao%2C+Chao+Zhang%2C+and+Hongyang+Zhang%2E+2023b%2E+Rain%3A+Your+language+models+can+align+themselves+without+finetuning%2E+arXiv+preprint+arXiv%3A2309%2E07124%2E+%E2%80%94+%5BarXiv%3A2309%2E07124%5D%28ht)
27. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2023. The unlocking spell on base LLMs: Rethinking alignment via in-context learning. arXiv preprint arXiv:2312.01552. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Abhilasha+Ravichander%2C+Ximing+Lu%2C+Nouha+Dziri%2C+Melanie+Sclar%2C+Khyathi+Chandu%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2023%2E+The+unlocking+spell+on+base+LLMs%3A+Rethinking+alignment+via+in%2D) [our card](https://md.research.elamur.ai/free/en/papers/lin-2023-unlocking-spell-base-llms-rethinking/)
28. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023a. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451. [find](https://scholar.google.com/scholar?q=Xiaogeng+Liu%2C+Nan+Xu%2C+Muhao+Chen%2C+and+Chaowei+Xiao%2E+2023a%2E+Autodan%3A+Generating+stealthy+jailbreak+prompts+on+aligned+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E04451%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-autodan-generating-stealthy-jailbreak-prompts/)
29. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023b. Jailbreaking ChatGPT via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860. [find](https://scholar.google.com/scholar?q=Yi+Liu%2C+Gelei+Deng%2C+Zhengzi+Xu%2C+Yuekang+Li%2C+Yaowen+Zheng%2C+Ying+Zhang%2C+Lida+Zhao%2C+Tianwei+Zhang%2C+and+Yang+Liu%2E+2023b%2E+Jailbreaking+ChatGPT+via+prompt+engineering%3A+An+empirical+study%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-jailbreaking-chatgpt-prompt-engineering-empirical/)
30. Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1–40. [find](https://scholar.google.com/scholar?q=Bonan+Min%2C+Hayley+Ross%2C+Elior+Sulem%2C+Amir+Pouran+Ben+Veyseh%2C+Thien+Huu+Nguyen%2C+Oscar+Sainz%2C+Eneko+Agirre%2C+Ilana+Heintz%2C+and+Dan+Roth%2E+2023%2E+Recent+advances+in+natural+language+processing+via+large+pre) [our card](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
31. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+2022%2E+Training+language+models+to+follow+instructi)
32. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023%2E+The+refinedweb+dataset+for+F)
33. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2019. Bpe-dropout: Simple and effective subword regularization. arXiv preprint arXiv:1910.13267. [find](https://scholar.google.com/scholar?q=Ivan+Provilkov%2C+Dmitrii+Emelianenko%2C+and+Elena+Voita%2E+2019%2E+Bpe%2Ddropout%3A+Simple+and+effective+subword+regularization%2E+arXiv+preprint+arXiv%3A1910%2E13267%2E) [our card](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/)
34. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xiangyu+Qi%2C+Yi+Zeng%2C+Tinghao+Xie%2C+Pin%2DYu+Chen%2C+Ruoxi+Jia%2C+Prateek+Mittal%2C+and+Peter+Henderson%2E+2024%2E+Fine%2Dtuning+aligned+language+models+compromises+safety%2C+even+when+users+do+not+intend+to%21+In+The+Tw) [our card](https://md.research.elamur.ai/free/en/papers/qi-2023-fine-tuning-aligned-language-models/)
35. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. SmoothLLM: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684. [find](https://scholar.google.com/scholar?q=Alexander+Robey%2C+Eric+Wong%2C+Hamed+Hassani%2C+and+George+J+Pappas%2E+2023%2E+SmoothLLM%3A+Defending+large+language+models+against+jailbreaking+attacks%2E+arXiv+preprint+arXiv%3A2310%2E03684%2E) [our card](https://md.research.elamur.ai/free/en/papers/robey-2023-smoothllm-defending-large-language-models/)
36. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. [find](https://scholar.google.com/scholar?q=Gemini+Team%2C+Rohan+Anil%2C+Sebastian+Borgeaud%2C+Yonghui+Wu%2C+Jean%2DBaptiste+Alayrac%2C+Jiahui+Yu%2C+Radu+Soricut%2C+Johan+Schalkwyk%2C+Andrew+M+Dai%2C+Anja+Hauth%2C+et+al%2E+2023%2E+Gemini%3A+a+family+of+highly+capable+mult) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/)
37. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
38. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966. [find](https://scholar.google.com/scholar?q=Yufei+Wang%2C+Wanjun+Zhong%2C+Liangyou+Li%2C+Fei+Mi%2C+Xingshan+Zeng%2C+Wenyong+Huang%2C+Lifeng+Shang%2C+Xin+Jiang%2C+and+Qun+Liu%2E+2023%2E+Aligning+large+language+models+with+human%3A+A+survey%2E+arXiv+preprint+arXiv%3A2307%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-aligning-large-language-models-human/)
39. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023a. Jailbroken: How does LLM safety training fail? arXiv preprint arXiv:2307.02483. [find](https://scholar.google.com/scholar?q=Alexander+Wei%2C+Nika+Haghtalab%2C+and+Jacob+Steinhardt%2E+2023a%2E+Jailbroken%3A+How+does+LLM+safety+training+fail%3F+arXiv+preprint+arXiv%3A2307%2E02483%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/)
40. Zeming Wei, Yifei Wang, and Yisen Wang. 2023b. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387. [find](https://scholar.google.com/scholar?q=Zeming+Wei%2C+Yifei+Wang%2C+and+Yisen+Wang%2E+2023b%2E+Jailbreak+and+guard+aligned+language+models+with+only+few+in%2Dcontext+demonstrations%2E+arXiv+preprint+arXiv%3A2310%2E06387%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbreak-guard-aligned-language-models/)
41. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. [find](https://scholar.google.com/scholar?q=Laura+Weidinger%2C+John+Mellor%2C+Maribeth+Rauh%2C+Conor+Griffin%2C+Jonathan+Uesato%2C+Po%2DSen+Huang%2C+Myra+Cheng%2C+Mia+Glaese%2C+Borja+Balle%2C+Atoosa+Kasirzadeh%2C+et+al%2E+2021%2E+Ethical+and+social+risks+of+harm+from+la) [our card](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)
42. Fangzhao Wu, Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, and Xing Xie. 2023a. Defending ChatGPT against jailbreak attack via self-reminder. [find](https://scholar.google.com/scholar?q=Fangzhao+Wu%2C+Yueqi+Xie%2C+Jingwei+Yi%2C+Jiawei+Shao%2C+Justin+Curl%2C+Lingjuan+Lyu%2C+Qifeng+Chen%2C+and+Xing+Xie%2E+2023a%2E+Defending+ChatGPT+against+jailbreak+attack+via+self%2Dreminder%2E)
43. Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. 2023b. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256. IEEE. [find](https://scholar.google.com/scholar?q=Jiayang+Wu%2C+Wensheng+Gan%2C+Zefeng+Chen%2C+Shicheng+Wan%2C+and+S+Yu+Philip%2E+2023b%2E+Multimodal+large+language+models%3A+A+survey%2E+In+2023+IEEE+International+Conference+on+Big+Data+%28BigData%29%2C+pages+2247%E2%80%932256%2E+I)
44. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridg)
45. Jiahao Yu, Xingwei Lin, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253. [find](https://scholar.google.com/scholar?q=Jiahao+Yu%2C+Xingwei+Lin%2C+and+Xinyu+Xing%2E+2023%2E+Gptfuzzer%3A+Red+teaming+large+language+models+with+auto%2Dgenerated+jailbreak+prompts%2E+arXiv+preprint+arXiv%3A2309%2E10253%2E) [our card](https://md.research.elamur.ai/free/en/papers/yu-2023-gptfuzzer-red-teaming-large-language/)
46. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How Johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing LLMs. arXiv preprint arXiv:2401.06373. [find](https://scholar.google.com/scholar?q=Yi+Zeng%2C+Hongpeng+Lin%2C+Jingwen+Zhang%2C+Diyi+Yang%2C+Ruoxi+Jia%2C+and+Weiyan+Shi%2E+2024%2E+How+Johnny+can+persuade+llms+to+jailbreak+them%3A+Rethinking+persuasion+to+challenge+AI+safety+by+humanizing+LLMs%2E+arXiv)
47. Zhexin Zhang, Junxiao Yang, Pei Ke, and Minlie Huang. 2023. Defending large language models against jailbreaking attacks through goal prioritization. arXiv preprint arXiv:2311.09096. [find](https://scholar.google.com/scholar?q=Zhexin+Zhang%2C+Junxiao+Yang%2C+Pei+Ke%2C+and+Minlie+Huang%2E+2023%2E+Defending+large+language+models+against+jailbreaking+attacks+through+goal+prioritization%2E+arXiv+preprint+arXiv%3A2311%2E09096%2E)
48. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging LLM-as-a-judge with MT-Bench and chatbot arena. arXiv preprint arXiv:2306.05685. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+2023%2E+Judging+LLM%2Das%2Da%2Djudge+with+MT%2DBench+and+chatbot+arena%2E+a)
49. Andy Zhou, Bo Li, and Haohan Wang. 2024. Robust prompt optimization for defending language models against jailbreaking attacks. arXiv preprint arXiv:2401.17263. — [arXiv:2401.17263](https://arxiv.org/abs/2401.17263) [find](https://scholar.google.com/scholar?q=Andy+Zhou%2C+Bo+Li%2C+and+Haohan+Wang%2E+2024%2E+Robust+prompt+optimization+for+defending+language+models+against+jailbreaking+attacks%2E+arXiv+preprint+arXiv%3A2401%2E17263%2E+%E2%80%94+%5BarXiv%3A2401%2E17263%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
50. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+2023%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2E)
51. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. Autodan: Automatic and interpretable adversarial attacks on large language models. arXiv preprint arXiv:2310.15140. [find](https://scholar.google.com/scholar?q=Sicheng+Zhu%2C+Ruiyi+Zhang%2C+Bang+An%2C+Gang+Wu%2C+Joe+Barrow%2C+Zichao+Wang%2C+Furong+Huang%2C+Ani+Nenkova%2C+and+Tong+Sun%2E+2023%2E+Autodan%3A+Automatic+and+interpretable+adversarial+attacks+on+large+language+models%2E+a)
52. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. [find](https://scholar.google.com/scholar?q=Andy+Zou%2C+Zifan+Wang%2C+J+Zico+Kolter%2C+and+Matt+Fredrikson%2E+2023%2E+Universal+and+transferable+adversarial+attacks+on+aligned+language+models%2E+arXiv+preprint+arXiv%3A2307%2E15043%2E) [our card](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/)