---
title: "Do-Not-Answer: Evaluating Safeguards in LLMs (2024)"
description: "This paper offers a cheap, public way to test whether a model refuses requests it should not fulfil. 26 references, cited by 74 papers here."
authors: ["Yuxia Wang", "Haonan Li", "Xudong Han", "Preslav Nakov", "Timothy Baldwin"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: EACL 2024"
source: "https://aclanthology.org/2024.findings-eacl.61/"
doi: "10.18653/v1/2024.findings-eacl.61"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-do-not-answer-evaluating-safeguards/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-do-not-answer-evaluating-safeguards/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Do-Not-Answer: Evaluating Safeguards in LLMs

# Do-Not-Answer: Evaluating Safeguards in LLMs

**Authors:** [Yuxia Wang](https://md.research.elamur.ai/free/en/authors/yuxia-wang/), [Haonan Li](https://md.research.elamur.ai/free/en/authors/haonan-li/), [Xudong Han](https://md.research.elamur.ai/free/en/authors/xudong-han/), [Preslav Nakov](https://md.research.elamur.ai/free/en/authors/preslav-nakov/) and [Timothy Baldwin](https://md.research.elamur.ai/free/en/authors/timothy-baldwin/) · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-eacl.61/) · [PDF](https://aclanthology.org/2024.findings-eacl.61.pdf) · [DOI 10.18653/v1/2024.findings-eacl.61](https://doi.org/10.18653/v1/2024.findings-eacl.61) · [OpenAlex](https://openalex.org/W7126420743)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · [74](https://md.research.elamur.ai/free/en/papers/wang-2024-do-not-answer-evaluating-safeguards/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Open-source LLMs often ship without the safety layers that commercial systems have. Existing harmful-prompt collections tend to be small, proprietary, or unlabeled. This paper offers a cheap, public way to test whether a model refuses requests it should not fulfil.

The authors extend the risk framework of Weidinger et al. (2021) into a three-tier taxonomy for text-only models. It has five risk areas, twelve harm types and 61 fine-grained risk categories. From this taxonomy they build Do-Not-Answer, a set of 939 prompts that a responsible assistant should decline or push back on. GPT-4 generated the prompts, coaxed through a fabricated multi-turn chat history so it would not refuse. The authors collected and manually labelled the responses of six popular LLMs using a scheme of response patterns. They then tested automatic safety judging with GPT-4 prompting and with fine-tuned pretrained classifiers. Compact BERT-style evaluators of at most 600M parameters performed about as well as GPT-4 overall. This makes routine safeguard auditing affordable for teams releasing open models.

## Key points
- **Dataset size:** 939 risky prompts and 5,634 manually annotated responses from six LLMs. Data and code are hosted at github.com/Libr-AI/do-not-answer.
- **Taxonomy:** the top level covers information hazards, malicious uses, discrimination/exclusion/toxicity, misinformation harms, and human–chatbot interaction harms. Weidinger's automation/environmental area is excluded because the work targets text-only models without API access.
- **Largest harm types:** sensitive organizational or government information (136 prompts) and help with illegal activities (132). The smallest is adult content (28).
- **Main result:** fine-tuned small classifiers (≤600M parameters) match GPT-4 as automatic safety evaluators at much lower cost.
- **Limitations:**
  - Every prompt is risky, so over-cautious models that refuse everything cannot be detected.
  - Evaluation covers only English, single-turn, zero-shot use.
  - Each response receives a single label, although several response categories can apply.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) — University, Melbourne, Australia · 60 works · 21,199 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 221 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works

## Cite

Cite as: Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov and Timothy Baldwin. 2024. Do-Not-Answer: Evaluating Safeguards in LLMs. Findings of the Association for Computational Linguistics: EACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2024do,
  title = {{Do-Not-Answer: Evaluating Safeguards in LLMs}},
  author = {Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov and Timothy Baldwin},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-eacl.61},
  url = {https://doi.org/10.18653/v1/2024.findings-eacl.61}
}
```

- [← previous LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models](https://md.research.elamur.ai/free/en/papers/parmar-2024-logicbench-systematic-evaluation-logical-reasoning/)
- [next → FineSurE: Fine-grained Summarization Evaluation using LLMs](https://md.research.elamur.ai/free/en/papers/song-2024-finesure-fine-grained-summarization-evaluation/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

26 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). CoRR, abs/2004.05150.
2. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. [Toxicity in ChatGPT: Analyzing persona-assigned language models](https://md.research.elamur.ai/free/en/papers/deshpande-2023-toxicity-chatgpt-analyzing-persona-assigned/). CoRR, abs/2304.05335.
3. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. BOLD: dataset and metrics for measuring biases in open-ended language generation. In FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 862–872. ACM. [find](https://scholar.google.com/scholar?q=Jwala+Dhamala%2C+Tony+Sun%2C+Varun+Kumar%2C+Satyapriya+Krishna%2C+Yada+Pruksachatkun%2C+Kai%2DWei+Chang%2C+and+Rahul+Gupta%2E+2021%2E+BOLD%3A+dataset+and+metrics+for+measuring+biases+in+open%2Dended+language+generation%2E+In)
4. Emilio Ferrara. 2023. Should ChatGPT be biased? challenges and risks of bias in large language models. CoRR, abs/2304.03738. [find](https://scholar.google.com/scholar?q=Emilio+Ferrara%2E+2023%2E+Should+ChatGPT+be+biased%3F+challenges+and+risks+of+bias+in+large+language+models%2E+CoRR%2C+abs%2F2304%2E03738%2E)
5. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. [Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/). arXiv preprint arXiv:2209.07858.
6. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. [RealToxicityPrompts: Evaluating neural toxic degeneration in language models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti…
7. Xudong Han, Timothy Baldwin, and Trevor Cohn. 2023a. Everybody needs good neighbours: An unsupervised locality-based method for bias mitigation. In The Eleventh International Conference on Learning Representations, ICLR 2023. OpenReview.net. [find](https://scholar.google.com/scholar?q=Xudong+Han%2C+Timothy+Baldwin%2C+and+Trevor+Cohn%2E+2023a%2E+Everybody+needs+good+neighbours%3A+An+unsupervised+locality%2Dbased+method+for+bias+mitigation%2E+In+The+Eleventh+International+Conference+on+Learning+Re)
8. Xudong Han, Timothy Baldwin, and Trevor Cohn. 2023b. Fair enough: Standardizing evaluation and model selection for fairness research in NLP. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, pages 297–312. Association for Compu… [find](https://scholar.google.com/scholar?q=Xudong+Han%2C+Timothy+Baldwin%2C+and+Trevor+Cohn%2E+2023b%2E+Fair+enough%3A+Standardizing+evaluation+and+model+selection+for+fairness+research+in+NLP%2E+In+Proceedings+of+the+17th+Conference+of+the+European+Chapt)
9. Xudong Han, Aili Shen, Trevor Cohn, Timothy Baldwin, and Lea Frermann. 2022. Systematic evaluation of predictive fairness. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language… [find](https://scholar.google.com/scholar?q=Xudong+Han%2C+Aili+Shen%2C+Trevor+Cohn%2C+Timothy+Baldwin%2C+and+Lea+Frermann%2E+2022%2E+Systematic+evaluation+of+predictive+fairness%2E+In+Proceedings+of+the+2nd+Conference+of+the+Asia%2DPacific+Chapter+of+the+Assoc)
10. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. [ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:…
11. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. [BeaverTails: Towards improved safety alignment of LLM via a human-preference dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/). arXiv preprint arXiv:2307.04657.
12. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
13. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
14. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2E+OpenReview%2Enet%2E)
15. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. [On measuring social biases in sentence encoders](https://md.research.elamur.ai/free/en/papers/may-2019-measuring-social-biases-sentence-encoders/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pag…
16. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. [Recipes for building an open-domain chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational…
17. Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul F. Christiano, and… [find](https://scholar.google.com/scholar?q=Toby+Shevlane%2C+Sebastian+Farquhar%2C+Ben+Garfinkel%2C+Mary+Phuong%2C+Jess+Whittlestone%2C+Jade+Leung%2C+Daniel+Kokotajlo%2C+Nahema+Marchal%2C+Markus+Anderljung%2C+Noam+Kolt%2C+Lewis+Ho%2C+Divya+Siddarth%2C+Shahar+Avin%2C+Wil)
18. Shivashankar Subramanian, Xudong Han, Timothy Baldwin, Trevor Cohn, and Lea Frermann. 2021. Evaluating debiasing techniques for intersectional biases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pages 2492–2498. Association for Computational… [find](https://scholar.google.com/scholar?q=Shivashankar+Subramanian%2C+Xudong+Han%2C+Timothy+Baldwin%2C+Trevor+Cohn%2C+and+Lea+Frermann%2E+2021%2E+Evaluating+debiasing+techniques+for+intersectional+biases%2E+In+Proceedings+of+the+2021+Conference+on+Empirica)
19. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)
20. Sander Van Der Linden. 2022. Misinformation: susceptibility, spread, and interventions to immunize the public. Nature Medicine, 28(3):460–467. [find](https://scholar.google.com/scholar?q=Sander+Van+Der+Linden%2E+2022%2E+Misinformation%3A+susceptibility%2C+spread%2C+and+interventions+to+immunize+the+public%2E+Nature+Medicine%2C+28%283%29%3A460%E2%80%93467%2E)
21. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Alham Fikri Aji, and Preslav Nakov. 2023. M4: Multi-generator, multi-domain, and multilingual black-box machine-generated text detection. arXiv:2305.14902. [find](https://scholar.google.com/scholar?q=Yuxia+Wang%2C+Jonibek+Mansurov%2C+Petar+Ivanov%2C+Jinyan+Su%2C+Artem+Shelmanov%2C+Akim+Tsvigun%2C+Chenxi+Whitehouse%2C+Osama+Mohammed+Afzal%2C+Tarek+Mahmoud%2C+Alham+Fikri+Aji%2C+and+Preslav+Nakov%2E+2023%2E+M4%3A+Multi%2Dgenera)
22. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. [Jailbroken: How does LLM safety training fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) arXiv preprint arXiv:2307.02483.
23. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). CoRR, abs/2201.11903.
24. [Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean L…](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)
25. Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2023. [FLASK: fine-grained language model evaluation based on alignment skill sets](https://md.research.elamur.ai/free/en/papers/ye-2023-flask-fine-grained-language-model/). CoRR, abs/2307.10928.
26. Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023. [Exploring AI ethics of ChatGPT: A diagnostic analysis](https://md.research.elamur.ai/free/en/papers/zhuo-2023-red-teaming-chatgpt-jailbreaking-bias/). arXiv preprint arXiv:2301.12867.

*Updated 24 October 2026*