---
title: "AEGIS2.0 — paper by Ghosh et al. (2025) — summary and ideas"
description: "Content moderation for LLMs needs openly licensed, human-labeled training data that covers many… 41 references, cited by 39 papers here."
authors: ["Shaona Ghosh", "Prasoon Varshney", "Makesh Narsimhan Sreedhar", "Aishwarya Padmakumar", "Traian Rebedea", "Jibin Rajan Varghese", "Christopher Parisien"]
year: 2025
venue: "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.naacl-long.306/"
doi: "10.18653/v1/2025.naacl-long.306"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ghosh-2025-aegis2-0-diverse-ai-safety/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ghosh-2025-aegis2-0-diverse-ai-safety/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails

# AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails

**Authors:** Shaona Ghosh, Prasoon Varshney, [Makesh Narsimhan Sreedhar](https://md.research.elamur.ai/free/en/authors/makesh-narsimhan-sreedhar/), [Aishwarya Padmakumar](https://md.research.elamur.ai/free/en/authors/aishwarya-padmakumar/), [Traian Rebedea](https://md.research.elamur.ai/free/en/authors/traian-rebedea/), Jibin Rajan Varghese and Christopher Parisien · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2025](https://md.research.elamur.ai/free/en/venues/naacl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.naacl-long.306/) · [PDF](https://aclanthology.org/2025.naacl-long.306.pdf) · [DOI 10.18653/v1/2025.naacl-long.306](https://doi.org/10.18653/v1/2025.naacl-long.306) · [OpenAlex](https://openalex.org/W4411119289)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [39](https://md.research.elamur.ai/free/en/papers/ghosh-2025-aegis2-0-diverse-ai-safety/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Content moderation for LLMs needs openly licensed, human-labeled training data that covers many kinds of harm. Existing resources fall short. Many are small benchmarks. Others depend on GPT-4 or other non-commercial generators, provide only binary labels, or are kept private, as with Llama Guard and ShieldGemma. NVIDIA researchers address this with AEGIS 2.0, an extension of AEGIS 1.0. It has two parts:

- **Taxonomy:** a tiered risk scheme with 12 core categories plus 9 finer ones. The finer categories were derived from free-text notes that annotators wrote when content fit no predefined class.
- **Dataset:** 34,248 human–LLM interactions. Prompts were drawn from commercially licensed sources, including adversarial jailbreak and culturally specific prompts. Responses were generated by Mistral-7B-v0.1. Professional annotators labeled whole dialogues, and a jury of several LLMs produced separate labels for prompts and responses.

LoRA fine-tuning of Llama 3.1-8B-Instruct on this data yields Llama3.1-AegisGuard. It outperforms Llama Guard 3-8B, which starts from the same backbone, and matches WildGuard, which was trained on a larger GPT-4-derived corpus. The work shows that a commercially usable resource can support state-of-the-art guard models.

## Key points
- The dataset is published on Hugging Face as `nvidia/Aegis-AI-Content-Safety-Dataset-2.0`. The guard model is available as LoRA adapters (`llama-3.1-nemoguard-8b-content-safety`) and as an NVIDIA NIM.
- For comparison, 85% of the training split of WildGuardMix (92K samples) was generated with GPT-4, which restricts commercial use. ToxicChat and WildGuardTrain provide no hazard categories.
- Adding dialogue topic-following data to the safety training mix improves robustness. It also helps guard models handle new risk policies that are only specified at inference time.
- Annotation was done in Label Studio by about 12 US-based annotators over six months.
- Acknowledged limitations:
  - English-only data.
  - A single model, Mistral-7B-v0.1, generated all responses.
  - Some categories, such as Sexual (minor) and Threat, are underrepresented.
  - Both the LLM jury and the human annotators may introduce bias.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese and Christopher Parisien. 2025. AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ghosh2025aegis2,
  title = {{AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails}},
  author = {Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese and Christopher Parisien},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.306},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.306}
}
```

- [← previous DeepSeek-OCR: Contexts Optical Compression](https://md.research.elamur.ai/free/en/papers/wei-2025-deepseek-ocr-contexts-optical/)
- [next → Why do language models perform worse for morphologically complex languages?](https://md.research.elamur.ai/free/en/papers/arnett-2025-why-do-language-models-perform/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

41 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. 2024. [Nemotron-4 340b technical report](https://md.research.elamur.ai/free/en/papers/adler-2024-nemotron-4-340b-technical-report/). arXiv preprint arXiv:2406.11704.
2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+2022a%2E+Training+a+helpful+and+harmless+assistant+with+rei)
3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073.
4. Rishabh Bhardwaj and Soujanya Poria. 2023. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662. [find](https://scholar.google.com/scholar?q=Rishabh+Bhardwaj+and+Soujanya+Poria%2E+2023%2E+Red%2Dteaming+large+language+models+using+chain+of+utterances+for+safety%2Dalignment%2E+arXiv+preprint+arXiv%3A2308%2E09662%2E)
5. Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2023. [Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions](https://md.research.elamur.ai/free/en/papers/bianchi-2023-safety-tuned-llamas-lessons-improving/). arXiv preprint arXiv:2309.07875.
6. Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. 2024. Aegis: Online adaptive ai content safety moderation with ensemble of llm experts. Preprint, arXiv:2404.05993. — [arXiv:2404.05993](https://arxiv.org/abs/2404.05993) [find](https://scholar.google.com/scholar?q=Shaona+Ghosh%2C+Prasoon+Varshney%2C+Erick+Galinkin%2C+and+Christopher+Parisien%2E+2024%2E+Aegis%3A+Online+adaptive+ai+content+safety+moderation+with+ensemble+of+llm+experts%2E+Preprint%2C+arXiv%3A2404%2E05993%2E+%E2%80%94+%5BarXiv%3A2)
7. Maarten Grootendorst. 2022. [Bertopic: Neural topic modeling with a class-based tf-idf procedure](https://md.research.elamur.ai/free/en/papers/grootendorst-2022-bertopic-neural-topic-modeling-class/). Preprint, arXiv:2203.05794.
8. Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. [Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms](https://md.research.elamur.ai/free/en/papers/han-2024-wildguard-open-one-stop-moderation/). arXiv preprint arXiv:2406.18495.
9. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. [Llama guard: Llm-based input-output safeguard for human-ai conversations](https://md.research.elamur.ai/free/en/papers/inan-2023-llama-guard-llm-based-input/). arXiv preprint arXiv:2312.06674.
10. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2024. [Beavertails: Towards improved safety alignment of llm via a human-preference dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/). Advances in Neural Information Processing Systems, 36.
11. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
12. Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovec, Jun-Yan Zhu, Li Fei-Fei, Jiajun Wu, Stefano Ermon, and Percy Liang. 2023. Holistic evaluation of text-… [find](https://scholar.google.com/scholar?q=Tony+Lee%2C+Michihiro+Yasunaga%2C+Chenlin+Meng%2C+Yifan+Mai%2C+Joon+Sung+Park%2C+Agrim+Gupta%2C+Yunzhi+Zhang%2C+Deepak+Narayanan%2C+Hannah+Benita+Teufel%2C+Marco+Bellagente%2C+Minguk+Kang%2C+Taesung+Park%2C+Jure+Leskovec%2C+Ju)
13. Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3197–3207. [find](https://scholar.google.com/scholar?q=Alyssa+Lees%2C+Vinh+Q+Tran%2C+Yi+Tay%2C+Jeffrey+Sorensen%2C+Jai+Gupta%2C+Donald+Metzler%2C+and+Lucy+Vasserman%2E+2022%2E+A+new+generation+of+perspective+api%3A+Efficient+multilingual+character%2Dlevel+transformers%2E+In+Pr)
14. Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. 2023. [Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation](https://md.research.elamur.ai/free/en/papers/lin-2023-toxicchat-unveiling-hidden-challenges-toxicity/). arXiv preprint arXiv:2310.17389.
15. AI @ Meta Llama Team. 2024a. The llama 3 family of models. [https://github.com/meta-llama/](https://github.com/meta-llama/) PurpleLlama/blob/main/Llama-Guard3/1B/ MODEL_CARD.md. [↗](https://github.com/meta-llama/)
16. AI @ Meta Llama Team. 2024b. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783.
17. Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. [A holistic approach to undesired content detection in the real world](https://md.research.elamur.ai/free/en/papers/markov-2022-holistic-approach-undesired-content-detection/). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15009–150…
18. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. [Harmbench: A standardized evaluation framework for automated red teaming and robust refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/). arXiv preprint arXiv:2402.04249.
19. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
20. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744.
21. Bhaktipriya Radharapu, Kevin Robinson, Lora Aroyo, and Preethi Lahoti. 2023. Aart: Ai-assisted red-teaming with diverse data generation for new llm-powered applications. arXiv preprint arXiv:2311.08592. [find](https://scholar.google.com/scholar?q=Bhaktipriya+Radharapu%2C+Kevin+Robinson%2C+Lora+Aroyo%2C+and+Preethi+Lahoti%2E+2023%2E+Aart%3A+Ai%2Dassisted+red%2Dteaming+with+diverse+data+generation+for+new+llm%2Dpowered+applications%2E+arXiv+preprint+arXiv%3A2311%2E0859)
22. Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. Preprint, arXiv:2310.10501. [find](https://scholar.google.com/scholar?q=Traian+Rebedea%2C+Razvan+Dinu%2C+Makesh+Sreedhar%2C+Christopher+Parisien%2C+and+Jonathan+Cohen%2E+2023%2E+Nemo+guardrails%3A+A+toolkit+for+controllable+and+safe+llm+applications+with+programmable+rails%2E+Preprint%2C+a)
23. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. Preprint, arXiv:2112.10752. [find](https://scholar.google.com/scholar?q=Robin+Rombach%2C+Andreas+Blattmann%2C+Dominik+Lorenz%2C+Patrick+Esser%2C+and+Bj%C3%B6rn+Ommer%2E+2022%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+Preprint%2C+arXiv%3A2112%2E10752%2E)
24. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023. [Xstest: A test suite for identifying exaggerated safety behaviours in large language models](https://md.research.elamur.ai/free/en/papers/rottger-2023-xstest-test-suite-identifying-exaggerated/). arXiv preprint arXiv:2308.01263.
25. Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. 2023. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. Preprint, arXiv:2211.05105. [find](https://scholar.google.com/scholar?q=Patrick+Schramowski%2C+Manuel+Brack%2C+Bj%C3%B6rn+Deiseroth%2C+and+Kristian+Kersting%2E+2023%2E+Safe+latent+diffusion%3A+Mitigating+inappropriate+degeneration+in+diffusion+models%2E+Preprint%2C+arXiv%3A2211%2E05105%2E)
26. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. ["do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models](https://md.research.elamur.ai/free/en/papers/shen-2023-anything-now-characterizing-evaluating-wild/). Preprint, arXiv:2308.03825.
27. Makesh Narsimhan Sreedhar, Traian Rebedea, Shaona Ghosh, Jiaqi Zeng, and Christopher Parisien. 2024. Canttalkaboutthis: Aligning language models to stay on topic in dialogues. Preprint, arXiv:2404.03820. [find](https://scholar.google.com/scholar?q=Makesh+Narsimhan+Sreedhar%2C+Traian+Rebedea%2C+Shaona+Ghosh%2C+Jiaqi+Zeng%2C+and+Christopher+Parisien%2E+2024%2E+Canttalkaboutthis%3A+Aligning+language+models+to+stay+on+topic+in+dialogues%2E+Preprint%2C+arXiv%3A2404%2E038)
28. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: a strong, replicable instruction-following model; 2023. URL [https://crfm.stanford](https://crfm.stanford). edu/2023/03/13/alpaca. html. [↗](https://crfm.stanford)
29. Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00… [find](https://scholar.google.com/scholar?q=Slav+Petrov%2C+Oriol+Vinyals%2C+Jeff+Dean%2C+Demis+Hassabis%2C+Koray+Kavukcuoglu%2C+Clement+Farabet%2C+Elena+Buchatskaya%2C+Sebastian+Borgeaud%2C+Noah+Fiedel%2C+Armand+Joulin%2C+Kathleen+Kenealy%2C+Robert+Dadashi%2C+and+Alek)
30. Gemma Team. 2024a. Gemma. [find](https://scholar.google.com/scholar?q=Gemma+Team%2E+2024a%2E+Gemma%2E)
31. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton T… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Ta)
32. Neeraj Varshney, Pavel Dolin, Agastya Seth, and Chitta Baral. 2023. The art of defending: A systematic evaluation and analysis of llm defense strategies on safety and over-defensiveness. arXiv preprint arXiv:2401.00287. [find](https://scholar.google.com/scholar?q=Neeraj+Varshney%2C+Pavel+Dolin%2C+Agastya+Seth%2C+and+Chitta+Baral%2E+2023%2E+The+art+of+defending%3A+A+systematic+evaluation+and+analysis+of+llm+defense+strategies+on+safety+and+over%2Ddefensiveness%2E+arXiv+preprin)
33. Bertie Vidgen, Adarsh Agrawal, Ahmed M Ahmed, Victor Akinwande, Namir Al-Nuaimi, Najla Alfaraj, Elie Alhajjar, Lora Aroyo, Trupti Bavalatti, Borhane Blili-Hamelin, et al. 2024. [Introducing v0](https://md.research.elamur.ai/free/en/papers/vidgen-2024-introducing-v0-5-ai-safety/). 5 of the ai safety benchmark from mlcommons. arXiv preprint arXiv:2404.12241.
34. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+arXiv+pr)
35. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023. [Do-not-answer: A dataset for evaluating safeguards in llms](https://md.research.elamur.ai/free/en/papers/wang-2023-answer-dataset-evaluating-safeguards-llms/). arXiv preprint arXiv:2308.13387.
36. Yijun Yang, Ruiyuan Gao, Xiaosen Wang, Tsung-Yi Ho, Nan Xu, and Qiang Xu. 2024a. MMA-Diffusion: MultiModal Attack on Diffusion Models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Yijun+Yang%2C+Ruiyuan+Gao%2C+Xiaosen+Wang%2C+Tsung%2DYi+Ho%2C+Nan+Xu%2C+and+Qiang+Xu%2E+2024a%2E+MMA%2DDiffusion%3A+MultiModal+Attack+on+Diffusion+Models%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Patt)
37. Yue Yang, Yuqi Lin, Hong Liu, Wenqi Shao, Runjian Chen, Hailong Shang, Yu Wang, Yu Qiao, Kaipeng Zhang, and Ping Luo. 2024b. Position: Towards implicit prompt for text-to-image models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learn… [find](https://scholar.google.com/scholar?q=Yue+Yang%2C+Yuqi+Lin%2C+Hong+Liu%2C+Wenqi+Shao%2C+Runjian+Chen%2C+Hailong+Shang%2C+Yu+Wang%2C+Yu+Qiao%2C+Kaipeng+Zhang%2C+and+Ping+Luo%2E+2024b%2E+Position%3A+Towards+implicit+prompt+for+text%2Dto%2Dimage+models%2E+In+Proceedings+)
38. Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. 2024. Shieldgemma: Generative ai content moderation based on gemma. Preprint, arXiv:2407.21772. — [arXiv:2407.21772](https://arxiv.org/abs/2407.21772) [find](https://scholar.google.com/scholar?q=Wenjun+Zeng%2C+Yuchi+Liu%2C+Ryan+Mullins%2C+Ludovic+Peran%2C+Joe+Fernandez%2C+Hamza+Harkous%2C+Karthik+Narasimhan%2C+Drew+Proud%2C+Piyush+Kumar%2C+Bhaktipriya+Radharapu%2C+Olivia+Sturman%2C+and+Oscar+Wahltinez%2E+2024%2E+Shiel)
39. Yiming Zhang, Sravani Nanduri, Liwei Jiang, Tongshuang Wu, and Maarten Sap. 2023. Biasx:" thinking slow" in toxic content moderation with explanations of implied social biases. arXiv preprint arXiv:2305.13589. [find](https://scholar.google.com/scholar?q=Yiming+Zhang%2C+Sravani+Nanduri%2C+Liwei+Jiang%2C+Tongshuang+Wu%2C+and+Maarten+Sap%2E+2023%2E+Biasx%3A%22+thinking+slow%22+in+toxic+content+moderation+with+explanations+of+implied+social+biases%2E+arXiv+preprint+arXiv%3A23)
40. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+2023%2E)
41. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36:46595–46623.

*Updated 31 October 2026*