---
title: "MART — paper by Ge et al. (2024) — summary and ideas"
description: "Making LLMs safe usually depends on human red-teaming, where many annotators spend months… 21 references, cited by 49 papers here."
authors: ["Suyu Ge", "Chunting Zhou", "Rui Hou", "Madian Khabsa", "Yi-Chia Wang", "Qifan Wang", "Jiawei Han", "Yuning Mao"]
year: 2024
venue: "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.naacl-long.107/"
doi: "10.18653/v1/2024.naacl-long.107"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ge-2024-mart-improving-llm-safety-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ge-2024-mart-improving-llm-safety-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / MART: Improving LLM Safety with Multi-round Automatic Red-Teaming

# MART: Improving LLM Safety with Multi-round Automatic Red-Teaming

**Authors:** [Suyu Ge](https://md.research.elamur.ai/free/en/authors/suyu-ge/), [Chunting Zhou](https://md.research.elamur.ai/free/en/authors/chunting-zhou/), [Rui Hou](https://md.research.elamur.ai/free/en/authors/rui-hou/), [Madian Khabsa](https://md.research.elamur.ai/free/en/authors/madian-khabsa/), [Yi-Chia Wang](https://md.research.elamur.ai/free/en/authors/yi-chia-wang/), [Qifan Wang](https://md.research.elamur.ai/free/en/authors/qifan-wang/), [Jiawei Han](https://md.research.elamur.ai/free/en/authors/jiawei-han/) and [Yuning Mao](https://md.research.elamur.ai/free/en/authors/yuning-mao/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2024](https://md.research.elamur.ai/free/en/venues/naacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.naacl-long.107/) · [PDF](https://aclanthology.org/2024.naacl-long.107.pdf) · [DOI 10.18653/v1/2024.naacl-long.107](https://doi.org/10.18653/v1/2024.naacl-long.107) · [OpenAlex](https://openalex.org/W4401042259)

**Citations:** 30 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/ge-2024-mart-improving-llm-safety-multi/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Making LLMs safe usually depends on human red-teaming, where many annotators spend months writing attack prompts and safe answers. Earlier automatic red-teaming methods could find weaknesses, but they did not fix them. MART is a framework that automates both steps. Two models start from LLaMA 65B instruction-tuned on LIMA and Open Assistant. One is an adversarial model, warmed up on a small hand-built seed set of attack prompts. The other is the target model being hardened.

In each round:
- The adversarial model writes new prompts modelled on its earlier successful attacks.
- The target model answers them, and separate safety and helpfulness reward models score the answers.
- Prompts that produced unsafe answers become training data for the attacker.
- Answers that are both safe and helpful are used to fine-tune the target model on its own outputs.

After four rounds, the target model's violation rate falls sharply and approaches ChatGPT's safety level. Helpfulness stays stable. This suggests that LLM-versus-LLM competition can greatly reduce the human labour needed for safety alignment.

## Key points
- The seed red-teaming set has about 2,400 manually written prompts (1,700 for training, 700 for evaluation). They are organised by violation category and attack style, following the Llama 2 protocol.
- On adversarial benchmarks, violation rate drops by up to 84.7% under reward-model evaluation and 53.7% under human evaluation, relative to a baseline with little safety tuning.
- Gains level off once violations on generated attacks reach about 10%, because too few unsafe answers remain to use as training data. The authors suggest adding outside methods such as GCG or human red-teamers.
- For comparison, Llama 2-Chat used over 350 red-teamers, and Anthropic gathered 42k prompts from more than 100 crowdworkers.
- Limitations:
  - Only single-turn prompts are studied.
  - Training relies on supervised fine-tuning and rejection sampling. Extending MART to DPO and RLHF is left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works

## Cite

Cite as: Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han and Yuning Mao. 2024. MART: Improving LLM Safety with Multi-round Automatic Red-Teaming. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ge2024mart,
  title = {{MART: Improving LLM Safety with Multi-round Automatic Red-Teaming}},
  author = {Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han and Yuning Mao},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.107},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.107}
}
```

- [← previous Generating Benchmarks for Factuality Evaluation of Language Models](https://md.research.elamur.ai/free/en/papers/muhlgay-2024-generating-benchmarks-factuality-evaluation-language/)
- [next → LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning](https://md.research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

21 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+2021%2E+A+general+language+assistant+as+a+laboratory+for+alignm)
2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+2022a%2E+Training+a+helpful+and+harmless+assistant+with+rei)
3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073.
4. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. [Jailbreaking black box large language models in twenty queries](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/). arXiv preprint arXiv:2310.08419.
5. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. [Build it break it fix it for dialogue safety: Robustness from adversarial human attack](https://md.research.elamur.ai/free/en/papers/dinan-2019-build-it-break-it-fix/). arXiv preprint arXiv:1908.06083.
6. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. [Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/). arXiv preprint arXiv:2209.07858.
7. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. [Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection](https://md.research.elamur.ai/free/en/papers/greshake-2023-ve-signed-up-compromising-real/). arXiv preprint arXiv:2302.12173.
8. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2019%2E+The+curious+case+of+neural+text+degeneration%2E+arXiv+preprint+arXiv%3A1904%2E09751%2E)
9. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. [Llama guard: Llm-based input-output safeguard for human-ai conversations](https://md.research.elamur.ai/free/en/papers/inan-2023-llama-guard-llm-based-input/).
10. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. 2023. [Openassistant conversations–democratizing large language model alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/). arXiv preprint arXiv:2304.07327.
11. Raz Lapid, Ron Langberg, and Moshe Sipper. 2023. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446. — [arXiv:2309.01446](https://arxiv.org/abs/2309.01446) [find](https://scholar.google.com/scholar?q=Raz+Lapid%2C+Ron+Langberg%2C+and+Moshe+Sipper%2E+2023%2E+Open+sesame%21+universal+black+box+jailbreaking+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E01446%2E+%E2%80%94+%5BarXiv%3A2309%2E01446%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F230)
12. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/](https://github.com/) tatsu-lab/alpaca_eval. [↗](https://github.com/)
13. Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. 2023. Flirt: Feedback loop in-context red teaming. arXiv preprint arXiv:2308.04265. [find](https://scholar.google.com/scholar?q=Ninareh+Mehrabi%2C+Palash+Goyal%2C+Christophe+Dupuy%2C+Qian+Hu%2C+Shalini+Ghosh%2C+Richard+Zemel%2C+Kai%2DWei+Chang%2C+Aram+Galstyan%2C+and+Rahul+Gupta%2E+2023%2E+Flirt%3A+Feedback+loop+in%2Dcontext+red+teaming%2E+arXiv+preprint)
14. R OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, pages 2303–08774.
15. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. [Red teaming language models with language models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419–3448.
16. Fábio Perez and Ian Ribeiro. 2022. [Ignore previous prompt: Attack techniques for language models](https://md.research.elamur.ai/free/en/papers/perez-2022-ignore-previous-prompt-attack-techniques/). In NeurIPS ML Safety Workshop.
17. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
18. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023b%2E+Llama+2%3A+Open+foundation+and+f)
19. Xiaodong Yu, Hao Cheng, Xiaodong Liu, Dan Roth, and Jianfeng Gao. 2023. Automatic hallucination assessment for aligned large language models via transferable adversarial attacks. arXiv preprint. [find](https://scholar.google.com/scholar?q=Xiaodong+Yu%2C+Hao+Cheng%2C+Xiaodong+Liu%2C+Dan+Roth%2C+and+Jianfeng+Gao%2E+2023%2E+Automatic+hallucination+assessment+for+aligned+large+language+models+via+transferable+adversarial+attacks%2E+arXiv+preprint%2E)
20. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+2023%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2E)
21. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). arXiv preprint arXiv:2307.15043.

*Updated 24 October 2026*