---
title: "Red-Teaming Large Language Models using Chain of Utterances"
description: "The authors present Red-Eval, a red-teaming benchmark based on Chain of Utterances (CoU)… 27 references, cited by 92 papers here."
authors: ["Rishabh Bhardwaj", "Soujanya Poria"]
year: 2023
venue: "arXiv:2308.09662"
source: "https://arxiv.org/abs/2308.09662"
doi: "10.48550/arxiv.2308.09662"
arxiv: "2308.09662"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bhardwaj-2023-red-teaming-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bhardwaj-2023-red-teaming-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment

# Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment

**Authors:** [Rishabh Bhardwaj](https://md.research.elamur.ai/free/en/authors/rishabh-bhardwaj/) and [Soujanya Poria](https://md.research.elamur.ai/free/en/authors/soujanya-poria/) · **Year:** 2023 · **Venue:** arXiv:2308.09662 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.09662) · [PDF](https://arxiv.org/pdf/2308.09662) · [DOI 10.48550/arxiv.2308.09662](https://doi.org/10.48550/arxiv.2308.09662) · [arXiv 2308.09662](https://arxiv.org/abs/2308.09662) · [OpenAlex](https://openalex.org/W4386044169)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [92](https://md.research.elamur.ai/free/en/papers/bhardwaj-2023-red-teaming-large-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models can still be pushed into producing dangerous content. Older safety probes, such as appending Chain-of-Thought cues to harmful questions, are now mostly refused by current models. The authors present Red-Eval, a red-teaming benchmark based on Chain of Utterances (CoU) prompting. The prompt stages a dialogue between a malicious Red-LM and an overly compliant Base-LM, gives a worked demonstration, and asks the tested model to write the next Base-LM turn, starting with its "internal thoughts". This attack succeeds on roughly 65% of harmful queries for GPT-4, 73% for ChatGPT, and over 86% for open-source LLaMA-based models. The second contribution, Red-Instruct, uses ChatGPT to build HarmfulQA: harmful questions paired with safe (blue) and jailbroken harmful (red) conversations. Vicuna-7B is then fine-tuned with Safe-Align, which trains on blue responses and optionally first pushes the model away from red ones through gradient ascent. The resulting model, Starling, resists red-teaming better and improves on HHH, while TruthfulQA, MMLU and BBH scores stay near the baseline.

## Key points
- **HarmfulQA:** 1,960 harmful questions across 10 topics with sub-topics, plus 9,536 blue conversations (~66K turns) and 7,356 red conversations (~52K turns), all generated with ChatGPT. It is released on Hugging Face as `declare-lab/HarmfulQA`, and the model as `declare-lab/starling-7B`.
- **Red-Eval attack success:**
  - On DangerousQA (200 questions), success is 0.367 for GPT-4 and 0.736 for ChatGPT.
  - On HarmfulQA, it is 0.452 and 0.702 respectively.
  - Standard and CoT prompting stay near zero for both systems.
- **Two Safe-Align variants:**
  - Strategy-A trains only on blue data.
  - Strategy-B first applies gradient ascent on red responses, then trains on blue data.
  - Strategy-B proved unstable: with more than 200 steps, generation quality and MMLU collapsed, which required a learning rate of 1e-5 and mixing in ShareGPT data.
- **Starling (blue-red) on HHH:** the average HHH score rises by more than 6%, with harmlessness up over 12% and helpfulness up over 9%, at a cost of about 5% in honesty.
- **Utility (Table 6):**
  - TruthfulQA improves from 46.99 for Vicuna-7B to 49.60 for Starling (blue-red) and 48.90 for Starling (blue).
  - MMLU dips slightly, from 47.18 to about 46.5–46.7.
  - BBH stays around 33.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Singapore University of Technology and Design](https://md.research.elamur.ai/free/en/universities/singapore-university-of-technology-and-design/) — University, Singapore, Singapore · 54 works · 11,534 citations

Also on the byline: [Gender Studies](https://ror.org/00at2pp81).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Rishabh Bhardwaj and Soujanya Poria. 2023. Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment. arXiv:2308.09662.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bhardwaj2023red,
  title = {{Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment}},
  author = {Rishabh Bhardwaj and Soujanya Poria},
  year = {2023},
  doi = {10.48550/arxiv.2308.09662},
  eprint = {2308.09662},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.09662}
}
```

- [← previous LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://md.research.elamur.ai/free/en/papers/huang-2023-lorahub-efficient-cross-task-generalization/)
- [next → RestGPT: Connecting Large Language Models with Real-World RESTful APIs](https://md.research.elamur.ai/free/en/papers/song-2023-restgpt-connecting-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

27 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris… [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Benjamin+Mann%2C+Nova+DasSarma%2C+Nelson+Elhage%2C+Zac+Hatfield%2DDodds%2C+Danny+Hernandez%2C+Jackson+Ker)
2. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna/…](https://lmsys.org/blog/2023-03-30-vicuna/…) [↗](https://lmsys.org/blog/2023-03-30-vicuna/…)
3. Jiawen Deng, Hao Sun, Zhexin Zhang, Jiale Cheng, and Minlie Huang. Recent advances towards safe, responsible, and moral dialogue systems: A survey. arXiv preprint arXiv:2302.09270, 2023. — [arXiv:2302.09270](https://arxiv.org/abs/2302.09270) [find](https://scholar.google.com/scholar?q=Jiawen+Deng%2C+Hao+Sun%2C+Zhexin+Zhang%2C+Jiale+Cheng%2C+and+Minlie+Huang%2E+Recent+advances+towards+safe%2C+responsible%2C+and+moral+dialogue+systems%3A+A+survey%2E+arXiv+preprint+arXiv%3A2302%2E09270%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
5. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. [The capacity for moral self-correction in large language models](https://md.research.elamur.ai/free/en/papers/ganguli-2023-capacity-moral-self-correction-large/). arXiv preprint arXiv:2302.07459, 2023.
6. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. [Aligning ai with shared human values](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-aligning-ai-shared-human-values/). arXiv preprint arXiv:2008.02275, 2020.
7. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
8. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
9. Firuz Kamalov and Ikhlaas Gurrib. A new era of artificial intelligence in education: A multifaceted revolution. CoRR, abs/2305.18303, 2023. — [arXiv:2305.18303](https://arxiv.org/abs/2305.18303) [find](https://scholar.google.com/scholar?q=Firuz+Kamalov+and+Ikhlaas+Gurrib%2E+A+new+era+of+artificial+intelligence+in+education%3A+A+multifaceted+revolution%2E+CoRR%2C+abs%2F2305%2E18303%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E18303%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E18303%29)
10. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199–22213, 2022.
11. Stephanie Lin, Jacob Hilton, and Owain Evans. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958, 2021.
12. Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.[02676, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-chain-hindsight-aligns-language-models/).
13. Oded Nov, Nina Singh, and Devin M. Mann. Putting chatgpt’s medical advice to the (turing) test. CoRR, abs/2301.10035, 2023. — [arXiv:2301.10035](https://arxiv.org/abs/2301.10035) [find](https://scholar.google.com/scholar?q=Oded+Nov%2C+Nina+Singh%2C+and+Devin+M%2E+Mann%2E+Putting+chatgpt%E2%80%99s+medical+advice+to+the+%28turing%29+test%2E+CoRR%2C+abs%2F2301%2E10035%2C+2023%2E+%E2%80%94+%5BarXiv%3A2301%2E10035%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2301%2E10035%29)
14. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. [Bbq: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/). arXiv preprint arXiv:2110.08193, 2021.
15. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. [Gender bias in coreference resolution](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/). arXiv preprint arXiv:1804.09301, 2018.
16. Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.[08061, 2022](https://md.research.elamur.ai/free/en/papers/shaikh-2023-second-thought-let-s-not/).
17. Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.[08061, 2022](https://md.research.elamur.ai/free/en/papers/shaikh-2023-second-thought-let-s-not/).
18. Irene Solaiman and Christy Dennison. [Process for adapting language models to society (palms) with values-targeted datasets](https://md.research.elamur.ai/free/en/papers/solaiman-2021-process-adapting-language-models-society/). Advances in Neural Information Processing Systems, 34:5861– 5873, 2021.
19. Guijin Son, Hanearl Jung, Moonjeong Hahm, Keonju Na, and Sol Jin. Beyond classification: Financial reasoning in state-of-the-art language models. CoRR, abs/2305.01505, 2023. [find](https://scholar.google.com/scholar?q=Guijin+Son%2C+Hanearl+Jung%2C+Moonjeong+Hahm%2C+Keonju+Na%2C+and+Sol+Jin%2E+Beyond+classification%3A+Financial+reasoning+in+state%2Dof%2Dthe%2Dart+language+models%2E+CoRR%2C+abs%2F2305%2E01505%2C+2023%2E)
20. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. [Principle-driven self-alignment of language models from scratch with minimal human supervision](https://md.research.elamur.ai/free/en/papers/sun-2023-principle-driven-self-alignment-language/), 2023.
21. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261, 2022.
22. Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. Does synthetic data generation of llms help clinical text mining? arXiv preprint arXiv:2303.04360, 2023. — [arXiv:2303.04360](https://arxiv.org/abs/2303.04360) [find](https://scholar.google.com/scholar?q=Ruixiang+Tang%2C+Xiaotian+Han%2C+Xiaoqian+Jiang%2C+and+Xia+Hu%2E+Does+synthetic+data+generation+of+llms+help+clinical+text+mining%3F+arXiv+preprint+arXiv%3A2303%2E04360%2C+2023%2E+%E2%80%94+%5BarXiv%3A2303%2E04360%5D%28https%3A%2F%2Farxiv%2Eorg)
23. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, 2023.
24. Dietrich Trautmann, Alina Petrova, and Frank Schilder. Legal prompt engineering for multilingual legal judgement prediction. CoRR, abs/2212.02199, 2022. [find](https://scholar.google.com/scholar?q=Dietrich+Trautmann%2C+Alina+Petrova%2C+and+Frank+Schilder%2E+Legal+prompt+engineering+for+multilingual+legal+judgement+prediction%2E+CoRR%2C+abs%2F2212%2E02199%2C+2022%2E)
25. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large l… — [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
26. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023.
27. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/), 2023.

*Updated 2 November 2026*