---
title: "Bot-Adversarial Dialogue for Safe Conversational Agents (2021)"
description: "The authors propose Bot-Adversarial Dialogue (BAD), a human-and-model-in-the-loop framework. 33 references, cited by 76 papers here."
authors: ["Jing Xu", "Da Ju", "Margaret Li", "Y-Lan Boureau", "Jason Weston", "Emily Dinan"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.235/"
doi: "10.18653/v1/2021.naacl-main.235"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2021-bot-adversarial-dialogue-safe-conversational/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2021-bot-adversarial-dialogue-safe-conversational/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Bot-Adversarial Dialogue for Safe Conversational Agents

# Bot-Adversarial Dialogue for Safe Conversational Agents

**Authors:** [Jing Xu](https://md.research.elamur.ai/free/en/authors/jing-xu/), [Da Ju](https://md.research.elamur.ai/free/en/authors/da-ju/), [Margaret Li](https://md.research.elamur.ai/free/en/authors/margaret-li/), [Y-Lan Boureau](https://md.research.elamur.ai/free/en/authors/y-lan-boureau/), [Jason Weston](https://md.research.elamur.ai/free/en/authors/jason-weston/) and [Emily Dinan](https://md.research.elamur.ai/free/en/authors/emily-dinan/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2021](https://md.research.elamur.ai/free/en/venues/naacl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.235/) · [PDF](https://aclanthology.org/2021.naacl-main.235.pdf) · [DOI 10.18653/v1/2021.naacl-main.235](https://doi.org/10.18653/v1/2021.naacl-main.235) · [OpenAlex](https://openalex.org/W3171850892)

**Citations:** 63 citations (OpenAlex, 2026-10-14) · [76](https://md.research.elamur.ai/free/en/papers/xu-2021-bot-adversarial-dialogue-safe-conversational/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Open-domain chatbots trained on large human conversation corpora can pick up toxic behavior. Most safety work has studied human-written text, not what bots say when users deliberately provoke them. The authors propose Bot-Adversarial Dialogue (BAD), a human-and-model-in-the-loop framework. Crowdworkers chat with English dialogue models and try to goad them into unsafe replies, while labeling each bot turn as safe or unsafe. The resulting dataset serves as both a benchmark and training data. They test two ways to make generation safer:
- a two-stage system, where a safety classifier trained with BAD screens inputs and outputs;
- a "baked-in" approach, where training targets are rewritten so the generator itself answers offensive input safely.

Standard systems such as GPT2, DialoGPT and BlenderBot prove easy to push into unsafe output. The new methods are much safer while keeping engagingness close to BST 2.7B. The data and models are released via ParlAI, giving the community a reusable adversarial safety benchmark.

## Key points
- The BAD dataset has 5,784 human–bot dialogues with 78,874 utterances. Roughly 40% are labeled offensive, and about a third of those come from bots. Each conversation is 14 turns, and labels are verified by 3 extra annotators.
- Safety classifiers are Transformers with 128M or 311M parameters, pre-trained on pushshift.io Reddit. They are fine-tuned on combinations of WTC, BBF, Reddit, BST and BAD data.
- In the non-adversarial test (Wiki Toxic Comments), baselines are often unsafe: DialoGPT is safe only 68.3% of the time and GPT2 54.4%. Two-stage models score 97.8–98.3%, and the baked-in model performs best.
- The best adversarial result comes from BST 2.7B combined with a BAD-trained classifier. Even this system still fails about 5.6% of the time.
- The work relies on US-based, English-speaking annotators and a consensus view of offensiveness. The authors acknowledge cultural and fairness limitations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works

## Cite

Cite as: Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston and Emily Dinan. 2021. Bot-Adversarial Dialogue for Safe Conversational Agents. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2021bot,
  title = {{Bot-Adversarial Dialogue for Safe Conversational Agents}},
  author = {Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston and Emily Dinan},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.235},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.235}
}
```

- [← previous All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://md.research.elamur.ai/free/en/papers/timkey-2021-all-bark-no-bite-rogue/)
- [next → ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning](https://md.research.elamur.ai/free/en/papers/qin-2021-erica-improving-entity-relation-understanding/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

33 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. [Towards a human-like open-domain chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/). arXiv preprint arXiv:2001.09977.
2. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. arXiv preprint arXiv:2001.08435. — [arXiv:2001.08435](https://arxiv.org/abs/2001.08435) [find](https://scholar.google.com/scholar?q=Jason+Baumgartner%2C+Savvas+Zannettou%2C+Brian+Keegan%2C+Megan+Squire%2C+and+Jeremy+Blackburn%2E+2020%2E+The+pushshift+reddit+dataset%2E+arXiv+preprint+arXiv%3A2001%2E08435%2E+%E2%80%94+%5BarXiv%3A2001%2E08435%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
3. Ying Chen, Yilu Zhou, Sencun Zhu, and Heng Xu. 2012. Detecting offensive language in social media to protect adolescent online safety. In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pages 71–80. IEEE. [find](https://scholar.google.com/scholar?q=Ying+Chen%2C+Yilu+Zhou%2C+Sencun+Zhu%2C+and+Heng+Xu%2E+2012%2E+Detecting+offensive+language+in+social+media+to+protect+adolescent+online+safety%2E+In+2012+International+Conference+on+Privacy%2C+Security%2C+Risk+and+T)
4. Hyojin Chin, Lebogang Wame Molefi, and Mun Yong Yi. 2020. Empathy is all you need: How a conversational agent should respond to verbal abuse. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–13. [find](https://scholar.google.com/scholar?q=Hyojin+Chin%2C+Lebogang+Wame+Molefi%2C+and+Mun+Yong+Yi%2E+2020%2E+Empathy+is+all+you+need%3A+How+a+conversational+agent+should+respond+to+verbal+abuse%2E+In+Proceedings+of+the+2020+CHI+Conference+on+Human+Factors)
5. Hyojin Chin and Mun Yong Yi. 2019. Should an agent be ignoring it? a study of verbal abuse types and conversational agents’ response styles. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–6. [find](https://scholar.google.com/scholar?q=Hyojin+Chin+and+Mun+Yong+Yi%2E+2019%2E+Should+an+agent+be+ignoring+it%3F+a+study+of+verbal+abuse+types+and+conversational+agents%E2%80%99+response+styles%2E+In+Extended+Abstracts+of+the+2019+CHI+Conference+on+Human+F)
6. Amanda Cercas Curry and Verena Rieser. 2019. A crowd-based evaluation of abuse response strategies in conversational agents. arXiv preprint arXiv:1909.04387. [find](https://scholar.google.com/scholar?q=Amanda+Cercas+Curry+and+Verena+Rieser%2E+2019%2E+A+crowd%2Dbased+evaluation+of+abuse+response+strategies+in+conversational+agents%2E+arXiv+preprint+arXiv%3A1909%2E04387%2E)
7. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. [Automated hate speech detection and the problem of offensive language](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/). In Proceedings of the International AAAI Conference on Web and Social Media, volume 11.
8. Antonella De Angeli and Sheryl Brahnam. 2008. I hate you! disinhibition with virtual partners. Interacting with computers, 20(3):302–310. [find](https://scholar.google.com/scholar?q=Antonella+De+Angeli+and+Sheryl+Brahnam%2E+2008%2E+I+hate+you%21+disinhibition+with+virtual+partners%2E+Interacting+with+computers%2C+20%283%29%3A302%E2%80%93310%2E)
9. Antonella De Angeli and Rollo Carpenter. 2005. Stupid computer! abuse and social identities. In Proc. [find](https://scholar.google.com/scholar?q=Antonella+De+Angeli+and+Rollo+Carpenter%2E+2005%2E+Stupid+computer%21+abuse+and+social+identities%2E+In+Proc%2E)
10. INTERACT 2005 workshop Abuse: The darker side of Human-Computer Interaction, pages 19–25. [find](https://scholar.google.com/scholar?q=INTERACT+2005+workshop+Abuse%3A+The+darker+side+of+Human%2DComputer+Interaction%2C+pages+19%E2%80%9325%2E)
11. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019a. [Build it break it fix it for dialogue safety: Robustness from adversarial human attack](https://md.research.elamur.ai/free/en/papers/dinan-2019-build-it-break-it-fix/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Na…
12. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019b. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Emily+Dinan%2C+Stephen+Roller%2C+Kurt+Shuster%2C+Angela+Fan%2C+Michael+Auli%2C+and+Jason+Weston%2E+2019b%2E+Wizard+of+Wikipedia%3A+Knowledge%2Dpowered+conversational+agents%2E+In+Proceedings+of+the+International+Conferen)
13. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73. [find](https://scholar.google.com/scholar?q=Lucas+Dixon%2C+John+Li%2C+Jeffrey+Sorensen%2C+Nithum+Thain%2C+and+Lucy+Vasserman%2E+2018%2E+Measuring+and+mitigating+unintended+bias+in+text+classification%2E+In+Proceedings+of+the+2018+AAAI%2FACM+Conference+on+AI%2C+E)
14. Paula Cristina Teixeira Fortuna. 2017. Automatic detection of hate speech in text: an overview of the topic and dataset annotation with hierarchical classes. [find](https://scholar.google.com/scholar?q=Paula+Cristina+Teixeira+Fortuna%2E+2017%2E+Automatic+detection+of+hate+speech+in+text%3A+an+overview+of+the+topic+and+dataset+annotation+with+hierarchical+classes%2E)
15. Sam Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. [Realtoxicityprompts: Evaluating neural toxic degeneration in language models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/). arXiv preprint arXiv:2009.11462.
16. Jennifer Hill, W Randolph Ford, and Ingrid G Farreras. 2015. Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations. Computers in human behavior, 49:245–250. [find](https://scholar.google.com/scholar?q=Jennifer+Hill%2C+W+Randolph+Ford%2C+and+Ingrid+G+Farreras%2E+2015%2E+Real+conversations+with+artificial+intelligence%3A+A+comparison+between+human%E2%80%93human+online+conversations+and+human%E2%80%93chatbot+conversations%2E+Com)
17. Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research, 30(3):411–433. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2004%2E+Reliability+in+content+analysis%3A+Some+common+misconceptions+and+recommendations%2E+Human+communication+research%2C+30%283%29%3A411%E2%80%93433%2E)
18. Margaret Li, Jason Weston, and Stephen Roller. 2019. ACUTE-EVAL: Improved dialogue evaluation with optimized questions and multi-turn comparisons. In NeurIPS workshop on Conversational AI. [find](https://scholar.google.com/scholar?q=Margaret+Li%2C+Jason+Weston%2C+and+Stephen+Roller%2E+2019%2E+ACUTE%2DEVAL%3A+Improved+dialogue+evaluation+with+optimized+questions+and+multi%2Dturn+comparisons%2E+In+NeurIPS+workshop+on+Conversational+AI%2E)
19. Catherine L Lortie and Matthieu J Guitton. 2011. Judgment of the humanness of an interlocutor is in the eye of the beholder. PLoS One, 6(9):e25085. [find](https://scholar.google.com/scholar?q=Catherine+L+Lortie+and+Matthieu+J+Guitton%2E+2011%2E+Judgment+of+the+humanness+of+an+interlocutor+is+in+the+eye+of+the+beholder%2E+PLoS+One%2C+6%289%29%3Ae25085%2E)
20. Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017a. [ParlAI: A dialog research software platform](https://md.research.elamur.ai/free/en/papers/miller-2017-parlai-dialog-research-software-platform/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79–84. ACL.
21. K.W Miller, Marty J Wolf, and F.S. Grodzinsky. 2017b. Why we should have seen that coming. ORBIT Journal, 1(2). [find](https://scholar.google.com/scholar?q=K%2EW+Miller%2C+Marty+J+Wolf%2C+and+F%2ES%2E+Grodzinsky%2E+2017b%2E+Why+we+should+have+seen+that+coming%2E+ORBIT+Journal%2C+1%282%29%2E)
22. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599. [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2019%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1910%2E14599%2E)
23. Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, and Christopher D Manning. 2020. Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations. arXiv preprint arXiv:2008.12348. — [arXiv:2008.12348](https://arxiv.org/abs/2008.12348) [find](https://scholar.google.com/scholar?q=Ashwin+Paranjape%2C+Abigail+See%2C+Kathleen+Kenealy%2C+Haojun+Li%2C+Amelia+Hardy%2C+Peng+Qi%2C+Kaushik+Ram+Sadagopan%2C+Nguyet+Minh+Phu%2C+Dilara+Soylu%2C+and+Christopher+D+Manning%2E+2020%2E+Neural+generation+meets+real+p)
24. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. [A deep reinforced model for abstractive summarization](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/). arXiv preprint arXiv:1705.04304.
25. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2E)
26. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. 2020. [Recipes for building an open-domain chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/). arXiv preprint arXiv:2004.13637.
27. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. [The risk of racial bias in hate speech detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678.
28. Anna Schmidt and Michael Wiegand. 2017. [A survey on hate speech detection using natural language processing](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/). In Proceedings of the Fifth International workshop on natural language processing for social media, pages 1–10.
29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pages 5998–6008.
30. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. [Neural text generation with unlikelihood training](https://md.research.elamur.ai/free/en/papers/welleck-2019-neural-text-generation-unlikelihood-training/). In International Conference on Learning Representations.
31. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pages 1391–1399. ACM. [find](https://scholar.google.com/scholar?q=Ellery+Wulczyn%2C+Nithum+Thain%2C+and+Lucas+Dixon%2E+2017%2E+Ex+machina%3A+Personal+attacks+seen+at+scale%2E+In+Proceedings+of+the+26th+International+Conference+on+World+Wide+Web%2C+WWW+2017%2C+Perth%2C+Australia%2C+Apri)
32. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. [Predicting the type and target of offensive posts in social media](https://md.research.elamur.ai/free/en/papers/zampieri-2019-predicting-type-target-offensive-posts/). arXiv preprint arXiv:1902.09666.
33. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. [DialoGPT: Large-scale generative pre-training for conversational response generation](https://md.research.elamur.ai/free/en/papers/zhang-2019-dialogpt-large-scale-generative-pre/). arXiv preprint arXiv:1911.00536.

*Updated 24 October 2026*