---
title: "Detoxifying Language Models Risks Marginalizing Minority Voices"
description: "Detoxification methods aim to stop language models from producing offensive text while keeping… 25 references, cited by 77 papers here."
authors: ["Albert Xu", "Eshaan Pathak", "Eric Wallace", "Suchin Gururangan", "Maarten Sap", "Dan Klein"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.190/"
doi: "10.18653/v1/2021.naacl-main.190"
arxiv: "2104.06390"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2021-detoxifying-language-models-risks-marginalizing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2021-detoxifying-language-models-risks-marginalizing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Detoxifying Language Models Risks Marginalizing Minority Voices

# Detoxifying Language Models Risks Marginalizing Minority Voices

**Authors:** Albert Xu, Eshaan Pathak, [Eric Wallace](https://md.research.elamur.ai/free/en/authors/eric-wallace/), [Suchin Gururangan](https://md.research.elamur.ai/free/en/authors/suchin-gururangan/), [Maarten Sap](https://md.research.elamur.ai/free/en/authors/maarten-sap/) and [Dan Klein](https://md.research.elamur.ai/free/en/authors/dan-klein/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2021](https://md.research.elamur.ai/free/en/venues/naacl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.190/) · [PDF](https://aclanthology.org/2021.naacl-main.190.pdf) · [DOI 10.18653/v1/2021.naacl-main.190](https://doi.org/10.18653/v1/2021.naacl-main.190) · [arXiv 2104.06390](https://arxiv.org/abs/2104.06390) · [OpenAlex](https://openalex.org/W3172314079)

**Citations:** 58 citations (OpenAlex, 2026-10-14) · [77](https://md.research.elamur.ai/free/en/papers/xu-2021-detoxifying-language-models-risks-marginalizing/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Detoxification methods aim to stop language models from producing offensive text while keeping them useful on benign input. This paper asks whether that protection is fair to everyone. The authors detoxified GPT-2 medium with four methods (DAPT, PPLM, GeDi and output Filtering), all trained on the Jigsaw Civil Comments dataset. They then measured the models in two ways. First, they computed perplexity on tweets written in White-Aligned English (WAE), African-American English (AAE) and tweets containing minority identity mentions (MIM). Second, crowdworkers judged the quality of generated continuations.

Detoxification does reduce toxicity, but it also makes the models worse on harmless text. The loss is much larger for AAE and MIM text, and the gap grows as detoxification gets stronger. The authors trace this to spurious correlations in toxicity data: AAE and identity terms are often labelled toxic because of annotator unfamiliarity and sampling bias. This matters because such systems may push minority users to code-switch or hide their identity. It also shows that controllable generation can quietly inherit dataset biases.

## Key points
- **Setup.** All four techniques use GPT-2 medium with top-k sampling (k = 50).
  - GeDi relies on a finetuned GPT-2 small as its guide model.
  - Filtering picks the least toxic of 10 samples.
  - Jigsaw examples with 10–50% toxic votes were dropped because annotators disagreed on them.
- **Evaluation data.**
  - Paired WAE/AAE tweets from Groenwold et al. (2020), split into toxic and nontoxic with the Perspective API.
  - 1,000 nontoxic MIM tweets collected through the Twitter API.
- **Perplexity results.**
  - With PPLM, perplexity on nontoxic WAE tweets rose about 2.1×, but on MIM tweets it rose about 4.3×.
  - Raising GeDi's ω pushed the AAE-to-WAE perplexity ratio from roughly 3 (no detoxification) to nearly 400.
- **Human evaluation.** Amazon Mechanical Turk workers rated 100 prompts (50 AAE, 50 WAE), each with a 30-token continuation and three raters per example.
  - Over 80% judged GeDi less topical than baseline GPT-2.
  - Every method except DAPT was rated less fluent.
  - On AAE prompts, quality dropped on every metric, most of all topicality.
- **Code release.** The code is public at github.com/albertkx/detoxifying-lms.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 127 works · 212,523 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 221 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap and Dan Klein. 2021. Detoxifying Language Models Risks Marginalizing Minority Voices. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2021detoxifying,
  title = {{Detoxifying Language Models Risks Marginalizing Minority Voices}},
  author = {Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap and Dan Klein},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.190},
  eprint = {2104.06390},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.190}
}
```

- [← previous ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning](https://md.research.elamur.ai/free/en/papers/qin-2021-erica-improving-entity-relation-understanding/)
- [next → Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs](https://md.research.elamur.ai/free/en/papers/chen-2021-structure-aware-abstractive-conversation-summarization/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. [Language (technology) is power: A critical survey of “bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/). In ACL.
2. Stigmatization of Language Detoxified models also have a propensity to avoid certain topics, e.g., Christopher Clark, Mark Yatskar, and Luke Zettle-mentioning a minority identity term. As a practi-moyer. 2019. Don’t take the easy way out: Ensem-cal example, the (detoxified) Microsoft Zo chatbot ble… [find](https://scholar.google.com/scholar?q=Stigmatization+of+Language+Detoxified+models+also+have+a+propensity+to+avoid+certain+topics%2C+e%2Eg%2E%2C+Christopher+Clark%2C+Mark+Yatskar%2C+and+Luke+Zettle%2Dmentioning+a+minority+identity+term%2E+As+a+practi%2Dmoy)
3. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. [RealToxicityPrompts: Evaluating neural toxic degeneration in language models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/). In EMNLP Findings.
4. Lisa Green. 2002. African American English: A linguistic introduction. [find](https://scholar.google.com/scholar?q=Lisa+Green%2E+2002%2E+African+American+English%3A+A+linguistic+introduction%2E)
5. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. [Investigating African-American vernacular English in transformer-based text generation](https://md.research.elamur.ai/free/en/papers/groenwold-2020-investigating-african-american-vernacular-english/). In EMNLP.
6. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In ACL. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+In+ACL%2E)
7. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In AIES. [find](https://scholar.google.com/scholar?q=Peter+Henderson%2C+Koustuv+Sinha%2C+Nicolas+Angelard%2DGontier%2C+Nan+Rosemary+Ke%2C+Genevieve+Fried%2C+Ryan+Lowe%2C+and+Joelle+Pineau%2E+2018%2E+Ethical+challenges+in+data%2Ddriven+dialogue+systems%2E+In+AIES%2E)
8. Chris J Kennedy, Geoff Bacon, Alexander Sahn, and Claudia von Vacano. 2020. Constructing interval variables via faceted Rasch measurement and multi-task deep learning: A hate speech application. arXiv preprint arXiv:2009.10277. — [arXiv:2009.10277](https://arxiv.org/abs/2009.10277) [find](https://scholar.google.com/scholar?q=Chris+J+Kennedy%2C+Geoff+Bacon%2C+Alexander+Sahn%2C+and+Claudia+von+Vacano%2E+2020%2E+Constructing+interval+variables+via+faceted+Rasch+measurement+and+multi%2Dtask+deep+learning%3A+A+hate+speech+application%2E+arXiv)
9. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. [CTRL: A conditional transformer language model for controllable generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/). arXiv preprint arXiv:1909.05858.
10. Svetlana Kiritchenko and Saif M Mohammad. 2018. [Examining gender and race bias in two hundred sentiment analysis systems](https://md.research.elamur.ai/free/en/papers/kiritchenko-2018-examining-gender-race-bias-two/). In *SEM.
11. Ben Krause, Akhilesh Deepak Gotmare, Bryan Mc-Cann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020. [GeDi: Generative discriminator guided sequence generation](https://md.research.elamur.ai/free/en/papers/krause-2021-gedi-generative-discriminator-guided-sequence/). arXiv preprint arXiv:2009.06367.
12. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. 2020. [Recipes for building an open-domain chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/). arXiv preprint arXiv:2004.13637.
13. RWJF. 2017. Discrimination in America: Experiences and views. [find](https://scholar.google.com/scholar?q=RWJF%2E+2017%2E+Discrimination+in+America%3A+Experiences+and+views%2E)
14. Maarten Sap, Dallas Card, Saadia Gabriel, Choi Yejin, and Noah Smith. 2019. [The risk of racial bias in hate speech detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/). In ACL.
15. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020. [Social bias frames: Reasoning about social and power implications of language](https://md.research.elamur.ai/free/en/papers/sap-2020-social-bias-frames-reasoning-about/). In ACL.
16. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2020. [Towards controllable biases in language generation](https://md.research.elamur.ai/free/en/papers/sheng-2020-controllable-biases-language-generation/). In EMNLP Findings.
17. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Auto-Prompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP. [find](https://scholar.google.com/scholar?q=Taylor+Shin%2C+Yasaman+Razeghi%2C+Robert+L+Logan+IV%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+2020%2E+Auto%2DPrompt%3A+Eliciting+knowledge+from+language+models+with+automatically+generated+prompts%2E+In+EMNLP%2E)
18. Chloe Rose Stuart-Ulin. 2018. Microsoft’s politically correct chatbot is even worse than its racist one. Quartz. [find](https://scholar.google.com/scholar?q=Chloe+Rose+Stuart%2DUlin%2E+2018%2E+Microsoft%E2%80%99s+politically+correct+chatbot+is+even+worse+than+its+racist+one%2E+Quartz%2E)
19. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In EMNLP. [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Shi+Feng%2C+Nikhil+Kandpal%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2019%2E+Universal+adversarial+triggers+for+attacking+and+analyzing+NLP%2E+In+EMNLP%2E)
20. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079. [find](https://scholar.google.com/scholar?q=Jing+Xu%2C+Da+Ju%2C+Margaret+Li%2C+Y%2DLan+Boureau%2C+Jason+Weston%2C+and+Emily+Dinan%2E+2020%2E+Recipes+for+safety+in+open%2Ddomain+chatbots%2E+arXiv+preprint+arXiv%3A2010%2E07079%2E)
21. Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2021. [Challenges in automated debiasing for toxic language detection](https://md.research.elamur.ai/free/en/papers/zhou-2021-challenges-automated-debiasing-toxic-language/). In EACL.
22. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. [find](https://scholar.google.com/scholar?q=Michael+McCloskey+and+Neal+J+Cohen%2E+1989%2E+Catastrophic+interference+in+connectionist+networks%3A+The+sequential+learning+problem%2E+In+Psychology+of+learning+and+motivation%2E)
23. Thiago Dias Oliva, Dennys Marcelo Antonialli, and Alessandra Gomes. 2020. Fighting hate speech, silencing drag queens? Artificial intelligence in content moderation and risks to LGBTQ voices online. In Sexuality & Culture. [find](https://scholar.google.com/scholar?q=Thiago+Dias+Oliva%2C+Dennys+Marcelo+Antonialli%2C+and+Alessandra+Gomes%2E+2020%2E+Fighting+hate+speech%2C+silencing+drag+queens%3F+Artificial+intelligence+in+content+moderation+and+risks+to+LGBTQ+voices+online%2E+I)
24. Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. 2019. Distributionally robust language modeling. In EMNLP. [find](https://scholar.google.com/scholar?q=Yonatan+Oren%2C+Shiori+Sagawa%2C+Tatsunori+B+Hashimoto%2C+and+Percy+Liang%2E+2019%2E+Distributionally+robust+language+modeling%2E+In+EMNLP%2E)
25. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2E)

*Updated 24 October 2026*