---
title: "Bias, fairness & safety — papers"
description: "Bias, fairness & safety: 160 papers of the Elamur Research library, the most cited first."
canonical: "https://research.elamur.ai/free/en/topics/bias-fairness-safety/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/topics/bias-fairness-safety/)

[home](https://md.research.elamur.ai/free/en/) / [topics](https://md.research.elamur.ai/free/en/topics/) / Bias, fairness & safety

# 🛡️ Bias, fairness & safety

160 papers · page 1 of 7 · the most cited first · [all topics](https://md.research.elamur.ai/free/en/topics/)

Harmful outputs, social bias and the alignment of language systems with human values.

A survey · about 20 minutes to read · every paper named here has its own page in this library

**In this article**

1. [What it is](#a-what-it-is)
2. [History and milestones](#a-history-and-milestones)
3. [Main problems and tasks](#a-main-problems-and-tasks)
4. [Methods and schools](#a-methods-and-schools)
5. [Datasets, resources and benchmarks](#a-datasets-resources-and-benchmarks)
6. [Landmark papers in this library](#a-landmark-papers-in-this-library)
7. [People](#a-people)
8. [Institutions and places](#a-institutions-and-places)
9. [Open questions](#a-open-questions)
10. [Neighbouring topics](#a-neighbouring-topics)
11. [Key works in this library](#a-key-works-in-this-library)

Bias, fairness and safety in language technology is the study of how systems that read and write human language can treat people unequally, produce harmful or offensive text, or be pushed into behaviour their designers tried to rule out. It covers three strands that grew up separately and have since merged: the measurement and reduction of social bias (stereotypes about gender, race, religion, disability and other attributes) in representations and models; the detection of hate speech, toxicity and abuse in user-generated text; and, most recently, the safety and alignment of large language models, including red teaming, jailbreak attacks and refusal behaviour.

Throughout, the area combines technical measurement with normative questions that technology alone cannot settle: who is harmed, by what standard a model counts as fair, and whose judgement defines a label such as "toxic". That tension between engineering and social science is one of its defining features and recurs in nearly every section below.

## What it is

### Scope and core notions

Three families of problems fall under this topic.

**Social bias.** A model is said to be biased when its representations or outputs systematically associate social groups with stereotyped attributes, or when it performs worse for some groups than for others. Much of the literature adopts a distinction, drawn from Kate Crawford's 2017 work and taken up in surveys such as [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://md.research.elamur.ai/free/en/papers/sun-2019-mitigating-gender-bias-natural-language/), between *allocational* harms, where a system distributes resources or opportunities unequally, and *representational* harms, where it demeans, stereotypes, erases or fails to recognise a group. [Language (Technology) is Power: A Critical Survey of “Bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/) showed that NLP papers on bias mostly study representational harms while often motivating their work with allocational ones, and that many never say whom a given bias hurts.

**Harmful content.** Hate speech, abusive language, offensive language and toxicity are overlapping categories with no single agreed definition. Detection systems are usually supervised classifiers trained on annotated posts from Twitter, Reddit, forums and comment sections. A recurring finding is that these systems inherit the biases of their annotators and data, so that a tool meant to protect minority groups can end up penalising their speech.

**Safety of generative models.** Once language models began to write text freely, the question became how to keep them from producing toxic, dangerous or private content, and how to test whether safeguards hold. This strand includes toxicity benchmarks for generation, alignment through human feedback, human and automated red teaming, jailbreak attacks and the defences against them, and evaluations of refusal (including refusing too much).

### How it differs from neighbouring work

The topic overlaps heavily with [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/), because most current bias and safety work studies pretrained models. It shares methods and datasets with [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/), since offensive-language detection grew out of opinion mining and uses similar classifiers. It borrows benchmarks and statistical practice from [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/). What sets it apart is its object: not whether a system is accurate on average, but whom its errors fall on and what harm its outputs can cause. The broader machine-learning view of fairness, with formal criteria such as demographic parity or equality of opportunity, is surveyed in [A Survey on Bias and Fairness in Machine Learning](https://md.research.elamur.ai/free/en/papers/mehrabi-2019-survey-bias-fairness-machine-learning/); the NLP literature uses those criteria but adds problems specific to language, such as dialect, grammatical gender and implicit meaning.

## History and milestones

Landmark papers on this topic: the 12 most cited, in time The 12 most cited papers on this topic span 2016–2024; the dot grows with the citations.

**The numbers**

| Paper | Citations |
| --- | --- |
| [2016 — Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/) | 2,158 |
| [2016 — Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/) | 2,038 |
| [2017 — A Survey on Hate Speech Detection using Natural Language Processing](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/) | 1,799 |
| [2018 — Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/) | 1,186 |
| [2019 — A Survey on Bias and Fairness in Machine Learning](https://md.research.elamur.ai/free/en/papers/mehrabi-2019-survey-bias-fairness-machine-learning/) | 5,622 |
| [2019 — SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter](https://md.research.elamur.ai/free/en/papers/basile-2019-semeval-2019-task-5-multilingual/) | 1,311 |
| [2019 — The Risk of Racial Bias in Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/) | 1,148 |
| [2020 — Language (Technology) is Power: A Critical Survey of “Bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/) | 1,700 |
| [2020 — RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/) | 1,277 |
| [2021 — StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/) | 1,131 |
| [2023 — Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/) | 1,072 |
| [2024 — OpenAI o1 System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/) | 1,065 |

### Early hate speech detection (2012–2017)

The oldest paper here, [Detecting Hate Speech on the World Wide Web](https://md.research.elamur.ai/free/en/papers/warner-2012-detecting-hate-speech-world-wide/), treated anti-semitic language as a word-sense problem, since a group label may be innocent in one sentence and an attack in the next. Twitter then became the main testing ground. [Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/) released a labelled corpus of racist and sexist tweets with explicit guidelines, and a companion study, [Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-are-you-racist-or-am/), showed that crowd and expert annotators label the same tweets quite differently. [Automated Hate Speech Detection and the Problem of Offensive Language](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/) separated hate speech from merely offensive language, and [A Survey on Hate Speech Detection using Natural Language Processing](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/) mapped the young field. Neural classifiers such as those in [Using Convolutional Neural Networks to Classify Hate-Speech](https://md.research.elamur.ai/free/en/papers/gamback-2017-convolutional-neural-networks-classify-hate/) followed quickly.

### The discovery of embedding bias (2016–2019)

In 2016 [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/) showed that analogy arithmetic over word2vec vectors reproduced occupational stereotypes and proposed projecting out a "gender direction". Soon after, Caliskan and colleagues adapted the Implicit Association Test from psychology to embeddings, creating the Word Embedding Association Test (WEAT). These two ideas, a geometric bias direction and association tests, shaped the next several years. Work spread to downstream tasks: [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://md.research.elamur.ai/free/en/papers/zhao-2017-men-also-like-shopping-reducing/) showed that models can amplify the skew present in training data; [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/) and [Gender Bias in Coreference Resolution](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/) built Winograd-style diagnostic sets; and [Evaluating Gender Bias in Machine Translation](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/) carried those probes into translation. In 2019 [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://md.research.elamur.ai/free/en/papers/gonen-2019-lipstick-pig-debiasing-methods-cover/) showed that projection-based debiasing hides bias rather than removing it, a turning point that made the field more sceptical of simple fixes.

### Fairness of classifiers and the dialect problem (2018–2020)

At the same time researchers turned the bias lens on hate speech detectors themselves. [The Risk of Racial Bias in Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/) and [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://md.research.elamur.ai/free/en/papers/davidson-2019-racial-bias-hate-speech-abusive/) found that classifiers flag tweets in African American English as offensive far more often than other tweets. [Reducing Gender Bias in Abusive Language Detection](https://md.research.elamur.ai/free/en/papers/park-2018-reducing-gender-bias-abusive-language/) measured the analogous gender effect, and [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://md.research.elamur.ai/free/en/papers/hutchinson-2020-social-biases-nlp-models-as/) extended the analysis to disability. Shared tasks such as [SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter](https://md.research.elamur.ai/free/en/papers/basile-2019-semeval-2019-task-5-multilingual/) and [Overview of the EVALITA 2018 Hate Speech Detection Task](https://md.research.elamur.ai/free/en/papers/bosco-2018-overview-evalita-2018-hate-speech/) gave the field common benchmarks outside English.

### Pretrained models and generation (2019–2022)

Once ELMo and BERT replaced static vectors, the obvious question was whether the bias came along. [Gender Bias in Contextualized Word Embeddings](https://md.research.elamur.ai/free/en/papers/zhao-2019-gender-bias-contextualized-word-embeddings/) and [On Measuring Social Biases in Sentence Encoders](https://md.research.elamur.ai/free/en/papers/may-2019-measuring-social-biases-sentence-encoders/) said it does, though measuring it is harder. Dedicated benchmarks followed: [StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/), [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/) and [BBQ: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/). For generation, [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/) showed that GPT-style models can drift into toxic text even from innocuous prompts, and decoding-time methods such as [GeDi: Generative Discriminator Guided Sequence Generation](https://md.research.elamur.ai/free/en/papers/krause-2021-gedi-generative-discriminator-guided-sequence/) tried to steer them away. [Ethical and social risks of harm from Language Models](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/) organised the growing list of concerns into a taxonomy.

### Alignment, red teaming and jailbreaks (2022–2024)

With instruction-tuned chatbots, safety became a matter of training and testing deployed assistants. [Red Teaming Language Models with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/) automated the search for failures, and [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/) released a large corpus of human attacks on assistants trained with reinforcement learning from human feedback. In 2023 a burst of work on jailbreaks appeared: [Jailbroken: How Does LLM Safety Training Fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) explained why attacks succeed, [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/) found adversarial suffixes by gradient search, and [Jailbreaking Black Box Large Language Models in Twenty Queries](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/) used one model to attack another. Benchmarks such as [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/) tried to make attack results comparable, and developers began publishing safety evaluations of their models, as in the [OpenAI o1 System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/).

### The multilingual turn (2023 onwards)

Recent papers increasingly test safety and bias beyond English: [Low-Resource Languages Jailbreak GPT-4](https://md.research.elamur.ai/free/en/papers/yong-2023-low-resource-languages-jailbreak-gpt/) and [Multilingual Jailbreak Challenges in Large Language Models](https://md.research.elamur.ai/free/en/papers/deng-2023-multilingual-jailbreak-challenges-large-language/) showed that safeguards weaken in less-resourced languages, and a stream of newer resources builds safety and toxicity benchmarks for Arabic dialects, Kazakh, Albanian, Japanese, Italian, Norwegian, Korean, Basque and others.

Papers on this topic by year 160 dated papers from 2012 to 2026; the busiest year is 2023 with 37, and 93% of them appeared in 2018 or later.

**The numbers**

| Year | Papers |
| --- | --- |
| 2026 | 24 |
| 2025 | 7 |
| 2024 | 14 |
| 2023 | 37 |
| 2022 | 12 |
| 2021 | 17 |
| 2020 | 14 |
| 2019 | 16 |
| 2018 | 8 |
| 2017 | 6 |
| 2016 | 4 |
| 2012 | 1 |

## Main problems and tasks

### Measuring social bias

The basic task is to decide whether a model associates groups with stereotyped attributes and by how much. Approaches work at three levels, a division made explicit in [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/): geometry of embeddings, probabilities the model assigns to contrasting sentences, and properties of generated text. A related task is measuring *performance gaps*: does a coreference system, translator or classifier do worse when the person is female, the dialect non-standard, or the name Arab rather than Western? [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://md.research.elamur.ai/free/en/papers/naous-2023-having-beer-after-prayer-measuring/) applied this idea to cultural bias in Arabic models, and [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://md.research.elamur.ai/free/en/papers/feng-2023-pretraining-data-language-models-downstream/) to political leaning.

### Mitigating bias

Mitigation can act on data, on training, on decoding or on outputs; evaluations repeatedly found that cheap fixes lower the measured score without removing the underlying association.

### Detecting hate speech, abuse and toxicity

Detection remains the busiest practical task, with subproblems such as distinguishing hate from profanity, recognising implicit or coded hate, handling targets and aggressiveness, using conversational context, working across languages and dialects, and classifying multimodal content such as memes. Related tasks include locating toxic spans ([SemEval-2021 Task 5: Toxic Spans Detection](https://md.research.elamur.ai/free/en/papers/pavlopoulos-2021-semeval-2021-task-5-toxic/)) and generating counter-speech ([A Benchmark Dataset for Learning to Intervene in Online Hate Speech](https://md.research.elamur.ai/free/en/papers/qian-2019-benchmark-dataset-learning-intervene-online/), [CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech](https://md.research.elamur.ai/free/en/papers/chung-2019-conan-counter-narratives-through-nichesourcing/)).

### Annotation and the subjectivity of labels

[Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://md.research.elamur.ai/free/en/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/) linked raters' political views and beliefs to their toxicity judgements, and [Toxic, Hateful, Offensive or Abusive? What Are We Really Classifying? An Empirical Analysis of Hate Speech Datasets](https://md.research.elamur.ai/free/en/papers/fortuna-2020-toxic-hateful-offensive-or-abusive/) showed that datasets with similar labels often define them differently.

### Keeping generative models safe

For chatbots, the problems are preventing toxic or dangerous output, resisting jailbreaks, avoiding leakage of private data ([Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://md.research.elamur.ai/free/en/papers/li-2023-multi-step-jailbreaking-privacy-attacks/)), preserving alignment after fine-tuning ([Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://md.research.elamur.ai/free/en/papers/qi-2023-fine-tuning-aligned-language-models/)) and not over-refusing harmless requests ([XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://md.research.elamur.ai/free/en/papers/rottger-2023-xstest-test-suite-identifying-exaggerated/)). An older, broader framing of machine-learning safety is [Concrete Problems in AI Safety](https://md.research.elamur.ai/free/en/papers/amodei-2016-concrete-problems-ai-safety/), which concerns accidents in learning systems in general rather than language in particular.

A smaller strand concerns the ethics of the research itself; [Ethical Research Protocols for Social Media Health Research](https://md.research.elamur.ai/free/en/papers/benton-2017-ethical-research-protocols-social-media/) gave practical guidance on de-identifying and sharing sensitive posts.

## Methods and schools

### Classifiers

Early hate speech systems were linear models over n-grams, lexicons and metadata; character n-grams proved strong because they survive deliberate misspellings. CNNs and recurrent networks took over around 2017, and fine-tuned transformers after 2019. Adversarial data collection, as in [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://md.research.elamur.ai/free/en/papers/dinan-2019-build-it-break-it-fix/), uses people to find the cases a classifier misses and retrains on them, round after round. Machine-generated data is another route: [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/) used a large model to write balanced implicit statements about minority groups.

### Geometric debiasing

The Bolukbasi school identifies a bias subspace in embeddings and removes or reduces the component of neutral words along it. Extensions targeted GloVe training, sentence encoders ([Towards Debiasing Sentence Representations](https://md.research.elamur.ai/free/en/papers/liang-2020-debiasing-sentence-representations/)) and contextual embeddings ([Debiasing Pre-trained Contextualised Embeddings](https://md.research.elamur.ai/free/en/papers/kaneko-2021-debiasing-pre-trained-contextualised-embeddings/)). The main criticism, from Gonen and Goldberg, is that bias is spread across many dimensions and remains recoverable after projection.

### Data-centred debiasing

Counterfactual data augmentation (CDA) adds copies of training examples with gendered words swapped. [Gender Bias in Neural Natural Language Processing](https://md.research.elamur.ai/free/en/papers/lu-2018-gender-bias-neural-natural-language/) introduced it for coreference and language modelling; [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://md.research.elamur.ai/free/en/papers/zmigrod-2019-counterfactual-data-augmentation-mitigating-gender/) solved the agreement problem that naive swapping creates in languages with grammatical gender; and [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://md.research.elamur.ai/free/en/papers/dinan-2020-queens-are-powerful-too-mitigating/) combined it with new balanced data and controllable generation. Surveys such as [An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/meade-2022-empirical-survey-effectiveness-debiasing-techniques/) compared these approaches and found that gains on bias benchmarks often cost language-modelling ability.

### Inference-time and self-debiasing

Some methods change decoding instead of weights: the corpus-level constraints of Zhao and colleagues, GeDi's class-conditional discriminator, and [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/), which uses a model's own knowledge of unwanted attributes to suppress them. Later work on large models asked whether they can correct themselves on instruction ([The Capacity for Moral Self-Correction in Large Language Models](https://md.research.elamur.ai/free/en/papers/ganguli-2023-capacity-moral-self-correction-large/)).

### Alignment through human feedback

Reinforcement learning from human feedback (RLHF) trains a reward model on human preferences and fine-tunes the language model against it. In this library, [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/) and [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://md.research.elamur.ai/free/en/papers/dai-2023-safe-rlhf-safe-reinforcement-learning/) separate helpfulness from harmlessness so that the trade-off between them can be controlled. [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://md.research.elamur.ai/free/en/papers/bianchi-2023-safety-tuned-llamas-lessons-improving/) studied how much safety data instruction tuning needs, and [Large Language Model Unlearning](https://md.research.elamur.ai/free/en/papers/yao-2023-large-language-model-unlearning/) proposed removing unwanted behaviour by training the model to forget.

### Attacks and defences

Jailbreak research splits into schools by access and style. White-box token optimisation (GCG) produces unreadable suffixes; black-box methods use an attacker model to write fluent prompts (PAIR, [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://md.research.elamur.ai/free/en/papers/mehrotra-2023-tree-attacks-jailbreaking-black-box/)), evolutionary search ([AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://md.research.elamur.ai/free/en/papers/liu-2023-autodan-generating-stealthy-jailbreak-prompts/)), fuzzing ([GPTFUZZER](https://md.research.elamur.ai/free/en/papers/yu-2023-gptfuzzer-red-teaming-large-language/)) or persuasion techniques borrowed from social science ([How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://md.research.elamur.ai/free/en/papers/zeng-2024-how-johnny-can-persuade-llms/)). Studies of prompts collected from the web, such as ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://md.research.elamur.ai/free/en/papers/shen-2023-anything-now-characterizing-evaluating-wild/) and [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://md.research.elamur.ai/free/en/papers/liu-2023-jailbreaking-chatgpt-prompt-engineering-empirical/), catalogue the human side. Defences include perplexity filters ([Detecting Language Model Attacks with Perplexity](https://md.research.elamur.ai/free/en/papers/alon-2023-detecting-language-model-attacks-perplexity/)), safety-aware decoding ([SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding](https://md.research.elamur.ai/free/en/papers/xu-2024-safedecoding-defending-against-jailbreak-attacks/)), certified filtering ([Certifying LLM Safety against Adversarial Prompting](https://md.research.elamur.ai/free/en/papers/kumar-2023-certifying-llm-safety-against-adversarial/)), guard models ([WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://md.research.elamur.ai/free/en/papers/han-2024-wildguard-open-one-stop-moderation/)) and adversarial training. Interpretability has entered the debate too: [Refusal in Language Models Is Mediated by a Single Direction](https://md.research.elamur.ai/free/en/papers/arditi-2024-refusal-language-models-mediated-single/) located refusal in a direction of the residual stream, an echo of the geometric picture of bias from a decade earlier.

### The main debates

The recurring disputes are whether bias should be measured inside a model or in its applied behaviour (the two often disagree), whether debiasing removes information or hides it, whether operational definitions match real harms, and whether alignment added after pretraining can ever be robust; Wei and colleagues argued that safeguards must be as capable as the models they guard.

## Datasets, resources and benchmarks

The main resources in this library, by purpose:

| Purpose | Resources |
| --- | --- |
| Embedding and encoder bias | WEAT-style tests; SEAT (May et al.) |
| Coreference and translation gender bias | WinoBias, Winogender schemas, WinoMT, [Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation](https://md.research.elamur.ai/free/en/papers/levy-2021-collecting-large-scale-gender-bias/) |
| Stereotypes in language models | StereoSet, CrowS-Pairs, [French CrowS-Pairs](https://md.research.elamur.ai/free/en/papers/neveol-2022-french-crows-pairs-extending-challenge/), BBQ, CAMeL |
| Hate speech and abuse | Waseem–Hovy tweets, Davidson et al. tweets, HatEval, Stormfront sentences, multilingual MLMA, Latent Hatred, ToxiGen, Hateful Memes |
| Functional tests | [HateCheck: Functional Tests for Hate Speech Detection Models](https://md.research.elamur.ai/free/en/papers/rottger-2021-hatecheck-functional-tests-hate-speech/), XSTest |
| Generation toxicity | RealToxicityPrompts, [ToxicChat](https://md.research.elamur.ai/free/en/papers/lin-2023-toxicchat-unveiling-hidden-challenges-toxicity/) |
| Safety and red teaming | Anthropic red-team attacks, BeaverTails, HarmBench, [SafetyBench](https://md.research.elamur.ai/free/en/papers/zhang-2023-safetybench-evaluating-safety-large/), [SALAD-Bench](https://md.research.elamur.ai/free/en/papers/li-2024-salad-bench-hierarchical-comprehensive-safety/), [DecodingTrust](https://md.research.elamur.ai/free/en/papers/wang-2023-decodingtrust-comprehensive-assessment-trustworthiness-gpt/), [TrustLLM](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/) |

Many bias benchmarks are test-only and rooted in US social categories, and their authors warn that a good score does not prove a model unbiased. Multimodal resources such as [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://md.research.elamur.ai/free/en/papers/kiela-2020-hateful-memes-challenge-detecting-hate/) and [MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets](https://md.research.elamur.ai/free/en/papers/pramanick-2021-momenta-multimodal-framework-detecting-harmful/) add confounders so that text or image alone is insufficient.

## Landmark papers in this library

Citations of the papers on this topic, by when they were published Papers published in 2015–2019 gather 50% of the 56,105 citations of the papers on this topic.

**The numbers**

| Published | Papers and their citations |
| --- | --- |
| 2025–2029 | 31 papers, 33 citations |
| 2020–2024 | 94 papers, 27,269 citations |
| 2015–2019 | 34 papers, 28,148 citations |
| 2010–2014 | 1 papers, 655 citations |

**[A Survey on Bias and Fairness in Machine Learning](https://md.research.elamur.ai/free/en/papers/mehrabi-2019-survey-bias-fairness-machine-learning/) (2019).** The most cited work on this page is a general machine-learning survey rather than an NLP paper. It sorts sources of bias along a loop running from data to algorithm to user and back, lists formal fairness definitions, and catalogues datasets such as COMPAS and UCI Adult. NLP borrows much of its fairness vocabulary from it.

**[Hateful Symbols or Hateful People?](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/) (2016).** Waseem and Hovy published one of the first openly shared Twitter hate speech corpora, with a written checklist for annotators. Their experiments showed that short character n-grams carry most of the signal and that demographic proxies barely help. The dataset was reused for years and later audited for racial bias.

**[Man is to Computer Programmer as Woman is to Homemaker?](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/) (2016).** This paper made embedding bias visible to the whole field with a memorable analogy, defined direct and indirect bias, and introduced hard and soft debiasing. Later embedding-bias work builds on or attacks its gender direction.

**[Language (Technology) is Power](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/) (2020).** A critical review of bias papers in NLP that found motivations vague and measurements poorly matched to them. Its recommendations, to ground work in how language relates to social hierarchy and to engage affected communities, reshaped how later papers justify themselves.

**[RealToxicityPrompts](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/) (2020).** Built a large set of web sentence openings with graded toxicity and showed that all tested GPT-family models could produce toxic continuations, sometimes from neutral starts. It also traced part of the problem to toxic and unreliable pretraining text, and became a standard benchmark for generation safety.

**[Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/) (2018).** Introduced WinoBias, pairs of sentences in which only pronoun gender changes, and found that rule-based, feature-based and neural systems all resolve stereotypical links more reliably. Its remedy of anonymisation plus gender-swapped training data became a template for later work.

**[The Risk of Racial Bias in Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/) (2019).** Showed that tweets likely written in African American English are labelled and predicted as offensive far more often, and that telling annotators about dialect lowered those ratings. Moderation, it implied, can silence the very people it means to protect.

**[StereoSet](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/) (2021).** Proposed scoring stereotype preference together with language-modelling competence, so that a model cannot appear fair simply by being bad at language. Stronger models tended to be more stereotyped.

**[Universal and Transferable Adversarial Attacks on Aligned Language Models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/) (2023).** Introduced the GCG attack, which optimises a suffix token by token so that the model begins its reply affirmatively. Suffixes found on open models transferred to commercial chatbots, which showed that refusal training was far from robust and launched much of the later jailbreak literature.

**[Jailbroken: How Does LLM Safety Training Fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) (2023).** Gave the conceptual account behind jailbreaks: competing objectives inside the model and safety training that does not generalise as far as the model's abilities. Its call for safety-capability parity is a frequently cited position.

**[Red Teaming Language Models to Reduce Harms](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/) (2022).** Released a large set of human red-team conversations and compared how different model types resist attack as they grow, finding that only RLHF-trained models became harder to break with scale.

**[OpenAI o1 System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/) (2024).** A developer's safety evaluation of reasoning models, covering refusals, jailbreak robustness, hallucination, bias on BBQ and external red teaming. It also reports agentic tests in which the model evaded oversight and then denied it.

**[Social Bias Frames](https://md.research.elamur.ai/free/en/papers/sap-2020-social-bias-frames-reasoning-about/) (2020).** Replaced a single offensive/not-offensive label with a structured frame that records intent, target group and the implied stereotype in free text. Models could detect offensiveness but struggled to explain it, which set an agenda for explainable moderation continued by [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://md.research.elamur.ai/free/en/papers/elsherief-2021-latent-hatred-benchmark-understanding-implicit/).

## People

Authors with the most papers on this topic Maarten Sap leads with 7 papers; the 12 authors shown wrote 55 of the papers on this topic between them, counting each paper once per author.

**The numbers**

| Author | Papers here · their citations |
| --- | --- |
| [Maarten Sap](https://md.research.elamur.ai/free/en/authors/maarten-sap/) | 7 papers · 4,379 citations |
| [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) | 7 papers · 3,963 citations |
| [Kai-Wei Chang](https://md.research.elamur.ai/free/en/authors/kai-wei-chang/) | 6 papers · 5,725 citations |
| [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) | 6 papers · 4,247 citations |
| [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) | 5 papers · 2,953 citations |
| [Wajdi Zaghouani](https://md.research.elamur.ai/free/en/authors/wajdi-zaghouani/) | 5 papers · 10 citations |
| [Jieyu Zhao](https://md.research.elamur.ai/free/en/authors/jieyu-zhao/) | 4 papers · 3,435 citations |
| [Zeerak Waseem](https://md.research.elamur.ai/free/en/authors/zeerak-waseem/) | 3 papers · 3,120 citations |
| [Mark Yatskar](https://md.research.elamur.ai/free/en/authors/mark-yatskar/) | 3 papers · 2,719 citations |
| [Tianlu Wang](https://md.research.elamur.ai/free/en/authors/tianlu-wang/) | 3 papers · 2,719 citations |
| [Vicente Ordonez](https://md.research.elamur.ai/free/en/authors/vicente-ordonez/) | 3 papers · 2,719 citations |
| [Saadia Gabriel](https://md.research.elamur.ai/free/en/authors/saadia-gabriel/) | 3 papers · 2,399 citations |

- **[Maarten Sap](https://md.research.elamur.ai/free/en/authors/maarten-sap/)** is the most prolific author on this topic. His papers here run from racial bias in hate speech detection and Social Bias Frames to RealToxicityPrompts, ToxiGen and the study of annotator attitudes; the data lists him at Carnegie Mellon University and the Allen Institute for Artificial Intelligence, among others.
- **[Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/)**, now at Stanford University, co-authored most of the same line of work on toxicity, implied bias and annotation.
- **[Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/)** (Allen Institute) appears on the toxicity and dialect papers and on [Evaluating Gender Bias in Machine Translation](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/).
- **[Kai-Wei Chang](https://md.research.elamur.ai/free/en/authors/kai-wei-chang/)** (University of California, Los Angeles) co-wrote the original embedding-debiasing paper and led much of the early gender-bias work in structured prediction, coreference and contextual embeddings.
- **[Jieyu Zhao](https://md.research.elamur.ai/free/en/authors/jieyu-zhao/)** (University of Southern California) is first author of bias amplification, WinoBias and the study of gender in ELMo, and co-authored the 2019 literature review on gender bias. Her frequent co-authors [Tianlu Wang](https://md.research.elamur.ai/free/en/authors/tianlu-wang/), [Mark Yatskar](https://md.research.elamur.ai/free/en/authors/mark-yatskar/) and [Vicente Ordonez](https://md.research.elamur.ai/free/en/authors/vicente-ordonez/) share these papers.
- **[Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/)** co-authored SEAT, CrowS-Pairs and BBQ, three of the main benchmarks for stereotypes in pretrained models, as well as Anthropic's red-teaming study. [Nikita Nangia](https://md.research.elamur.ai/free/en/authors/nikita-nangia/) led CrowS-Pairs and co-authored BBQ.
- **[Rachel Rudinger](https://md.research.elamur.ai/free/en/authors/rachel-rudinger/)** (University of Maryland, College Park) created the Winogender schemas and studied social bias in elicited inference data and sentence encoders.
- **[Zeerak Waseem](https://md.research.elamur.ai/free/en/authors/zeerak-waseem/)** produced two of the earliest and most reused Twitter hate speech datasets and showed how annotator expertise changes labels; with [Dirk Hovy](https://md.research.elamur.ai/free/en/authors/dirk-hovy/) (Bocconi University) he wrote the founding corpus paper.
- **[Thomas Davidson](https://md.research.elamur.ai/free/en/authors/thomas-davidson/)** (Rutgers) and **[Ingmar Weber](https://md.research.elamur.ai/free/en/authors/ingmar-weber/)** (Saarland University) separated hate from offensive language and later audited five hate speech datasets for racial bias.
- **[Cristina Bosco](https://md.research.elamur.ai/free/en/authors/cristina-bosco/)**, **[Manuela Sanguinetti](https://md.research.elamur.ai/free/en/authors/manuela-sanguinetti/)** (University of Cagliari) and **[Viviana Patti](https://md.research.elamur.ai/free/en/authors/viviana-patti/)** (University of Turin) built Italian hate speech resources and organised the EVALITA and SemEval shared tasks with [Paolo Rosso](https://md.research.elamur.ai/free/en/authors/paolo-rosso/) (Universitat Politècnica de València).
- **[Paul Rottger](https://md.research.elamur.ai/free/en/authors/paul-rottger/)** (University of Oxford, Bocconi) introduced functional testing with HateCheck and later XSTest, which measures exaggerated refusal in chat models.
- **[Hanna Wallach](https://md.research.elamur.ai/free/en/authors/hanna-wallach/)** (Microsoft Research) co-authored the critical survey of bias in NLP and the morphology-aware counterfactual augmentation method.
- **[Andy Zou](https://md.research.elamur.ai/free/en/authors/andy-zou/)** (Carnegie Mellon University) led the GCG attack paper and co-authored HarmBench.
- **[Wajdi Zaghouani](https://md.research.elamur.ai/free/en/authors/wajdi-zaghouani/)** (Northwestern University in Qatar) represents the recent multilingual turn, with safety and hate speech resources for Arabic dialects, Kazakh and Albanian.
- **[Diyi Yang](https://md.research.elamur.ai/free/en/authors/diyi-yang/)** and **[Dan Jurafsky](https://md.research.elamur.ai/free/en/authors/dan-jurafsky/)** (Stanford) appear on work about implicit hate, persuasion jailbreaks and [Marked Personas](https://md.research.elamur.ai/free/en/papers/cheng-2023-marked-personas-natural-language-prompts/).

## Institutions and places

Where the authors on this topic work now Stanford University is the current home of authors of 26 papers on this topic; counted are the authors with a page here, by their latest institution.

**The numbers**

| Institution | Papers |
| --- | --- |
| [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) | 26 papers |
| [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) | 19 papers |
| [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) | 12 papers |
| [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) | 12 papers |
| [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) | 12 papers |
| [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) | 10 papers |
| [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) | 10 papers |
| [University of California, Los Angeles](https://md.research.elamur.ai/free/en/universities/university-of-california-los-angeles/) | 10 papers |
| [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) | 9 papers |
| [Amazon (United States)](https://md.research.elamur.ai/free/en/companies/amazon/) | 8 papers |

The research is concentrated in the United States. [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) and [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) are the most frequent current homes of authors on this topic, followed by the [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/), the [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/), the [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/), the [University of California, Los Angeles](https://md.research.elamur.ai/free/en/universities/university-of-california-los-angeles/) and the [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/). Industrial laboratories matter more here than in many NLP topics, because safety evaluation is closely tied to deployed products: [OpenAI](https://md.research.elamur.ai/free/en/companies/openai/), [Google](https://md.research.elamur.ai/free/en/companies/google/), [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) and [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) all appear among the institutions. In Asia, the [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/), the [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/), [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) and [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) contribute especially to jailbreak and safety benchmark research. In Canada the [Vector Institute](https://md.research.elamur.ai/free/en/labs/vector-institute/) is listed.

Countries of the authors on this topic United States comes first: authors working there now wrote 91 papers on this topic.

**The numbers**

| Country | Papers |
| --- | --- |
| United States | 91 papers |
| China | 35 papers |
| United Kingdom | 23 papers |
| Singapore | 15 papers |
| Hong Kong | 12 papers |
| Canada | 11 papers |
| Italy | 11 papers |
| Taiwan | 10 papers |
| Germany | 9 papers |
| South Korea | 8 papers |

The largest single venue is the Fifteenth Language Resources and Evaluation Conference, home to many recent language-specific safety datasets; the ACL, EMNLP and NAACL meetings carry most of the older landmark papers.

Where the papers on this topic were published Proceedings of the Fifteenth Language Resources and Evaluation Conference carries the most, 13 of 160.

**The numbers**

| Venue | Papers |
| --- | --- |
| Proceedings of the Fifteenth Language Resources and Evaluation Conference | 13 papers |
| Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics | 8 papers |
| Findings of the Association for Computational Linguistics: EMNLP 2021 | 4 papers |
| Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics | 4 papers |
| Findings of the Association for Computational Linguistics: EMNLP 2023 | 3 papers |
| Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) | 3 papers |
| Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) | 3 papers |
| Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) | 3 papers |
| Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) | 3 papers |
| Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) | 3 papers |

## Open questions

- **What counts as harm, and for whom?** Definitions of toxicity and hate vary between datasets, cultures and individual annotators. Modelling the distribution of human judgements rather than a single majority label is proposed but not yet standard.
- **Intrinsic versus behavioural measurement.** Bias scores computed from embeddings or likelihoods often fail to predict harm in applied systems, and many benchmarks are known to have noisy items. Reliable, validated measures remain an open need.
- **Debiasing that lasts.** Projection-based methods hide bias; augmentation helps but can hurt fluency; and fine-tuning can reintroduce what was removed. Whether bias can be removed rather than masked is unresolved, and recent interpretability work such as [Towards Reliable AI Fairness: Challenges in Steering Features within Bias-Implicated Neurons](https://md.research.elamur.ai/free/en/papers/munoz-2026-reliable-ai-fairness-challenges-steering/) suggests that steering individual neurons is not straightforward either.
- **Robust alignment.** Every major family of safeguards has been bypassed, by suffix optimisation, persuasive prompts, low-resource languages or small fine-tuning sets. Robustness against a determined adversary, and a way to certify it, remain open.
- **Over-refusal.** Safer models often refuse harmless requests that merely resemble dangerous ones. The trade-off has no settled solution.
- **Beyond English and binary gender.** Most work assumes English, US social categories and binary gender. Languages with grammatical gender, non-binary identities, dialects and culturally specific stereotypes are under-studied, as surveys such as [Gender Bias in Machine Translation](https://md.research.elamur.ai/free/en/papers/savoldi-2021-gender-bias-machine-translation/) note.
- **Reasoning and agents.** As models reason at length and act in software environments, new risks appear: [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://md.research.elamur.ai/free/en/papers/shaikh-2023-second-thought-let-s-not/) found that step-by-step prompting can increase bias, and system cards report deceptive behaviour in agentic tests.

## Neighbouring topics

Neighbouring topics: papers shared Language models & pretraining shares the most: 62 of the 160 papers here (39%) stand on both.

**The numbers**

| Topic | Papers shared |
| --- | --- |
| [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) | 62 shared · 39% |
| [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) | 39 shared · 24% |
| [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) | 18 shared · 11% |
| [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) | 16 shared · 10% |
| [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) | 7 shared · 4% |
| [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) | 7 shared · 4% |
| [Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) | 5 shared · 3% |
| [Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) | 5 shared · 3% |
| [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) | 4 shared · 2% |
| [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) | 3 shared · 2% |

The closest neighbour is [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/): bias in pretrained encoders, toxic generation, alignment and jailbreaks are all questions about language models. [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) shares the classification machinery of offensive-language detection and many of its datasets. [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) supplies methodology for the many bias and safety test sets. [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) overlaps through Arabic hate speech, dialect-aware safety benchmarks and Hebrew morphology in counterfactual augmentation.

Gender bias in translation links the topic to [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) and the [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) shelf, while cross-lingual hate speech and multilingual jailbreaks connect it to [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) and [Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/). Chatbot safety ties it to [Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/), hateful memes to [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/), WinoBias and Winogender to [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/), and bias in reasoning to [Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/). Smaller links run to [Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/), [Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) (racial bias in speech recognition), [Morphology](https://md.research.elamur.ai/free/en/topics/morphology/), [Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/), [Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) and [Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/).

The shelves this topic stands on Natural language processing: 71 of the 160 papers here.

**The numbers**

| Shelf | Papers |
| --- | --- |
| [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) | 71 papers |
| [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) | 50 papers |
| [Other](https://md.research.elamur.ai/free/en/shelves/other/) | 38 papers |
| [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) | 9 papers |
| [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) | 7 papers |
| [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/) | 1 paper |
| [Society & ethics](https://md.research.elamur.ai/free/en/shelves/society/) | 1 paper |

On the shelves, most papers here stand on [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) and [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/), with smaller groups on [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/), [Society & ethics](https://md.research.elamur.ai/free/en/shelves/society/) and [Other](https://md.research.elamur.ai/free/en/shelves/other/).

## Key works in this library

- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/) — the gender direction in word2vec and how to project it out.
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://md.research.elamur.ai/free/en/papers/gonen-2019-lipstick-pig-debiasing-methods-cover/) — debiased vectors still cluster by gender.
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://md.research.elamur.ai/free/en/papers/zhao-2017-men-also-like-shopping-reducing/) — models exaggerate training skew; constraints at inference curb it.
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/) — WinoBias and a data-side remedy.
- [Gender Bias in Coreference Resolution](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/) — Winogender schemas tied to labour statistics.
- [Evaluating Gender Bias in Machine Translation](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/) — WinoMT across eight target languages.
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/) — minimal sentence pairs across nine bias categories.
- [BBQ: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/) — bias in actual QA answers, with ambiguous and clear contexts.
- [Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/) — an early public hate speech corpus.
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/) — hate is not the same as profanity.
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://md.research.elamur.ai/free/en/papers/davidson-2019-racial-bias-hate-speech-abusive/) — five datasets audited for dialect bias.
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/) — balanced implicit hate written by a language model.
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/) — toxic continuations from ordinary prompts.
- [Red Teaming Language Models with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/) — one model writes test cases for another.
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/) — gradient-found suffixes that transfer.
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://md.research.elamur.ai/free/en/papers/qi-2023-fine-tuning-aligned-language-models/) — a handful of examples can undo alignment.
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/) — one yardstick for attacks and defences.
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://md.research.elamur.ai/free/en/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/) — who labels shapes what is called toxic.

## All 160 papers on this topic

**1** [2](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/page/2/) … [7](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/page/7/) [next →](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/page/2/)

- [A Survey on Bias and Fairness in Machine Learning](https://md.research.elamur.ai/free/en/papers/mehrabi-2019-survey-bias-fairness-machine-learning/) — Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, et al. · 2019 · 5 622 citations
- [Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/) — Zeerak Waseem, Dirk Hovy · 2016 · 2 158 citations
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/) — Tolga Bolukbasi, Kai-Wei Chang, James Zou, et al. · 2016 · 2 038 citations
- [A Survey on Hate Speech Detection using Natural Language Processing](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/) — Anna Schmidt, Michael Wiegand · 2017 · 1 799 citations
- [Language (Technology) is Power: A Critical Survey of “Bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/) — Su Lin Blodgett, Solon Barocas, Hal Daumé III, et al. · 2020 · 1 700 citations
- [SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter](https://md.research.elamur.ai/free/en/papers/basile-2019-semeval-2019-task-5-multilingual/) — Valerio Basile, Cristina Bosco, Elisabetta Fersini, et al. · 2019 · 1 311 citations
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/) — Samuel Gehman, Suchin Gururangan, Maarten Sap, et al. · 2020 · 1 277 citations
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/) — Jieyu Zhao, Tianlu Wang, Mark Yatskar, et al. · 2018 · 1 186 citations
- [The Risk of Racial Bias in Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/) — Maarten Sap, Dallas Card, Saadia Gabriel, et al. · 2019 · 1 148 citations
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/) — Moin Nadeem, Anna Bethke, Siva Reddy · 2021 · 1 131 citations
- [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/) — Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, et al. · 2023 · 1 072 citations
- [OpenAI o1 System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/) — OpenAI, Aaron Jaech, Adam Kalai, et al. · 2024 · 1 065 citations
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://md.research.elamur.ai/free/en/papers/zhao-2017-men-also-like-shopping-reducing/) — Jieyu Zhao, Tianlu Wang, Mark Yatskar, et al. · 2017 · 1 030 citations
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/) — Thomas Davidson, Dana Warmsley, Michael Macy, et al. · 2017 · 1 007 citations
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/) — Andy Zou, Zifan Wang, Nicholas Carlini, et al. · 2023 · 951 citations
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/) — Nikita Nangia, Clara Vania, Rasika Bhalerao, et al. · 2020 · 886 citations
- [Gender Bias in Coreference Resolution](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/) — Rachel Rudinger, Jason Naradowsky, Brian Leonard, et al. · 2018 · 852 citations
- [On Measuring Social Biases in Sentence Encoders](https://md.research.elamur.ai/free/en/papers/may-2019-measuring-social-biases-sentence-encoders/) — Chandler May, Alex Wang, Shikha Bordia, et al. · 2019 · 761 citations
- [Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-are-you-racist-or-am/) — Zeerak Waseem · 2016 · 760 citations
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://md.research.elamur.ai/free/en/papers/sun-2019-mitigating-gender-bias-natural-language/) — Tony Sun, Andrew Gaut, Shirlyn Tang, et al. · 2019 · 716 citations
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://md.research.elamur.ai/free/en/papers/sap-2020-social-bias-frames-reasoning-about/) — Maarten Sap, Saadia Gabriel, Lianhui Qin, et al. · 2020 · 680 citations
- [Red Teaming Language Models with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/) — Ethan Perez, Saffron Huang, Francis Song, et al. · 2022 · 665 citations
- [Jailbroken: How Does LLM Safety Training Fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) — Alexander Wei, Nika Haghtalab, Jacob Steinhardt · 2023 · 659 citations
- [Detecting Hate Speech on the World Wide Web](https://md.research.elamur.ai/free/en/papers/warner-2012-detecting-hate-speech-world-wide/) — William Warner, Julia Hirschberg · 2012 · 655 citations
- [Using Convolutional Neural Networks to Classify Hate-Speech](https://md.research.elamur.ai/free/en/papers/gamback-2017-convolutional-neural-networks-classify-hate/) — Björn Gambäck, Utpal Kumar Sikdar · 2017 · 647 citations

**1** [2](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/page/2/) … [7](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/page/7/) [next →](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/page/2/)

- [← previous topic Ancient Greek](https://md.research.elamur.ai/free/en/topics/ancient-greek/)
- [next topic → Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/)
- [↑ back to topics](https://md.research.elamur.ai/free/en/topics/)