---
title: "Identifying and Reducing Gender Bias in — Bordia 2019"
description: "Text corpora used to train language models carry gender skew."
authors: ["Shikha Bordia", "Samuel R. Bowman"]
year: 2019
venue: "arXiv:1904.03035"
source: "https://arxiv.org/abs/1904.03035"
doi: "10.48550/arxiv.1904.03035"
arxiv: "1904.03035"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bordia-2019-identifying-reducing-gender-bias-word/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bordia-2019-identifying-reducing-gender-bias-word/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Identifying and Reducing Gender Bias in Word-Level Language Models

# Identifying and Reducing Gender Bias in Word-Level Language Models

**Authors:** [Shikha Bordia](https://md.research.elamur.ai/free/en/authors/shikha-bordia/), [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) · **Year:** 2019 · **Venue:** arXiv:1904.03035 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1904.03035) · [PDF](https://arxiv.org/pdf/1904.03035) · [DOI 10.48550/arxiv.1904.03035](https://doi.org/10.48550/arxiv.1904.03035) · [arXiv 1904.03035](https://arxiv.org/abs/1904.03035) · [OpenAlex](https://openalex.org/W2942370121)

**Citations:** 211 citations (openalex-snapshot · 2026-10-14) · [152](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Text corpora used to train language models carry gender skew. For example, occupation words appear mostly near male pronouns, and models can reproduce or strengthen this pattern. Bordia and Bowman study the problem in word-level language models. They define a per-word bias score that compares how often a word co-occurs with female versus male terms. A score of zero means neutral, and the sign shows the direction. The score is computed either over a fixed context window or with exponentially weighted co-occurrences. They measure bias in three corpora (Penn Treebank, WikiText-2, CNN/Daily Mail) and in text sampled from an AWD-LSTM trained on each. To reduce bias, they add a regularizer adapted from the soft debiasing of Bolukbasi et al. (2016). It penalizes the projection of the learned input embeddings onto a gender subspace, scaled by a weight λ. Moderate λ lowers bias in generated text across all three corpora. Larger values make the model unstable and tend to raise perplexity, which reveals a bias–perplexity trade-off. The work offers a reproducible way to audit and partly reduce bias in models used in applications such as predictive keyboards.

## Key points
- Model: three-layer AWD-LSTM (1150 hidden units, embedding size 400, PyTorch). Baseline perplexities are 62.56 (PTB), 67.67 (WikiText-2) and 118.01 (CNN/Daily Mail, subsampled by a factor of 100).
- Corpus-level mean absolute bias µ (fixed window) is 0.83 for Penn Treebank, 0.80 for WikiText-2 and 0.72 for CNN/Daily Mail. PTB is the most male-skewed corpus.
- In generated text, µ falls to about 0.63 (PTB) and 0.61 (WikiText-2) at λ = 0.01, and to 0.34 (CNN/Daily Mail) at λ = 0.5. Bias rises again at higher λ.
- Three evaluation measures are used: µ, standard deviation σ, and β. β is the slope from regressing word-level bias in generated text on bias in the training corpus. The authors argue that word-level β tracks debiasing better than corpus-level µ and σ.
- Debiasing input and output embeddings together sharply raised perplexity, so the reported results debias only the input embeddings. Code is available at github.com/BordiaS/language-model-bias.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

152 papers of our library cite it (of the 140884 whose bibliographies we read); 24 of them have a page here, 128 not yet.

- [The Arabic Parallel Gender Corpus 2.0: Extensions and Analyses](https://md.research.elamur.ai/free/en/papers/alhafni-2022-arabic-parallel-gender-corpus-2/)
- [Language (Technology) is Power: A Critical Survey of “Bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/)
- [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/)
- [Debiasing Pre-trained Contextualised Embeddings](https://md.research.elamur.ai/free/en/papers/kaneko-2021-debiasing-pre-trained-contextualised-embeddings/)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://md.research.elamur.ai/free/en/papers/krause-2021-gedi-generative-discriminator-guided-sequence/)
- [Sustainable Modular Debiasing of Language Models](https://md.research.elamur.ai/free/en/papers/lauscher-2021-sustainable-modular-debiasing-language-models/)
- [UNQOVERing Stereotyping Biases via Underspecified Questions](https://md.research.elamur.ai/free/en/papers/li-2020-unqovering-stereotyping-biases-underspecified-questions/)
- [Towards Debiasing Sentence Representations](https://md.research.elamur.ai/free/en/papers/liang-2020-debiasing-sentence-representations/)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://md.research.elamur.ai/free/en/papers/lu-2022-quark-controllable-text-generation-reinforced/)
- [A Survey on Bias and Fairness in Machine Learning](https://md.research.elamur.ai/free/en/papers/mehrabi-2019-survey-bias-fairness-machine-learning/)
- [BBQ: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/)
- [Towards Controllable Biases in Language Generation](https://md.research.elamur.ai/free/en/papers/sheng-2020-controllable-biases-language-generation/)
- [Societal Biases in Language Generation: Progress and Challenges](https://md.research.elamur.ai/free/en/papers/sheng-2021-societal-biases-language-generation-progress/)
- [“I'm sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://md.research.elamur.ai/free/en/papers/smith-2022-i-m-sorry-hear-that/)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
- [LaMDA: Language Models for Dialog Applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
- [Challenges in Detoxifying Language Models](https://md.research.elamur.ai/free/en/papers/welbl-2021-challenges-detoxifying-language-models/)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://md.research.elamur.ai/free/en/papers/yao-2023-survey-large-language-model-llm/)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://md.research.elamur.ai/free/en/papers/yoo-2021-gpt3mix-leveraging-large-scale-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bordia2019identifying,
  title = {{Identifying and Reducing Gender Bias in Word-Level Language Models}},
  author = {Bordia, Shikha and Bowman, Samuel R.},
  year = {2019},
  doi = {10.48550/arxiv.1904.03035},
  eprint = {1904.03035},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.03035}
}
```

- [← previous How well do NLI models capture verb veridicality?](https://md.research.elamur.ai/free/en/papers/ross-2019-how-well-do-nli-models/)
- [next → Implicit Discourse Relation Identification for Open-domain Dialogues](https://md.research.elamur.ai/free/en/papers/ma-2019-implicit-discourse-relation-identification-open/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

19 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems… [find](https://scholar.google.com/scholar?q=Tolga+Bolukbasi%2C+Kai%2DWei+Chang%2C+James+Y+Zou%2C+Venkatesh+Saligrama%2C+and+Adam+T+Kalai%2E+2016%2E+Man+is+to+computer+programmer+as+woman+is+to+homemaker%3F+Debiasing+word+embeddings%2E+In+D%2E+D%2E+Lee%2C+M%2E+Sugiyama%2C+) [our card](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/)
2. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. [find](https://scholar.google.com/scholar?q=Aylin+Caliskan%2C+Joanna+J%2E+Bryson%2C+and+Arvind+Narayanan%2E+2017%2E+Semantics+derived+automatically+from+language+corpora+contain+human%2Dlike+biases%2E+Science%2C+356%286334%29%3A183%E2%80%93186%2E)
3. Joel Escudé Font and Marta R. Costa-Jussà. 2019. Equalizing gender biases in neural machine translation with word embeddings techniques. CoRR, abs/1901.03116. — [arXiv:1901.03116](https://arxiv.org/abs/1901.03116) [find](https://scholar.google.com/scholar?q=Joel+Escude%CC%81+Font+and+Marta+R%2E+Costa%2DJussa%CC%80%2E+2019%2E+Equalizing+gender+biases+in+neural+machine+translation+with+word+embeddings+techniques%2E+CoRR%2C+abs%2F1901%2E03116%2E+%E2%80%94+%5BarXiv%3A1901%2E03116%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
4. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Ass… [find](https://scholar.google.com/scholar?q=Hila+Gonen+and+Yoav+Goldberg%2E+2019%2E+Lipstick+on+a+pig%3A+Debiasing+methods+cover+up+systematic+gender+biases+in+word+embeddings+but+do+not+remove+them%2E+In+Proceedings+of+the+2019+Conference+of+the+North) [our card](https://md.research.elamur.ai/free/en/papers/gonen-2019-lipstick-pig-debiasing-methods-cover/)
5. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28,… [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+C%2E+Cortes%2C+N%2E+D%2E+Lawrence%2C+D%2E+) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
6. Dirk Hovy. 2015. Demographic factors improve classification performance. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 752–762. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2E+2015%2E+Demographic+factors+improve+classification+performance%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+7th+International+Joint+Conf) [our card](https://md.research.elamur.ai/free/en/papers/hovy-2015-demographic-factors-improve-classification-performance/)
7. Anja Lambrecht and Catherine E Tucker. 2018. Algorithmic bias? an empirical study into apparent gender-based discrimination in the display of stem career ads. Social Science Research Network (SSRN). [find](https://scholar.google.com/scholar?q=Anja+Lambrecht+and+Catherine+E+Tucker%2E+2018%2E+Algorithmic+bias%3F+an+empirical+study+into+apparent+gender%2Dbased+discrimination+in+the+display+of+stem+career+ads%2E+Social+Science+Research+Network+%28SSRN%29%2E)
8. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 302–308. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014%2E+Dependency%2Dbased+word+embeddings%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+Papers%29%2C+volume+2%2C+page)
9. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=Mitchell+P%2E+Marcus%2C+Beatrice+Santorini%2C+and+Mary+Ann+Marcinkiewicz%2E+1993%2E+Building+a+large+annotated+corpus+of+english%3A+The+penn+treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
10. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social bias in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Compu… [find](https://scholar.google.com/scholar?q=Chandler+May%2C+Alex+Wang%2C+Shikha+Bordia%2C+Samuel+R%2E+Bowman%2C+and+Rachel+Rudinger%2E+2019%2E+On+measuring+social+bias+in+sentence+encoders%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+)
11. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Nitish+Shirish+Keskar%2C+and+Richard+Socher%2E+2018%2E+Regularizing+and+optimizing+LSTM+language+models%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/)
12. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Caiming+Xiong%2C+James+Bradbury%2C+and+Richard+Socher%2E+2016%2E+Pointer+sentinel+mixture+models%2E+arXiv+preprint+arXiv%3A1609%2E07843%2E) [our card](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/)
13. Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH, pages 1045–1048. ISCA. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Martin+Karafia%CC%81t%2C+Luka%CC%81s+Burget%2C+Jan+Cernocky%CC%81%2C+and+Sanjeev+Khudanpur%2E+2010%2E+Recurrent+neural+network+based+language+model%2E+In+IN%2DTERSPEECH%2C+pages+1045%E2%80%931048%2E+ISCA%2E)
14. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+neural+information+processing+)
15. Mei Ngan and Patrick Grother. 2015. Face recognition vendor test (FRVT) performance of automated gender classification algorithms. US Department of Commerce, National Institute of Standards and Technology. [find](https://scholar.google.com/scholar?q=Mei+Ngan+and+Patrick+Grother%2E+2015%2E+Face+recognition+vendor+test+%28FRVT%29+performance+of+automated+gender+classification+algorithms%2E+US+Department+of+Commerce%2C+National+Institute+of+Standards+and+Techno)
16. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1650–1659. [find](https://scholar.google.com/scholar?q=Marta+Recasens%2C+Cristian+Danescu%2DNiculescu%2DMizil%2C+and+Dan+Jurafsky%2E+2013%2E+Linguistic+models+for+analyzing+and+detecting+biased+language%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+fo) [our card](https://md.research.elamur.ai/free/en/papers/recasens-2013-linguistic-models-analyzing-detecting-biased/)
17. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana. Associatio… [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Jason+Naradowsky%2C+Brian+Leonard%2C+and+Benjamin+Van+Durme%2E+2018%2E+Gender+bias+in+coreference+resolution%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Associ) [our card](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/)
18. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In EMNLP, pages 2979– 2989. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+2017%2E+Men+also+like+shopping%3A+Reducing+gender+bias+amplification+using+corpus%2Dlevel+constraints%2E+In+EMNLP%2C+pages+2979%E2%80%93+2989%2E+) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2017-men-also-like-shopping-reducing/)
19. Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018. Learning gender-neutral word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4847–4853. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Yichao+Zhou%2C+Zeyu+Li%2C+Wei+Wang%2C+and+Kai%2DWei+Chang%2E+2018%2E+Learning+gender%2Dneutral+word+embeddings%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2018-learning-gender-neutral-word-embeddings/)