--- title: "Gender Bias in Neural Natural Language Processing — Lu 2018" description: "The paper asks whether neural NLP systems absorb gender stereotypes from their training text and how to measure and reduce them." authors: ["Kaiji Lu", "Piotr Mardziel", "Fangjing Wu", "Preetam Amancharla", "Anupam Datta"] year: 2018 venue: "arXiv:1807.11714" source: "https://arxiv.org/abs/1807.11714" doi: "10.48550/arxiv.1807.11714" arxiv: "1807.11714" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lu-2018-gender-bias-neural-natural-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lu-2018-gender-bias-neural-natural-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / Gender Bias in Neural Natural Language Processing # Gender Bias in Neural Natural Language Processing **Authors:** Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, Anupam Datta · **Year:** 2018 · **Venue:** arXiv:1807.11714 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1807.11714) · [PDF](https://arxiv.org/pdf/1807.11714) · [DOI 10.48550/arxiv.1807.11714](https://doi.org/10.48550/arxiv.1807.11714) · [arXiv 1807.11714](https://arxiv.org/abs/1807.11714) · [OpenAlex](https://openalex.org/W3128232076) **Citations:** 250 citations (openalex-snapshot · 2026-10-14) · [144](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper asks whether neural NLP systems absorb gender stereotypes from their training text and how to measure and reduce them. The authors propose a general bias benchmark inspired by causal testing. Sentence pairs that differ only in a gendered word are fed to a model, and the gap in its internal scores is treated as bias. For coreference this means coreference scores, and for language models it means log-likelihoods. The benchmark is used to examine occupation words in two neural coreference systems trained on CoNLL-2012, the approaches of Lee et al. (2017) and of Clark and Manning (2016b), as well as in an LSTM language model (Zaremba et al. 2014) trained on Wikitext-2. All of these show measurable stereotypical bias. As a remedy they introduce counterfactual data augmentation (CDA), which adds gender-swapped copies of training instances while keeping the original labels. They compare CDA with word embedding debiasing (WED) and with combinations of the two. CDA cuts bias with little accuracy loss and beats WED, especially when embeddings are trained jointly with the model. Bias also tends to rise as training loss falls, and CDA limits this growth. ## Key points - Bias is measured as aggregate occupation bias (AOB) over matched gendered pairs, for example "he/she" alongside "doctor/nurse". - Language model results (2-layer LSTM, 1500 units, Wikitext-2): - The baseline has perplexity 83.39 and AOB 0.054. - Post-training WED pushes perplexity to 1128.15. - CDA with naive swapping lowers bias by 85% for only +0.24 perplexity. - Grammatical CDA lowers bias by 46%. - WED applied to pre-trained embeddings before training is largely undone, because the bias is relearned by the end of training. - Combining CDA with post-training WED in the Clark and Manning model (configurations 2.7 and 2.8) overcorrects, flipping the bias toward females and hurting accuracy. - CDA combined with pre-training WED gives complementary debiasing for models that use pre-trained embeddings. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) Topics: [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) ## Cited by Cited by 144 papers of our library cite it (of the 140884 whose bibliographies we read); 21 of them have a page here, 123 not yet. - [The Arabic Parallel Gender Corpus 2.0: Extensions and Analyses](https://md.research.elamur.ai/free/en/papers/alhafni-2022-arabic-parallel-gender-corpus-2/) - [The User-Aware Arabic Gender Rewriter](https://md.research.elamur.ai/free/en/papers/alhafni-2023-user-aware-arabic-gender-rewriter/) - [Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus](https://md.research.elamur.ai/free/en/papers/bentivogli-2020-gender-danger-evaluating-speech-translation/) - [Language (Technology) is Power: A Critical Survey of “Bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/) - [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/) - [Evaluating Models' Local Decision Boundaries via Contrast Sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/) - [Sustainable Modular Debiasing of Language Models](https://md.research.elamur.ai/free/en/papers/lauscher-2021-sustainable-modular-debiasing-language-models/) - [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-trustworthy-llms-survey-guideline-evaluating/) - [BBQ: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/) - [Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem](https://md.research.elamur.ai/free/en/papers/saunders-2020-reducing-gender-bias-neural-machine/) - [Gender Bias in Machine Translation](https://md.research.elamur.ai/free/en/papers/savoldi-2021-gender-bias-machine-translation/) - [The Woman Worked as a Babysitter: On Biases in Language Generation](https://md.research.elamur.ai/free/en/papers/sheng-2019-woman-worked-as-babysitter-biases/) - [Towards Controllable Biases in Language Generation](https://md.research.elamur.ai/free/en/papers/sheng-2020-controllable-biases-language-generation/) - [Societal Biases in Language Generation: Progress and Challenges](https://md.research.elamur.ai/free/en/papers/sheng-2021-societal-biases-language-generation-progress/) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/) - [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://md.research.elamur.ai/free/en/papers/sun-2019-mitigating-gender-bias-natural-language/) - [A Multiscale Visualization of Attention in the Transformer Model](https://md.research.elamur.ai/free/en/papers/vig-2019-multiscale-visualization-attention-transformer-model/) - [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) - [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://md.research.elamur.ai/free/en/papers/zmigrod-2019-counterfactual-data-augmentation-mitigating-gender/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{lu2018gender, title = {{Gender Bias in Neural Natural Language Processing}}, author = {Lu, Kaiji and Mardziel, Piotr and Wu, Fangjing and Amancharla, Preetam and Datta, Anupam}, year = {2018}, doi = {10.48550/arxiv.1807.11714}, eprint = {1807.11714}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1807.11714} } ``` - [← previous Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/) - [next → Gendered Speech: A Sociolinguistic Study of Conversations between Men and Women in Biblical Narrative](https://md.research.elamur.ai/free/en/papers/hbo-2018-hare-gendered-speech-sociolinguistic-study-conversations/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) ## References 19 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Kevin Clark and Christopher D Manning. Improving coreference resolution by learning entity-level distributed representations. arXiv preprint arXiv:1606.01323, 2016a. [find](https://scholar.google.com/scholar?q=Kevin+Clark+and+Christopher+D+Manning%2E+Improving+coreference+resolution+by+learning+entity%2Dlevel+distributed+representations%2E+arXiv+preprint+arXiv%3A1606%2E01323%2C+2016a%2E) [our card](https://md.research.elamur.ai/free/en/papers/clark-2016-improving-coreference-resolution-learning-entity/) 2. Kevin Clark and Christopher D Manning. Deep reinforcement learning for mention-ranking coreference models. arXiv preprint arXiv:1609.08667, 2016b. [find](https://scholar.google.com/scholar?q=Kevin+Clark+and+Christopher+D+Manning%2E+Deep+reinforcement+learning+for+mention%2Dranking+coreference+models%2E+arXiv+preprint+arXiv%3A1609%2E08667%2C+2016b%2E) [our card](https://md.research.elamur.ai/free/en/papers/clark-2016-deep-reinforcement-learning-mention-ranking/) 3. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference resolution. arXiv preprint arXiv:1707.07045, 2017. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Luheng+He%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+End%2Dto%2Dend+neural+coreference+resolution%2E+arXiv+preprint+arXiv%3A1707%2E07045%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/lee-2017-end-end-neural-coreference-resolution/) 4. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2C+2016%2E) 5. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google’s multilingual neural machine translation system: Enabling zero-shot translation. TACL, 5: 339–351, 2017. UR… [find](https://scholar.google.com/scholar?q=Melvin+Johnson%2C+Mike+Schuster%2C+Quoc+V%2E+Le%2C+Maxim+Krikun%2C+Yonghui+Wu%2C+Zhifeng+Chen%2C+Nikhil+Thorat%2C+Fernanda+B%2E+Vi%C3%A9gas%2C+Martin+Wattenberg%2C+Greg+Corrado%2C+Macduff+Hughes%2C+and+Jeffrey+Dean%2E+Google%E2%80%99s+multil) 6. Issie Lapowsky. Google autocomplete still has a hitler problem, Feb 2018. URL [https://www.wired.com/story/google-autocomplete-vile-suggestions/](https://www.wired.com/story/google-autocomplete-vile-suggestions/). [↗](https://www.wired.com/story/google-autocomplete-vile-suggestions/) 7. Rachael Tatman. Gender and dialect bias in youtube’s automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 53–59, 2017. [find](https://scholar.google.com/scholar?q=Rachael+Tatman%2E+Gender+and+dialect+bias+in+youtube%E2%80%99s+automatic+captions%2E+In+Proceedings+of+the+First+ACL+Workshop+on+Ethics+in+Natural+Language+Processing%2C+pages+53%E2%80%9359%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/tatman-2017-gender-dialect-bias-youtube-s/) 8. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, pages 4349–4357, 2016. [find](https://scholar.google.com/scholar?q=Tolga+Bolukbasi%2C+Kai%2DWei+Chang%2C+James+Y+Zou%2C+Venkatesh+Saligrama%2C+and+Adam+T+Kalai%2E+Man+is+to+computer+programmer+as+woman+is+to+homemaker%3F+debiasing+word+embeddings%2E+In+Advances+in+Neural+Information) [our card](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/) 9. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, 2017. [find](https://scholar.google.com/scholar?q=Aylin+Caliskan%2C+Joanna+J+Bryson%2C+and+Arvind+Narayanan%2E+Semantics+derived+automatically+from+language+corpora+contain+human%2Dlike+biases%2E+Science%2C+356%286334%29%3A183%E2%80%93186%2C+2017%2E) 10. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+arXiv+preprint+arXiv%3A1301%2E3781%2C+2013%2E) 11. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. [find](https://scholar.google.com/scholar?q=Wojciech+Zaremba%2C+Ilya+Sutskever%2C+and+Oriol+Vinyals%2E+Recurrent+neural+network+regularization%2E+arXiv+preprint+arXiv%3A1409%2E2329%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/) 12. Jiaping Zheng, Wendy W Chapman, Rebecca S Crowley, and Guergana K Savova. Coreference resolution: A review of general methodologies and applications in the clinical domain. Journal of biomedical informatics, 44(6):1113–1122, 2011. [find](https://scholar.google.com/scholar?q=Jiaping+Zheng%2C+Wendy+W+Chapman%2C+Rebecca+S+Crowley%2C+and+Guergana+K+Savova%2E+Coreference+resolution%3A+A+review+of+general+methodologies+and+applications+in+the+clinical+domain%2E+Journal+of+biomedical+infor) 13. Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. [find](https://scholar.google.com/scholar?q=Alex+Graves%2E+Generating+sequences+with+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1308%2E0850%2C+2013%2E) 14. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In Acoustics, speech and signal processing (icassp), 2013 ieee international conference on, pages 6645–6649. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Abdel%2Drahman+Mohamed%2C+and+Geoffrey+Hinton%2E+Speech+recognition+with+deep+recurrent+neural+networks%2E+In+Acoustics%2C+speech+and+signal+processing+%28icassp%29%2C+2013+ieee+international+conference+) 15. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+arXiv+preprint+arXiv%3A1409%2E0473%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) 16. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1–40. Association for Computational Linguistics, 2012. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+Conll%2D2012+shared+task%3A+Modeling+multilingual+unrestricted+coreference+in+ontonotes%2E+In+Joint+Conference+on+EMNLP+an) [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/) 17. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Caiming+Xiong%2C+James+Bradbury%2C+and+Richard+Socher%2E+Pointer+sentinel+mixture+models%2E+arXiv+preprint+arXiv%3A1609%2E07843%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/) 18. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876, 2018. [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+Gender+bias+in+coreference+resolution%3A+Evaluation+and+debiasing+methods%2E+arXiv+preprint+arXiv%3A1804%2E06876%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/) 19. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301, 2018. [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Jason+Naradowsky%2C+Brian+Leonard%2C+and+Benjamin+Van+Durme%2E+Gender+bias+in+coreference+resolution%2E+arXiv+preprint+arXiv%3A1804%2E09301%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/)