--- title: "The Woman Worked as a Babysitter — Sheng 2019" description: "Sheng et al. study how open-domain language generators portray different social groups. They found that off-the-shelf sentiment tools often miss socially…" authors: ["Emily Sheng", "Kai-Wei Chang", "Premkumar Natarajan", "Nanyun Peng"] year: 2019 venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)" source: "https://aclanthology.org/D19-1339/" doi: "10.18653/v1/d19-1339" arxiv: "1909.01326" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/sheng-2019-woman-worked-as-babysitter-biases/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sheng-2019-woman-worked-as-babysitter-biases/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / The Woman Worked as a Babysitter: On Biases in Language Generation # The Woman Worked as a Babysitter: On Biases in Language Generation **Authors:** [Emily Sheng](https://md.research.elamur.ai/free/en/authors/emily-sheng/), [Kai-Wei Chang](https://md.research.elamur.ai/free/en/authors/kai-wei-chang/), Premkumar Natarajan, [Nanyun Peng](https://md.research.elamur.ai/free/en/authors/nanyun-peng/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/D19-1339/) · [PDF](https://aclanthology.org/D19-1339.pdf) · [DOI 10.18653/v1/d19-1339](https://doi.org/10.18653/v1/d19-1339) · [arXiv 1909.01326](https://arxiv.org/abs/1909.01326) · [OpenAlex](https://openalex.org/W2971307358) **Citations:** 419 citations (openalex-snapshot · 2026-10-14) · [332](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Sheng et al. study how open-domain language generators portray different social groups. They found that off-the-shelf sentiment tools often miss socially loaded content: a sentence saying someone worked as a pimp is scored as neutral. So they propose **regard** as the bias metric. Regard is the social perception a sentence creates of a demographic. To measure it, they wrote 10 prefix templates in two bias contexts, respect and occupation. Each template was filled with one of six groups: man/woman, Black/White and gay/straight. They sampled GPT-2 continuations, and three annotators labeled 360 of them for both sentiment and regard. This left 302 usable examples, which they used to train a BERT-based regard classifier. They then scored 500 generations per group from GPT-2 and LM 1B. Both models treat groups unequally, and regard shows larger gaps in negative scores than sentiment does. This means sentiment-only audits can understate bias, which matters for every downstream system built on these models. ## Key points - **Models and tools:** the models audited were GPT-2 (small) and Google's LM 1B, trained on the One Billion Word Benchmark. VADER was the main sentiment analyzer, and TextBlob was a baseline. - **Annotation agreement:** on the positive/neutral/negative classes, Fleiss' kappa was 0.60 for sentiment and 0.67 for regard. Spearman's correlation between annotators was 0.76 and 0.80. The dataset was split 212/60/30 for train/dev/test. - **Sentiment vs. regard:** VADER predictions match regard labels poorly in the occupation context (Spearman 0.54) but better in the respect context (0.69). The two human-annotated labels agree at 0.95 for respect but only 0.70 for occupation. - **Classifier results:** the BERT regard classifier reached about 0.8 accuracy. That is more than 20% above LSTM models and above a sentiment analyzer reused as a regard classifier. - **Bias findings:** with GPT-2, respect prompts led to more negative regard for Black, man and gay groups. Occupation prompts led to more negative regard for Black, woman and gay groups ("prostitute" came up often). LM 1B looked less biased overall. The annotations are public at github.com/ewsheng/nlg-bias. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 332 papers of our library cite it (of the 140673 whose bibliographies we read); 42 of them have a page here, 290 not yet. - [Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training](https://md.research.elamur.ai/free/en/papers/agarwal-2021-knowledge-graph-based-synthetic-corpus/) - [The Arabic Parallel Gender Corpus 2.0: Extensions and Analyses](https://md.research.elamur.ai/free/en/papers/alhafni-2022-arabic-parallel-gender-corpus-2/) - [Investigating Cultural Alignment of Large Language Models](https://md.research.elamur.ai/free/en/papers/alkhamissi-2024-investigating-cultural-alignment-large-language/) - [Language (Technology) is Power: A Critical Survey of “Bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Toxicity in chatgpt: Analyzing persona-assigned language models](https://md.research.elamur.ai/free/en/papers/deshpande-2023-toxicity-chatgpt-analyzing-persona-assigned/) - [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://md.research.elamur.ai/free/en/papers/dinan-2020-queens-are-powerful-too-mitigating/) - [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/) - [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://md.research.elamur.ai/free/en/papers/fernandes-2023-bridging-gap-survey-integrating-human/) - [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/) - [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/) - [Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts](https://md.research.elamur.ai/free/en/papers/guo-2022-auto-debias-debiasing-masked-language/) - [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/) - [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/) - [The Importance of Modeling Social Factors of Language: Theory and Practice](https://md.research.elamur.ai/free/en/papers/hovy-2021-importance-modeling-social-factors-language/) - [UNQOVERing Stereotyping Biases via Underspecified Questions](https://md.research.elamur.ai/free/en/papers/li-2020-unqovering-stereotyping-biases-underspecified-questions/) - [Towards Understanding and Mitigating Social Biases in Language Models](https://md.research.elamur.ai/free/en/papers/liang-2021-towards-understanding-mitigating-social-biases/) - [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://md.research.elamur.ai/free/en/papers/liu-2021-dexperts-decoding-time-controlled-text/) - [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/) - [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/) - [Gender and Representation Bias in GPT-3 Generated Stories](https://md.research.elamur.ai/free/en/papers/lucy-2021-gender-representation-bias-gpt-3/) - [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://md.research.elamur.ai/free/en/papers/mehrotra-2023-tree-attacks-jailbreaking-black-box/) - [StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/) - [HONEST: Measuring Hurtful Sentence Completion in Language Models](https://md.research.elamur.ai/free/en/papers/nozza-2021-honest-measuring-hurtful-sentence-completion/) - [BBQ: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/) - [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/) - [Discovering Language Model Behaviors with Model-Written Evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/) - [Red Teaming Language Models with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/) - [How many data points is a prompt worth?](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/) - [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/) - [Towards Controllable Biases in Language Generation](https://md.research.elamur.ai/free/en/papers/sheng-2020-controllable-biases-language-generation/) - [Societal Biases in Language Generation: Progress and Challenges](https://md.research.elamur.ai/free/en/papers/sheng-2021-societal-biases-language-generation-progress/) - [“I'm sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://md.research.elamur.ai/free/en/papers/smith-2022-i-m-sorry-hear-that/) - [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/) - [Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://md.research.elamur.ai/free/en/papers/tarrant-2021-intrinsic-bias-metrics-do-not/) - [Galactica: A Large Language Model for Science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/) - [How Far Can Bias Go? Tracing Bias from Pre-Training Data to Alignment](https://md.research.elamur.ai/free/en/papers/thaler-2026-how-far-can-bias-go/) - [LaMDA: Language Models for Dialog Applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/) - [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/) - [Measuring and Reducing Gendered Correlations in Pre-trained Models](https://md.research.elamur.ai/free/en/papers/webster-2020-measuring-reducing-gendered-correlations-pre/) - [Challenges in Detoxifying Language Models](https://md.research.elamur.ai/free/en/papers/welbl-2021-challenges-detoxifying-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{sheng2019woman, title = {{The Woman Worked as a Babysitter: On Biases in Language Generation}}, author = {Sheng, Emily and Chang, Kai-Wei and Natarajan, Premkumar and Peng, Nanyun}, booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)}, year = {2019}, doi = {10.18653/v1/d19-1339}, eprint = {1909.01326}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/d19-1339} } ``` - [← previous The Utility of Discourse Parsing Features for Predicting Argumentation Structure](https://md.research.elamur.ai/free/en/papers/hewett-2019-utility-discourse-parsing-features-predicting/) - [next → Topic Tensor Network for Implicit Discourse Relation Recognition in Chinese](https://md.research.elamur.ai/free/en/papers/xu-2019-topic-tensor-network-implicit-discourse/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) ## References 15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, pages 4349–4357. [find](https://scholar.google.com/scholar?q=Tolga+Bolukbasi%2C+Kai%2DWei+Chang%2C+James+Y+Zou%2C+Venkatesh+Saligrama%2C+and+Adam+T+Kalai%2E+2016%2E+Man+is+to+computer+programmer+as+woman+is+to+homemaker%3F+debiasing+word+embeddings%2E+In+Advances+in+Neural+Infor) [our card](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/) 2. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 123–129. ACM. [find](https://scholar.google.com/scholar?q=Peter+Henderson%2C+Koustuv+Sinha%2C+Nicolas+Angelard%2DGontier%2C+Nan+Rosemary+Ke%2C+Genevieve+Fried%2C+Ryan+Lowe%2C+and+Joelle+Pineau%2E+2018%2E+Ethical+challenges+in+data%2Ddriven+dialogue+systems%2E+In+Proceedings+of+th) 3. Clayton J Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Eighth international AAAI conference on weblogs and social media. [find](https://scholar.google.com/scholar?q=Clayton+J+Hutto+and+Eric+Gilbert%2E+2014%2E+Vader%3A+A+parsimonious+rule%2Dbased+model+for+sentiment+analysis+of+social+media+text%2E+In+Eighth+international+AAAI+conference+on+weblogs+and+social+media%2E) 4. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+2016%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2E+end%2Dto%2Dend+dialogue+systems+using+genera) 5. Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7378–7385. [find](https://scholar.google.com/scholar?q=Lili+Yao%2C+Nanyun+Peng%2C+Ralph+Weischedel%2C+Kevin+Knight%2C+Dongyan+Zhao%2C+and+Rui+Yan%2E+2019%2E+Plan%2Dand%2Dwrite%3A+Towards+better+automatic+storytelling%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intel) 6. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+2018%2E+Gender+bias+in+coreference+resolution%3A+Evaluation+and+debiasing+methods%2E+In+Proceedings+of+the+2018+Conference+of+the+N) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/) 7. Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical gender. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Pei+Zhou%2C+Weijia+Shi%2C+Jieyu+Zhao%2C+Kuan%2DHao+Huang%2C+Muhao+Chen%2C+Ryan+Cotterell%2C+and+Kai%2DWei+Chang%2E+2019%2E+Examining+gender+bias+in+languages+with+grammatical+gender%2E+In+Proceedings+of+the+Conference+on+E) 8. Svetlana Kiritchenko and Saif M Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In 7th Joint Conference on Lexical and Computational Semantics (SEM‘18). [find](https://scholar.google.com/scholar?q=Svetlana+Kiritchenko+and+Saif+M+Mohammad%2E+2018%2E+Examining+Gender+and+Race+Bias+in+Two+Hundred+Sentiment+Analysis+Systems%2E+In+7th+Joint+Conference+on+Lexical+and+Computational+Semantics+%28SEM%E2%80%9818%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/kiritchenko-2018-examining-gender-race-bias-two/) 9. Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2009%2E+Statistical+machine+translation%2E+Cambridge+University+Press%2E) 10. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Aman-charla, and Anupam Datta. 2018. Gender bias in neural natural language processing. arXiv preprint arXiv:1807.11714. [find](https://scholar.google.com/scholar?q=Kaiji+Lu%2C+Piotr+Mardziel%2C+Fangjing+Wu%2C+Preetam+Aman%2Dcharla%2C+and+Anupam+Datta%2E+2018%2E+Gender+bias+in+neural+natural+language+processing%2E+arXiv+preprint+arXiv%3A1807%2E11714%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2018-gender-bias-neural-natural-language/) 11. Saif Mohammad. 2016. A practical guide to sentiment annotation: Challenges and solutions. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 174–179. [find](https://scholar.google.com/scholar?q=Saif+Mohammad%2E+2016%2E+A+practical+guide+to+sentiment+annotation%3A+Challenges+and+solutions%2E+In+Proceedings+of+the+7th+Workshop+on+Computational+Approaches+to+Subjectivity%2C+Sentiment+and+Social+Media+Ana) 12. Marcelo OR Prates, Pedro H Avelar, and Luı́s C Lamb. 2018. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, pages 1–19. [find](https://scholar.google.com/scholar?q=Marcelo+OR+Prates%2C+Pedro+H+Avelar%2C+and+Lu%C4%B1%CC%81s+C+Lamb%2E+2018%2E+Assessing+gender+bias+in+machine+translation%3A+a+case+study+with+google+translate%2E+Neural+Computing+and+Applications%2C+pages+1%E2%80%9319%2E) 13. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2E) 14. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1… [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Jason+Naradowsky%2C+Brian+Leonard%2C+and+Benjamin+Van+Durme%2E+2018%2E+Gender+bias+in+coreference+resolution%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Associ) [our card](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/) 15. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building [find](https://scholar.google.com/scholar?q=Iulian+V+Serban%2C+Alessandro+Sordoni%2C+Yoshua+Bengio%2C+Aaron+Courville%2C+and+Joelle+Pineau%2E+2016%2E+Building)