---
title: "Predictive Biases in Natural Language Processing — Shah 2019"
description: "Research on bias in NLP has grown as a set of isolated studies."
authors: ["Deven Shah", "H. Andrew Schwartz", "Dirk Hovy"]
year: 2019
venue: "arXiv:1912.11078 · Association for Computational Linguistics. (2020) 5248--5264"
source: "https://arxiv.org/abs/1912.11078"
doi: "10.18653/v1/2020.acl-main.468"
arxiv: "1912.11078"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shah-2019-predictive-biases-natural-language-processing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shah-2019-predictive-biases-natural-language-processing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview

# Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview

**Authors:** Deven Shah, [H. Andrew Schwartz](https://md.research.elamur.ai/free/en/authors/h-andrew-schwartz/), [Dirk Hovy](https://md.research.elamur.ai/free/en/authors/dirk-hovy/) · **Year:** 2019 · **Venue:** arXiv:1912.11078 · Association for Computational Linguistics. (2020) 5248--5264 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://doi.org/10.18653/v1/2020.acl-main.468) · [PDF](https://arxiv.org/pdf/1912.11078) · [DOI 10.18653/v1/2020.acl-main.468](https://doi.org/10.18653/v1/2020.acl-main.468) · [arXiv 1912.11078](https://arxiv.org/abs/1912.11078) · [OpenAlex](https://openalex.org/W3035377849)

**Citations:** 99 citations (openalex-snapshot · 2026-10-14) · [102](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Research on bias in NLP has grown as a set of isolated studies. Each one defines and fixes its own kind of "bias" without shared terminology, so work often treats symptoms rather than causes. Shah, Schwartz and Hovy propose a unifying conceptual framework built from a survey of NLP literature and selected social-science work. They adapt Swinton's (1981) notion of predictive bias from standardized testing and formalize two measurable consequences:
- **Outcome disparity:** predictions conditioned on a human attribute deviate from an application-specific ideal distribution.
- **Error disparity:** errors are unevenly distributed across attribute values, whether discrete or continuous.

They then locate four sources of bias within a standard supervised pipeline: bias in the annotated labels, bias in how the data sample is chosen, the model itself amplifying bias beyond what the data contain, and semantic bias carried by pre-trained embeddings. Existing countermeasures are grouped under these origins, and the paper ends with a practical checklist. A central claim is that bias belongs to a model *in a specific application*, not to the model alone. Naming where a bias comes from makes biases and mitigations easier to identify and compare across papers.

## Key points
- **Outcome disparity**, formally Q(Ŷ|A) ≁ P(Y|A). Example: image captioning models over-predict women in images with ovens and men in images with snowboards (Zhao et al., 2017; Hendricks et al., 2018).
- **Error disparity**, formally Q(ε|A_i) ≁ Q(ε|A_j). Example: the "Wall Street Journal Effect" (Hovy and Søgaard, 2015), where POS tagging errors increase as authors differ demographically from the 1980s–90s WSJ writers whose texts trained the taggers. This is a selection-bias case.
- **Origins mapped to pipeline stages:** embedding corpus (semantic bias), source sample (selection bias), annotator labels (label bias), and model fitting (over-amplification, e.g., MT outputs sounding older and more male).
- **Suggested remedies by origin:**
  - Label bias: post-stratification or annotator retraining.
  - Selection bias: stratified sampling or re-weighting.
  - Over-amplification: synthetic distribution matching or adding disparity to the cost function.
  - Semantic bias: hard/soft embedding de-biasing (Bolukbasi et al., 2016).
- **Community-level documentation:** Data Statements (Bender and Friedman, 2018), Datasheets (Gebru et al., 2018) and model reporting across user groups (Mitchell et al., 2019) are presented as tools for surfacing label and selection bias.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

102 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 89 not yet.

- [Language (Technology) is Power: A Critical Survey of “Bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/)
- [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/)
- [Evaluating Models' Local Decision Boundaries via Contrast Sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
- [The Importance of Modeling Social Factors of Language: Theory and Practice](https://md.research.elamur.ai/free/en/papers/hovy-2021-importance-modeling-social-factors-language/)
- [Why Can't Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversity](https://md.research.elamur.ai/free/en/papers/liu-2023-why-can-t-discourse-parsing/)
- [HONEST: Measuring Hurtful Sentence Completion in Language Models](https://md.research.elamur.ai/free/en/papers/nozza-2021-honest-measuring-hurtful-sentence-completion/)
- [Neural Unsupervised Domain Adaptation in NLP–-A Survey](https://md.research.elamur.ai/free/en/papers/ramponi-2020-neural-unsupervised-domain-adaptation-nlp/)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://md.research.elamur.ai/free/en/papers/rottger-2021-hatecheck-functional-tests-hate-speech/)
- [Gender Bias in Machine Translation](https://md.research.elamur.ai/free/en/papers/savoldi-2021-gender-bias-machine-translation/)
- [Large Language Model Alignment: A Survey](https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-model-alignment-survey/)
- [Societal Biases in Language Generation: Progress and Challenges](https://md.research.elamur.ai/free/en/papers/sheng-2021-societal-biases-language-generation-progress/)
- [A Second Wave of UD Hebrew Treebanking and Cross-Domain Parsing](https://md.research.elamur.ai/free/en/papers/ud-2022-zeldes-second-wave-ud-hebrew-treebanking/)
- [Ethical and social risks of harm from Language Models](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shah2019predictive,
  title = {{Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview}},
  author = {Shah, Deven and Schwartz, H. Andrew and Hovy, Dirk},
  year = {2019},
  doi = {10.18653/v1/2020.acl-main.468},
  eprint = {1912.11078},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.468}
}
```

- [← previous Predicting the Type and Target of Offensive Posts in Social Media](https://md.research.elamur.ai/free/en/papers/zampieri-2019-predicting-type-target-offensive-posts/)
- [next → Prepositions and the Grammaticalization of Ancient Hebrew Bipartite Reciprocal Markers](https://md.research.elamur.ai/free/en/papers/hbo-2019-staps-prepositions-grammaticalization-ancient-hebrew-bipartite/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

94 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jussara M Almeida, Gisele L Pappa, et al. 2015. Twitter population sample bias and its impact on predictive outcomes: A case study on elections. In Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015, pages 1254–1261. ACM. [find](https://scholar.google.com/scholar?q=Jussara+M+Almeida%2C+Gisele+L+Pappa%2C+et+al%2E+2015%2E+Twitter+population+sample+bias+and+its+impact+on+predictive+outcomes%3A+A+case+study+on+elections%2E+In+Proceedings+of+the+2015+IEEE%2FACM+International+Confe)
2. Fatemeh Almodaresi, Lyle Ungar, Vivek Kulkarni, Mohsen Zakeri, Salvatore Giorgi, and H. Andrew Schwartz. 2017. On the distribution of lexical features at multiple levels of analysis. In The 55th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Fatemeh+Almodaresi%2C+Lyle+Ungar%2C+Vivek+Kulkarni%2C+Mohsen+Zakeri%2C+Salvatore+Giorgi%2C+and+H%2E+Andrew+Schwartz%2E+2017%2E+On+the+distribution+of+lexical+features+at+multiple+levels+of+analysis%2E+In+The+55th+Annua)
3. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. [find](https://scholar.google.com/scholar?q=Ron+Artstein+and+Massimo+Poesio%2E+2008%2E+Inter%2Dcoder+agreement+for+computational+linguistics%2E+Computational+Linguistics%2C+34%284%29%3A555%E2%80%93596%2E)
4. Reg Baker, J Michael Brick, Nancy A Bates, Mike Battaglia, Mick P Couper, Jill A Dever, Krista J Gile, and Roger Tourangeau. 2013. Summary report of the aapor task force on non-probability sampling. Journal of Survey Statistics and Methodology, 1(2):90–143. [find](https://scholar.google.com/scholar?q=Reg+Baker%2C+J+Michael+Brick%2C+Nancy+A+Bates%2C+Mike+Battaglia%2C+Mick+P+Couper%2C+Jill+A+Dever%2C+Krista+J+Gile%2C+and+Roger+Tourangeau%2E+2013%2E+Summary+report+of+the+aapor+task+force+on+non%2Dprobability+sampling%2E+J)
5. Roy F Baumeister, Kathleen D Vohs, and David C Funder. 2007. Psychology as the science of self-reports and finger movements: Whatever happened to actual behavior? Perspectives on Psychological Science, 2(4):396–403. [find](https://scholar.google.com/scholar?q=Roy+F+Baumeister%2C+Kathleen+D+Vohs%2C+and+David+C+Funder%2E+2007%2E+Psychology+as+the+science+of+self%2Dreports+and+finger+movements%3A+Whatever+happened+to+actual+behavior%3F+Perspectives+on+Psychological+Science)
6. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. [find](https://scholar.google.com/scholar?q=Emily+M+Bender+and+Batya+Friedman%2E+2018%2E+Data+statements+for+natural+language+processing%3A+Toward+mitigating+system+bias+and+enabling+better+science%2E+Transactions+of+the+Association+for+Computational+L)
7. Adrian Benton, Margaret Mitchell, and Dirk Hovy. 2017. Multitask learning for mental health conditions with limited social media data. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, volume 1, pages 152–162. [find](https://scholar.google.com/scholar?q=Adrian+Benton%2C+Margaret+Mitchell%2C+and+Dirk+Hovy%2E+2017%2E+Multitask+learning+for+mental+health+conditions+with+limited+social+media+data%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+)
8. Richard A Berk. 1983. An introduction to sample selection bias in sociological data. American Sociological Review, pages 386–398. [find](https://scholar.google.com/scholar?q=Richard+A+Berk%2E+1983%2E+An+introduction+to+sample+selection+bias+in+sociological+data%2E+American+Sociological+Review%2C+pages+386%E2%80%93398%2E)
9. Sudeep Bhatia. 2017. Associative judgment and vector space semantics. Psychological review, 124(1):1. [find](https://scholar.google.com/scholar?q=Sudeep+Bhatia%2E+2017%2E+Associative+judgment+and+vector+space+semantics%2E+Psychological+review%2C+124%281%29%3A1%2E)
10. Su Lin Blodgett and Brendan O’Connor. 2017. Racial disparity in natural language processing: A case study of social media African-American English. arXiv preprint arXiv:1707.00061. [find](https://scholar.google.com/scholar?q=Su+Lin+Blodgett+and+Brendan+O%E2%80%99Connor%2E+2017%2E+Racial+disparity+in+natural+language+processing%3A+A+case+study+of+social+media+African%2DAmerican+English%2E+arXiv+preprint+arXiv%3A1707%2E00061%2E)
11. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems, pages 4349–4357. [find](https://scholar.google.com/scholar?q=Tolga+Bolukbasi%2C+Kai%2DWei+Chang%2C+James+Y+Zou%2C+Venkatesh+Saligrama%2C+and+Adam+T+Kalai%2E+2016%2E+Man+is+to+computer+programmer+as+woman+is+to+homemaker%3F+debiasing+word+embeddings%2E+In+Advances+in+neural+infor) [our card](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/)
12. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. [find](https://scholar.google.com/scholar?q=Aylin+Caliskan%2C+Joanna+J+Bryson%2C+and+Arvind+Narayanan%2E+2017%2E+Semantics+derived+automatically+from+language+corpora+contain+human%2Dlike+biases%2E+Science%2C+356%286334%29%3A183%E2%80%93186%2E)
13. Murphy Choy, Michelle Cheong, Ma Nang Laik, and Koo Ping Shung. 2012. Us presidential election 2012 prediction using census corrected twitter model. arXiv preprint arXiv:1211.0938. — [arXiv:1211.0938](https://arxiv.org/abs/1211.0938) [find](https://scholar.google.com/scholar?q=Murphy+Choy%2C+Michelle+Cheong%2C+Ma+Nang+Laik%2C+and+Koo+Ping+Shung%2E+2012%2E+Us+presidential+election+2012+prediction+using+census+corrected+twitter+model%2E+arXiv+preprint+arXiv%3A1211%2E0938%2E+%E2%80%94+%5BarXiv%3A1211%2E0938%5D)
14. Murphy Choy, Michelle LF Cheong, Ma Nang Laik, and Koo Ping Shung. 2011. A sentiment analysis of singapore presidential election 2011 using twitter data with census correction. arXiv preprint arXiv:1108.5520. — [arXiv:1108.5520](https://arxiv.org/abs/1108.5520) [find](https://scholar.google.com/scholar?q=Murphy+Choy%2C+Michelle+LF+Cheong%2C+Ma+Nang+Laik%2C+and+Koo+Ping+Shung%2E+2011%2E+A+sentiment+analysis+of+singapore+presidential+election+2011+using+twitter+data+with+census+correction%2E+arXiv+preprint+arXiv%3A11)
15. Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. DonâĂŹt take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lang… [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Mark+Yatskar%2C+and+Luke+Zettlemoyer%2E+2019%2E+Don%C3%A2A%CC%86Z%CC%81t+take+the+easy+way+out%3A+Ensemble+based+methods+for+avoiding+known+dataset+biases%2E+In+Proceedings+of+the+2019+Conference+on+Empiric)
16. Maximin Coavoux, Shashi Narayan, and Shay B Cohen. 2018. Privacy-preserving neural representations of text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1–10. [find](https://scholar.google.com/scholar?q=Maximin+Coavoux%2C+Shashi+Narayan%2C+and+Shay+B+Cohen%2E+2018%2E+Privacy%2Dpreserving+neural+representations+of+text%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+p)
17. Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. 2015. [find](https://scholar.google.com/scholar?q=Glen+Coppersmith%2C+Mark+Dredze%2C+Craig+Harman%2C+Kristy+Hollingshead%2C+and+Margaret+Mitchell%2E+2015%2E)
18. Clpsych 2015 shared task: Depression and ptsd on twitter. In CLPsych@ HLT-NAACL, pages 31–39. [find](https://scholar.google.com/scholar?q=Clpsych+2015+shared+task%3A+Depression+and+ptsd+on+twitter%2E+In+CLPsych%40+HLT%2DNAACL%2C+pages+31%E2%80%9339%2E) [our card](https://md.research.elamur.ai/free/en/papers/coppersmith-2015-clpsych-2015-shared-task-depression/)
19. Sam Corbett-Davies and Sharad Goel. 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023. — [arXiv:1808.00023](https://arxiv.org/abs/1808.00023) [find](https://scholar.google.com/scholar?q=Sam+Corbett%2DDavies+and+Sharad+Goel%2E+2018%2E+The+measure+and+mismeasure+of+fairness%3A+A+critical+review+of+fair+machine+learning%2E+arXiv+preprint+arXiv%3A1808%2E00023%2E+%E2%80%94+%5BarXiv%3A1808%2E00023%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
20. Fintan Costello and Paul Watts. 2014. Surprisingly rational: Probability theory plus noise explains biases in judgment. Psychological review, 121(3):463. [find](https://scholar.google.com/scholar?q=Fintan+Costello+and+Paul+Watts%2E+2014%2E+Surprisingly+rational%3A+Probability+theory+plus+noise+explains+biases+in+judgment%2E+Psychological+review%2C+121%283%29%3A463%2E)
21. Mick P Couper. 2013. Is the sky falling? new technology, changing media, and the future of surveys. In Survey Research Methods, volume 7, pages 145– 156. [find](https://scholar.google.com/scholar?q=Mick+P+Couper%2E+2013%2E+Is+the+sky+falling%3F+new+technology%2C+changing+media%2C+and+the+future+of+surveys%2E+In+Survey+Research+Methods%2C+volume+7%2C+pages+145%E2%80%93+156%2E)
22. Aron Culotta. 2014. Reducing sampling bias in social media data for county health inference. In Joint Statistical Meetings Proceedings, pages 1–12. [find](https://scholar.google.com/scholar?q=Aron+Culotta%2E+2014%2E+Reducing+sampling+bias+in+social+media+data+for+county+health+inference%2E+In+Joint+Statistical+Meetings+Proceedings%2C+pages+1%E2%80%9312%2E)
23. Cristian Danescu-Niculescu-Mizil, Robert West, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. No country for old members: User lifecycle and linguistic change in online communities. In Proceedings of the 22nd international conference on World Wide Web, pages 307–318. [find](https://scholar.google.com/scholar?q=Cristian+Danescu%2DNiculescu%2DMizil%2C+Robert+West%2C+Dan+Jurafsky%2C+Jure+Leskovec%2C+and+Christopher+Potts%2E+2013%2E+No+country+for+old+members%3A+User+lifecycle+and+linguistic+change+in+online+communities%2E+In+Proc)
24. Hal Daume III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256–263. [find](https://scholar.google.com/scholar?q=Hal+Daume+III%2E+2007%2E+Frustratingly+easy+domain+adaptation%2E+In+Proceedings+of+the+45th+Annual+Meeting+of+the+Association+of+Computational+Linguistics%2C+pages+256%E2%80%93263%2E) [our card](https://md.research.elamur.ai/free/en/papers/iii-2007-frustratingly-easy-domain-adaptation/)
25. Jacob Eisenstein, Brendan O’Connor, Noah A Smith, and Eric P Xing. 2010. A latent variable model for geographic lexical variation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1277–1287. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jacob+Eisenstein%2C+Brendan+O%E2%80%99Connor%2C+Noah+A+Smith%2C+and+Eric+P+Xing%2E+2010%2E+A+latent+variable+model+for+geographic+lexical+variation%2E+In+Proceedings+of+the+2010+Conference+on+Empirical+Methods+in+Natural)
26. Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11–21. [find](https://scholar.google.com/scholar?q=Yanai+Elazar+and+Yoav+Goldberg%2E+2018%2E+Adversarial+removal+of+demographic+attributes+from+text+data%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+11%E2%80%93) [our card](https://md.research.elamur.ai/free/en/papers/elazar-2018-adversarial-removal-demographic-attributes-text/)
27. Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2016. On the (im) possibility of fairness. corr abs/1609.07236 (2016). — [arXiv:1609.07236](https://arxiv.org/abs/1609.07236) [find](https://scholar.google.com/scholar?q=Sorelle+A+Friedler%2C+Carlos+Scheidegger%2C+and+Suresh+Venkatasubramanian%2E+2016%2E+On+the+%28im%29+possibility+of+fairness%2E+corr+abs%2F1609%2E07236+%282016%29%2E+%E2%80%94+%5BarXiv%3A1609%2E07236%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1609%2E07236%29)
28. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644. [find](https://scholar.google.com/scholar?q=Nikhil+Garg%2C+Londa+Schiebinger%2C+Dan+Jurafsky%2C+and+James+Zou%2E+2018%2E+Word+embeddings+quantify+100+years+of+gender+and+ethnic+stereotypes%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+115%2816%29%3AE3635%E2%80%93E)
29. Aparna Garimella, Carmen Banea, Dirk Hovy, and Rada Mihalcea. 2019. Women’s syntactic resilience and men’s grammatical luck: Gender-bias in part-of-speech tagging and dependency parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3493– 3498, Flo… [find](https://scholar.google.com/scholar?q=Aparna+Garimella%2C+Carmen+Banea%2C+Dirk+Hovy%2C+and+Rada+Mihalcea%2E+2019%2E+Women%E2%80%99s+syntactic+resilience+and+men%E2%80%99s+grammatical+luck%3A+Gender%2Dbias+in+part%2Dof%2Dspeech+tagging+and+dependency+parsing%2E+In+Proceeding)
30. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets. In Proceedings of the 5 th Workshop on Fairness, Accountability, and Transparency in Machine Learning, Stockholm, Sweden. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daum%C3%A9+III%2C+and+Kate+Crawford%2E+2018%2E+Datasheets+for+datasets%2E+In+Proceedings+of+the+5+th+Workshop+on+Fair)
31. Salvatore Giorgi, Veronica Lynn, Keshav Gupta, Sandra Matz, Lyle Ungar, and H.A. Schwartz. 2019. Correcting sociodemographic selection biases for population prediction. ArXiv. [find](https://scholar.google.com/scholar?q=Salvatore+Giorgi%2C+Veronica+Lynn%2C+Keshav+Gupta%2C+Sandra+Matz%2C+Lyle+Ungar%2C+and+H%2EA%2E+Schwartz%2E+2019%2E+Correcting+sociodemographic+selection+biases+for+population+prediction%2E+ArXiv%2E)
32. Bruce Glymour and Jonathan Herington. 2019. Measuring the biases that matter: The ethical and casual foundations for measures of fairness in algorithms. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, pages 269–278, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Bruce+Glymour+and+Jonathan+Herington%2E+2019%2E+Measuring+the+biases+that+matter%3A+The+ethical+and+casual+foundations+for+measures+of+fairness+in+algorithms%2E+In+Proceedings+of+the+Conference+on+Fairness%2C+A)
33. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol… [find](https://scholar.google.com/scholar?q=Hila+Gonen+and+Yoav+Goldberg%2E+2019%2E+Lipstick+on+a+pig%3A+Debiasing+methods+cover+up+systematic+gender+biases+in+word+embeddings+but+do+not+remove+them%2E+In+Proceedings+of+the+2019+Conference+of+the+North) [our card](https://md.research.elamur.ai/free/en/papers/gonen-2019-lipstick-pig-debiasing-methods-cover/)
34. Ron D Hays, Honghu Liu, and Arie Kapteyn. 2015. Use of internet panels to conduct surveys. Behavior research methods, 47(3):685–690. [find](https://scholar.google.com/scholar?q=Ron+D+Hays%2C+Honghu+Liu%2C+and+Arie+Kapteyn%2E+2015%2E+Use+of+internet+panels+to+conduct+surveys%2E+Behavior+research+methods%2C+47%283%29%3A685%E2%80%93690%2E)
35. Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018. Women also snowboard: Overcoming bias in captioning models. In European Conference on Computer Vision, pages 793–811. Springer. [find](https://scholar.google.com/scholar?q=Lisa+Anne+Hendricks%2C+Kaylee+Burns%2C+Kate+Saenko%2C+Trevor+Darrell%2C+and+Anna+Rohrbach%2E+2018%2E+Women+also+snowboard%3A+Overcoming+bias+in+captioning+models%2E+In+European+Conference+on+Computer+Vision%2C+pages+79)
36. Joseph Henrich, Steven J Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and brain sciences, 33(2-3):61–83. [find](https://scholar.google.com/scholar?q=Joseph+Henrich%2C+Steven+J+Heine%2C+and+Ara+Norenzayan%2E+2010%2E+The+weirdest+people+in+the+world%3F+Behavioral+and+brain+sciences%2C+33%282%2D3%29%3A61%E2%80%9383%2E)
37. Miguel A Hernán, Sonia Hernández-Díaz, and James M Robins. 2004. A structural approach to selection bias. Epidemiology, pages 615–625. [find](https://scholar.google.com/scholar?q=Miguel+A+Hern%C3%A1n%2C+Sonia+Hern%C3%A1ndez%2DD%C3%ADaz%2C+and+James+M+Robins%2E+2004%2E+A+structural+approach+to+selection+bias%2E+Epidemiology%2C+pages+615%E2%80%93625%2E)
38. Yasmeen Hitti, Eunbee Jang, Ines Moreno, and Carolyne Pelletier. 2019. Proposed taxonomy for gender bias in text; a filtering methodology for the gender generalization subtype. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 8–17, Florence, Italy. Associatio… [find](https://scholar.google.com/scholar?q=Yasmeen+Hitti%2C+Eunbee+Jang%2C+Ines+Moreno%2C+and+Carolyne+Pelletier%2E+2019%2E+Proposed+taxonomy+for+gender+bias+in+text%3B+a+filtering+methodology+for+the+gender+generalization+subtype%2E+In+Proceedings+of+the+F)
39. Dirk Hovy. 2015. Demographic factors improve classification performance. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 752–762. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2E+2015%2E+Demographic+factors+improve+classification+performance%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+7th+International+Joint+Conf) [our card](https://md.research.elamur.ai/free/en/papers/hovy-2015-demographic-factors-improve-classification-performance/)
40. Dirk Hovy. 2018. The social and the neural network: How to make natural language processing about people again. In Proceedings of the Second Workshop on Computational Modeling of Peo-pleâĂŹs Opinions, Personality, and Emotions in Social Media, pages 42–49. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2E+2018%2E+The+social+and+the+neural+network%3A+How+to+make+natural+language+processing+about+people+again%2E+In+Proceedings+of+the+Second+Workshop+on+Computational+Modeling+of+Peo%2Dple%C3%A2A%CC%86Z%CC%81s+Opinion)
41. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with mace. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Taylor+Berg%2DKirkpatrick%2C+Ashish+Vaswani%2C+and+Eduard+Hovy%2E+2013%2E+Learning+whom+to+trust+with+mace%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/)
42. Dirk Hovy, Federico Bianchi, and Tommaso Fornaciari. 2020. Can you translate that into man? commercial machine translation systems include stylistic biases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, WA, USA. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Federico+Bianchi%2C+and+Tommaso+Fornaciari%2E+2020%2E+Can+you+translate+that+into+man%3F+commercial+machine+translation+systems+include+stylistic+biases%2E+In+Proceedings+of+the+58th+Annual+Meeting+o)
43. Dirk Hovy and Anders Søgaard. 2015. Tagging performance correlates with author age. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 483–488. [find](https://scholar.google.com/scholar?q=Dirk+Hovy+and+Anders+S%C3%B8gaard%2E+2015%2E+Tagging+performance+correlates+with+author+age%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+7th+International)
44. Dirk Hovy and Shannon L Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 591–598. [find](https://scholar.google.com/scholar?q=Dirk+Hovy+and+Shannon+L+Spruit%2E+2016%2E+The+social+impact+of+natural+language+processing%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+Pape) [our card](https://md.research.elamur.ai/free/en/papers/hovy-2016-social-impact-natural-language-processing/)
45. Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in nlp. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 264–271. [find](https://scholar.google.com/scholar?q=Jing+Jiang+and+ChengXiang+Zhai%2E+2007%2E+Instance+weighting+for+domain+adaptation+in+nlp%2E+In+Proceedings+of+the+45th+annual+meeting+of+the+association+of+computational+linguistics%2C+pages+264%E2%80%93271%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2007-instance-weighting-domain-adaptation-nlp/)
46. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2015. Challenges of studying and processing dialects in social media. In Proceedings of the Workshop on Noisy User-generated Text, pages 9–18. [find](https://scholar.google.com/scholar?q=Anna+J%C3%B8rgensen%2C+Dirk+Hovy%2C+and+Anders+S%C3%B8gaard%2E+2015%2E+Challenges+of+studying+and+processing+dialects+in+social+media%2E+In+Proceedings+of+the+Workshop+on+Noisy+User%2Dgenerated+Text%2C+pages+9%E2%80%9318%2E)
47. Kenneth Joseph, Lisa Friedland, William Hobbs, Oren Tsur, and David Lazer. 2017. Constance: Modeling annotation contexts to improve stance classification. arXiv preprint arXiv:1708.06309. [find](https://scholar.google.com/scholar?q=Kenneth+Joseph%2C+Lisa+Friedland%2C+William+Hobbs%2C+Oren+Tsur%2C+and+David+Lazer%2E+2017%2E+Constance%3A+Modeling+annotation+contexts+to+improve+stance+classification%2E+arXiv+preprint+arXiv%3A1708%2E06309%2E)
48. David Jurgens. 2013. That’s what friends are for: Inferring location in online social media platforms based on social relationships. In Seventh International AAAI Conference on Weblogs and Social Media. [find](https://scholar.google.com/scholar?q=David+Jurgens%2E+2013%2E+That%E2%80%99s+what+friends+are+for%3A+Inferring+location+in+online+social+media+platforms+based+on+social+relationships%2E+In+Seventh+International+AAAI+Conference+on+Weblogs+and+Social+Medi)
49. David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equitable language identification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 51–57. [find](https://scholar.google.com/scholar?q=David+Jurgens%2C+Yulia+Tsvetkov%2C+and+Dan+Jurafsky%2E+2017%2E+Incorporating+dialectal+variability+for+socially+equitable+language+identification%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+)
50. ML Kern, G Park, JC Eichstaedt, HA Schwartz, M Sap, LK Smith, and LH Ungar. 2016. Gaining insights from social media language: Methodologies and challenges. Psychological methods, 21(4):507–525. [find](https://scholar.google.com/scholar?q=ML+Kern%2C+G+Park%2C+JC+Eichstaedt%2C+HA+Schwartz%2C+M+Sap%2C+LK+Smith%2C+and+LH+Ungar%2E+2016%2E+Gaining+insights+from+social+media+language%3A+Methodologies+and+challenges%2E+Psychological+methods%2C+21%284%29%3A507%E2%80%93525%2E)
51. Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43–53. [find](https://scholar.google.com/scholar?q=Svetlana+Kiritchenko+and+Saif+Mohammad%2E+2018%2E+Examining+gender+and+race+bias+in+two+hundred+sentiment+analysis+systems%2E+In+Proceedings+of+the+Seventh+Joint+Conference+on+Lexical+and+Computational+Sema) [our card](https://md.research.elamur.ai/free/en/papers/kiritchenko-2018-examining-gender-race-bias-two/)
52. Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Cass R Sunstein. 2018. Discrimination in the age of algorithms. Journal of Legal Analysis, 10. [find](https://scholar.google.com/scholar?q=Jon+Kleinberg%2C+Jens+Ludwig%2C+Sendhil+Mullainathan%2C+and+Cass+R+Sunstein%2E+2018%2E+Discrimination+in+the+age+of+algorithms%2E+Journal+of+Legal+Analysis%2C+10%2E)
53. Austin C Kozlowski, Matt Taddy, and James A Evans. 2018. The geometry of culture: Analyzing meaning through word embeddings. arXiv preprint arXiv:1803.09288. — [arXiv:1803.09288](https://arxiv.org/abs/1803.09288) [find](https://scholar.google.com/scholar?q=Austin+C+Kozlowski%2C+Matt+Taddy%2C+and+James+A+Evans%2E+2018%2E+The+geometry+of+culture%3A+Analyzing+meaning+through+word+embeddings%2E+arXiv+preprint+arXiv%3A1803%2E09288%2E+%E2%80%94+%5BarXiv%3A1803%2E09288%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
54. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. arXiv preprint arXiv:1906.07337. [find](https://scholar.google.com/scholar?q=Keita+Kurita%2C+Nidhi+Vyas%2C+Ayush+Pareek%2C+Alan+W+Black%2C+and+Yulia+Tsvetkov%2E+2019%2E+Measuring+bias+in+contextualized+word+representations%2E+arXiv+preprint+arXiv%3A1906%2E07337%2E) [our card](https://md.research.elamur.ai/free/en/papers/kurita-2019-measuring-bias-contextualized-word-representations/)
55. Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2020. A general framework for implicit and explicit debiasing of distributional word vector spaces. [find](https://scholar.google.com/scholar?q=Anne+Lauscher%2C+Goran+Glava%C5%A1%2C+Simone+Paolo+Ponzetto%2C+and+Ivan+Vulic%CC%81%2E+2020%2E+A+general+framework+for+implicit+and+explicit+debiasing+of+distributional+word+vector+spaces%2E)
56. Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018. Towards robust and privacy-preserving text representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 25–30. [find](https://scholar.google.com/scholar?q=Yitong+Li%2C+Timothy+Baldwin%2C+and+Trevor+Cohn%2E+2018%2E+Towards+robust+and+privacy%2Dpreserving+text+representations%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistic) [our card](https://md.research.elamur.ai/free/en/papers/li-2018-robust-privacy-preserving-text-representations/)
57. Veronica E. Lynn, Youngseo Son, Vivek Kulkarni, Niranjan Balasubramanian, and H. Andrew Schwartz. 2017. Human centered nlp with user-factor adaptation. In Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Veronica+E%2E+Lynn%2C+Youngseo+Son%2C+Vivek+Kulkarni%2C+Niranjan+Balasubramanian%2C+and+H%2E+Andrew+Schwartz%2E+2017%2E+Human+centered+nlp+with+user%2Dfactor+adaptation%2E+In+Empirical+Methods+in+Natural+Language+Process)
58. Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and Alan W Black. 2019. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. arXiv preprint arXiv:1904.04047. [find](https://scholar.google.com/scholar?q=Thomas+Manzini%2C+Yao+Chong+Lim%2C+Yulia+Tsvetkov%2C+and+Alan+W+Black%2E+2019%2E+Black+is+to+criminal+as+caucasian+is+to+police%3A+Detecting+and+removing+multiclass+bias+in+word+embeddings%2E+arXiv+preprint+arXiv%3A1) [our card](https://md.research.elamur.ai/free/en/papers/manzini-2019-black-is-criminal-as-caucasian/)
59. Robert R. McCrae and Paul T. Costa Jr. 1989. Reinterpreting the Myers-Briggs type indicator from the perspective of the five-factor model of personality. Journal of Personality, 57(1):17–40. [find](https://scholar.google.com/scholar?q=Robert+R%2E+McCrae+and+Paul+T%2E+Costa+Jr%2E+1989%2E+Reinterpreting+the+Myers%2DBriggs+type+indicator+from+the+perspective+of+the+five%2Dfactor+model+of+personality%2E+Journal+of+Personality%2C+57%281%29%3A17%E2%80%9340%2E)
60. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+In+Proceedings+of+ICLR%2E)
61. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 220–229. ACM. [find](https://scholar.google.com/scholar?q=Margaret+Mitchell%2C+Simone+Wu%2C+Andrew+Zaldivar%2C+Parker+Barnes%2C+Lucy+Vasserman%2C+Ben+Hutchinson%2C+Elena+Spitzer%2C+Inioluwa+Deborah+Raji%2C+and+Timnit+Gebru%2E+2019%2E+Model+cards+for+model+reporting%2E+In+Proceedi)
62. Ehsan Mohammady and Aron Culotta. 2014. Using county demographics to infer attributes of twitter users. In Proceedings of the joint workshop on social dynamics and personal attributes in social media, pages 7–16. [find](https://scholar.google.com/scholar?q=Ehsan+Mohammady+and+Aron+Culotta%2E+2014%2E+Using+county+demographics+to+infer+attributes+of+twitter+users%2E+In+Proceedings+of+the+joint+workshop+on+social+dynamics+and+personal+attributes+in+social+media%2C)
63. Malvina Nissim, Rik van Noord, and Rob van der Goot. 2019. Fair is better than sensational: Man is to doctor as woman is to doctor. arXiv preprint arXiv:1905.09866. [find](https://scholar.google.com/scholar?q=Malvina+Nissim%2C+Rik+van+Noord%2C+and+Rob+van+der+Goot%2E+2019%2E+Fair+is+better+than+sensational%3A+Man+is+to+doctor+as+woman+is+to+doctor%2E+arXiv+preprint+arXiv%3A1905%2E09866%2E)
64. David K Park, Andrew Gelman, and Joseph Bafumi. 2006. State-level opinions from national surveys: Poststratification using multilevel logistic regression. Public opinion in state politics, pages 209–28. [find](https://scholar.google.com/scholar?q=David+K+Park%2C+Andrew+Gelman%2C+and+Joseph+Bafumi%2E+2006%2E+State%2Dlevel+opinions+from+national+surveys%3A+Poststratification+using+multilevel+logistic+regression%2E+Public+opinion+in+state+politics%2C+pages+209%E2%80%932)
65. Rebecca J Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311–326. [find](https://scholar.google.com/scholar?q=Rebecca+J+Passonneau+and+Bob+Carpenter%2E+2014%2E+The+benefits+of+a+model+of+annotation%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+2%3A311%E2%80%93326%2E) [our card](https://md.research.elamur.ai/free/en/papers/passonneau-2014-benefits-model-annotation/)
66. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6:571–585. [find](https://scholar.google.com/scholar?q=Silviu+Paun%2C+Bob+Carpenter%2C+Jon+Chamberlain%2C+Dirk+Hovy%2C+Udo+Kruschwitz%2C+and+Massimo+Poesio%2E+2018%2E+Comparing+bayesian+models+of+annotation%2E+Transactions+of+the+Association+for+Computational+Linguistics)
67. Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch. 2014. The language demographics of amazon mechanical turk. Transactions of the Association for Computational Linguistics, 2:79–92. [find](https://scholar.google.com/scholar?q=Ellie+Pavlick%2C+Matt+Post%2C+Ann+Irvine%2C+Dmitry+Kachaev%2C+and+Chris+Callison%2DBurch%2E+2014%2E+The+language+demographics+of+amazon+mechanical+turk%2E+Transactions+of+the+Association+for+Computational+Linguistics)
68. James W Pennebaker. 2011. The secret life of pronouns. New Scientist, 211(2828):42–45. [find](https://scholar.google.com/scholar?q=James+W+Pennebaker%2E+2011%2E+The+secret+life+of+pronouns%2E+New+Scientist%2C+211%282828%29%3A42%E2%80%9345%2E)
69. James W. Pennebaker and Lori D. Stone. 2003. Words of wisdom: Language use over the life span. Journal of Personality and Social Psychology, 85(2):291. [find](https://scholar.google.com/scholar?q=James+W%2E+Pennebaker+and+Lori+D%2E+Stone%2E+2003%2E+Words+of+wisdom%3A+Language+use+over+the+life+span%2E+Journal+of+Personality+and+Social+Psychology%2C+85%282%29%3A291%2E)
70. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751. [find](https://scholar.google.com/scholar?q=Barbara+Plank%2C+Dirk+Hovy%2C+and+Anders+S%C3%B8gaard%2E+2014%2E+Learning+part%2Dof%2Dspeech+taggers+with+inter%2Dannotator+agreement+loss%2E+In+Proceedings+of+the+14th+Conference+of+the+European+Chapter+of+the+Associatio) [our card](https://md.research.elamur.ai/free/en/papers/plank-2014-learning-part-speech-taggers-inter/)
71. Marcelo OR Prates, Pedro H Avelar, and Luís C Lamb. 2018. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, pages 1–19. [find](https://scholar.google.com/scholar?q=Marcelo+OR+Prates%2C+Pedro+H+Avelar%2C+and+Lu%C3%ADs+C+Lamb%2E+2018%2E+Assessing+gender+bias+in+machine+translation%3A+a+case+study+with+google+translate%2E+Neural+Computing+and+Applications%2C+pages+1%E2%80%9319%2E)
72. Daniel Preoţiuc-Pietro, Johannes Eichstaedt, Gregory Park, Maarten Sap, Laura Smith, Victoria Tobolsky, H Andrew Schwartz, and Lyle Ungar. 2015. The role of personality, age, and gender in tweeting about mental illness. In Proceedings of the 2nd workshop on computational linguistics and clinical ps… [find](https://scholar.google.com/scholar?q=Daniel+Preot%CC%A7iuc%2DPietro%2C+Johannes+Eichstaedt%2C+Gregory+Park%2C+Maarten+Sap%2C+Laura+Smith%2C+Victoria+Tobolsky%2C+H+Andrew+Schwartz%2C+and+Lyle+Ungar%2E+2015%2E+The+role+of+personality%2C+age%2C+and+gender+in+tweeting+a)
73. Daniel Preotiuc-Pietro, Maarten Sap, H Andrew Schwartz, and Lyle Ungar. 2015. Mental illness detection at the world well-being project for the clpsych 2015 shared task. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, NA… [find](https://scholar.google.com/scholar?q=Daniel+Preotiuc%2DPietro%2C+Maarten+Sap%2C+H+Andrew+Schwartz%2C+and+Lyle+Ungar%2E+2015%2E+Mental+illness+detection+at+the+world+well%2Dbeing+project+for+the+clpsych+2015+shared+task%2E+In+Proceedings+of+the+Workshop+)
74. Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-François Bonnefon, Cynthia Breazeal, Jacob W Crandall, Nicholas A Christakis, Iain D Couzin, Matthew O Jackson, et al. 2019. Machine behaviour. Nature, 568(7753):477. [find](https://scholar.google.com/scholar?q=Iyad+Rahwan%2C+Manuel+Cebrian%2C+Nick+Obradovich%2C+Josh+Bongard%2C+Jean%2DFran%C3%A7ois+Bonnefon%2C+Cynthia+Breazeal%2C+Jacob+W+Crandall%2C+Nicholas+A+Christakis%2C+Iain+D+Couzin%2C+Matthew+O+Jackson%2C+et+al%2E+2019%2E+Machine+be)
75. Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Anna Rumshisky, and Adam Tauman Kalai. 2019. What’s in a name? reducing bias in bios without access to protected attributes. arXiv preprint arXiv:1904.05233. [find](https://scholar.google.com/scholar?q=Alexey+Romanov%2C+Maria+De%2DArteaga%2C+Hanna+Wallach%2C+Jennifer+Chayes%2C+Christian+Borgs%2C+Alexandra+Chouldechova%2C+Sahin+Geyik%2C+Krishnaram+Kenthapadi%2C+Anna+Rumshisky%2C+and+Adam+Tauman+Kalai%2E+2019%2E+What%E2%80%99s+in+a+)
76. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2,… [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Jason+Naradowsky%2C+Brian+Leonard%2C+and+Benjamin+Van+Durme%2E+2018%2E+Gender+bias+in+coreference+resolution%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Associ) [our card](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/)
77. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Dallas+Card%2C+Saadia+Gabriel%2C+Yejin+Choi%2C+and+Noah+A%2E+Smith%2E+2019%2E+The+risk+of+racial+bias+in+hate+speech+detection%2E+In+Proceedings+of+the+57th+Conference+of+the+Association+for+Computatio) [our card](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/)
78. H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Megha Agrawal, Gregory J Park, Shrinidhi K Lakshmikanth, Sneha Jha, Martin EP Seligman, Lyle Ungar, et al. 2013. Characterizing geographic variation in well-being using tweets. In Seventh International AAAI Conference on… [find](https://scholar.google.com/scholar?q=H+Andrew+Schwartz%2C+Johannes+C+Eichstaedt%2C+Margaret+L+Kern%2C+Lukasz+Dziurzynski%2C+Megha+Agrawal%2C+Gregory+J+Park%2C+Shrinidhi+K+Lakshmikanth%2C+Sneha+Jha%2C+Martin+EP+Seligman%2C+Lyle+Ungar%2C+et+al%2E+2013%2E+Characte)
79. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gabriel+Stanovsky%2C+Noah+A%2E+Smith%2C+and+Luke+Zettlemoyer%2E+2019%2E+Evaluating+gender+bias+in+machine+translation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C) [our card](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/)
80. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Conference of the Association for Computational… [find](https://scholar.google.com/scholar?q=Tony+Sun%2C+Andrew+Gaut%2C+Shirlyn+Tang%2C+Yuxin+Huang%2C+Mai+ElSherief%2C+Jieyu+Zhao%2C+Diba+Mirza%2C+Elizabeth+Belding%2C+Kai%2DWei+Chang%2C+and+William+Yang+Wang%2E+2019%2E+Mitigating+gender+bias+in+natural+language+proce) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-mitigating-gender-bias-natural-language/)
81. Harini Suresh and John V Guttag. 2019. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002. — [arXiv:1901.10002](https://arxiv.org/abs/1901.10002) [find](https://scholar.google.com/scholar?q=Harini+Suresh+and+John+V+Guttag%2E+2019%2E+A+framework+for+understanding+unintended+consequences+of+machine+learning%2E+arXiv+preprint+arXiv%3A1901%2E10002%2E+%E2%80%94+%5BarXiv%3A1901%2E10002%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1901%2E10002)
82. Chris Sweeney and Maryam Najafian. 2019. A transparent framework for evaluating unintended demographic bias in word embeddings. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1662–1667, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chris+Sweeney+and+Maryam+Najafian%2E+2019%2E+A+transparent+framework+for+evaluating+unintended+demographic+bias+in+word+embeddings%2E+In+Proceedings+of+the+57th+Conference+of+the+Association+for+Computation)
83. Spencer S Swinton. 1981. Predictive bias in graduate admissions tests. ETS Research Report Series, 1981(1):i–53. [find](https://scholar.google.com/scholar?q=Spencer+S+Swinton%2E+1981%2E+Predictive+bias+in+graduate+admissions+tests%2E+ETS+Research+Report+Series%2C+1981%281%29%3Ai%E2%80%9353%2E)
84. Peter Trudgill. 2000. Sociolinguistics: An introduction to language and society. Penguin UK. [find](https://scholar.google.com/scholar?q=Peter+Trudgill%2E+2000%2E+Sociolinguistics%3A+An+introduction+to+language+and+society%2E+Penguin+UK%2E)
85. Amos Tversky and Daniel Kahneman. 1973. Availability: A heuristic for judging frequency and probability. Cognitive psychology, 5(2):207–232. [find](https://scholar.google.com/scholar?q=Amos+Tversky+and+Daniel+Kahneman%2E+1973%2E+Availability%3A+A+heuristic+for+judging+frequency+and+probability%2E+Cognitive+psychology%2C+5%282%29%3A207%E2%80%93232%2E)
86. Svitlana Volkova, Theresa Wilson, and David Yarowsky. 2013. Exploring demographic language variations to improve multilingual sentiment analysis in social media. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1815–1827. [find](https://scholar.google.com/scholar?q=Svitlana+Volkova%2C+Theresa+Wilson%2C+and+David+Yarowsky%2E+2013%2E+Exploring+demographic+language+variations+to+improve+multilingual+sentiment+analysis+in+social+media%2E+In+Proceedings+of+the+2013+Conference+)
87. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. [find](https://scholar.google.com/scholar?q=Kellie+Webster%2C+Marta+Recasens%2C+Vera+Axelrod%2C+and+Jason+Baldridge%2E+2018%2E+Mind+the+gap%3A+A+balanced+corpus+of+gendered+ambiguous+pronouns%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+) [our card](https://md.research.elamur.ai/free/en/papers/webster-2018-mind-gap-balanced-corpus-gendered/)
88. Thomas A. Widiger and Douglas B. Samuel. 2005. Diagnostic categories or dimensions? A question for the Diagnostic and Statistical Manual of Mental Disorders—Fifth Edition. Journal of Abnormal Psychology, 114(4):494. [find](https://scholar.google.com/scholar?q=Thomas+A%2E+Widiger+and+Douglas+B%2E+Samuel%2E+2005%2E+Diagnostic+categories+or+dimensions%3F+A+question+for+the+Diagnostic+and+Statistical+Manual+of+Mental+Disorders%E2%80%94Fifth+Edition%2E+Journal+of+Abnormal+Psycholo)
89. Diyi Yang, Robert E Kraut, Tenbroeck Smith, Elijah Mayfield, and Dan Jurafsky. 2019. Seekers, providers, welcomers, and storytellers: Modeling social roles in online health communities. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–14. [find](https://scholar.google.com/scholar?q=Diyi+Yang%2C+Robert+E+Kraut%2C+Tenbroeck+Smith%2C+Elijah+Mayfield%2C+and+Dan+Jurafsky%2E+2019%2E+Seekers%2C+providers%2C+welcomers%2C+and+storytellers%3A+Modeling+social+roles+in+online+health+communities%2E+In+Proceedings)
90. Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5534–5542. [find](https://scholar.google.com/scholar?q=Mark+Yatskar%2C+Luke+Zettlemoyer%2C+and+Ali+Farhadi%2E+2016%2E+Situation+recognition%3A+Visual+semantic+role+labeling+for+image+understanding%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Patter)
91. Emilio Zagheni and Ingmar Weber. 2015. Demographic research with non-representative internet data. International Journal of Manpower, 36(1):13– 25. [find](https://scholar.google.com/scholar?q=Emilio+Zagheni+and+Ingmar+Weber%2E+2015%2E+Demographic+research+with+non%2Drepresentative+internet+data%2E+International+Journal+of+Manpower%2C+36%281%29%3A13%E2%80%93+25%2E)
92. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979–2989. [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+2017%2E+Men+also+like+shopping%3A+Reducing+gender+bias+amplification+using+corpus%2Dlevel+constraints%2E+In+Proceedings+of+the+2017+C) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2017-men-also-like-shopping-reducing/)
93. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876. [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+2018%2E+Gender+bias+in+coreference+resolution%3A+Evaluation+and+debiasing+methods%2E+arXiv+preprint+arXiv%3A1804%2E06876%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/)
94. Ran Zmigrod, Sebastian J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1651–1661. [find](https://scholar.google.com/scholar?q=Ran+Zmigrod%2C+Sebastian+J%2E+Mielke%2C+Hanna+Wallach%2C+and+Ryan+Cotterell%2E+2019%2E+Counterfactual+data+augmentation+for+mitigating+gender+stereotypes+in+languages+with+rich+morphology%2E+In+Proceedings+of+the+5) [our card](https://md.research.elamur.ai/free/en/papers/zmigrod-2019-counterfactual-data-augmentation-mitigating-gender/)