---
title: "Fair Is Better than Sensational — paper by Nissim et al. (2020)"
description: "This squib examines why word-analogy queries have exaggerated, and sometimes hidden, social… 27 references, cited by 32 papers here."
authors: ["Malvina Nissim", "Rik van Noord", "Rob van der Goot"]
year: 2020
venue: "Computational Linguistics"
source: "https://aclanthology.org/2020.cl-2.7/"
doi: "10.1162/coli_a_00379"
licence: "CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)"
licence_url: "https://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nissim-2020-fair-is-better-than-sensational/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nissim-2020-fair-is-better-than-sensational/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor

# Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor

**Authors:** [Malvina Nissim](https://md.research.elamur.ai/free/en/authors/malvina-nissim/), [Rik van Noord](https://md.research.elamur.ai/free/en/authors/rik-van-noord/) and [Rob van der Goot](https://md.research.elamur.ai/free/en/authors/rob-van-der-goot/) · **Year:** 2020 · **Venue:** [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/) ([Computational Linguistics 2020](https://md.research.elamur.ai/free/en/venues/cl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)](https://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.cl-2.7/) · [PDF](https://aclanthology.org/2020.cl-2.7.pdf) · [DOI 10.1162/coli\_a\_00379](https://doi.org/10.1162/coli_a_00379) · [OpenAlex](https://openalex.org/W2945576249)

**Citations:** 88 citations (OpenAlex, 2026-10-14) · [32](https://md.research.elamur.ai/free/en/papers/nissim-2020-fair-is-better-than-sensational/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This squib examines why word-analogy queries have exaggerated, and sometimes hidden, social biases in static word embeddings. The authors agree that embeddings do encode bias. Their argument is that analogies are a weak diagnostic and that common usage made matters worse. The central issue is an implementation detail: standard analogy code never allows the answer D to equal an input term. For man:doctor :: woman:X, the model therefore cannot output doctor, even when that would be the unbiased answer. The authors compare three scoring functions: 3COSADD, 3COSMUL and the pair-based formula of Bolukbasi et al. (2016). They then re-run published biased analogies, mainly those of Manzini et al. (2019). Many reported examples depend on the input-exclusion rule or on hand-picked terms from lower in the ranking, and several cannot be reproduced. In the Bolukbasi method, the threshold δ and the vocabulary cutoff also change the result drastically. Bias claims based on analogies therefore need transparent reporting of query choices, constraints and full top-N outputs.

## Key points
- Typical implementations silently exclude A, B and C as candidate answers. This is a problem for homographs (split/split, Singapore/Singapore), hypernyms, ordered concepts and gender-neutral job titles like doctor.
- The paper distinguishes "factual" analogies, which have a single expected answer (mostly morphosyntactic), from open-ended semantic ones. For the open-ended type, claiming bias requires specifying what an unbiased completion would be.
- Re-running the second version of Manzini et al. showed problems in most examples. In 5 of 6 cases the input word B would rank above the reported answer, and in 3 of 6 the reported term came from the top-10 rather than the top-1. For their third version, 4 of 6 analogies could not be reproduced with any algorithm, even within the top 10.
- For man:doctor :: woman:X under the Bolukbasi formula, the answer depends on the settings. The original setup (δ = 1.0, 50,000-word vocabulary) gives midwife. Other settings return doctors, nurse, gynecologist or woman. Extreme δ values with query words allowed yield doctor (δ ≤ 0.5) or she (δ ≥ 1.5).
- Arbitrary preprocessing affects which answers are reachable. Examples include limiting the vocabulary to the 50k most frequent words, lowercasing, and dropping tokens with punctuation, capitals or more than 20 characters.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Groningen](https://md.research.elamur.ai/free/en/universities/university-of-groningen/) — University, Groningen, Netherlands · 70 works · 5,142 citations
- [IT University of Copenhagen](https://md.research.elamur.ai/free/en/universities/it-university-of-copenhagen/) — University, Copenhagen, Denmark · 27 works · 2,406 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

## Cite

Cite as: Malvina Nissim, Rik van Noord and Rob van der Goot. 2020. Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{nissim2020fair,
  title = {{Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor}},
  author = {Malvina Nissim, Rik {van Noord} and Rob {van der Goot}},
  journal = {Computational Linguistics},
  year = {2020},
  doi = {10.1162/coli\_a\_00379},
  url = {https://doi.org/10.1162/coli_a_00379}
}
```

- [← previous Task-Oriented Dialogue as Dataflow Synthesis](https://md.research.elamur.ai/free/en/papers/andreas-2020-task-oriented-dialogue-as-dataflow/)
- [next → MAGPIE: A Large Corpus of Potentially Idiomatic Expressions](https://md.research.elamur.ai/free/en/papers/haagsma-2020-magpie-large-corpus-potentially-idiomatic/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

27 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bolukbasi, Tolga, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. [Man is to computer programmer as woman is to homemaker? Debiasing word embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/). In Advances in Neural Information Processing Systems 29, pages 4349–4357, Curran Associates, Inc.
2. Caliskan, Aylin, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. [find](https://scholar.google.com/scholar?q=Caliskan%2C+Aylin%2C+Joanna+J%2E+Bryson%2C+and+Arvind+Narayanan%2E+2017%2E+Semantics+derived+automatically+from+language+corpora+contain+human%2Dlike+biases%2E+Science%2C+356%286334%29%3A183%E2%80%93186%2E)
3. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
4. Drozd, Aleksandr, Anna Gladkova, and Satoshi Matsuoka. 2016. Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3519–3530, Osaka. [find](https://scholar.google.com/scholar?q=Drozd%2C+Aleksandr%2C+Anna+Gladkova%2C+and+Satoshi+Matsuoka%2E+2016%2E+Word+embeddings%2C+analogies%2C+and+machine+learning%3A+Beyond+king+%2D+man+%2B+woman+%3D+queen%2E+In+Proceedings+of+COLING+2016%2C+the+26th+International+)
5. Fischer, Olga. 2019. Analogy in Language and Linguistics, Oxford Bibliographies in Linguistics, Oxford University Press. [find](https://scholar.google.com/scholar?q=Fischer%2C+Olga%2E+2019%2E+Analogy+in+Language+and+Linguistics%2C+Oxford+Bibliographies+in+Linguistics%2C+Oxford+University+Press%2E)
6. Garg, Nikhil, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644. [find](https://scholar.google.com/scholar?q=Garg%2C+Nikhil%2C+Londa+Schiebinger%2C+Dan+Jurafsky%2C+and+James+Zou%2E+2018%2E+Word+embeddings+quantify+100+years+of+gender+and+ethnic+stereotypes%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+115%2816%29%3AE3635%E2%80%93)
7. Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumeé III, and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010 V4. [find](https://scholar.google.com/scholar?q=Gebru%2C+Timnit%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daumee%CC%81+III%2C+and+Kate+Crawford%2E+2018%2E+Datasheets+for+datasets%2E+arXiv+preprint+arXiv%3A1803%2E09010+V4%2E)
8. Gladkova, Anna, Aleksandr Drozd, and Satoshi Matsuoka. 2016. [Analogy-based detection of morphological and semantic relations with word embeddings: What works and what doesn’t](https://md.research.elamur.ai/free/en/papers/gladkova-2016-analogy-based-detection-morphological-semantic/). In Proceedings of the NAACL Student Research Workshop, pages 8–15, San Diego, CA.
9. Goldberg, Yoav. 2017. Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies, 10(1):1–309. [find](https://scholar.google.com/scholar?q=Goldberg%2C+Yoav%2E+2017%2E+Neural+network+methods+for+natural+language+processing%2E+Synthesis+Lectures+on+Human+Language+Technologies%2C+10%281%29%3A1%E2%80%93309%2E)
10. Gonen, Hila and Yoav Goldberg. 2019. [Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them](https://md.research.elamur.ai/free/en/papers/gonen-2019-lipstick-pig-debiasing-methods-cover/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
11. Hall Maudslay, Rowan, Hila Gonen, Ryan Cotterell, and Simone Teufel. 2019. [Its all in the name: Mitigating gender bias with name-based counterfactual data substitution](https://md.research.elamur.ai/free/en/papers/maudslay-2019-it-s-all-name-mitigating/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferenc…
12. Hovy, Dirk and Shannon L. Spruit. 2016. [The social impact of natural language processing](https://md.research.elamur.ai/free/en/papers/hovy-2016-social-impact-natural-language-processing/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598.
13. Jha, Akshita and Radhika Mamidi. 2017. When does a compliment become sexist? Analysis and classification of ambivalent sexism using Twitter data. In Proceedings of the Second Workshop on NLP and Computational Social Science, pages 7–16, Vancouver. [find](https://scholar.google.com/scholar?q=Jha%2C+Akshita+and+Radhika+Mamidi%2E+2017%2E+When+does+a+compliment+become+sexist%3F+Analysis+and+classification+of+ambivalent+sexism+using+Twitter+data%2E+In+Proceedings+of+the+Second+Workshop+on+NLP+and+Compu)
14. Kozlowski, Austin C., Matt Taddy, and James A. Evans. 2019. The geometry of culture: Analyzing the meanings of class through word embeddings. American Sociological Review, 84(5):905–949. [find](https://scholar.google.com/scholar?q=Kozlowski%2C+Austin+C%2E%2C+Matt+Taddy%2C+and+James+A%2E+Evans%2E+2019%2E+The+geometry+of+culture%3A+Analyzing+the+meanings+of+class+through+word+embeddings%2E+American+Sociological+Review%2C+84%285%29%3A905%E2%80%93949%2E)
15. Levy, Omer and Yoav Goldberg. 2014. [Linguistic regularities in sparse and explicit word representations](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/). In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 171–180, Ann Arbor, MI.
16. Linzen, Tal. 2016. Issues in evaluating semantic spaces using word analogies. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 13–18, Berlin. [find](https://scholar.google.com/scholar?q=Linzen%2C+Tal%2E+2016%2E+Issues+in+evaluating+semantic+spaces+using+word+analogies%2E+In+Proceedings+of+the+1st+Workshop+on+Evaluating+Vector%2DSpace+Representations+for+NLP%2C+pages+13%E2%80%9318%2C+Berlin%2E)
17. Manzini, Thomas, Yao Chong Lim, Yulia Tsvetkov, and Alan W. Black. 2019b. [Black is to criminal as Caucasian is to police: Detecting and removing multiclass bias in word embeddings](https://md.research.elamur.ai/free/en/papers/manzini-2019-black-is-criminal-as-caucasian/). arXiv preprint arXiv:1904.04047 V2.
18. Manzini, Thomas, Yao Chong Lim, Yulia Tsvetkov, and Alan W. Black. 2019c. [Black is to criminal as Caucasian is to police: Detecting and removing multiclass bias in word embeddings](https://md.research.elamur.ai/free/en/papers/manzini-2019-black-is-criminal-as-caucasian/). arXiv preprint arXiv:1904.04047 V3.
19. Manzini, Thomas, Lim Yao Chong, Alan W. Black, and Yulia Tsvetkov. 2019a. [Black is to criminal as Caucasian is to police: Detecting and removing multiclass bias in word embeddings](https://md.research.elamur.ai/free/en/papers/manzini-2019-black-is-criminal-as-caucasian/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: H…
20. McQuillan, Dan. 2018. People’s councils for ethical machine learning. Social Media+ Society, 4(2):1–10. [find](https://scholar.google.com/scholar?q=McQuillan%2C+Dan%2E+2018%2E+People%E2%80%99s+councils+for+ethical+machine+learning%2E+Social+Media%2B+Society%2C+4%282%29%3A1%E2%80%9310%2E)
21. SAGE Publications, London, England. Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). In Proceedings of Workshop at ICLR, Scottsdale, AZ.
22. Mohammad, Saif, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. [SemEval-2018 Task 1: Affect in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2018-semeval-2018-task-1-affect/). In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, LA.
23. Peters, Matthew, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo…
24. Rogers, Anna, Aleksandr Drozd, and Bofang Li. 2017. The (too many) problems of analogical reasoning with word vectors. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 135–148, Vancouver. [find](https://scholar.google.com/scholar?q=Rogers%2C+Anna%2C+Aleksandr+Drozd%2C+and+Bofang+Li%2E+2017%2E+The+%28too+many%29+problems+of+analogical+reasoning+with+word+vectors%2E+In+Proceedings+of+the+6th+Joint+Conference+on+Lexical+and+Computational+Semantics)
25. Schluter, Natalie. 2018. The word analogy testing caveat. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 242–246, New Orleans, LA. [find](https://scholar.google.com/scholar?q=Schluter%2C+Natalie%2E+2018%2E+The+word+analogy+testing+caveat%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technolog)
26. Turney, Peter D. 2012. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research, 44:533–585. [find](https://scholar.google.com/scholar?q=Turney%2C+Peter+D%2E+2012%2E+Domain+and+function%3A+A+dual%2Dspace+model+of+semantic+relations+and+compositions%2E+Journal+of+Artificial+Intelligence+Research%2C+44%3A533%E2%80%93585%2E)
27. Zou, James and Londa Schiebinger. 2018. AI can be sexist and racist—it’s time to make it fair. Nature, 559(7714):324. [find](https://scholar.google.com/scholar?q=Zou%2C+James+and+Londa+Schiebinger%2E+2018%2E+AI+can+be+sexist+and+racist%E2%80%94it%E2%80%99s+time+to+make+it+fair%2E+Nature%2C+559%287714%29%3A324%2E)

*Updated 30 October 2026*