---
title: "On the Intrinsic and Extrinsic Fairness Evaluation Metrics for"
description: "Fairness in NLP is measured in two ways. 26 references, cited by 57 papers here. Intrinsic metrics probe a pre-trained language model directly."
authors: ["Yang Trista Cao", "Yada Pruksachatkun", "Kai-Wei Chang", "Rahul Gupta", "Varun Kumar", "Jwala Dhamala", "Aram Galstyan"]
year: 2022
venue: "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2022.acl-short.62/"
doi: "10.18653/v1/2022.acl-short.62"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cao-2022-intrinsic-extrinsic-fairness-evaluation-metrics/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cao-2022-intrinsic-extrinsic-fairness-evaluation-metrics/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations

# On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations

**Authors:** [Yang Trista Cao](https://md.research.elamur.ai/free/en/authors/yang-trista-cao/), [Yada Pruksachatkun](https://md.research.elamur.ai/free/en/authors/yada-pruksachatkun/), [Kai-Wei Chang](https://md.research.elamur.ai/free/en/authors/kai-wei-chang/), [Rahul Gupta](https://md.research.elamur.ai/free/en/authors/rahul-gupta/), [Varun Kumar](https://md.research.elamur.ai/free/en/authors/varun-kumar/), [Jwala Dhamala](https://md.research.elamur.ai/free/en/authors/jwala-dhamala/) and [Aram Galstyan](https://md.research.elamur.ai/free/en/authors/aram-galstyan/) · **Year:** 2022 · **Venue:** [Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2022](https://md.research.elamur.ai/free/en/venues/acl/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.acl-short.62/) · [PDF](https://aclanthology.org/2022.acl-short.62.pdf) · [DOI 10.18653/v1/2022.acl-short.62](https://doi.org/10.18653/v1/2022.acl-short.62) · [OpenAlex](https://openalex.org/W4285294416)

**Citations:** 43 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/cao-2022-intrinsic-extrinsic-fairness-evaluation-metrics/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Fairness in NLP is measured in two ways. Intrinsic metrics probe a pre-trained language model directly. Extrinsic metrics look at group disparities in the outputs of a downstream system. People often assume a model that looks fair intrinsically will also behave fairly downstream, but this had not been checked broadly for contextualized models.

The authors tested this assumption on 19 pre-trained models: ALBERT, BERT, RoBERTa, DistilRoBERTa, GPT2, DistilGPT2, EleutherAI/gpt-neo and XLNet. They covered toxicity, sentiment and stereotype bias across gender, race and religion. Intrinsic scores (CEAT, ILPS, StereoSet) were compared with extrinsic ones from two task types:
- classification: Jigsaw toxicity, HateXPlain;
- generation: BOLD-based sentiment, toxicity, regard and stereotype.

For each metric, bias is the variance of scores across protected groups. These variances are then correlated across models.

The correlations are mostly weak and inconsistent in sign. Aligning protected groups or data sources, or removing loaded prompts, improves them only slightly. Extrinsic scores also shift with configuration choices such as decoding temperature. The study therefore warns against treating a less biased upstream model as proof of a fairer downstream system.

## Key points
- The extrinsic B-STEREO metric relies on a BERT classifier fine-tuned on StereoSet, CrowS-Pairs and Social Bias Frames (validation F1 0.80). ALBERT, DistilRoBERTa and XLNet were left out of the BOLD generation metrics.
- Restricting race to overlapping groups (White, Black, Hispanic, Asian) raised two correlations, neither significantly:
  - StereoSet vs B-REGARD: −0.08 to 0.19 (p 0.56);
  - StereoSet vs B-STEREO: −0.18 to 0.08 (p 0.80).
- Some BOLD religion prompts were themselves toxic or stereotyped. Filtering out the polarized ones moved CEAT vs B-SENT to 0.11 and StereoSet vs B-STEREO to 0.10.
- Raising GPT-2 sampling temperature from 0.5 to 1.0 increased the share of negative-sentiment generations from 4.6% to 15.6%. Word swaps and deletions in 500 generations changed classifier outputs moderately.
- Code is published at github.com/pruksmhc/fairness-metrics-correlations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 104 works · 33,876 citations
- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 83 works · 23,784 citations
- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 70 works · 39,977 citations
- [University of California, Los Angeles](https://md.research.elamur.ai/free/en/universities/university-of-california-los-angeles/) — University, Los Angeles, United States · 54 works · 53,688 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 221 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works

## Cite

Cite as: Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala and Aram Galstyan. 2022. On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cao2022intrinsic,
  title = {{On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations}},
  author = {Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala and Aram Galstyan},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.62},
  url = {https://doi.org/10.18653/v1/2022.acl-short.62}
}
```

- [← previous Time Waits for No One! Analysis and Challenges of Temporal Misalignment](https://md.research.elamur.ai/free/en/papers/luu-2022-time-waits-no-one-analysis/)
- [next → Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://md.research.elamur.ai/free/en/papers/bulian-2022-tomayto-tomahto-beyond-token-level/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

26 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Robert Adragna, Elliot Creager, David Madras, and Richard S. Zemel. 2020. Fairness and robustness in invariant learning: A case study in toxicity classification. ArXiv, abs/2011.06485. — [arXiv:2011.06485](https://arxiv.org/abs/2011.06485) [find](https://scholar.google.com/scholar?q=Robert+Adragna%2C+Elliot+Creager%2C+David+Madras%2C+and+Richard+S%2E+Zemel%2E+2020%2E+Fairness+and+robustness+in+invariant+learning%3A+A+case+study+in+toxicity+classification%2E+ArXiv%2C+abs%2F2011%2E06485%2E+%E2%80%94+%5BarXiv%3A2011%2E0)
2. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The Problem With Bias: Allocative Versus Representational Harms in Machine Learning. In Proceedings of SIGCIS. [find](https://scholar.google.com/scholar?q=Solon+Barocas%2C+Kate+Crawford%2C+Aaron+Shapiro%2C+and+Hanna+Wallach%2E+2017%2E+The+Problem+With+Bias%3A+Allocative+Versus+Representational+Harms+in+Machine+Learning%2E+In+Proceedings+of+SIGCIS%2E)
3. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Leo+Gao%2C+Phil+Wang%2C+Connor+Leahy%2C+and+Stella+Biderman%2E+2021%2E+GPT%2DNeo%3A+Large+Scale+Autoregressive+Language+Modeling+with+Mesh%2DTensorflow%2E)
4. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna M. Wallach. 2021. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In ACL/IJCNLP. [find](https://scholar.google.com/scholar?q=Su+Lin+Blodgett%2C+Gilsinia+Lopez%2C+Alexandra+Olteanu%2C+Robert+Sim%2C+and+Hanna+M%2E+Wallach%2E+2021%2E+Stereotyping+norwegian+salmon%3A+An+inventory+of+pitfalls+in+fairness+benchmark+datasets%2E+In+ACL%2FIJCNLP%2E)
5. Daniel Borkan, Lucas Dixon, Jeffrey Scott Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. Companion Proceedings of The 2019 World Wide Web Conference. with application bias. In Proceedings of the 59th Annual Meet… [find](https://scholar.google.com/scholar?q=Daniel+Borkan%2C+Lucas+Dixon%2C+Jeffrey+Scott+Sorensen%2C+Nithum+Thain%2C+and+Lucy+Vasserman%2E+2019%2E+Nuanced+metrics+for+measuring+unintended+bias+with+real+data+for+text+classification%2E+Companion+Proceedings+)
6. Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, page 122–133. ArXiv: 2006.03955. — [arXiv:2006.03955](https://arxiv.org/abs/2006.03955) [find](https://scholar.google.com/scholar?q=Wei+Guo+and+Aylin+Caliskan%2E+2021%2E+Detecting+emergent+intersectional+biases%3A+Contextualized+word+embeddings+contain+a+distribution+of+human%2Dlike+biases%2E+Proceedings+of+the+2021+AAAI%2FACM+Conference+on+A)
7. Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack W. Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020. Reducing sentiment bias in language models via counterfactual evaluation. ArXiv, abs/1911.03064. [find](https://scholar.google.com/scholar?q=Po%2DSen+Huang%2C+Huan+Zhang%2C+Ray+Jiang%2C+Robert+Stanforth%2C+Johannes+Welbl%2C+Jack+W%2E+Rae%2C+Vishal+Maini%2C+Dani+Yogatama%2C+and+Pushmeet+Kohli%2E+2020%2E+Reducing+sentiment+bias+in+language+models+via+counterfactual)
8. Conversational AI Jigsaw. 2019. Jigsaw unintended bias in toxicity classification. Kaggle. [find](https://scholar.google.com/scholar?q=Conversational+AI+Jigsaw%2E+2019%2E+Jigsaw+unintended+bias+in+toxicity+classification%2E+Kaggle%2E)
9. Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren. 2021. On transferability of bias mitigation effects in language model fine-tuning. arXiv:2010.12864 [cs, stat]. ArXiv: 2010.12864. [find](https://scholar.google.com/scholar?q=Xisen+Jin%2C+Francesco+Barbieri%2C+Brendan+Kennedy%2C+Aida+Mostafazadeh+Davani%2C+Leonardo+Neves%2C+and+Xiang+Ren%2E+2021%2E+On+transferability+of+bias+mitigation+effects+in+language+model+fine%2Dtuning%2E+arXiv%3A2010%2E1)
10. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. [Measuring bias in contextualized word representations](https://md.research.elamur.ai/free/en/papers/kurita-2019-measuring-bias-contextualized-word-representations/). In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, page 166–172. Association for Computational Linguistics.
11. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+ArXiv%2C+abs%2F1909%2E11942%2E)
12. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. Proceedings of the Conference on Fairness,… — [arXiv:1901.09451](https://arxiv.org/abs/1901.09451) [find](https://scholar.google.com/scholar?q=Maria+De%2DArteaga%2C+Alexey+Romanov%2C+Hanna+Wallach%2C+Jennifer+Chayes%2C+Christian+Borgs%2C+Alexandra+Chouldechova%2C+Sahin+Geyik%2C+Krishnaram+Kenthapadi%2C+and+Adam+Tauman+Kalai%2E+2019%2E+Bias+in+bios%3A+A+case+study+o)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL.
14. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, page 862–872. Ar… — [arXiv:2101.11718](https://arxiv.org/abs/2101.11718) [find](https://scholar.google.com/scholar?q=Jwala+Dhamala%2C+Tony+Sun%2C+Varun+Kumar%2C+Satyapriya+Krishna%2C+Yada+Pruksachatkun%2C+Kai%2DWei+Chang%2C+and+Rahul+Gupta%2E+2021%2E+Bold%3A+Dataset+and+metrics+for+measuring+biases+in+open%2Dended+language+generation%2E+Pr)
15. Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2020. Hatexplain: A benchmark dataset for explainable ha… — [arXiv:2012.10289](https://arxiv.org/abs/2012.10289) [find](https://scholar.google.com/scholar?q=Seraphina+Goldfarb%2DTarrant%2C+Rebecca+Marchant%2C+Ricardo+Mu%C3%B1oz+S%C3%A1nchez%2C+Mugdha+Pandya%2C+and+Adam+Lopez%2E+2021%2E+Intrinsic+bias+metrics+do+not+correlate+Binny+Mathew%2C+Punyajoy+Saha%2C+Seid+Muhie+Yimam%2C+Chris+B)
16. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. [Stereoset: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/). arXiv:2004.09456 [cs]. ArXiv: 2004.09456.
17. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. [Crows-pairs: A challenge dataset for measuring social biases in masked language models](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/). arXiv:2010.00133 [cs]. ArXiv: 2010.00133.
18. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
19. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+ArXiv%2C+abs%2F1910%2E01108%2E)
20. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. [Social bias frames: Reasoning about social and power implications of language](https://md.research.elamur.ai/free/en/papers/sap-2020-social-bias-frames-reasoning-about/). arXiv:1911.03891 [cs]. ArXiv: 1911.03891.
21. Anna Sotnikova, Yang Trista Cao, Hal Daumé III, and Rachel Rudinger. 2021. Analyzing stereotypes in generative text inference tasks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4052–4065, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anna+Sotnikova%2C+Yang+Trista+Cao%2C+Hal+Daum%C3%A9+III%2C+and+Rachel+Rudinger%2E+2021%2E+Analyzing+stereotypes+in+generative+text+inference+tasks%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DI)
22. Alex Wang and Kyunghyun Cho. 2019. [BERT has a mouth, and it must speak: BERT as a Markov random field language model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/). In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30–36, Minneapolis, Minnesota. Association for Computational Linguistics.
23. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In NeurIPS.
24. Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Conghui Zhu, and Tiejun Zhao. 2020. Demographics should not be the reason of toxicity: Mitigating discrimination in text classifications with instance weighting. In ACL. [find](https://scholar.google.com/scholar?q=Guanhua+Zhang%2C+Bing+Bai%2C+Junqi+Zhang%2C+Kun+Bai%2C+Conghui+Zhu%2C+and+Tiejun+Zhao%2E+2020%2E+Demographics+should+not+be+the+reason+of+toxicity%3A+Mitigating+discrimination+in+text+classifications+with+instance+we)
25. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. [Gender bias in contextualized word embeddings](https://md.research.elamur.ai/free/en/papers/zhao-2019-gender-bias-contextualized-word-embeddings/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1…
26. Warning: This appendix contains examples which are offensive in nature. [find](https://scholar.google.com/scholar?q=Warning%3A+This+appendix+contains+examples+which+are+offensive+in+nature%2E)

*Updated 24 October 2026*