---
title: "Mapping the Multilingual Margins — paper by Câmara et al. (2022)"
description: "The paper asks whether sentiment and emotion systems treat people differently depending on gender… 35 references, cited by 14 papers here."
authors: ["António Câmara", "Nina Taneja", "Tamjeed Azad", "Emily Allaway", "Richard Zemel"]
year: 2022
venue: "Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion"
source: "https://aclanthology.org/2022.ltedi-1.11/"
doi: "10.18653/v1/2022.ltedi-1.11"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/camara-2022-mapping-multilingual-margins-intersectional-biases/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/camara-2022-mapping-multilingual-margins-intersectional-biases/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Mapping the Multilingual Margins: Intersectional Biases of Sentiment Analysis Systems in English, Spanish, and Arabic

# Mapping the Multilingual Margins: Intersectional Biases of Sentiment Analysis Systems in English, Spanish, and Arabic

**Authors:** António Câmara, Nina Taneja, Tamjeed Azad, [Emily Allaway](https://md.research.elamur.ai/free/en/authors/emily-allaway/) and [Richard Zemel](https://md.research.elamur.ai/free/en/authors/richard-s-zemel/) · **Year:** 2022 · **Venue:** [Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion](https://md.research.elamur.ai/free/en/venues/ltedi/) ([Workshop on Language Technology for Equality, Diversity, Inclusion 2022](https://md.research.elamur.ai/free/en/venues/ltedi/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.ltedi-1.11/) · [PDF](https://aclanthology.org/2022.ltedi-1.11.pdf) · [DOI 10.18653/v1/2022.ltedi-1.11](https://doi.org/10.18653/v1/2022.ltedi-1.11) · [OpenAlex](https://openalex.org/W4285163830)

**Citations:** 19 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/camara-2022-mapping-multilingual-margins-intersectional-biases/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether sentiment and emotion systems treat people differently depending on gender, race or ethnicity, and on combinations of these. It also asks whether this holds beyond English. Earlier work mostly examined one identity axis at a time, mainly in embeddings and mainly in English. The authors extend the Equity Evaluation Corpus (EEC) idea of Kiritchenko and Mohammad with four new template-based test sets. Two of them, English and Spanish, contrast Latino and Anglo names. The other two, English and Arabic, contrast Arab and Anglo names. The templates were translated by volunteers. Five emotion-regression models (BERT+, mBERT, XLM-RoBERTa, SVM-tfidf, fastText) are run on these sentences. A Beta regression then models each predicted intensity using separate gender and race/ethnicity terms plus an interaction term, and each coefficient is tested for significance. Many model–emotion–language combinations show significant single-axis bias, intersectional bias, or both. The patterns vary widely across settings. This suggests bias audits need to be done per language and per identity combination.

## Key points
- Each EEC follows the original structure of eleven templates with a [person] slot. The slot is filled with group-typical given names or gendered noun phrases such as "my mother." Seven templates also contain an emotion state word or emotion situation word.
- The original English EEC (8,640 sentences, Black/white names) is complemented by four new corpora: en/es Latino–Anglo and en/ar Arab–Anglo.
- Bias is measured with Beta regression coefficients for race/ethnicity, gender and their intersection. Results are reported separately for anger, joy, fear and sadness, with significance marked at p ≤ 0.01/0.05/0.10.
- In the Anglo–Arab results, the largest race/ethnicity effects are around 0.06 on the intensity scale. Many other coefficients are near zero but still significant.
- The authors list several limitations:
  - The framework uses binary gender and single-race categories.
  - Treating the majority identity as the baseline implicitly centers it.
  - No fixed sentence set can capture bias fully.

## Interesting ideas
- **Minimal-pair templates:** the sentences differ only in the identity slot, so any change in output can be attributed to that slot alone.
- **Interaction term:** fitting an explicit term for the combination, beside the per-axis terms, reveals effects that are more than additive. Effects visible only for a combination would stay hidden if each axis were tested alone.
- **Bounded-output regression:** Beta regression suits outputs confined to the 0–1 range, and significance testing separates real shifts from noise.
- **Culturally adapted parallel templates:** the templates and emotion words stay fixed across languages, while the names are chosen by humans to fit each target culture.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Build negative-control suites where swapping a name or pronoun must leave the UD tree, frames and final answer unchanged. Any difference flags a spurious dependence.
- **I2** (learn): Regress parser or reader errors on construction features (e.g., passive, relative clause) with interaction terms. This would find failures that appear only when constructions co-occur.
- **I3** (answer): Model answer-confidence scores the same way when comparing reader variants, rather than eyeballing mean differences.
- **I4** (back): Use fixed parallel templates with locally appropriate entities to test whether translation through the meta-world preserves frames and emotional content.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 241 works · 54,053 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (4): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: António Câmara, Nina Taneja, Tamjeed Azad, Emily Allaway and Richard Zemel. 2022. Mapping the Multilingual Margins: Intersectional Biases of Sentiment Analysis Systems in English, Spanish, and Arabic. Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{camara2022mapping,
  title = {{Mapping the Multilingual Margins: Intersectional Biases of Sentiment Analysis Systems in English, Spanish, and Arabic}},
  author = {António Câmara, Nina Taneja, Tamjeed Azad, Emily Allaway and Richard Zemel},
  booktitle = {Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion},
  year = {2022},
  doi = {10.18653/v1/2022.ltedi-1.11},
  url = {https://doi.org/10.18653/v1/2022.ltedi-1.11}
}
```

- [← previous Compositional Evaluation on Japanese Textual Entailment and Similarity](https://md.research.elamur.ai/free/en/papers/yanaka-2022-compositional-evaluation-japanese-textual-entailment/)
- [next → ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages](https://md.research.elamur.ai/free/en/papers/chai-2022-ernie-code-english-centric/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

35 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Artem Domnich and Gholamreza Anbarjafari. 2021. Responsible ai: Gender bias assessment in emotion recognition. arXiv preprint arXiv:2103.11436. [↗](https://arxiv.org/abs/2103.11436)
2. Kimberle Crenshaw. 1990. Mapping the margins: Intersectionality, identity politics, and violence against women of color. Stan. L. Rev., 43:1241. [↗](https://doi.org/10.2307/1229039)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
4. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 1999. Machine bias: There’s software used across the country to predict future criminals. and it’s biased against blacks. [find](https://scholar.google.com/scholar?q=Julia+Angwin%2C+Jeff+Larson%2C+Surya+Mattu%2C+and+Lauren+Kirchner%2E+1999%2E+Machine+bias%3A+There%E2%80%99s+software+used+across+the+country+to+predict+future+criminals%2E+and+it%E2%80%99s+biased+against+blacks%2E)
5. Hillary Anger Elfenbein and Nalini Ambady. 2002. On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychological bulletin, 128(2):203. [↗](https://doi.org/10.1037/0033-2909.128.2.203)
6. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery. [↗](https://doi.org/10.1145/3442188.3445922)
7. Silvia Ferrari and Francisco Cribari-Neto. 2004. Beta regression for modelling rates and proportions. Journal of applied statistics, 31(7):799–815. [↗](https://doi.org/10.1080/0266476042000214501)
8. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635– E3644. [↗](https://doi.org/10.1073/pnas.1720347115)
9. Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1–4. Springer. [↗](https://doi.org/10.1007/978-3-642-00296-0_5)
10. Hila Gonen and Yoav Goldberg. 2019. [Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them](https://md.research.elamur.ai/free/en/papers/gonen-2019-lipstick-pig-debiasing-methods-cover/). arXiv preprint arXiv:1903.03862. [↗](https://arxiv.org/abs/1903.03862)
11. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the association for computational linguistics, 5:135–146. [↗](https://doi.org/10.1162/tacl_a_00051)
12. Anthony G Greenwald, Debbie E McGhee, and Jordan LK Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology, 74(6):1464. [↗](https://doi.org/10.1037/0022-3514.74.6.1464)
13. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. [Man is to computer programmer as woman is to homemaker? debiasing word embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/). Advances in neural information processing systems, 29. [↗](https://doi.org/10.48550/arxiv.1607.06520)
14. Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 122–133. [find](https://scholar.google.com/scholar?q=Wei+Guo+and+Aylin+Caliskan%2E+2021%2E+Detecting+emergent+intersectional+biases%3A+Contextualized+word+embeddings+contain+a+distribution+of+human%2Dlike+biases%2E+In+Proceedings+of+the+2021+AAAI%2FACM+Conference+o)
15. Lisa Bowleg. 2008. When black+ lesbian+ woman̸ = black lesbian woman: The methodological challenges of qualitative and quantitative intersectionality research. Sex roles, 59(5):312–325. [↗](https://doi.org/10.1007/s11199-008-9400-z)
16. Aurélie Herbelot, Eva Von Redecker, and Johanna Müller. 2012. Distributional techniques for philosophical enquiry. In Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 45–54. [find](https://scholar.google.com/scholar?q=Aur%C3%A9lie+Herbelot%2C+Eva+Von+Redecker%2C+and+Johanna+M%C3%BCller%2E+2012%2E+Distributional+techniques+for+philosophical+enquiry%2E+In+Proceedings+of+the+6th+Workshop+on+Language+Technology+for+Cultural+Heritage%2C+Soci)
17. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91. PMLR. [find](https://scholar.google.com/scholar?q=Joy+Buolamwini+and+Timnit+Gebru%2E+2018%2E+Gender+shades%3A+Intersectional+accuracy+disparities+in+commercial+gender+classification%2E+In+Conference+on+fairness%2C+accountability+and+transparency%2C+pages+77%E2%80%9391%2E+)
18. Svetlana Kiritchenko and Saif M. Mohammad. 2018. [Examining gender and race bias in two hundred sentiment analysis systems](https://md.research.elamur.ai/free/en/papers/kiritchenko-2018-examining-gender-race-bias-two/). CoRR, abs/1805.04508. [↗](https://arxiv.org/abs/1805.04508)
19. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. [↗](https://doi.org/10.1126/science.aal4230)
20. José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. 2020. Spanish pre-trained bert model and evaluation data. In PML4DC at ICLR 2020. [find](https://scholar.google.com/scholar?q=Jos%C3%A9+Ca%C3%B1ete%2C+Gabriel+Chaperon%2C+Rodrigo+Fuentes%2C+Jou%2DHui+Ho%2C+Hojin+Kang%2C+and+Jorge+P%C3%A9rez%2E+2020%2E+Spanish+pre%2Dtrained+bert+model+and+evaluation+data%2E+In+PML4DC+at+ICLR+2020%2E)
21. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. [Measuring bias in contextualized word representations](https://md.research.elamur.ai/free/en/papers/kurita-2019-measuring-bias-contextualized-word-representations/). In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics.
22. Patricia Hill Collins. 2004. Black sexual politics: African Americans, gender, and the new racism. Routledge. [find](https://scholar.google.com/scholar?q=Patricia+Hill+Collins%2E+2004%2E+Black+sexual+politics%3A+African+Americans%2C+gender%2C+and+the+new+racism%2E+Routledge%2E)
23. Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and Alan W Black. 2019. [Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings](https://md.research.elamur.ai/free/en/papers/manzini-2019-black-is-criminal-as-caucasian/). arXiv preprint arXiv:1904.04047. [↗](https://arxiv.org/abs/1904.04047)
24. Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical gender. arXiv preprint arXiv:1909.02224. [↗](https://arxiv.org/abs/1909.02224)
25. Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. 2019. [On measuring social biases in sentence encoders](https://md.research.elamur.ai/free/en/papers/may-2019-measuring-social-biases-sentence-encoders/). arXiv preprint arXiv:1903.10561. [↗](https://arxiv.org/abs/1903.10561)
26. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. [SemEval-2018 task 1: Affect in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2018-semeval-2018-task-1-affect/). In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana. Association for Computational Linguistics.
27. Jaihyun Park, Karla Felix, and Grace Lee. 2007. Implicit attitudes toward arab-muslims and the moderating effects of social information. Basic and Applied Social Psychology, 29(1):35–45. [↗](https://doi.org/10.1080/01973530701330942)
28. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in neural information processing systems, 32. [↗](https://doi.org/10.48550/arxiv.1912.01703)
29. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. [↗](https://doi.org/10.48550/arxiv.1201.0490)
30. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. [KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/). In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
31. Yi Chern Tan and L Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. Advances in Neural Information Processing Systems, 32. [↗](https://doi.org/10.48550/arxiv.1911.01485)
32. J Celeste Walley-Jean. 2009. Debunking the myth of the “angry black woman”: An exploration of anger in young african american women. Black Women, Gender & Families, 3(2):68–86. [find](https://scholar.google.com/scholar?q=J+Celeste+Walley%2DJean%2E+2009%2E+Debunking+the+myth+of+the+%E2%80%9Cangry+black+woman%E2%80%9D%3A+An+exploration+of+anger+in+young+african+american+women%2E+Black+Women%2C+Gender+%26+Families%2C+3%282%29%3A68%E2%80%9386%2E)
33. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. [Huggingface’s transformers: State-ofthe-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771. [↗](https://arxiv.org/abs/1910.03771)
34. Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender bias in multilingual embeddings and cross-lingual transfer. arXiv preprint arXiv:2005.00699. [↗](https://arxiv.org/abs/2005.00699)
35. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. [Gender bias in contextualized word embeddings](https://md.research.elamur.ai/free/en/papers/zhao-2019-gender-bias-contextualized-word-embeddings/). arXiv preprint arXiv:1904.03310. Black Female Male Ebony Alonzo Jasmine Alphonse Lakisha Darnell Latisha Jamel Latoya Jerome Nichelle Lamar Shaniqua Leroy Shereen Malik Tanisha Terrence Tia Torrance White Female Male Amanda Adam Betsy Alan Courtney Andrew Ellen Frank Heather Harry Katie Jack Kristin Josh Melanie Justin Nancy Roger Stephanie Ryan Anglo Female Male Ellen Adam Emily Andrew Heather Chip Rachel Frank Katie Jonathan Betsy Justin Nancy Harry Amanda Matthew Megan Roger Stephanie Stephen *Table 6: Given names used in original EEC* Anglo Female Jessica Ashley Emily Sarah Samantha Amanda Brittany Elizabeth Taylor Megan Male Michael Christopher Matthew Joshua Jacob Nicholas Andrew Daniel Tyler Joseph Anglo Female Male  á ÊK @ ÐX @ ú ÎÖß @ ðPY K @ QJ ë I. ƒ  ÉJ ƒ@  P ½ K@Q ¯ ø ú GA¿ àAKA Kð ú æ„J K. á J‚k. ú æ„ A K ø PAë @Y KAÓ @ ñJ KAÓ àA ªJ Ó Qk. ðP  á ®J ƒ  ú GA ®J ƒ [↗](https://arxiv.org/abs/1904.03310)

*Updated 1 November 2026*