---
title: "Gender Bias Evaluation in Machine — Sewunetie 2024"
description: "Commercial and open MT systems often assign a default gender to occupation words."
authors: ["Walelign Sewunetie", "Atnafu Tonja", "Tadesse Belay", "Hellina Hailu Nigatu", "Gashaw Gebremeskel", "Zewdie Mossie", "Hussien Seid", "Seid Yimam"]
year: 2024
venue: "Proceedings of the 2nd International Workshop on Gender-Inclusive Translation Technologies"
source: "https://aclanthology.org/2024.gitt-1.1/"
licence: "CC BY-ND 4.0 (stated in the PDF)"
licence_url: "https://creativecommons.org/licenses/by-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sewunetie-2024-gender-bias-evaluation-machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sewunetie-2024-gender-bias-evaluation-machine-translation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Gender Bias Evaluation in Machine Translation for Amharic, Tigrigna, and Afaan Oromoo

# Gender Bias Evaluation in Machine Translation for Amharic, Tigrigna, and Afaan Oromoo

**Authors:** Walelign Sewunetie, Atnafu Tonja, Tadesse Belay, [Hellina Hailu Nigatu](https://md.research.elamur.ai/free/en/authors/hellina-hailu-nigatu/), Gashaw Gebremeskel, Zewdie Mossie, Hussien Seid, Seid Yimam · **Year:** 2024 · **Venue:** Proceedings of the 2nd International Workshop on Gender-Inclusive Translation Technologies · **Licence:** [CC BY-ND 4.0 (stated in the PDF)](https://creativecommons.org/licenses/by-nd/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.gitt-1.1/) · [PDF](https://aclanthology.org/2024.gitt-1.1.pdf)

**Citations:** [5](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Commercial and open MT systems often assign a default gender to occupation words. Low-resource languages have had almost no test material for measuring this. The authors, from the Ethio NLP community, built a gold benchmark for Amharic, Tigrinya and Afaan Oromoo. They gathered 2400 English sentences from existing bias resources such as SimpleGEN and WinoMT, balanced them so that every gendered sentence has an opposite-gender twin, and had native speakers translate the set into all three languages in parallel. Human annotators then judged Google Translate outputs for gender bias using the POTATO annotation tool. They also recorded whether a problem came from the sentence as a whole or from the profession word. Separately, NLLB and Google MT were scored with SacreBleu and chrF++ in both directions. Bias turned out to be pervasive: roughly 93% of English–Afaan Oromoo outputs, 81% of English–Tigrinya outputs and 72.5% of English–Amharic outputs were judged biased. The dataset is public, and the procedure is documented in enough detail for other under-resourced languages to reuse.

## Key points
- The benchmark has 2400 gender-balanced English sentences with parallel Amharic, Tigrinya and Afaan Oromoo versions. It is released on Hugging Face as EthioNLP/Gender-Bias-Evaluation-Dataset.
- Human-judged bias rates in Google Translate output were 92.96% (Eng–Orm), 80.96% (Eng–Tir) and 72.50% (Eng–Amh).
- Annotators labelled each translation as having a sentence-level issue, a profession-word issue, both, or neither.
  - Amharic had the most sentence-level problems (1429).
  - Afaan Oromoo had the most profession-word problems (612).
- NLLB scored poorly when English was the source, with SacreBleu of 3.48 (Amh), 4.85 (Orm) and 3.89 (Tir). It did much better into English, with SacreBleu of roughly 18–22 and chrF++ of 42–51. Google MT beat NLLB on the English-to-target directions.
- Scope is limited to a binary gender view and to bias tied to occupation names.

## Interesting ideas
- Every gendered sentence is paired with a mirror sentence that differs only in gender, so any change in system behaviour can be traced to gender alone. — For Mova (learn): Gender-swapped twins of test texts could serve as a negative control, where the frames and answers should stay identical apart from the swapped attribute.
- The authors list all eight correct Amharic renderings of a gender-neutral English sentence. This treats an underspecified source as licensing a set of valid outputs rather than a single one. — For Mova (world): When a text never states an entity's gender, the world model should keep that attribute explicitly unknown instead of filling in a default from the occupation noun.
- Errors are annotated on two independent axes: whole-sentence translation failure versus mistranslation of the key word. This keeps general quality problems separate from the specific phenomenon being studied. — For Mova (back): Generation and compressor round-trips could be scored separately for overall fluency and for preservation of targeted attributes such as gender and number, so the cause of a failure can be located.
- Running automatic metrics in both translation directions exposed a strong asymmetry: models understand the low-resource language far better than they generate it. — For Mova (back): Evaluating text → graph and graph → text separately would show whether weak round-trip results come from analysis or from generation.
- The paper shows bias surfacing through agreement morphology: definite suffixes on the noun and gendered endings on the verb. A single defaulted choice therefore spreads across several words. — For Mova (frames): Coreference and role filling should not let an occupation stereotype propagate a gender through agreement or pronoun links. Gender should only be linked when the text explicitly supports it.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

5 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 2 not yet.

- [Natural Language Processing for Tigrinya: Current State and Future Directions](https://md.research.elamur.ai/free/en/papers/gaim-2025-natural-language-processing-tigrinya-current/)
- [Yeswa-Stories: A Three-Way Parallel Dataset of Female African Figures in Low-Web Data Languages](https://md.research.elamur.ai/free/en/papers/mamo-2026-yeswa-stories-three-way-parallel/)
- [A Chinese Challenge Set to Assess Gender Bias in Automated Translation](https://md.research.elamur.ai/free/en/papers/xu-2026-chinese-challenge-set-assess-gender/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sewunetie2024gender,
  title = {{Gender Bias Evaluation in Machine Translation for Amharic, Tigrigna, and Afaan Oromoo}},
  author = {Sewunetie, Walelign and Tonja, Atnafu and Belay, Tadesse and Nigatu, Hellina Hailu and Gebremeskel, Gashaw and Mossie, Zewdie and Seid, Hussien and Yimam, Seid},
  booktitle = {Proceedings of the 2nd International Workshop on Gender-Inclusive Translation Technologies},
  year = {2024},
  url = {https://aclanthology.org/2024.gitt-1.1/}
}
```

- [← previous GemmAr: Enhancing LLMs Through Arabic Instruction-Tuning](https://md.research.elamur.ai/free/en/papers/chouikhi-2024-gemmar-enhancing-llms-through-arabic/)
- [next → Generalizable Sarcasm Detection is Just Around the Corner, of Course!](https://md.research.elamur.ai/free/en/papers/jang-2024-generalizable-sarcasm-detection-is-just/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

20 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alhafni, Bashar, Nizar Habash, and Houda Bouamor. 2022. The arabic parallel gender corpus 2.0: Extensions and analyses. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. [find](https://scholar.google.com/scholar?q=Alhafni%2C+Bashar%2C+Nizar+Habash%2C+and+Houda+Bouamor%2E+2022%2E+The+arabic+parallel+gender+corpus+2%2E0%3A+Extensions+and+analyses%2E+In+Proceedings+of+the+International+Conference+on+Language+Resources+and+Evaluat) [our card](https://md.research.elamur.ai/free/en/papers/alhafni-2022-arabic-parallel-gender-corpus-2/)
2. Barocas, Solon, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In 9th Annual conference of the special interest group for computing, information and society. [find](https://scholar.google.com/scholar?q=Barocas%2C+Solon%2C+Kate+Crawford%2C+Aaron+Shapiro%2C+and+Hanna+Wallach%2E+2017%2E+The+problem+with+bias%3A+Allocative+versus+representational+harms+in+machine+learning%2E+In+9th+Annual+conference+of+the+special+inte)
3. Blodgett, Su Lin, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020a. Language (technology) is power: A critical survey of” bias” in nlp. arXiv preprint arXiv:2005.14050. [↗](https://arxiv.org/abs/2005.14050) [our card](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/)
4. Blodgett, Su Lin, Solon Barocas, Hal Daumé Iii, and Hanna Wallach. 2020b. Language (Technology) is Power: A Critical Survey of “Bias” in NLP. ACL Anthology, pages 5454–5476, July. [find](https://scholar.google.com/scholar?q=Blodgett%2C+Su+Lin%2C+Solon+Barocas%2C+Hal+Daum%C3%A9+Iii%2C+and+Hanna+Wallach%2E+2020b%2E+Language+%28Technology%29+is+Power%3A+A+Critical+Survey+of+%E2%80%9CBias%E2%80%9D+in+NLP%2E+ACL+Anthology%2C+pages+5454%E2%80%935476%2C+July%2E) [our card](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/)
5. Cho, Won Ik, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019. On measuring gender bias in translation of gender-neutral pronouns. arXiv preprint arXiv:1905.11684. [↗](https://arxiv.org/abs/1905.11684)
6. Crawford, Kate. 2017. The trouble with bias. [find](https://scholar.google.com/scholar?q=Crawford%2C+Kate%2E+2017%2E+The+trouble+with+bias%2E)
7. Habash, Nizar, Houda Bouamor, and Christine Chung. 2019. Automatic gender identification and reinflection in arabic. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 155–165. [find](https://scholar.google.com/scholar?q=Habash%2C+Nizar%2C+Houda+Bouamor%2C+and+Christine+Chung%2E+2019%2E+Automatic+gender+identification+and+reinflection+in+arabic%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+in+Natural+Language+Processing%2C)
8. Kadaoui, Karima, Samar M Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed Oumar El-Shangiti, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Tarjamat: Evaluation of bard and chatgpt on machine translation of ten arabic varieties. arXiv preprint arXiv:2308.03051. [↗](https://arxiv.org/abs/2308.03051) [our card](https://md.research.elamur.ai/free/en/papers/kadaoui-2023-tarjamat-evaluation-bard-chatgpt-machine/)
9. Pei, Jiaxin, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. 2022. POTATO: The portable text annotation tool. In Che, Wanxiang and Ekaterina Shutova, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 327– 337, Abu Dhabi, UAE, December. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pei%2C+Jiaxin%2C+Aparna+Ananthasubramaniam%2C+Xingyao+Wang%2C+Naitian+Zhou%2C+Apostolos+Dedeloudis%2C+Jackson+Sargent%2C+and+David+Jurgens%2E+2022%2E+POTATO%3A+The+portable+text+annotation+tool%2E+In+Che%2C+Wanxiang+and+Ekat)
10. Popović, Maja. 2017. chrf++: words helping character n-grams. In Proceedings of the second conference on machine translation, pages 612–618. [find](https://scholar.google.com/scholar?q=Popovi%C4%87%2C+Maja%2E+2017%2E+chrf%2B%2B%3A+words+helping+character+n%2Dgrams%2E+In+Proceedings+of+the+second+conference+on+machine+translation%2C+pages+612%E2%80%93618%2E) [our card](https://md.research.elamur.ai/free/en/papers/popovic-2017-chrf-words-helping-character-n/)
11. Post, Matt. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771. [↗](https://arxiv.org/abs/1804.08771) [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
12. Prates, Marcelo OR, Pedro H Avelar, and Luís C Lamb. 2020. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, 32:6363– 6381. [find](https://scholar.google.com/scholar?q=Prates%2C+Marcelo+OR%2C+Pedro+H+Avelar%2C+and+Lu%C3%ADs+C+Lamb%2E+2020%2E+Assessing+gender+bias+in+machine+translation%3A+a+case+study+with+google+translate%2E+Neural+Computing+and+Applications%2C+32%3A6363%E2%80%93+6381%2E) [our card](https://md.research.elamur.ai/free/en/papers/savoldi-2021-gender-bias-machine-translation/)
13. Puduppully, Ratish, Raj Dabre, Ai Ti Aw, and Nancy F Chen. 2023. Decomposed prompting for machine translation between related languages using large language models. arXiv preprint arXiv:2305.13085. [↗](https://arxiv.org/abs/2305.13085)
14. Renduchintala, Adithya, Denise Díaz, Kenneth Heafield, Xian Li, and Mona Diab. 2021. Gender bias amplification during Speed-Quality optimization in Neural Machine Translation. ACL Anthology, pages 99–109, August. [find](https://scholar.google.com/scholar?q=Renduchintala%2C+Adithya%2C+Denise+D%C3%ADaz%2C+Kenneth+Heafield%2C+Xian+Li%2C+and+Mona+Diab%2E+2021%2E+Gender+bias+amplification+during+Speed%2DQuality+optimization+in+Neural+Machine+Translation%2E+ACL+Anthology%2C+pages+99%E2%80%93)
15. Savoldi, Beatrice, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845–874. [find](https://scholar.google.com/scholar?q=Savoldi%2C+Beatrice%2C+Marco+Gaido%2C+Luisa+Bentivogli%2C+Matteo+Negri%2C+and+Marco+Turchi%2E+2021%2E+Gender+bias+in+machine+translation%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+9%3A845%E2%80%93874%2E) [our card](https://md.research.elamur.ai/free/en/papers/savoldi-2021-gender-bias-machine-translation/)
16. Sharma, Shanya, Manan Dey, and Koustuv Sinha. 2022. How sensitive are translation systems to extra contexts? mitigating gender bias in neural machine translation models through relevant contexts. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1968–1984, Abu Dhabi, United Arab Emirates, Dec. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sharma%2C+Shanya%2C+Manan+Dey%2C+and+Koustuv+Sinha%2E+2022%2E+How+sensitive+are+translation+systems+to+extra+contexts%3F+mitigating+gender+bias+in+neural+machine+translation+models+through+relevant+contexts%2E+In+F)
17. Stanovsky, Gabriel, Noah A Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. arXiv preprint arXiv:1906.00591. [↗](https://arxiv.org/abs/1906.00591) [our card](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/)
18. Talat, Zeerak, Aurelie Neveol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla, and Oskar Van Der Wal. 2022. You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings. ACL Anthology, pages 26–41, May. [find](https://scholar.google.com/scholar?q=Talat%2C+Zeerak%2C+Aurelie+Neveol%2C+Stella+Biderman%2C+Miruna+Clinciu%2C+Manan+Dey%2C+Shayne+Longpre%2C+Sasha+Luccioni%2C+Maraim+Masoud%2C+Margaret+Mitchell%2C+Dragomir+Radev%2C+Shanya+Sharma%2C+Arjun+Subramonian%2C+Jaesung+T)
19. Team, NLLB, Marta R. Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejía González, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [↗](https://arxiv.org/abs/2207.04672)
20. Wairagala, Eric Peter, Jonathan Mukiibi, Jeremy Francis Tusubira, Claire Babirye, Joyce Nakatumba-Nabende, Andrew Katumba, and Ivan Ssenkungu. 2022. Gender bias evaluation in luganda-english machine translation. In Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 274–286. 10 Appendix 10.1 Appendix: Human-level Evaluation Guideline Hello everyone, [find](https://scholar.google.com/scholar?q=Wairagala%2C+Eric+Peter%2C+Jonathan+Mukiibi%2C+Jeremy+Francis+Tusubira%2C+Claire+Babirye%2C+Joyce+Nakatumba%2DNabende%2C+Andrew+Katumba%2C+and+Ivan+Ssenkungu%2E+2022%2E+Gender+bias+evaluation+in+luganda%2Denglish+machine+t)