---
title: "A Chinese Challenge Set to Assess Gender Bias in — Xu 2026"
description: "Most studies of gender bias in machine translation use English as the source language and Indo-European target languages."
authors: ["Xiaolan Xu", "Sara Mendes", "Yu-Yin Hsu"]
year: 2026
venue: "Proceedings of the 4th Workshop on Gender-Inclusive Translation Technologies (GITT 2026)"
source: "https://aclanthology.org/2026.gitt-1.8/"
licence: "CC BY-ND 4.0 (stated in the PDF)"
licence_url: "https://creativecommons.org/licenses/by-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/xu-2026-chinese-challenge-set-assess-gender/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/xu-2026-chinese-challenge-set-assess-gender/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/) / A Chinese Challenge Set to Assess Gender Bias in Automated Translation

# A Chinese Challenge Set to Assess Gender Bias in Automated Translation

**Authors:** Xiaolan Xu, Sara Mendes, Yu-Yin Hsu · **Year:** 2026 · **Venue:** Proceedings of the 4th Workshop on Gender-Inclusive Translation Technologies (GITT 2026) · **Licence:** [CC BY-ND 4.0 (stated in the PDF)](https://creativecommons.org/licenses/by-nd/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.gitt-1.8/) · [PDF](https://aclanthology.org/2026.gitt-1.8.pdf)

## Summary
Most studies of gender bias in machine translation use English as the source language and Indo-European target languages. This paper examines Chinese→Portuguese. In Chinese, gender appears only in written third-person pronouns (他/她). In Portuguese, gender must be marked on articles, adjectives and participles. The authors built a controlled challenge set by inserting 45 occupation nouns into 11 sentence templates, which yields 495 Chinese sentences. The templates vary the gender cue: no cue, an explicit 男/女 ("male/female") modifier before the noun, or a coreferential pronoun placed before or after the noun in simple or complex sentences. Two commercial NMT engines and six LLMs translated the set. Gender in the occupation noun phrase was annotated, and the results were analysed with a mixed-effects model.

Explicit modifiers were always rendered correctly. Without cues, systems mostly chose masculine forms. Complex pronoun conditions exposed a strong failure to carry feminine cues, especially in the NMT engines, while LLMs treated masculine and feminine cues more evenly. The work offers a reusable recipe for bias testing from source languages without grammatical gender, which matters in practice in places such as Macao.

## Key points
- The resource contains 495 sentences (45 occupations × 11 templates). The 8 systems tested were Google Translate, DeepL and six LLMs accessed via OpenRouter, including GPT, Claude and Qwen, for 3,960 queries in total. Data and code are available on GitHub.
- Accuracy varied by cue type:
  - Explicit 男/女 modifiers gave 100% accuracy for every system.
  - Gender-indeterminate sentences were translated as masculine in 77.8–93.3% of cases.
  - Feminine accuracy in complex sentences fell to 24.4% for DeepL in the "ante cpx f" condition.
- Mixed-effects odds ratios for correct gender:
  - Complex syntax: about 0.16, the strongest predictor of failure.
  - Feminine target: 0.24.
  - Pronoun preceding the noun: 2.53.
  - LLM vs. NMT: 2.17.
  - Masculine-stereotyped occupations: 0.52.
  - The gender × complexity interaction was not significant (p = .219).
- Google Translate and DeepL often produced masculine output despite an explicit feminine pronoun in the source. GPT, Claude and Qwen stayed nearly symmetric across coreference conditions.
- Stated limitations:
  - The framework is binary and follows European Portuguese.
  - Stereotype labels come from U.S. statistics.
  - Annotation covers only the target noun phrase.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2026chinese,
  title = {{A Chinese Challenge Set to Assess Gender Bias in Automated Translation}},
  author = {Xu, Xiaolan and Mendes, Sara and Hsu, Yu-Yin},
  booktitle = {Proceedings of the 4th Workshop on Gender-Inclusive Translation Technologies (GITT 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.gitt-1.8/}
}
```

- [← previous A Bilingual Bimodal Benchmark for Arabic-English NLP across Grammatical Correction, Essay Scoring, Morphological Tagging, and Speech Recognition](https://md.research.elamur.ai/free/papers/alhafni-2026-bilingual-bimodal-benchmark-arabic-english/)
- [next → A Comparative Evaluation of Open-Source Models for Russian-Kazakh Translation](https://md.research.elamur.ai/free/papers/shanshin-2026-comparative-evaluation-open-source-models/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/)

## References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Blodgett, Su Lin, Solon Barocas, Hal Daumé Iii, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in nlp. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5454–5476. [find](https://scholar.google.com/scholar?q=Blodgett%2C+Su+Lin%2C+Solon+Barocas%2C+Hal+Daume%CC%81+Iii%2C+and+Hanna+Wallach%2E+2020%2E+Language+%28technology%29+is+power%3A+A+critical+survey+of+%E2%80%9Cbias%E2%80%9D+in+nlp%2E+In+Proceedings+of+the+58th+annual+meeting+of+the+associati) [our card](https://md.research.elamur.ai/free/papers/blodgett-2020-language-technology-is-power-critical/)
2. Bonifácio, Sofia, Helena Moniz, and Luisa Coheur. 2025. Diving into gender translation bias for the portuguese language. In 28th European Conference on Artificial Intelligence, ECAI 2025, including 14th Conference on Prestigious Applications of Intelligent Systems, PAIS 2025, pages 1067–1074. IOS Press BV. [↗](https://doi.org/10.3233/faia250916)
3. Chao, Lidia S, Derek F Wong, Chi Hong Ao, and Ana Luısa Leal. 2018. Um-pcorpus: a large portuguese-chinese parallel corpus. In Proceedings of the LREC 2018 Workshop “Belt & Road: Language Resources and Evaluation, pages 38–43. [find](https://scholar.google.com/scholar?q=Chao%2C+Lidia+S%2C+Derek+F+Wong%2C+Chi+Hong+Ao%2C+and+Ana+Lu%C4%B1sa+Leal%2E+2018%2E+Um%2Dpcorpus%3A+a+large+portuguese%2Dchinese+parallel+corpus%2E+In+Proceedings+of+the+LREC+2018+Workshop+%E2%80%9CBelt+%26+Road%3A+Language+Resources+an)
4. Cho, Won Ik, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019. On measuring gender bias in translation of gender-neutral pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 173–181. [find](https://scholar.google.com/scholar?q=Cho%2C+Won+Ik%2C+Ji+Won+Kim%2C+Seok+Min+Kim%2C+and+Nam+Soo+Kim%2E+2019%2E+On+measuring+gender+bias+in+translation+of+gender%2Dneutral+pronouns%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+in+Natural+Languag)
5. Corbett, Greville G. 1991. Gender. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Corbett%2C+Greville+G%2E+1991%2E+Gender%2E+Cambridge+University+Press%2E)
6. Costa-Jussà, Marta R, Carlos Escolano, Christine Basta, Javier Ferrando, Roser Batlle, and Ksenia Kharitonova. 2022. Interpreting gender bias in neural machine translation: Multilingual architecture matters. In Proceedings of the aaai conference on artificial intelligence, volume 36, pages 11855– 11863. [find](https://scholar.google.com/scholar?q=Costa%2DJussa%CC%80%2C+Marta+R%2C+Carlos+Escolano%2C+Christine+Basta%2C+Javier+Ferrando%2C+Roser+Batlle%2C+and+Ksenia+Kharitonova%2E+2022%2E+Interpreting+gender+bias+in+neural+machine+translation%3A+Multilingual+architecture+)
7. Fleisig, Eve and Christiane Fellbaum. 2022. Mitigating gender bias in machine translation through adversarial learning. arXiv preprint arXiv:2203.10675. [↗](https://arxiv.org/abs/2203.10675) [our card](https://md.research.elamur.ai/free/papers/savoldi-2021-gender-bias-machine-translation/)
8. Font, Joel Escudé and Marta R Costa-Jussà. 2019. Equalizing gender bias in neural machine translation with word embeddings techniques. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147–154. [find](https://scholar.google.com/scholar?q=Font%2C+Joel+Escude%CC%81+and+Marta+R+Costa%2DJussa%CC%80%2E+2019%2E+Equalizing+gender+bias+in+neural+machine+translation+with+word+embeddings+techniques%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+in+Natural+)
9. Gkovedarou, Eleni, Joke Daems, and Luna De Bruyne. 2025. Gender bias in English-to-Greek machine translation. In Hackenbuchner, Janiça, Luisa Bentivogli, Joke Daems, Chiara Manna, Beatrice Savoldi, and Eva Vanmassenhove, editors, Proceedings of the 3rd Workshop on Gender-Inclusive Translation Technologies (GITT 2025), pages 17– 45, Geneva, Switzerland, June. European Association for Machine Translation. [↗](https://doi.org/10.48550/arxiv.2506.09558)
10. Hackenbuchner, Janiça, Joke Daems, Arda Tezcan, and Aaron Maladry. 2024. You shall know a word’s gender by the company it keeps: Comparing the role of context in human gender assumptions with MT. In Savoldi, Beatrice, Janiça Hackenbuchner, Luisa Bentivogli, Joke Daems, Eva Vanmassenhove, and Jasmijn Bastings, editors, Proceedings of the 2nd International Workshop on Gender-Inclusive Translation Technologies, pages 31–41, Sheffield, United Kingdom, June. European Association for Machine Translation (EAMT). [find](https://scholar.google.com/scholar?q=Hackenbuchner%2C+Janic%CC%A7a%2C+Joke+Daems%2C+Arda+Tezcan%2C+and+Aaron+Maladry%2E+2024%2E+You+shall+know+a+word%E2%80%99s+gender+by+the+company+it+keeps%3A+Comparing+the+role+of+context+in+human+gender+assumptions+with+MT%2E+In+)
11. Kaneko, Masahiro, Danushka Bollegala, Naoaki Okazaki, and Timothy Baldwin. 2024. Evaluating gender bias in large language models via chain-of-thought prompting. arXiv preprint arXiv:2401.15585. [↗](https://arxiv.org/abs/2401.15585)
12. Kibort, Anna and Greville G Corbett. 2008. Grammatical features inventory. Typology of grammatical features. [find](https://scholar.google.com/scholar?q=Kibort%2C+Anna+and+Greville+G+Corbett%2E+2008%2E+Grammatical+features+inventory%2E+Typology+of+grammatical+features%2E)
13. Levy, Shahar, Koren Lazar, and Gabriel Stanovsky. 2021. Collecting a large-scale gender bias dataset for coreference resolution and machine translation. In Findings of the association for computational linguistics: EMNLP 2021, pages 2470–2480. [find](https://scholar.google.com/scholar?q=Levy%2C+Shahar%2C+Koren+Lazar%2C+and+Gabriel+Stanovsky%2E+2021%2E+Collecting+a+large%2Dscale+gender+bias+dataset+for+coreference+resolution+and+machine+translation%2E+In+Findings+of+the+association+for+computationa) [our card](https://md.research.elamur.ai/free/papers/levy-2021-collecting-large-scale-gender-bias/)
14. Piergentili, Andrea, Beatrice Savoldi, Dennis Fucci, Matteo Negri, and Luisa Bentivogli. 2023. Hi guys or hi folks? benchmarking gender-neutral machine translation with the gente corpus. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14124–14140. [find](https://scholar.google.com/scholar?q=Piergentili%2C+Andrea%2C+Beatrice+Savoldi%2C+Dennis+Fucci%2C+Matteo+Negri%2C+and+Luisa+Bentivogli%2E+2023%2E+Hi+guys+or+hi+folks%3F+benchmarking+gender%2Dneutral+machine+translation+with+the+gente+corpus%2E+In+Proceeding)
15. Popović, Maja and Ekaterina Lapshinova-Koltunski. 2024. Gender and bias in amazon review translations: by humans, mt systems and chatgpt. In Proceedings of the 2nd International Workshop on Gender-Inclusive Translation Technologies, pages 22–30. [find](https://scholar.google.com/scholar?q=Popovic%CC%81%2C+Maja+and+Ekaterina+Lapshinova%2DKoltunski%2E+2024%2E+Gender+and+bias+in+amazon+review+translations%3A+by+humans%2C+mt+systems+and+chatgpt%2E+In+Proceedings+of+the+2nd+International+Workshop+on+Gender%2DIn)
16. Prates, Marcelo OR, Pedro H Avelar, and Luı́s C Lamb. 2020. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, 32(10):6363–6381. [↗](https://doi.org/10.1007/s00521-019-04144-6) [our card](https://md.research.elamur.ai/free/papers/savoldi-2021-gender-bias-machine-translation/)
17. Quirk, Randolph and David Crystal. 2010. A comprehensive grammar of the English language. Pearson Education India. [find](https://scholar.google.com/scholar?q=Quirk%2C+Randolph+and+David+Crystal%2E+2010%2E+A+comprehensive+grammar+of+the+English+language%2E+Pearson+Education+India%2E)
18. Raposo, Eduardo Buzaglo, Maria Fernanda Nascimento, Maria Antónia Mota, Luı́sa Segura, and Amália Mendes. 2013. Gramática do Português. Fundação Calouste Gulbenkian, Lisboa. [find](https://scholar.google.com/scholar?q=Raposo%2C+Eduardo+Buzaglo%2C+Maria+Fernanda+Nascimento%2C+Maria+Anto%CC%81nia+Mota%2C+Lu%C4%B1%CC%81sa+Segura%2C+and+Ama%CC%81lia+Mendes%2E+2013%2E+Grama%CC%81tica+do+Portugue%CC%82s%2E+Fundac%CC%A7a%CC%83o+Calouste+Gulbenkian%2C+Lisboa%2E)
19. Sant, Aleix, Carlos Escolano, Audrey Mash, Francesca De Luca Fornaciari, and Maite Melero. 2024. The power of prompts: Evaluating and mitigating gender bias in mt with llms. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 94–139. [find](https://scholar.google.com/scholar?q=Sant%2C+Aleix%2C+Carlos+Escolano%2C+Audrey+Mash%2C+Francesca+De+Luca+Fornaciari%2C+and+Maite+Melero%2E+2024%2E+The+power+of+prompts%3A+Evaluating+and+mitigating+gender+bias+in+mt+with+llms%2E+In+Proceedings+of+the+5th+)
20. Saunders, Danielle and Bill Byrne. 2020. Reducing gender bias in neural machine translation as a domain adaptation problem. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics, pages 7724–7736. [find](https://scholar.google.com/scholar?q=Saunders%2C+Danielle+and+Bill+Byrne%2E+2020%2E+Reducing+gender+bias+in+neural+machine+translation+as+a+domain+adaptation+problem%2E+In+Proceedings+of+the+58th+annual+meeting+of+the+Association+for+Computation)
21. Saunders, Danielle and Katrina Olsen. 2023. Gender, names and other mysteries: Towards the ambiguous for gender-inclusive translation. In Proceedings of the First Workshop on Gender-Inclusive Translation Technologies, pages 85–93. [↗](https://doi.org/10.48550/arxiv.2306.04573)
22. Savoldi, Beatrice, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845–874. [↗](https://doi.org/10.1162/tacl_a_00401) [our card](https://md.research.elamur.ai/free/papers/savoldi-2021-gender-bias-machine-translation/)
23. Savoldi, Beatrice, Sara Papi, Matteo Negri, Ana Guerberof-Arenas, and Luisa Bentivogli. 2024. What the harm? quantifying the tangible impact of gender bias in machine translation with a human-centered study. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18048–18076. [find](https://scholar.google.com/scholar?q=Savoldi%2C+Beatrice%2C+Sara+Papi%2C+Matteo+Negri%2C+Ana+Guerberof%2DArenas%2C+and+Luisa+Bentivogli%2E+2024%2E+What+the+harm%3F+quantifying+the+tangible+impact+of+gender+bias+in+machine+translation+with+a+human%2Dcentered) [our card](https://md.research.elamur.ai/free/papers/savoldi-2021-gender-bias-machine-translation/)
24. Sewunetie, Walelign, Atnafu Tonja, Tadesse Belay, Hellina Hailu Nigatu, Gashaw Gebremeskel, Zewdie Mossie, Hussien Seid, and Seid Yimam. 2024. Gender bias evaluation in machine translation for Amharic, Tigrigna, and afaan oromoo. In Savoldi, Beatrice, Janiça Hackenbuchner, Luisa Bentivogli, Joke Daems, Eva Vanmassenhove, and Jasmijn Bastings, editors, Proceedings of the 2nd International Workshop on Gender-Inclusive Translation Technologies, pages 1–11, Sheffield, United Kingdom, June. European Association for Machine Translation (EAMT). [find](https://scholar.google.com/scholar?q=Sewunetie%2C+Walelign%2C+Atnafu+Tonja%2C+Tadesse+Belay%2C+Hellina+Hailu+Nigatu%2C+Gashaw+Gebremeskel%2C+Zewdie+Mossie%2C+Hussien+Seid%2C+and+Seid+Yimam%2E+2024%2E+Gender+bias+evaluation+in+machine+translation+for+Amharic)
25. Sharma, Shanya, Manan Dey, and Koustuv Sinha. 2022. How sensitive are translation systems to extra contexts? mitigating gender bias in neural machine translation models through relevant contexts. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1968–1984. [find](https://scholar.google.com/scholar?q=Sharma%2C+Shanya%2C+Manan+Dey%2C+and+Koustuv+Sinha%2E+2022%2E+How+sensitive+are+translation+systems+to+extra+contexts%3F+mitigating+gender+bias+in+neural+machine+translation+models+through+relevant+contexts%2E+In+F)
26. Stanovsky, Gabriel, Noah A Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1679–1684. [find](https://scholar.google.com/scholar?q=Stanovsky%2C+Gabriel%2C+Noah+A+Smith%2C+and+Luke+Zettlemoyer%2E+2019%2E+Evaluating+gender+bias+in+machine+translation%2E+In+Proceedings+of+the+57th+annual+meeting+of+the+association+for+computational+linguistics%2C) [our card](https://md.research.elamur.ai/free/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/)
27. Zhao, Jieyu, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 2979–2989. [find](https://scholar.google.com/scholar?q=Zhao%2C+Jieyu%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+2017%2E+Men+also+like+shopping%3A+Reducing+gender+bias+amplification+using+corpus%2Dlevel+constraints%2E+In+Proceedings+of+the+2017+) [our card](https://md.research.elamur.ai/free/papers/zhao-2017-men-also-like-shopping-reducing/)