--- title: "On Measuring Gender Bias in Translation of Gender-neutral" description: "The authors build an equity evaluation corpus (EEC) from template sentences of the form \"S/he is [xx]\". 3 references, cited by 67 papers here." authors: ["Won Ik Cho", "Ji Won Kim", "Seok Min Kim", "Nam Soo Kim"] year: 2019 venue: "Proceedings of the First Workshop on Gender Bias in Natural Language Processing" source: "https://aclanthology.org/W19-3824/" doi: "10.18653/v1/w19-3824" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/cho-2019-measuring-gender-bias-translation-gender/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cho-2019-measuring-gender-bias-translation-gender/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / On Measuring Gender Bias in Translation of Gender-neutral Pronouns # On Measuring Gender Bias in Translation of Gender-neutral Pronouns **Authors:** [Won Ik Cho](https://md.research.elamur.ai/free/en/authors/won-ik-cho/), Ji Won Kim, Seok Min Kim and [Nam Soo Kim](https://md.research.elamur.ai/free/en/authors/nam-soo-kim/) · **Year:** 2019 · **Venue:** [Proceedings of the First Workshop on Gender Bias in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/gebnlp/) ([Workshop on Gender Bias in Natural Language Processing 2019](https://md.research.elamur.ai/free/en/venues/gebnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-3824/) · [PDF](https://aclanthology.org/W19-3824.pdf) · [DOI 10.18653/v1/w19-3824](https://doi.org/10.18653/v1/w19-3824) · [OpenAlex](https://openalex.org/W2947432670) **Citations:** 53 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/cho-2019-measuring-gender-bias-translation-gender/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary The paper tackles a gap in fairness research: there was no systematic way to check whether Korean-to-English MT systems invent a gender when translating Korean third-person pronouns that carry none. The authors build an equity evaluation corpus (EEC) from template sentences of the form "S/he is [xx]". The slot is filled with sentiment expressions or occupation names, and two further dimensions are varied: the pronoun used (informal kyay versus formal ku salam) and the presence or absence of a politeness suffix. The result is 4,236 test sentences grouped into seven subsets. They also define a measure, the translation gender bias index (TGBI). For each subset, it scores how often outputs are female, male or neutral, and it rewards neutral or balanced renderings. They apply it to three commercial translators. Google Translator scores highest overall and Kakao lowest, while Papago appears to handle occupations differently. The work matters because it gives a reusable, mechanically scorable resource and metric for comparing systems on gender-neutrality preservation. ## Key points - The lexicon has 1,059 entries: - 324 sentiment items (124 positive, 200 negative), taken from the Korean Sentiment Word Dictionary of Kunsan National University, which contains 14,843 items. - 735 occupation terms, taken from an official Korean government employment website. - Each entry is combined with two pronouns (kyay, ku salam) and with politeness on or off. This gives 4,236 utterances in seven subsets covering (in)formality, (im)politeness, polarity and occupation. - Words were filtered to avoid confounds: - Sentiment items were kept only if three native speakers fully agreed on their polarity. - Sentiment items touching on appearance, wealth, sexual orientation, disability, education or status were removed. - Gender-marked jobs (e.g., ballerino, woman diver) and hateful terms were dropped. - The metric works in two steps: - Per subset, P_S = √(p_w·p_m) + p_n, where p_w, p_m and p_n are the shares of female, male and neutral translations. It peaks when all outputs are neutral and bottoms out when all are one gender. - TGBI is the average of P_S across the seven subsets. - The systems evaluated were Google Translator, Naver Papago and Kakao Translator. The corpus and code are released at github.com/nolongerprejudice/tgbi. ## Interesting ideas - **Template-based probes from a gender-neutral source language.** Starting from a language whose pronouns lack gender turns any gendered output into measurable bias. The same probe design could be used for other language pairs where the source is less marked than the target. - **A metric with a built-in ideal.** Combining a geometric mean of the female and male shares with the neutral share means both balanced guessing and neutral wording earn credit, while one-sided output is penalized. This bounded score is easy to adapt for other binary attributes that a system should not hallucinate. - **Varying register as a separate axis.** Treating formality and politeness as their own dimensions shows whether bias shifts with style rather than content. This is especially useful for languages with rich honorific or register systems. - **Filtering lexicon items for hidden confounds.** Removing words tied to appearance, wealth, disability and similar traits keeps the test focused on gender alone. This cleaning step makes any bias-probing word list more trustworthy. - **Context-sensitive sanity checks.** Sentences where only one gender is physiologically possible, such as being on a period, test whether a system can tell when guessing a gender is appropriate and when it is required. This complements neutrality tests in other evaluation suites. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Seoul National University](https://md.research.elamur.ai/free/en/universities/seoul-national-university/) — University, Seoul, South Korea · 20 works · 45,282 citations 📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 221 works · [🤝 Gender Studies](https://md.research.elamur.ai/free/en/shelves/social-sciences/gender-studies/) · 30 works 🧭 Topics (2): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 609 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works ## Cite Cite as: Won Ik Cho, Ji Won Kim, Seok Min Kim and Nam Soo Kim. 2019. On Measuring Gender Bias in Translation of Gender-neutral Pronouns. Proceedings of the First Workshop on Gender Bias in Natural Language Processing. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{cho2019measuring, title = {{On Measuring Gender Bias in Translation of Gender-neutral Pronouns}}, author = {Won Ik Cho, Ji Won Kim, Seok Min Kim and Nam Soo Kim}, booktitle = {Proceedings of the First Workshop on Gender Bias in Natural Language Processing}, year = {2019}, doi = {10.18653/v1/w19-3824}, url = {https://doi.org/10.18653/v1/w19-3824} } ``` - [← previous Neural Fuzzy Repair: Integrating Fuzzy Matches into Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/bulte-2019-neural-fuzzy-repair-integrating-fuzzy/) - [next → ChID: A Large-scale Chinese IDiom Dataset for Cloze Test](https://md.research.elamur.ai/free/en/papers/zheng-2019-chid-large-scale-chinese/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) ## References 3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Shlomo Argamon, Moshe Koppel, Jonathan Fine, and Anat Rachel Shimoni. 2003. Gender, genre, and writing style in formal written texts. Text-The Hague Then Amsterdam Then Berlin-, 23(3):321–346. [find](https://scholar.google.com/scholar?q=Shlomo+Argamon%2C+Moshe+Koppel%2C+Jonathan+Fine%2C+and+Anat+Rachel+Shimoni%2E+2003%2E+Gender%2C+genre%2C+and+writing+style+in+formal+written+texts%2E+Text%2DThe+Hague+Then+Amsterdam+Then+Berlin%2D%2C+23%283%29%3A321%E2%80%93346%2E) 2. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. [Men also like shopping: Reducing gender bias amplification using corpus-level constraints](https://md.research.elamur.ai/free/en/papers/zhao-2017-men-also-like-shopping-reducing/). arXiv preprint arXiv:1707.09457. 3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473. *Updated 24 October 2026*