---
title: "Gendered Ambiguous Pronoun (GAP) Shared Task at the"
description: "This shared task, held at the first ACL workshop on Gender Bias in NLP, asked whether… 0.18397 F1. 28 references, cited by 11 papers here."
authors: ["Kellie Webster", "Marta R. Costa-jussà", "Christian Hardmeier", "Will Radford"]
year: 2019
venue: "Proceedings of the First Workshop on Gender Bias in Natural Language Processing"
source: "https://aclanthology.org/W19-3801/"
doi: "10.18653/v1/w19-3801"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/webster-2019-gendered-ambiguous-pronoun-gap-shared/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/webster-2019-gendered-ambiguous-pronoun-gap-shared/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/) / Gendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias in NLP Workshop 2019

# Gendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias in NLP Workshop 2019

**Authors:** [Kellie Webster](https://md.research.elamur.ai/free/en/authors/kellie-webster/), [Marta R. Costa-jussà](https://md.research.elamur.ai/free/en/authors/marta-r-costa-jussa/), [Christian Hardmeier](https://md.research.elamur.ai/free/en/authors/christian-hardmeier/) and Will Radford · **Year:** 2019 · **Venue:** [Proceedings of the First Workshop on Gender Bias in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/gebnlp/) ([Workshop on Gender Bias in Natural Language Processing 2019](https://md.research.elamur.ai/free/en/venues/gebnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W19-3801/) · [PDF](https://aclanthology.org/W19-3801.pdf) · [DOI 10.18653/v1/w19-3801](https://doi.org/10.18653/v1/w19-3801) · [OpenAlex](https://openalex.org/W2972708930)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/webster-2019-gendered-ambiguous-pronoun-gap-shared/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This shared task, held at the first ACL workshop on Gender Bias in NLP, asked whether gender-balanced training and test data are enough to get fair pronoun resolution. It used the GAP coreference challenge (Webster et al., 2018). Given a Wikipedia snippet, a target pronoun and two candidate names, a system must give probabilities that the pronoun refers to the first name, the second name, or neither. The organizers built 760 new examples that three raters agreed on unanimously. They mixed these into a much larger unlabeled pool, so hand-labeling the test set was not practical. The competition ran on Kaggle and was scored by logloss. 263 teams took part, and eleven described their systems in papers. Every described system relied on BERT, either fine-tuned or used as a feature extractor, and ensembles were common. The best entry reached logloss 0.13667 and F1 96.2. Its feminine-to-masculine F1 ratio was 0.99, close to parity, even though no team used special debiasing. This suggests balanced data can go a long way toward fair results. The open problem is getting fair outcomes when the training data itself is skewed.

## Key points
- Setting: gold-two-mention with page-context. Systems were given both names and the source Wikipedia URL, so no mention detection was needed. Each submission was a probability over three classes (Name A / Name B / Neither).
- Test data: 760 clean examples hidden among 11,599 unlabeled ones, giving 12,359 to rate. The masculine:feminine ratio was 1:1 but kept secret from participants.
- Top three (Attree, Wang, Abzaliev) had logloss 0.13667 / 0.17289 / 0.18397, F1 96.2 / 95.7 / 95.4, and Bias 0.99 each. Logloss and F1 ranked them in the same order. Prizes totaled USD 25,000.
- 8 of 11 systems fine-tuned BERT, mostly using the original GAP release as training data. Abzaliev also used OntoNotes 5.0, WinoBias, WinoGender and the Definite Pronoun Resolution Dataset. Feature-extraction teams pooled subtokens into span vectors, often with AllenAI's SelfAttentionSpanExtractor.
- A handcrafted pronoun-gender feature made no difference (Abzaliev). BERT encodings showed no significant bias on WEAT or SEAT (Bao and Qiao).

## Interesting ideas
- Measure fairness as a ratio: compute F1 separately on feminine and masculine items and divide, so 1.0 means parity and a single number exposes skew that overall accuracy hides.
- Hide a small set of gold items inside a big pool of unlabeled decoys, and keep the class balance secret, so the test cannot be gamed or hand-annotated.
- The winner's "evidence pooling" fed the decisions of several off-the-shelf coreference resolvers in as features. This sits between ensembling and feature crossing and stays simple.
- Augment scarce data by swapping person names, and synthesize extra examples of the rare "neither" class to balance labels.
- Treat reference as a probability over A / B / neither and score it with logloss, which rewards calibration rather than just the top guess.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Mova's evaluation could report per-gender (or per-group) scores and their ratio for pronoun resolution, as a standing check next to the negative controls.
- **I2** (learn): when testing LLM-teacher tiers or self-play outputs, Mova could bury gold items in unscored filler so no component can overfit to the known test set.
- **I3** (frames): Mova's coreference step could treat outside resolvers as witnesses, recording which ones support each antecedent, so the final link stays explainable.
- **I4** (learn): silver coreference data could be enlarged by name substitution, plus generated cases where no candidate is the antecedent, to stop the resolver from always picking a name.
- **I5** (world): entity linking in the text world could keep calibrated weights over candidate referents, including "none of them", and pass that uncertainty on to the answering readers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [Uppsala University](https://md.research.elamur.ai/free/en/universities/uppsala-university/) — University, Uppsala, Sweden · 56 works · 13,046 citations
- [Universitat Politècnica de Catalunya](https://md.research.elamur.ai/free/en/universities/universitat-politecnica-de-catalunya/) — University, Barcelona, Spain · 23 works · 15,102 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🔗 Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) · 550 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works

## Cite

Cite as: Kellie Webster, Marta R. Costa-jussà, Christian Hardmeier and Will Radford. 2019. Gendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias in NLP Workshop 2019. Proceedings of the First Workshop on Gender Bias in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{webster2019gendered,
  title = {{Gendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias in NLP Workshop 2019}},
  author = {Kellie Webster, Marta R. Costa-jussà, Christian Hardmeier and Will Radford},
  booktitle = {Proceedings of the First Workshop on Gender Bias in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/w19-3801},
  url = {https://doi.org/10.18653/v1/w19-3801}
}
```

- [← previous Implicit Discourse Relation Identification for Open-domain Dialogues](https://md.research.elamur.ai/free/en/papers/ma-2019-implicit-discourse-relation-identification-open/)
- [next → Concord and agreement features in Modern Standard Arabic](https://md.research.elamur.ai/free/en/papers/winchester-2019-concord-agreement-features-modern-standard/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/)

## References

28 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gender It is encouraging to see submitted systems improve the gender gap so close to parity at 0.99, particularly as no special modeling strategies were required. Indeed, Abzaliev (2019) reported that a handcrafted pronoun gender feature had no impact. [find](https://scholar.google.com/scholar?q=Gender+It+is+encouraging+to+see+submitted+systems+improve+the+gender+gap+so+close+to+parity+at+0%2E99%2C+particularly+as+no+special+modeling+strategies+were+required%2E+Indeed%2C+Abzaliev+%282019%29+reported+that)
2. Moreover, Bao and Qiao (2019) report that BERT encodings show no significant gender bias on either WEAT (Caliskan et al., 2017) or SEAT (May et al., 2019). We look forward to studies considering potential biases in BERT across more tasks and dimensions of diversity. [find](https://scholar.google.com/scholar?q=Moreover%2C+Bao+and+Qiao+%282019%29+report+that+BERT+encodings+show+no+significant+gender+bias+on+either+WEAT+%28Caliskan+et+al%2E%2C+2017%29+or+SEAT+%28May+et+al%2E%2C+2019%29%2E+We+look+forward+to+studies+considering+poten)
3. Attree (2019) for a good analysis), particularly when compared to hand-crafted features (Abzaliev, 2019). We also believe the widespread use of BERT is related to the low rate of external data usage, as it is easier for most teams to reuse an existing model than to clean and integrate new data. As well as the phenomenal modeling power of BERT , one possible reason for this observation is that the public releases of BERT are trained on the same domain as the GAP examples, Wikipedia. Future work could benchmark non-Wikipedia BERT models on the shared task examples, or collect more GAP examples from different domains. [find](https://scholar.google.com/scholar?q=Attree+%282019%29+for+a+good+analysis%29%2C+particularly+when+compared+to+hand%2Dcrafted+features+%28Abzaliev%2C+2019%29%2E+We+also+believe+the+widespread+use+of+BERT+is+related+to+the+low+rate+of+external+data+usage%2C+)
4. BERT Artem Abzaliev. 2019. On GAP coreference resolution shared task: insights from the 3rd place solution. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=BERT+Artem+Abzaliev%2E+2019%2E+On+GAP+coreference+resolution+shared+task%3A+insights+from+the+3rd+place+solution%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+for+Natural+Language+Processing%2C+Florenc)
5. Sandeep Attree. 2019. Gendered Pronoun Resolution Shared Task: Boosting Model Confidence by Evidence Pooling. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sandeep+Attree%2E+2019%2E+Gendered+Pronoun+Resolution+Shared+Task%3A+Boosting+Model+Confidence+by+Evidence+Pooling%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+for+Natural+Language+Processing%2C+Flore)
6. Xingce Bao and Qianqian Qiao. 2019. Transfer Learning from Pre-trained BERT for Pronoun Resolution. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xingce+Bao+and+Qianqian+Qiao%2E+2019%2E+Transfer+Learning+from+Pre%2Dtrained+BERT+for+Pronoun+Resolution%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+for+Natural+Language+Processing%2C+Florence%2C+Italy)
7. Jayadev Bhaskaran and Isha Bhallamudi. 2019. [Good Secretaries, Bad Truck Drivers?](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/) Occupational Gender Stereotypes in Sentiment Analysis. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics.
8. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. [↗](https://doi.org/10.1126/science.aal4230)
9. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. [On measuring social biases in sentence encoders](https://md.research.elamur.ai/free/en/papers/may-2019-measuring-social-biases-sentence-encoders/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622–628, Minneapolis, Minnesota. Association for Computational Linguistics.
10. Rakesh Chada. 2019. Gendered Pronoun Resolution using BERT and an extractive question answering formulation. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rakesh+Chada%2E+2019%2E+Gendered+Pronoun+Resolution+using+BERT+and+an+extractive+question+answering+formulation%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+for+Natural+Language+Processing%2C+Floren)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
12. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. [Reducing gender bias in abusive language detection](https://md.research.elamur.ai/free/en/papers/park-2018-reducing-gender-bias-abusive-language/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2799–2804, Brussels, Belgium. Association for Computational Linguistics.
13. Joel Escudé Font and Marta R. Costa-jussà. 2019. Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniques. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1901.03116)
14. Sameer S. Pradhan and Nianwen Xue. 2009. [OntoNotes: The 90% solution](https://md.research.elamur.ai/free/en/papers/pradhan-2009-ontonotes-90-solution/). In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Tutorial Abstracts, pages 11–12, Boulder, Colorado. Association for Computational Linguistics.
15. Hila Gonen and Yoav Goldberg. 2019. [Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them](https://md.research.elamur.ai/free/en/papers/gonen-2019-lipstick-pig-debiasing-methods-cover/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 609–614, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1903.03862)
16. Altaf Rahman and Vincent Ng. 2012. [Resolving complex cases of definite pronouns: The Winograd schema challenge](https://md.research.elamur.ai/free/en/papers/rahman-2012-resolving-complex-cases-definite-pronouns/). In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789, Jeju Island, Korea. Association for Computational Linguistics.
17. Matei Ionita, Yury Kashnitsky, Ken Krige, Vladimir Larin, and Atanas Atanasov. 2019. Gender-unbiased BERT-based Pronoun Resolution. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matei+Ionita%2C+Yury+Kashnitsky%2C+Ken+Krige%2C+Vladimir+Larin%2C+and+Atanas+Atanasov%2E+2019%2E+Gender%2Dunbiased+BERT%2Dbased+Pronoun+Resolution%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+for+Natural+Lang)
18. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. [Gender bias in coreference resolution](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2002)
19. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural Questions: a Benchmark for Question Answering Research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association of Computational Linguistics. [↗](https://doi.org/10.1162/tacl_a_00276)
20. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [BERT Rediscovers the Classical NLP Pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). [↗](https://doi.org/10.48550/arxiv.1905.05950)
21. Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. [Getting gender right in neural machine translation](https://md.research.elamur.ai/free/en/papers/vanmassenhove-2018-getting-gender-right-neural-machine/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003–3008, Brussels, Belgium. Association for Computational Linguistics.
22. Bo Liu. 2019. Anonymized BERT: An Augmentation Approach to the Gendered Pronoun Resolution Challenge. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bo+Liu%2E+2019%2E+Anonymized+BERT%3A+An+Augmentation+Approach+to+the+Gendered+Pronoun+Resolution+Challenge%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+for+Natural+Language+Processing%2C+Florence%2C+Ita)
23. Zili Wang. 2019. MSnet: A BERT-based Network for Gendered Pronoun Resolution. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zili+Wang%2E+2019%2E+MSnet%3A+A+BERT%2Dbased+Network+for+Gendered+Pronoun+Resolution%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+for+Natural+Language+Processing%2C+Florence%2C+Italy%2E+Association+for+Comp)
24. Felipe Alfaro Lois, José A. R. Fonollosa, and Marta R. Costa-jussà. 2019. BERT Masked Language Modeling for Coreference Resolution. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Felipe+Alfaro+Lois%2C+Jose%CC%81+A%2E+R%2E+Fonollosa%2C+and+Marta+R%2E+Costa%2Djussa%CC%80%2E+2019%2E+BERT+Masked+Language+Modeling+for+Coreference+Resolution%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+for+Natural+La)
25. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. [Mind the GAP: A balanced corpus of gendered ambiguous pronouns](https://md.research.elamur.ai/free/en/papers/webster-2018-mind-gap-balanced-corpus-gendered/). Transactions of the Association for Computational Linguistics, 6:605–617. guistics. [↗](https://doi.org/10.1162/tacl_a_00240)
26. Yinchuan Xu and Junlin Yang. 2019. Look Again at the Syntax: Relational Graph Convolutional Network for Gendered Ambiguous Pronoun Resolution. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yinchuan+Xu+and+Junlin+Yang%2E+2019%2E+Look+Again+at+the+Syntax%3A+Relational+Graph+Convolutional+Network+for+Gendered+Ambiguous+Pronoun+Resolution%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+for+N)
27. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. [Gender bias in coreference resolution: Evaluation and debiasing methods](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15– 20, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2003)
28. Kai-Chou Yang, Timothy Niven, and Hung-Yu Kao. 2019. Fill the GAP: Exploiting BERT for Pronoun Resolution. In Proceedings of the First Workshop on Gender Bias for Natural Language Processing, Florence, Italy. Association for Computational Lin- [find](https://scholar.google.com/scholar?q=Kai%2DChou+Yang%2C+Timothy+Niven%2C+and+Hung%2DYu+Kao%2E+2019%2E+Fill+the+GAP%3A+Exploiting+BERT+for+Pronoun+Resolution%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+for+Natural+Language+Processing%2C+Florence)

*Updated 24 October 2026*