---
title: "WikiCREM: A Large Unsupervised Corpus for — Kocijan 2019"
description: "Hard pronoun resolution is when syntax alone cannot reveal the referent."
authors: ["Vid Kocijan", "Oana-Maria Camburu", "Ana-Maria Cretu", "Yordan Yordanov", "Phil Blunsom", "Thomas Lukasiewicz"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1439/"
doi: "10.18653/v1/d19-1439"
arxiv: "1908.08025"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/kocijan-2019-wikicrem-large-unsupervised-corpus-coreference/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/kocijan-2019-wikicrem-large-unsupervised-corpus-coreference/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2019](https://md.research.elamur.ai/free/papers/2019/page/4/) / WikiCREM: A Large Unsupervised Corpus for Coreference Resolution

# WikiCREM: A Large Unsupervised Corpus for Coreference Resolution

**Authors:** Vid Kocijan, [Oana-Maria Camburu](https://md.research.elamur.ai/free/authors/oana-maria-camburu/), Ana-Maria Cretu, Yordan Yordanov, [Phil Blunsom](https://md.research.elamur.ai/free/authors/phil-blunsom/), [Thomas Lukasiewicz](https://md.research.elamur.ai/free/authors/thomas-lukasiewicz/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1439/) · [PDF](https://aclanthology.org/D19-1439.pdf) · [DOI 10.18653/v1/d19-1439](https://doi.org/10.18653/v1/d19-1439) · [arXiv 1908.08025](https://arxiv.org/abs/1908.08025) · [OpenAlex](https://openalex.org/W2967584709)

**Citations:** 26 citations (openalex-snapshot · 2026-10-14) · [15](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Hard pronoun resolution is when syntax alone cannot reveal the referent. Neural models need large training sets for it, but existing hard benchmarks are small and often serve only as test sets. The authors propose an unsupervised way to build such data automatically. They search English Wikipedia for one- or two-sentence passages that mention at least two different personal names, with one name repeated. A later occurrence of the repeated name is masked, and the task is to choose between the true name and a distractor that appears earlier in the text. Because the answer comes from the original text, labels need no human annotation.

The resulting corpus, WikiCREM, is used to fine-tune BERT. The model scores each candidate by how probable it is in the masked slot. Experiments cover three settings: no target training data, available target training data, and a transductive setup. Training on WikiCREM helps consistently and reaches or beats prior state of the art on 6 of 7 benchmarks. This shows that cheap, automatically labelled data can substantially improve coreference systems.

## Key points
- **Corpus construction:** WikiCREM has 2,438,897 examples taken from the enwiki-20181201 dump, with 10,000 held out for validation. Names were detected with the Spacy NER library.
- **Annotation check (100 sampled items):**
  - 18 items were judged unsolvable.
  - Annotators answered 95.1% of the remaining items correctly.
  - About 63% of items would read naturally with a pronoun in place of the mask, compared with only 7 of 100 in the earlier MaskedWiki corpus.
- **Results:** The authors report gains over earlier state of the art of 5.9% on GAP, 8.4% on DPR and 12.7% on PDP. The combined model BERT WikiCREM ALL set new best results on GAP, DPR and WinoBias. BERT WikiCREM DPR led on WinoGender, and WNLI was also matched or exceeded. WSC was the one benchmark where prior results were not reached.
- **Gender bias:** Correct answers are skewed toward male names (female-to-male ratio about 0.53). Even so, WikiCREM-trained models performed better and showed less bias on WinoGender and WinoBias.
- **Released resources:** The dataset, trained models and code (repository vid-koci/bert-commonsense) are publicly available.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/)

## Cited by

Cited by

15 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 14 not yet.

- [Free the Plural: Unrestricted Split-Antecedent Anaphora Resolution](https://md.research.elamur.ai/free/papers/yu-2020-free-plural-unrestricted-split-antecedent/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kocijan2019wikicrem,
  title = {{WikiCREM: A Large Unsupervised Corpus for Coreference Resolution}},
  author = {Kocijan, Vid and Camburu, Oana-Maria and Cretu, Ana-Maria and Yordanov, Yordan and Blunsom, Phil and Lukasiewicz, Thomas},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1439},
  eprint = {1908.08025},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1439}
}
```

- [← previous WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://md.research.elamur.ai/free/papers/pilehvar-2019-wic-word-context-dataset-evaluating/)
- [next → WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://md.research.elamur.ai/free/papers/schwenk-2019-wikimatrix-mining-135m-parallel-sentences/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/papers/2019/page/4/)

## References

27 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ernest Davis, Leora Morgenstern, and Charles L. Ortiz. 2017. The first Winograd Schema Challenge at IJCAI-16. AI Magazine, 38(3):97–98. [↗](https://doi.org/10.1609/aimag.v38i4.2734)
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. Computing Research Repository, arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
3. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1971–1982, Seattle, Washington, USA. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d13-1203) [our card](https://md.research.elamur.ai/free/papers/durrett-2013-easy-victories-uphill-battles-coreference/)
4. Ali Emami, Noelia De La Cruz, Adam Trischler, Kaheer Suleman, and Jackie Chi Kit Cheung. 2018. A knowledge hunting framework for common sense reasoning. Computing Research Repository, arXiv:1810.01375. [↗](https://arxiv.org/abs/1810.01375)
5. Ali Emami, Paul Trichelair, Adam Trischler, Kaheer Suleman, Hannes Schulz, and Jackie Chi Kit Cheung. 2019. The K NOW R EF Coreference Corpus: Removing gender and number cues for difficult pronominal anaphora resolution. In ACL 2019. [find](https://scholar.google.com/scholar?q=Ali+Emami%2C+Paul+Trichelair%2C+Adam+Trischler%2C+Kaheer+Suleman%2C+Hannes+Schulz%2C+and+Jackie+Chi+Kit+Cheung%2E+2019%2E+The+K+NOW+R+EF+Coreference+Corpus%3A+Removing+gender+and+number+cues+for+difficult+pronominal+)
6. Johannes Fähndrich, Sabine Weber, and Hannes Kan-thak. 2018. A marker passing approach to Winograd schemas. In Semantic Technology, pages 165–181, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=Johannes+Fa%CC%88hndrich%2C+Sabine+Weber%2C+and+Hannes+Kan%2Dthak%2E+2018%2E+A+marker+passing+approach+to+Winograd+schemas%2E+In+Semantic+Technology%2C+pages+165%E2%80%93181%2C+Cham%2E+Springer+International+Publishing%2E)
7. Abbas Ghaddar and Philippe Langlais. 2016. Wiki-Coref: An English coreference-annotated corpus of Wikipedia articles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia. European Language Resources Association (ELRA), European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Abbas+Ghaddar+and+Philippe+Langlais%2E+2016%2E+Wiki%2DCoref%3A+An+English+coreference%2Dannotated+corpus+of+Wikipedia+articles%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and+Eva) [our card](https://md.research.elamur.ai/free/papers/ghaddar-2016-wikicoref-english-coreference-annotated-corpus/)
8. Liane Guillou. 2012. Improving pronoun translation for statistical machine translation. In Proceedings of the Student Research Workshop at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 1–10. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Liane+Guillou%2E+2012%2E+Improving+pronoun+translation+for+statistical+machine+translation%2E+In+Proceedings+of+the+Student+Research+Workshop+at+the+13th+Conference+of+the+European+Chapter+of+the+Associatio)
9. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for Winograd Schema Challenge. Computing Research Repository, arXiv:1905.06290. [↗](https://arxiv.org/abs/1905.06290)
10. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. [↗](https://doi.org/10.1162/coli_a_00152) [our card](https://md.research.elamur.ai/free/papers/lee-2013-deterministic-coreference-resolution-based-entity/)
11. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d17-1018) [our card](https://md.research.elamur.ai/free/papers/lee-2017-end-end-neural-coreference-resolution/)
12. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd Schema Challenge. AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 46. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2011%2E+The+Winograd+Schema+Challenge%2E+AAAI+Spring+Symposium%3A+Logical+Formalizations+of+Commonsense+Reasoning%2C+46%2E)
13. Fei Liu, Luke S. Zettlemoyer, and Jacob Eisenstein. 2019. The referential reader: A recurrent entity network for anaphora resolution. Computing Research Repository, abs/1902.01541. [↗](https://arxiv.org/abs/1902.01541) [our card](https://md.research.elamur.ai/free/papers/liu-2019-referential-reader-recurrent-entity-network/)
14. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+J%2E+Bethard%2C+and+David+Mc%2DClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Association+for+Computation) [our card](https://md.research.elamur.ai/free/papers/manning-2014-stanford-corenlp-natural-language-processing/)
15. Kotaro Nakayama. 2019. Wikipedia mining for triple extraction enhanced by co-reference resolution. [find](https://scholar.google.com/scholar?q=Kotaro+Nakayama%2E+2019%2E+Wikipedia+mining+for+triple+extraction+enhanced+by+co%2Dreference+resolution%2E)
16. Haoruo Peng, Daniel Khashabi, and Dan Roth. 2015. Solving hard coreference problems. In HLT-NAACL. [find](https://scholar.google.com/scholar?q=Haoruo+Peng%2C+Daniel+Khashabi%2C+and+Dan+Roth%2E+2015%2E+Solving+hard+coreference+problems%2E+In+HLT%2DNAACL%2E) [our card](https://md.research.elamur.ai/free/papers/peng-2015-solving-hard-coreference-problems/)
17. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, CoNLL ’12, pages 1–40, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+CoNLL%2D2012+Shared+Task%3A+Modeling+multilingual+unrestricted+coreference+in+OntoNotes%2E+In+Joint+Conference+on+EM) [our card](https://md.research.elamur.ai/free/papers/pradhan-2012-conll-2012-shared-task-modeling/)
18. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
19. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The Winograd Schema Challenge. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Altaf+Rahman+and+Vincent+Ng%2E+2012%2E+Resolving+complex+cases+of+definite+pronouns%3A+The+Winograd+Schema+Challenge%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/papers/rahman-2012-resolving-complex-cases-definite-pronouns/)
20. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. Computing Research Repository, abs/1804.09301. [↗](https://arxiv.org/abs/1804.09301) [our card](https://md.research.elamur.ai/free/papers/rudinger-2018-gender-bias-coreference-resolution/)
21. Ulrich Schäfer, Christian Spurk, and Jörg Steffen. 2012. A fully coreference-annotated corpus of scholarly papers from the ACL Anthology. Proceedings of COLING 2012: Posters, pages 1059–1070. [find](https://scholar.google.com/scholar?q=Ulrich+Scha%CC%88fer%2C+Christian+Spurk%2C+and+Jo%CC%88rg+Steffen%2E+2012%2E+A+fully+coreference%2Dannotated+corpus+of+scholarly+papers+from+the+ACL+Anthology%2E+Proceedings+of+COLING+2012%3A+Posters%2C+pages+1059%E2%80%931070%2E) [our card](https://md.research.elamur.ai/free/papers/schafer-2012-fully-coreference-annotated-corpus-scholarly/)
22. T. H. Trinh and Q. V. Le. 2018. A Simple Method for Commonsense Reasoning. Computing Research Repository, arXiv:1806.02847. [↗](https://arxiv.org/abs/1806.02847)
23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Computing Research Repository, arXiv:1706.03762. [↗](https://arxiv.org/abs/1706.03762)
24. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR. [↗](https://doi.org/10.48550/arxiv.1804.07461) [our card](https://md.research.elamur.ai/free/papers/wang-2018-glue-multi-task-benchmark-analysis/)
25. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. In Transactions of the ACL. [↗](https://doi.org/10.1162/tacl_a_00240) [our card](https://md.research.elamur.ai/free/papers/webster-2018-mind-gap-balanced-corpus-gendered/)
26. Terry Winograd. 1972. Understanding Natural Language. Academic Press, Inc., Orlando, FL, USA. [↗](https://doi.org/10.1016/0010-0285(72)90002-3)
27. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15– 20, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2003) [our card](https://md.research.elamur.ai/free/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/)