---
title: "Realistic Evaluation Principles for — Cattan 2021"
description: "Cross-document (CD) coreference resolution links mentions of the same entity or event across separate texts."
authors: ["Arie Cattan", "Alon Eirew", "Gabriel Stanovsky", "Mandar Joshi", "Ido Dagan"]
year: 2021
venue: "Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics"
source: "https://aclanthology.org/2021.starsem-1.13/"
doi: "10.18653/v1/2021.starsem-1.13"
arxiv: "2106.04192"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cattan-2021-realistic-evaluation-principles-cross-document/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cattan-2021-realistic-evaluation-principles-cross-document/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Realistic Evaluation Principles for Cross-document Coreference Resolution

# Realistic Evaluation Principles for Cross-document Coreference Resolution

**Authors:** [Arie Cattan](https://md.research.elamur.ai/free/en/authors/arie-cattan/), [Alon Eirew](https://md.research.elamur.ai/free/en/authors/alon-eirew/), [Gabriel Stanovsky](https://md.research.elamur.ai/free/en/authors/gabriel-stanovsky/), [Mandar Joshi](https://md.research.elamur.ai/free/en/authors/mandar-joshi/), [Ido Dagan](https://md.research.elamur.ai/free/en/authors/ido-dagan/) · **Year:** 2021 · **Venue:** Proceedings of \*SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.starsem-1.13/) · [PDF](https://aclanthology.org/2021.starsem-1.13.pdf) · [DOI 10.18653/v1/2021.starsem-1.13](https://doi.org/10.18653/v1/2021.starsem-1.13) · [arXiv 2106.04192](https://arxiv.org/abs/2106.04192) · [OpenAlex](https://openalex.org/W3187020651)

**Citations:** 12 citations (openalex-snapshot · 2026-10-14) · [19](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Cross-document (CD) coreference resolution links mentions of the same entity or event across separate texts. Reported scores on the ECB+ benchmark reach about 80 F1. The authors argue these scores come from overly lenient evaluation habits. Prior work typically feeds models gold mentions, counts trivial singleton clusters inside coreference metrics, and runs models within ECB+'s artificial subtopics, which avoids the lexical ambiguity the corpus was built to test.

They propose two principles:
- **Evaluate on predicted mentions.** Mention detection is scored separately with span F1, including singletons. Coreference linking is scored only on non-singleton clusters, following CoNLL-2012.
- **Report results at the topic level.** Models should be evaluated without relying on subtopic document clustering.

To provide baselines, they build an end-to-end CD model derived from e2e-coref. Under the old protocol it beats earlier systems. Under the realistic one its score falls by 33 F1. This shows that current results overstate real-world ability and leaves substantial room for progress.

## Key points
- Data: the ECB+ corpus groups documents into topics, each made of two lexically similar subtopics. Previous systems compared were Barhom et al. (2019) and Meged et al. (2020).
- A toy example shows the "singleton effect." Once singletons are counted, B3, CEAFe and LEA can favor the system with weaker coreference links, while the link-based MUC metric is unaffected.
- Model pipeline:
  - span encoding in the style of Lee et al. (2017)
  - mention detector
  - pairwise scorer
  - agglomerative clustering
- Mention detection reaches 80.1 F1 (precision 84.7, recall 76).
- Moving from gold to predicted mentions lowers CoNLL F1 from 71.1 to 54.4 at the subtopic level. Topic-level scores are lower still: 62.0 with gold mentions and 48.6 with predicted mentions.
- Code is available at github.com/ariecattan/coref. Experiments ran on a single 12GB GPU.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/)

## Cited by

Cited by

19 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 14 not yet.

- [Cross-document Coreference Resolution over Predicted Mentions](https://md.research.elamur.ai/free/en/papers/cattan-2021-cross-document-coreference-resolution-over/)
- [Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text](https://md.research.elamur.ai/free/en/papers/coref-2026-greenfeld-beyond-word-boundaries-hebrew-coreference/)
- [Contrastive Representation Learning for Cross-Document Coreference Resolution of Events and Entities](https://md.research.elamur.ai/free/en/papers/hsu-2022-contrastive-representation-learning-cross-document/)
- [Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models](https://md.research.elamur.ai/free/en/papers/min-2024-synergetic-event-understanding-collaborative-approach/)
- [What happens before and after: Multi-Event Commonsense in Event Coreference Resolution](https://md.research.elamur.ai/free/en/papers/ravi-2023-what-happens-before-after-multi/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cattan2021realistic,
  title = {{Realistic Evaluation Principles for Cross-document Coreference Resolution}},
  author = {Cattan, Arie and Eirew, Alon and Stanovsky, Gabriel and Joshi, Mandar and Dagan, Ido},
  booktitle = {Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics},
  year = {2021},
  doi = {10.18653/v1/2021.starsem-1.13},
  eprint = {2106.04192},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.starsem-1.13}
}
```

- [← previous QuestEval: Summarization Asks for Fact-based Evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/)
- [next → Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

23 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 79–85, Montreal, Quebec, Canada. Associa… [find](https://scholar.google.com/scholar?q=Amit+Bagga+and+Breck+Baldwin%2E+1998%2E+Entity%2Dbased+cross%2Ddocument+coreferencing+using+the+vector+space+model%2E+In+36th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+17th+Internation)
2. Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019. Revisiting joint modeling of cross-document entity and event coreference resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4179–4189, Florence, Ital… [find](https://scholar.google.com/scholar?q=Shany+Barhom%2C+Vered+Shwartz%2C+Alon+Eirew%2C+Michael+Bugert%2C+Nils+Reimers%2C+and+Ido+Dagan%2E+2019%2E+Revisiting+joint+modeling+of+cross%2Ddocument+entity+and+event+coreference+resolution%2E+In+Proceedings+of+the+5) [our card](https://md.research.elamur.ai/free/en/papers/barhom-2019-revisiting-joint-modeling-cross-document/)
3. C. Bejan and Sanda M. Harabagiu. 2014. Unsupervised event coreference resolution. Computational Linguistics, 40:311–347. [find](https://scholar.google.com/scholar?q=C%2E+Bejan+and+Sanda+M%2E+Harabagiu%2E+2014%2E+Unsupervised+event+coreference+resolution%2E+Computational+Linguistics%2C+40%3A311%E2%80%93347%2E) [our card](https://md.research.elamur.ai/free/en/papers/bejan-2014-unsupervised-event-coreference-resolution/)
4. Cosmin Bejan and Sanda Harabagiu. 2010. Unsupervised event coreference resolution with rich linguistic features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1412–1422, Uppsala, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cosmin+Bejan+and+Sanda+Harabagiu%2E+2010%2E+Unsupervised+event+coreference+resolution+with+rich+linguistic+features%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguist) [our card](https://md.research.elamur.ai/free/en/papers/bejan-2010-unsupervised-event-coreference-resolution-rich/)
5. Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2021. Cross-document coreference resolution over predicted mentions. In Findings of the Association for Computational Linguistics: ACL 2021, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Arie+Cattan%2C+Alon+Eirew%2C+Gabriel+Stanovsky%2C+Mandar+Joshi%2C+and+Ido+Dagan%2E+2021%2E+Cross%2Ddocument+coreference+resolution+over+predicted+mentions%2E+In+Findings+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/cattan-2021-cross-document-coreference-resolution-over/)
6. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 172–181, Brussels, Belgium. Association for Computational… [find](https://scholar.google.com/scholar?q=Hong+Chen%2C+Zhenhua+Fan%2C+Hao+Lu%2C+Alan+Yuille%2C+and+Shu+Rong%2E+2018%2E+PreCo%3A+A+large%2Dscale+dataset+in+preschool+vocabulary+for+coreference+resolution%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Met) [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-preco-large-scale-dataset-preschool/)
7. Prafulla Kumar Choubey and Ruihong Huang. 2017. Event coreference resolution by iteratively unfolding inter-dependencies among events. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2124–2133, Copenhagen, Denmark. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Prafulla+Kumar+Choubey+and+Ruihong+Huang%2E+2017%2E+Event+coreference+resolution+by+iteratively+unfolding+inter%2Ddependencies+among+events%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Nat) [our card](https://md.research.elamur.ai/free/en/papers/choubey-2017-event-coreference-resolution-iteratively-unfolding/)
8. Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4545– 4552, Reykjavik, Iceland. European Language Resources Associ… [find](https://scholar.google.com/scholar?q=Agata+Cybulska+and+Piek+Vossen%2E+2014%2E+Using+a+sledgehammer+to+crack+a+nut%3F+lexical+diversity+and+event+coreference+resolution%2E+In+Proceedings+of+the+Ninth+International+Conference+on+Language+Resource) [our card](https://md.research.elamur.ai/free/en/papers/cybulska-2014-sledgehammer-crack-nut-lexical-diversity/)
9. Agata Cybulska and Piek Vossen. 2015. Translating granularity of event slots into features for event coreference resolution. In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 1–10, Denver, Colorado. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Agata+Cybulska+and+Piek+Vossen%2E+2015%2E+Translating+granularity+of+event+slots+into+features+for+event+coreference+resolution%2E+In+Proceedings+of+the+The+3rd+Workshop+on+EVENTS%3A+Definition%2C+Detection%2C+Co) [our card](https://md.research.elamur.ai/free/en/papers/cybulska-2015-translating-granularity-event-slots-into/)
10. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP… [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Omer+Levy%2C+Luke+Zettlemoyer%2C+and+Daniel+Weld%2E+2019%2E+BERT+for+coreference+resolution%3A+Baselines+and+analysis%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Languag) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-bert-coreference-resolution-baselines-analysis/)
11. Kian Kenyon-Dean, Jackie Chi Kit Cheung, and Doina Precup. 2018. Resolving event coreference with supervised representation learning and clustering-oriented regularization. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 1–10, New Orleans, Louisiana. Asso… [find](https://scholar.google.com/scholar?q=Kian+Kenyon%2DDean%2C+Jackie+Chi+Kit+Cheung%2C+and+Doina+Precup%2E+2018%2E+Resolving+event+coreference+with+supervised+representation+learning+and+clustering%2Doriented+regularization%2E+In+Proceedings+of+the+Seven) [our card](https://md.research.elamur.ai/free/en/papers/dean-2018-resolving-event-coreference-supervised-representation/)
12. Heeyoung Lee, Marta Recasens, Angel Chang, Mihai Surdeanu, and Dan Jurafsky. 2012. Joint entity and event coreference resolution across documents. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 489–50… [find](https://scholar.google.com/scholar?q=Heeyoung+Lee%2C+Marta+Recasens%2C+Angel+Chang%2C+Mihai+Surdeanu%2C+and+Dan+Jurafsky%2E+2012%2E+Joint+entity+and+event+coreference+resolution+across+documents%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empiri) [our card](https://md.research.elamur.ai/free/en/papers/lee-2012-joint-entity-event-coreference-resolution/)
13. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Luheng+He%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2017%2E+End%2Dto%2Dend+neural+coreference+resolution%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/lee-2017-end-end-neural-coreference-resolution/)
14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
15. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiaoqiang+Luo%2E+2005%2E+On+coreference+resolution+performance+metrics%2E+In+Proceedings+of+Human+Language+Technology+Conference+and+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+25%E2%80%93) [our card](https://md.research.elamur.ai/free/en/papers/luo-2005-coreference-resolution-performance-metrics/)
16. Yehudit Meged, Avi Caciularu, Vered Shwartz, and Ido Dagan. 2020. Paraphrasing vs coreferring: Two sides of the same coin. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4897–4907, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yehudit+Meged%2C+Avi+Caciularu%2C+Vered+Shwartz%2C+and+Ido+Dagan%2E+2020%2E+Paraphrasing+vs+coreferring%3A+Two+sides+of+the+same+coin%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pag) [our card](https://md.research.elamur.ai/free/en/papers/meged-2020-paraphrasing-vs-coreferring-two-sides/)
17. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642, Berlin, Germany. Associ… [find](https://scholar.google.com/scholar?q=Nafise+Sadat+Moosavi+and+Michael+Strube%2E+2016%2E+Which+coreference+evaluation+metric+do+you+trust%3F+a+proposal+for+a+link%2Dbased+entity+aware+metric%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Assoc) [our card](https://md.research.elamur.ai/free/en/papers/moosavi-2016-which-coreference-evaluation-metric-do/)
18. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+CoNLL%2D2012+shared+task%3A+Modeling+multilingual+unrestricted+coreference+in+OntoNotes%2E+In+Joint+Conference+on+EM) [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/)
19. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 968–977, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Altaf+Rahman+and+Vincent+Ng%2E+2009%2E+Supervised+models+for+coreference+resolution%2E+In+Proceedings+of+the+2009+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+968%E2%80%93977%2C+Singapore%2E+As) [our card](https://md.research.elamur.ai/free/en/papers/rahman-2009-supervised-models-coreference-resolution/)
20. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language… [find](https://scholar.google.com/scholar?q=Veselin+Stoyanov%2C+Nathan+Gilbert%2C+Claire+Cardie%2C+and+Ellen+Riloff%2E+2009%2E+Conundrums+in+noun+phrase+coreference+resolution%3A+Making+sense+of+the+state%2Dof%2Dthe%2Dart%2E+In+Proceedings+of+the+Joint+Conference+) [our card](https://md.research.elamur.ai/free/en/papers/stoyanov-2009-conundrums-noun-phrase-coreference-resolution/)
21. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang+and+Fien+De+Meulder%2E+2003%2E+Introduction+to+the+CoNLL%2D2003+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+In+Proceedings+of+the+Seventh+Conference+on+Natural+Languag)
22. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995. [find](https://scholar.google.com/scholar?q=Marc+Vilain%2C+John+Burger%2C+John+Aberdeen%2C+Dennis+Connolly%2C+and+Lynette+Hirschman%2E+1995%2E+A+model%2Dtheoretic+coreference+scoring+scheme%2E+In+Sixth+Message+Understanding+Conference+%28MUC%2D6%29%3A+Proceedings+of+a) [our card](https://md.research.elamur.ai/free/en/papers/vilain-1995-model-theoretic-coreference-scoring-scheme/)
23. Bishan Yang, Claire Cardie, and Peter Frazier. 2015. A hierarchical distance-dependent Bayesian model for event coreference resolution. Transactions of the Association for Computational Linguistics, 3:517–528. [find](https://scholar.google.com/scholar?q=Bishan+Yang%2C+Claire+Cardie%2C+and+Peter+Frazier%2E+2015%2E+A+hierarchical+distance%2Ddependent+Bayesian+model+for+event+coreference+resolution%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+3)