---
title: "Evaluation Metrics For End-to-End Coreference — Cai 2010"
description: "End-to-end coreference systems must detect mentions automatically, so their output contains \"twinless\" mentions: system mentions absent from the gold…"
authors: ["Jie Cai", "Michael Strube"]
year: 2010
venue: "Proceedings of the SIGDIAL 2010 Conference"
source: "https://aclanthology.org/W10-4305/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/cai-2010-evaluation-metrics-end-end-coreference/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/cai-2010-evaluation-metrics-end-end-coreference/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2010](https://md.research.elamur.ai/free/papers/2010/) / Evaluation Metrics For End-to-End Coreference Resolution Systems

# Evaluation Metrics For End-to-End Coreference Resolution Systems

**Authors:** [Jie Cai](https://md.research.elamur.ai/free/authors/jie-cai/), [Michael Strube](https://md.research.elamur.ai/free/authors/michael-strube/) · **Year:** 2010 · **Venue:** Proceedings of the SIGDIAL 2010 Conference · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W10-4305/) · [PDF](https://aclanthology.org/W10-4305.pdf) · [OpenAlex](https://openalex.org/W118383491)

**Citations:** 56 citations (openalex-snapshot · 2026-10-14) · [31](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
End-to-end coreference systems must detect mentions automatically, so their output contains "twinless" mentions: system mentions absent from the gold key, and gold mentions the system missed. Standard metrics assume key and response share the same mention set. Researchers have patched them in inconsistent and loosely documented ways, so published scores cannot be compared. Cai and Strube analyse MUC, B3 and CEAF and the existing B3 variants (B3 0 and B3 all from Stoyanov et al. 2009, and B3 r&n from Rahman & Ng 2009). Using small constructed examples, they show these variants are either too lenient or shift with the mention tagger rather than with resolution quality. They propose B3 sys and CEAF sys, which handle twinless mentions differently for precision and for recall. They then test the metrics with the BART reimplementation of Soon et al. (2001), fed by two mention taggers, and compare against Bengtson & Roth (2008). The new variants rank systems consistently with MUC and intuition, giving the field a reproducible basis for comparing end-to-end systems.

## Key points
- **How B3 sys treats twinless mentions:**
  - Missed gold mentions are inserted into the response as singletons.
  - Spurious system mentions left unresolved are dropped.
  - Spurious system mentions that were wrongly linked are added to the key, so they lower precision.
  - Recall is computed only over the original key chains.
- **Worked example (key {a b c} vs. response {a b d}):**
  - B3 0 scores P 1.0 / F 0.615, because it ignores the spurious mention d.
  - B3 sys scores P 0.667 / R 0.556 / F 0.606.
- **Weakness of B3 all:** it awards higher precision when a mention tagger merely extracts additional mentions, such as extra singletons, even if the coreference links are identical.
- **Realistic setting (Soon, SM2 tagger vs. SM1 tagger):**
  - SM2 is better under B3 sys (F 73.9 vs. 70.8) and under CEAF sys (66.3 vs. 63.7).
  - B3 0 reverses this ranking (68.4 vs. 70.1).
  - CEAF orig gives very low, unstable scores (58.2 vs. 48.5).
- **Code availability:** implementations of both metrics, with extended examples, are released open source at h-its.org/nlp/download.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/) · [Part-of-speech tagging](https://md.research.elamur.ai/free/topics/part-of-speech-tagging/)

## Cited by

Cited by

31 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 18 not yet.

- [Sequential Cross-Document Coreference Resolution](https://md.research.elamur.ai/free/papers/allaway-2021-sequential-cross-document-coreference-resolution/)
- [Model-based Annotation of Coreference](https://md.research.elamur.ai/free/papers/aralikatte-2020-model-based-annotation-coreference/)
- [End-to-End Coreference Resolution via Hypergraph Partitioning](https://md.research.elamur.ai/free/papers/cai-2010-end-end-coreference-resolution-hypergraph/)
- [Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle*](https://md.research.elamur.ai/free/papers/cao-2021-gender-inclusive-coreference-resolution-analysis/)
- [Linguistically Aware Coreference Evaluation Metrics](https://md.research.elamur.ai/free/papers/chen-2013-linguistically-aware-coreference-evaluation-metrics/)
- [Easy Victories and Uphill Battles in Coreference Resolution](https://md.research.elamur.ai/free/papers/durrett-2013-easy-victories-uphill-battles-coreference/)
- [Cross-Document Co-Reference Resolution using Sample-Based Clustering with Knowledge Enrichment](https://md.research.elamur.ai/free/papers/dutta-2015-cross-document-co-reference-resolution/)
- [End-to-end Deep Reinforcement Learning Based Coreference Resolution](https://md.research.elamur.ai/free/papers/fei-2019-end-end-deep-reinforcement-learning/)
- [An Incremental Entity-Mention Model for Coreference Resolution with Restrictive Antecedent Accessibility](https://md.research.elamur.ai/free/papers/klenner-2011-incremental-entity-mention-model-coreference/)
- [CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes](https://md.research.elamur.ai/free/papers/pradhan-2011-conll-2011-shared-task-modeling/)
- [CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes](https://md.research.elamur.ai/free/papers/pradhan-2012-conll-2012-shared-task-modeling/)
- [Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation](https://md.research.elamur.ai/free/papers/pradhan-2014-scoring-coreference-partitions-predicted-mentions/)
- [RelaxCor Participation in CoNLL Shared Task on Coreference Resolution](https://md.research.elamur.ai/free/papers/sapena-2011-relaxcor-participation-conll-shared-task/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cai2010evaluation,
  title = {{Evaluation Metrics For End-to-End Coreference Resolution Systems}},
  author = {Cai, Jie and Strube, Michael},
  booktitle = {Proceedings of the SIGDIAL 2010 Conference},
  year = {2010},
  url = {https://aclanthology.org/W10-4305/}
}
```

- [← previous Evaluating FrameNet-style semantic parsing: the role of coverage gaps in FrameNet](https://md.research.elamur.ai/free/papers/palmer-2010-evaluating-framenet-style-semantic-parsing/)
- [next → Evaluation of Dependency Parsers on Unbounded Dependencies](https://md.research.elamur.ai/free/papers/aclbulk-2010-nivre-evaluation-dependency-parsers-unbounded-dependencies/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/papers/2010/)

## References

22 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bagga, Amit & Breck Baldwin (1998). Algorithms for scoring coreference chains. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain, 28–30 May 1998, pp. 563–566. [find](https://scholar.google.com/scholar?q=Bagga%2C+Amit+%26+Breck+Baldwin+%281998%29%2E+Algorithms+for+scoring+coreference+chains%2E+In+Proceedings+of+the+1st+International+Conference+on+Language+Resources+and+Evaluation%2C+Granada%2C+Spain%2C+28%E2%80%9330+May+1998%2C+) [our card](https://md.research.elamur.ai/free/papers/bagga-1998-algorithms-scoring-coreference-chains/)
2. Bengtson, Eric & Dan Roth (2008). Understanding the value of features for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Waikiki, Honolulu, Hawaii, 25-27 October 2008, pp. 294–303. [find](https://scholar.google.com/scholar?q=Bengtson%2C+Eric+%26+Dan+Roth+%282008%29%2E+Understanding+the+value+of+features+for+coreference+resolution%2E+In+Proceedings+of+the+2008+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+Waikiki%2C+Ho) [our card](https://md.research.elamur.ai/free/papers/bengtson-2008-understanding-value-features-coreference-resolution/)
3. Culotta, Aron, Michael Wick & Andrew McCallum (2007). First-order probabilistic models for coreference resolution. In Proceedings of Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics, Rochester, N.Y., 22–27 April 2007, pp.… [find](https://scholar.google.com/scholar?q=Culotta%2C+Aron%2C+Michael+Wick+%26+Andrew+McCallum+%282007%29%2E+First%2Dorder+probabilistic+models+for+coreference+resolution%2E+In+Proceedings+of+Human+Language+Technologies+2007%3A+The+Conference+of+the+North+Ameri) [our card](https://md.research.elamur.ai/free/papers/culotta-2007-first-order-probabilistic-models-coreference/)
4. Denis, Pascal & Jason Baldridge (2007). Joint determination of anaphoricity and coreference resolution using integer programming. In Proceedings of Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics, Rochester, N.Y., 22–27… [find](https://scholar.google.com/scholar?q=Denis%2C+Pascal+%26+Jason+Baldridge+%282007%29%2E+Joint+determination+of+anaphoricity+and+coreference+resolution+using+integer+programming%2E+In+Proceedings+of+Human+Language+Technologies+2007%3A+The+Conference+of+) [our card](https://md.research.elamur.ai/free/papers/denis-2007-joint-determination-anaphoricity-coreference-resolution/)
5. Haghighi, Aria & Dan Klein (2007). Unsupervised coreference resolution in a nonparametric Bayesian model. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, Prague, Czech Republic, 23–30 June 2007, pp. 848–855. [find](https://scholar.google.com/scholar?q=Haghighi%2C+Aria+%26+Dan+Klein+%282007%29%2E+Unsupervised+coreference+resolution+in+a+nonparametric+Bayesian+model%2E+In+Proceedings+of+the+45th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+Pr) [our card](https://md.research.elamur.ai/free/papers/haghighi-2007-unsupervised-coreference-resolution-nonparametric-bayesian/)
6. Luo, Xiaoqiang (2005). On coreference resolution performance metrics. In Proceedings of the Human Language Technology Conference and the 2005 Conference on Empirical Methods in Natural Language Processing, Vancouver, B.C., Canada, 6–8 October 2005, pp. 25–32. [find](https://scholar.google.com/scholar?q=Luo%2C+Xiaoqiang+%282005%29%2E+On+coreference+resolution+performance+metrics%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+and+the+2005+Conference+on+Empirical+Methods+in+Natural+Language+Proces) [our card](https://md.research.elamur.ai/free/papers/luo-2005-coreference-resolution-performance-metrics/)
7. Luo, Xiaoqiang, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla & Salim Roukos (2004). A mention-synchronous coreference resolution algorithm based on the Bell Tree. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain, 21–26 July 2004, pp. 136–… [find](https://scholar.google.com/scholar?q=Luo%2C+Xiaoqiang%2C+Abe+Ittycheriah%2C+Hongyan+Jing%2C+Nanda+Kambhatla+%26+Salim+Roukos+%282004%29%2E+A+mention%2Dsynchronous+coreference+resolution+algorithm+based+on+the+Bell+Tree%2E+In+Proceedings+of+the+42nd+Annual+M) [our card](https://md.research.elamur.ai/free/papers/luo-2004-mention-synchronous-coreference-resolution-algorithm/)
8. Mitchell, Alexis, Stephanie Strassel, Shudong Huang & Ramez Zakhary (2004). [find](https://scholar.google.com/scholar?q=Mitchell%2C+Alexis%2C+Stephanie+Strassel%2C+Shudong+Huang+%26+Ramez+Zakhary+%282004%29%2E)
9. ACE 2004 Multilingual Training B 0 3 Pr 91.1 91.3 3 [find](https://scholar.google.com/scholar?q=ACE+2004+Multilingual+Training+B+0+3+Pr+91%2E1+91%2E3+3)
10. B all Pr 85.8 87.1 3 B r&n Pr 78.7 84.9 F 70.1 68.4 R 65.1 64.3 F 74.0 73.9 R 65.1 64.3 F 71.2 73.2 CEAF r&n Pr 59.8 66.2 CEAF orig Pr 39.9 56.6 R 62.1 60.0 R 62.0 60.0 F 48.5 58.2 F 60.9 62.9 Corpus. LDC2005T09, Philadelphia, Penn.: Linguistic Data Consortium. [find](https://scholar.google.com/scholar?q=B+all+Pr+85%2E8+87%2E1+3+B+r%26n+Pr+78%2E7+84%2E9+F+70%2E1+68%2E4+R+65%2E1+64%2E3+F+74%2E0+73%2E9+R+65%2E1+64%2E3+F+71%2E2+73%2E2+CEAF+r%26n+Pr+59%2E8+66%2E2+CEAF+orig+Pr+39%2E9+56%2E6+R+62%2E1+60%2E0+R+62%2E0+60%2E0+F+48%2E5+58%2E2+F+60%2E9+62%2E9+Corpus%2E)
11. Nicolae, Cristina & Gabriel Nicolae (2006). BestCut: A graph algorithm for coreference resolution. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, Sydney, Australia, 22–23 July 2006, pp. 275–283. [find](https://scholar.google.com/scholar?q=Nicolae%2C+Cristina+%26+Gabriel+Nicolae+%282006%29%2E+BestCut%3A+A+graph+algorithm+for+coreference+resolution%2E+In+Proceedings+of+the+2006+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+Sydney%2C+Au) [our card](https://md.research.elamur.ai/free/papers/nicolae-2006-bestcut-graph-algorithm-coreference-resolution/)
12. Ponzetto, Simone Paolo & Michael Strube (2006). Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, New York, N.Y., 4–9 June 200… [find](https://scholar.google.com/scholar?q=Ponzetto%2C+Simone+Paolo+%26+Michael+Strube+%282006%29%2E+Exploiting+semantic+role+labeling%2C+WordNet+and+Wikipedia+for+coreference+resolution%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+N) [our card](https://md.research.elamur.ai/free/papers/ponzetto-2006-exploiting-semantic-role-labeling-wordnet/)
13. Rahman, Altaf & Vincent Ng (2009). Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, Singapore, 6-7 August 2009, pp. 968–977. [find](https://scholar.google.com/scholar?q=Rahman%2C+Altaf+%26+Vincent+Ng+%282009%29%2E+Supervised+models+for+coreference+resolution%2E+In+Proceedings+of+the+2009+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+Singapore%2C+6%2D7+August+2009%2C+) [our card](https://md.research.elamur.ai/free/papers/rahman-2009-supervised-models-coreference-resolution/)
14. Rand, William R. (1971). Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(336):846–850. [find](https://scholar.google.com/scholar?q=Rand%2C+William+R%2E+%281971%29%2E+Objective+criteria+for+the+evaluation+of+clustering+methods%2E+Journal+of+the+American+Statistical+Association%2C+66%28336%29%3A846%E2%80%93850%2E)
15. Recasens, Marta & Eduard Hovy (2010). BLANC: Implementing the Rand index for coreference evaluation. Submitted. [find](https://scholar.google.com/scholar?q=Recasens%2C+Marta+%26+Eduard+Hovy+%282010%29%2E+BLANC%3A+Implementing+the+Rand+index+for+coreference+evaluation%2E+Submitted%2E)
16. Soon, Wee Meng, Hwee Tou Ng & Daniel Chung Yong Lim (2001). A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4):521–544. [find](https://scholar.google.com/scholar?q=Soon%2C+Wee+Meng%2C+Hwee+Tou+Ng+%26+Daniel+Chung+Yong+Lim+%282001%29%2E+A+machine+learning+approach+to+coreference+resolution+of+noun+phrases%2E+Computational+Linguistics%2C+27%284%29%3A521%E2%80%93544%2E) [our card](https://md.research.elamur.ai/free/papers/soon-2001-machine-learning-approach-coreference-resolution/)
17. Stoyanov, Veselin, Nathan Gilbert, Claire Cardie & Ellen Riloff (2009). Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International… [find](https://scholar.google.com/scholar?q=Stoyanov%2C+Veselin%2C+Nathan+Gilbert%2C+Claire+Cardie+%26+Ellen+Riloff+%282009%29%2E+Conundrums+in+noun+phrase+coreference+resolution%3A+Making+sense+of+the+state%2Dof%2Dthe%2Dart%2E+In+Proceedings+of+the+Joint+Conference+o) [our card](https://md.research.elamur.ai/free/papers/stoyanov-2009-conundrums-noun-phrase-coreference-resolution/)
18. Versley, Yannick, Simone Paolo Ponzetto, Massimo Poesio, Vladimir Eidelman, Alan Jern, Jason Smith, Xiaofeng Yang & Alessandro Moschitti (2008). BART: A modular toolkit for coreference resolution. In Companion Volume to the Proceedings of the 46th Annual Meeting of the Association for Computational… [find](https://scholar.google.com/scholar?q=Versley%2C+Yannick%2C+Simone+Paolo+Ponzetto%2C+Massimo+Poesio%2C+Vladimir+Eidelman%2C+Alan+Jern%2C+Jason+Smith%2C+Xiaofeng+Yang+%26+Alessandro+Moschitti+%282008%29%2E+BART%3A+A+modular+toolkit+for+coreference+resolution%2E+In+) [our card](https://md.research.elamur.ai/free/papers/versley-2008-bart-modular-toolkit-coreference-resolution/)
19. Vilain, Marc, John Burger, John Aberdeen, Dennis Connolly & Lynette Hirschman (1995). A model-theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference (MUC-6), pp. 45–52. San Mateo, Cal.: Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Vilain%2C+Marc%2C+John+Burger%2C+John+Aberdeen%2C+Dennis+Connolly+%26+Lynette+Hirschman+%281995%29%2E+A+model%2Dtheoretic+coreference+scoring+scheme%2E+In+Proceedings+of+the+6th+Message+Understanding+Conference+%28MUC%2D6%29%2C+) [our card](https://md.research.elamur.ai/free/papers/vilain-1995-model-theoretic-coreference-scoring-scheme/)
20. Witten, Ian H. & Eibe Frank (2005). Data Mining: Practical Machine Learning Tools and Techniques (2nd ed.). San Francisco, Cal.: Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Witten%2C+Ian+H%2E+%26+Eibe+Frank+%282005%29%2E+Data+Mining%3A+Practical+Machine+Learning+Tools+and+Techniques+%282nd+ed%2E%29%2E+San+Francisco%2C+Cal%2E%3A+Morgan+Kaufmann%2E)
21. Yang, Xiaofeng, Jian Su & Chew Lim Tan (2008). A twin-candidate model for learning-based anaphora resolution. Computational Linguistics, 34(3):327–356. [find](https://scholar.google.com/scholar?q=Yang%2C+Xiaofeng%2C+Jian+Su+%26+Chew+Lim+Tan+%282008%29%2E+A+twin%2Dcandidate+model+for+learning%2Dbased+anaphora+resolution%2E+Computational+Linguistics%2C+34%283%29%3A327%E2%80%93356%2E) [our card](https://md.research.elamur.ai/free/papers/yang-2008-twin-candidate-model-learning-based/)
22. A B 3 sys Example Output [find](https://scholar.google.com/scholar?q=A+B+3+sys+Example+Output)