---
title: "Coreference Resolution Evaluation for Higher — Tuggener 2014"
description: "Standard coreference metrics (MUC, BCUB, CEAF, BLANC) score system output as a clustering problem and ignore the linear order of mentions."
authors: ["Don Tuggener"]
year: 2014
venue: "Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers"
source: "https://aclanthology.org/E14-4045/"
doi: "10.3115/v1/e14-4045"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/tuggener-2014-coreference-resolution-evaluation-higher-level/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/tuggener-2014-coreference-resolution-evaluation-higher-level/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2014](https://md.research.elamur.ai/free/papers/2014/) / Coreference Resolution Evaluation for Higher Level Applications

# Coreference Resolution Evaluation for Higher Level Applications

**Authors:** [Don Tuggener](https://md.research.elamur.ai/free/authors/don-tuggener/) · **Year:** 2014 · **Venue:** Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/E14-4045/) · [PDF](https://aclanthology.org/E14-4045.pdf) · [DOI 10.3115/v1/e14-4045](https://doi.org/10.3115/v1/e14-4045) · [OpenAlex](https://openalex.org/W4252231743)

**Citations:** 14 citations (openalex-snapshot · 2026-10-14) · [9](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Standard coreference metrics (MUC, BCUB, CEAF, BLANC) score system output as a clustering problem and ignore the linear order of mentions. This makes the scores hard to translate into benefit for downstream tasks. Tuggener proposes an evaluation framework built around three ways applications consume coreference:

- **Entity-distribution models** such as entity grids or event sequences. These need a gapless chain, so a mention counts as correct only if its immediate antecedent matches the key.
- **Applications needing a local nominal antecedent** such as summarization or MT gender agreement. Here the nearest preceding noun in the response must be a true antecedent.
- **Query-driven tasks** such as sentiment analysis or biomedical relation mining. Response chains are aligned to key chains through an anchor (the first nominal mention), and the metric measures entity detection and entity mention quality.

Recall and Precision both include a "wrong linkage" error class, so they cover all key and system mentions. Scores can also be broken down by any mention feature, such as PoS or named entity class. Tested on three systems on the CoNLL 2012 English test set, the metrics give detailed diagnostics while broadly preserving the system ranking from the MELA score.

## Key points
- Recall is defined as tp/(tp+wl+fn) and Precision as tp/(tp+wl+fp). The wrong linkage (wl) concept comes from Durrett and Klein (2013). Singleton entities are excluded.
- The evaluated systems were Berkeley (Durrett and Klein, 2013), IMS (Björkelund and Farkas, 2012) and Stanford (Lee et al., 2013). Their MELA scores were 61.62, 57.42 and 55.69 respectively.
- Total F for immediate antecedents was 56.76 (Berkeley), 55.48 (IMS) and 52.19 (Stanford). For inferred nominal antecedents it was 55.37, 50.25 and 46.97.
- Nouns score better under the nominal-antecedent criterion, while pronouns score better under the immediate-antecedent criterion. IMS beats Berkeley on pronouns for immediate antecedents. Its lower Recall comes mostly from false negatives, not wrong linkages.
- In the anchor-based evaluation, overall Fφ was 63.41 (Berkeley), 55.24 (IMS) and 55.27 (Stanford). By entity class, IMS is weak on PERSON but best on GPE. Of the 4532 key chains, 476 (10.05%) contain no nominal mention. The scorers are freely available.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/)

## Cited by

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 5 not yet.

- [Findings of the Second Shared Task on Multilingual Coreference Resolution](https://md.research.elamur.ai/free/papers/abokrtsky-2023-findings-second-shared-task-multilingual/)
- [Recall Error Analysis for Coreference Resolution](https://md.research.elamur.ai/free/papers/martschat-2014-recall-error-analysis-coreference-resolution/)
- [Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric](https://md.research.elamur.ai/free/papers/moosavi-2016-which-coreference-evaluation-metric-do/)
- [The Universal Anaphora Scorer 2.0](https://md.research.elamur.ai/free/papers/yu-2023-universal-anaphora-scorer-2-0/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tuggener2014coreference,
  title = {{Coreference Resolution Evaluation for Higher Level Applications}},
  author = {Tuggener, Don},
  booktitle = {Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers},
  year = {2014},
  doi = {10.3115/v1/e14-4045},
  url = {https://doi.org/10.3115/v1/e14-4045}
}
```

- [← previous Convolutional Neural Networks for Sentence Classification](https://md.research.elamur.ai/free/papers/kim-2014-convolutional-neural-networks-sentence-classification/)
- [next → Creating a massively parallel Bible corpus](https://md.research.elamur.ai/free/papers/mayer-2014-creating-massively-parallel-bible-corpus/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/papers/2014/)

## References

11 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Comput. Linguist., 34(1):1–34, March. [find](https://scholar.google.com/scholar?q=Regina+Barzilay+and+Mirella+Lapata%2E+2008%2E+Modeling+local+coherence%3A+An+entity%2Dbased+approach%2E+Comput%2E+Linguist%2E%2C+34%281%29%3A1%E2%80%9334%2C+March%2E) [our card](https://md.research.elamur.ai/free/papers/barzilay-2008-modeling-local-coherence-entity-based/)
2. Anders Björkelund and Richárd Farkas. 2012. Data-driven multilingual coreference resolution using resolver stacking. In Joint Conference on EMNLP and CoNLL - Shared Task, CoNLL ’12, pages 49–55, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anders+Bjo%CC%88rkelund+and+Richa%CC%81rd+Farkas%2E+2012%2E+Data%2Ddriven+multilingual+coreference+resolution+using+resolver+stacking%2E+In+Joint+Conference+on+EMNLP+and+CoNLL+%2D+Shared+Task%2C+CoNLL+%E2%80%9912%2C+pages+49%E2%80%9355%2C+Str) [our card](https://md.research.elamur.ai/free/papers/bjorkelund-2012-data-driven-multilingual-coreference-resolution/)
3. Chen Chen and Vincent Ng. 2013. Linguistically aware coreference evaluation metrics. In Proceedings of the 6th International Joint Conference on Natural Language Processing, pages 1366–1374. [find](https://scholar.google.com/scholar?q=Chen+Chen+and+Vincent+Ng%2E+2013%2E+Linguistically+aware+coreference+evaluation+metrics%2E+In+Proceedings+of+the+6th+International+Joint+Conference+on+Natural+Language+Processing%2C+pages+1366%E2%80%931374%2E) [our card](https://md.research.elamur.ai/free/papers/chen-2013-linguistically-aware-coreference-evaluation-metrics/)
4. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Greg+Durrett+and+Dan+Klein%2E+2013%2E+Easy+victories+and+uphill+battles+in+coreference+resolution%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+Seattle%2C+Washington) [our card](https://md.research.elamur.ai/free/papers/durrett-2013-easy-victories-uphill-battles-coreference/)
5. Micha Elsner and Eugene Charniak. 2011. Extending the entity grid with entity-specific features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2, HLT ’11, pages 125–129, Stroudsburg, PA, USA. Association… [find](https://scholar.google.com/scholar?q=Micha+Elsner+and+Eugene+Charniak%2E+2011%2E+Extending+the+entity+grid+with+entity%2Dspecific+features%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Langu)
6. Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. 1995. Centering: a framework for modeling the local coherence of discourse. Comput. Linguist., 21(2):203–225, June. [find](https://scholar.google.com/scholar?q=Barbara+J%2E+Grosz%2C+Scott+Weinstein%2C+and+Aravind+K%2E+Joshi%2E+1995%2E+Centering%3A+a+framework+for+modeling+the+local+coherence+of+discourse%2E+Comput%2E+Linguist%2E%2C+21%282%29%3A203%E2%80%93225%2C+June%2E) [our card](https://md.research.elamur.ai/free/papers/cl-1995-grosz-centering-framework-modeling-local-coherence/)
7. Joseph Irwin, Mamoru Komachi, and Yuji Matsumoto. 2011. Narrative schema as world knowledge for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, CONLL Shared Task ’11, pages 86–92, Stroudsburg, PA, USA. Association for Comput… [find](https://scholar.google.com/scholar?q=Joseph+Irwin%2C+Mamoru+Komachi%2C+and+Yuji+Matsumoto%2E+2011%2E+Narrative+schema+as+world+knowledge+for+coreference+resolution%2E+In+Proceedings+of+the+Fifteenth+Conference+on+Computational+Natural+Language+Lea) [our card](https://md.research.elamur.ai/free/papers/irwin-2011-narrative-schema-as-world-knowledge/)
8. June-Jei Kuo and Hsin-Hsi Chen. 2007. Cross-document event clustering using knowledge mining from co-reference chains. Inf. Process. Manage., 43(2):327–343, March. [find](https://scholar.google.com/scholar?q=June%2DJei+Kuo+and+Hsin%2DHsi+Chen%2E+2007%2E+Cross%2Ddocument+event+clustering+using+knowledge+mining+from+co%2Dreference+chains%2E+Inf%2E+Process%2E+Manage%2E%2C+43%282%29%3A327%E2%80%93343%2C+March%2E)
9. Heeyoung Lee, Marta Recasens, Angel Chang, Mihai Surdeanu, and Dan Jurafsky. 2012. Joint entity and event coreference resolution across documents. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL… [find](https://scholar.google.com/scholar?q=Heeyoung+Lee%2C+Marta+Recasens%2C+Angel+Chang%2C+Mihai+Surdeanu%2C+and+Dan+Jurafsky%2E+2012%2E+Joint+entity+and+event+coreference+resolution+across+documents%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empiri) [our card](https://md.research.elamur.ai/free/papers/lee-2012-joint-entity-event-coreference-resolution/)
10. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4). [find](https://scholar.google.com/scholar?q=Heeyoung+Lee%2C+Angel+Chang%2C+Yves+Peirsman%2C+Nathanael+Chambers%2C+Mihai+Surdeanu%2C+and+Dan+Jurafsky%2E+2013%2E+Deterministic+coreference+resolution+based+on+entity%2Dcentric%2C+precision%2Dranked+rules%2E+Computationa) [our card](https://md.research.elamur.ai/free/papers/lee-2013-deterministic-coreference-resolution-based-entity/)
11. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+CoNLL%2D2012+shared+task%3A+Modeling+multilingual+unrestricted+coreference+in+OntoNotes%2E+In+Proceedings+of+the+Six) [our card](https://md.research.elamur.ai/free/papers/pradhan-2012-conll-2012-shared-task-modeling/)