---
title: "Analyzing and Visualizing Coreference Resolution Errors (2015)"
description: "The authors present cort, an open-source Python toolkit that implements their earlier graph-based error… 11 references, cited by 5 papers here."
authors: ["Sebastian Martschat", "Thierry Göckel", "Michael Strube"]
year: 2015
venue: "Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations"
source: "https://aclanthology.org/N15-3002/"
doi: "10.3115/v1/n15-3002"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/martschat-2015-analyzing-visualizing-coreference-resolution-errors/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/martschat-2015-analyzing-visualizing-coreference-resolution-errors/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/) / Analyzing and Visualizing Coreference Resolution Errors

# Analyzing and Visualizing Coreference Resolution Errors

**Authors:** [Sebastian Martschat](https://md.research.elamur.ai/free/en/authors/sebastian-martschat/), Thierry Göckel and [Michael Strube](https://md.research.elamur.ai/free/en/authors/michael-strube/) · **Year:** 2015 · **Venue:** [Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2015](https://md.research.elamur.ai/free/en/venues/naacl/2015/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N15-3002/) · [PDF](https://aclanthology.org/N15-3002.pdf) · [DOI 10.3115/v1/n15-3002](https://doi.org/10.3115/v1/n15-3002) · [OpenAlex](https://openalex.org/W2250347727)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/martschat-2015-analyzing-visualizing-coreference-resolution-errors/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Coreference resolution produces sets of mentions, so it is not obvious how to name the specific errors when a system's clusters differ from the gold ones. The authors present cort, an open-source Python toolkit that implements their earlier graph-based error analysis framework (Martschat and Strube, 2014). Each entity is treated as a complete graph over its mentions, and a spanning tree is computed for it. Recall errors are the reference spanning-tree edges that are absent from the system output. Precision errors come from the same procedure with system and reference swapped. The choice of spanning-tree algorithm defines what counts as an error. cort reads and writes CoNLL-2012 data and includes a rule-based mention extractor and two baseline resolvers. It also offers tools to categorize, filter, plot and compare errors, plus a browser-based viewer that shows errors in document context. A worked use case shows an engineer locating weak noun–name recall, finding that it stems from missed is-a relations, and checking a new feature by comparing errors. The toolkit makes coreference debugging systematic and interpretable.

## Key points
- The toolkit has three modules: `core` (mention extraction and attributes such as head, POS, NE tag, gender, number, semantic class, grammatical function, and mention type), `coreference` (resolution approaches), and `analysis` (the `ErrorAnalysis` class plus visualization).
- It ships two baselines: multigraph, a deterministic resolver using a few strong features, and a mention-pair model following Soon et al. (2001) that relies on an extensive set of features, trained with a perceptron on CoNLL'12 English training data.
- Average CoNLL scores on CoNLL'12 English dev:
  - multigraph: 61.28 (MUC 69.13, B³ 58.61, CEAFe 56.06)
  - mention-pair: 60.16
  - BerkeleyCoref: 61.89
  - StanfordSieve: 56.90
- For reference entities, spanning trees are built with an algorithm based on Ariel's accessibility theory. For system entities, every output pair is used as an edge.
- Compared with Kummerfeld and Klein (2013), whose errors are transformation-based, and ICE (Gärtner et al., 2014), cort uses a pairwise notion of error, lets users plug in their own error definitions, and visualizes both data and documents.

## Interesting ideas
- Turning set-level mistakes into concrete pairwise edges via spanning trees makes cluster errors countable and explainable, with the tree algorithm acting as a swappable definition of "error"
- Error counts are reported against a ceiling: for recall, the errors of a system that leaves every mention as a singleton; for precision, all pairwise decisions of the model. This shows how bad a category really is relative to what was possible
- Errors are bucketed by the coarse types of anaphor and antecedent (name, noun, pronoun, demonstrative, verb), and the most frequent weak bucket is then refined by cues such as token overlap
- A new model's errors are loaded next to the baseline's, and the user inspects errors that are shared versus unique to each system to judge whether a feature truly helps
- The finding that noun–name misses were mostly unrecognized is-a relations (e.g., "the U.S.-backed rebels" ↔ "the Contras") motivates adding world knowledge

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): Coreference output in prag could be scored by listing missed and spurious antecedent links per entity, so each failure points to one explainable mention pair.
- **I2** (learn): Evaluation reports could pair every error count with such a trivial-baseline ceiling, which works as a built-in negative control.
- **I3** (frames): Coreference failures could be grouped by mention-type pair to decide whether to invest in pronoun rules or in nominal-to-name linking.
- **I4** (learn): When self-induced repair rules or new training runs are tried, diffing error sets against the previous version would reveal what was fixed and what newly broke, beyond the score change.
- **I5** (world): Ontology and knowledge-graph facts from WordNet/Wikidata could supply is-a links that let coreference connect descriptive nouns to named entities.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Heidelberg Institute for Theoretical Studies](https://md.research.elamur.ai/free/en/labs/heidelberg-institute-for-theoretical-studies/) — Research institute, Heidelberg, Germany · 27 works · 1,842 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔗 Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) · 550 works

## Cite

Cite as: Sebastian Martschat, Thierry Göckel and Michael Strube. 2015. Analyzing and Visualizing Coreference Resolution Errors. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{martschat2015analyzing,
  title = {{Analyzing and Visualizing Coreference Resolution Errors}},
  author = {Sebastian Martschat, Thierry Göckel and Michael Strube},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations},
  year = {2015},
  doi = {10.3115/v1/n15-3002},
  url = {https://doi.org/10.3115/v1/n15-3002}
}
```

- [← previous A Study of the Effect of Resolving Negation and Sentiment Analysis in Recognizing Text Entailment for Arabic](https://md.research.elamur.ai/free/en/papers/al-khawaldeh-2019-study-effect-resolving-negation-sentiment/)
- [next → CBAS: context based arabic stemmer](https://md.research.elamur.ai/free/en/papers/el-defrawy-2015-cbas-context-based-arabic-stemmer/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/)

## References

11 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mira Ariel. 1990. Accessing Noun Phrase Antecedents. Routledge, London, U.K.; New York, N.Y. [find](https://scholar.google.com/scholar?q=Mira+Ariel%2E+1990%2E+Accessing+Noun+Phrase+Antecedents%2E+Routledge%2C+London%2C+U%2EK%2E%3B+New+York%2C+N%2EY%2E)
2. Greg Durrett and Dan Klein. 2013. [Easy victories and uphill battles in coreference resolution](https://md.research.elamur.ai/free/en/papers/durrett-2013-easy-victories-uphill-battles-coreference/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Wash., 18–21 October 2013, pages 1971–1982.
3. Markus Gärtner, Anders Björkelund, Gregor Thiele, Wolfgang Seeker, and Jonas Kuhn. 2014. Visualization, search, and error analysis for coreference annotations. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Baltimore, Md., 22–27 June 2014, pages 7–12. [find](https://scholar.google.com/scholar?q=Markus+Ga%CC%88rtner%2C+Anders+Bjo%CC%88rkelund%2C+Gregor+Thiele%2C+Wolfgang+Seeker%2C+and+Jonas+Kuhn%2E+2014%2E+Visualization%2C+search%2C+and+error+analysis+for+coreference+annotations%2E+In+Proceedings+of+52nd+Annual+Meeting+)
4. John D. Hunter. 2007. Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3):90–95. [↗](https://doi.org/10.1109/mcse.2007.55)
5. Jonathan K. Kummerfeld and Dan Klein. 2013. [Error-driven analysis of challenges in coreference resolution](https://md.research.elamur.ai/free/en/papers/kummerfeld-2013-error-driven-analysis-challenges-coreference/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Wash., 18–21 October 2013, pages 265–277.
6. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. [Deterministic coreference resolution based on entity-centric, precision-ranked rules](https://md.research.elamur.ai/free/en/papers/lee-2013-deterministic-coreference-resolution-based-entity/). Computational Linguistics, 39(4):885–916. [↗](https://doi.org/10.1162/coli_a_00152)
7. Sebastian Martschat and Michael Strube. 2014. [Recall error analysis for coreference resolution](https://md.research.elamur.ai/free/en/papers/martschat-2014-recall-error-analysis-coreference-resolution/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25–29 October 2014, pages 2070–2081.
8. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. [CoNLL-2012 Shared Task: Modeling multilingual unrestricted coreference in OntoNotes](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/). In Proceedings of the Shared Task of the 16th Conference on Computational Natural Language Learning, Jeju Island, Korea, 12–14 July 2012, pages 1–40.
9. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. [Scoring coreference partitions of predicted mentions: A reference implementation](https://md.research.elamur.ai/free/en/papers/pradhan-2014-scoring-coreference-partitions-predicted-mentions/). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Baltimore, Md., 22–27 June 2014, pages 30–35.
10. Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2011. [Large-scale cross-document coreference using distributed inference and hierarchical models](https://md.research.elamur.ai/free/en/papers/singh-2011-large-scale-cross-document-coreference/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Portland, Oreg., 19–24 June 2011, pages 793–803.
11. Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. [A machine learning approach to coreference resolution of noun phrases](https://md.research.elamur.ai/free/en/papers/soon-2001-machine-learning-approach-coreference-resolution/). Computational Linguistics, 27(4):521–544. [↗](https://doi.org/10.1162/089120101753342653)

*Updated 24 October 2026*