---
title: "Error Analysis for Learning-based — Uryupina 2008"
description: "Coreference resolvers trained on MUC-7 tend to plateau at around 60% F-measure."
authors: ["Olga Uryupina"]
year: 2008
venue: "Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)"
source: "https://aclanthology.org/L08-1049/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/uryupina-2008-error-analysis-learning-based-coreference/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/uryupina-2008-error-analysis-learning-based-coreference/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2008](https://md.research.elamur.ai/free/papers/2008/) / Error Analysis for Learning-based Coreference Resolution

# Error Analysis for Learning-based Coreference Resolution

**Authors:** [Olga Uryupina](https://md.research.elamur.ai/free/authors/olga-uryupina/) · **Year:** 2008 · **Venue:** Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/L08-1049/) · [PDF](https://aclanthology.org/L08-1049.pdf) · [OpenAlex](https://openalex.org/W1607177814)

**Citations:** 8 citations (openalex-snapshot · 2026-10-14) · [5](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Coreference resolvers trained on MUC-7 tend to plateau at around 60% F-measure. This paper asks why, by closely examining the errors of one strong learning-based system. The author's engine uses a large, linguistically motivated feature set covering surface, syntactic, semantic and salience information. It beats the Soon et al. (2001) baseline with several learners and reaches 65.4% F with SVM, which the paper reports as the best published MUC-7 result at the time.

The missed and spurious links on the MUC-7 test set were categorized by hand. The main sources of trouble are:
- markables lost during preprocessing, including those in auxiliary document fields;
- full-NP (nominal) anaphora that requires inference;
- same-sentence and 1st/2nd person pronouns;
- the local, pair-by-pair resolution scheme, which spreads one mistake into others.

The author argues that the feature set is already rich but redundant. Progress should therefore come from more training data, better coupling with preprocessing modules, feature selection, and chain-level resolution. Because most comparable systems score similarly, the analysis is presented as relevant to the field as a whole.

## Key points
- **Features:** 351 nominal features (1096 boolean/continuous), split into surface (122), syntactic (64), semantic (29) and salience (136). They were tested with SVM, C4.5, Ripper, Slipper and MaxEnt.
- **SVM results on 20 MUC-7 "formal test" documents:** F 65.4%, recall 63.9%, precision 67.0%. The system missed 469 of 1299 gold links and proposed 408 spurious ones. Learning curves had not yet converged.
- **Recall errors:** missing markables account for about 35%, nominal anaphora for 22%, pronouns for 16% and name-matching for about 7%. ORGANIZATION names cause most of the matching failures.
- **Precision errors:** nominal anaphora (mostly same-head NPs) accounts for about 45%, pronouns for 19% and misleading markables for 19%.
- **Proposed directions:** improve training material, integrate preprocessing modules more closely, and move to global chain-based resolution as in Luo et al. (2004).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/)

## Cited by

Cited by

5 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 3 not yet.

- [Recall Error Analysis for Coreference Resolution](https://md.research.elamur.ai/free/papers/martschat-2014-recall-error-analysis-coreference-resolution/)
- [Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective](https://md.research.elamur.ai/free/papers/porada-2024-challenges-evaluating-generalization-coreference-resolution/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{uryupina2008error,
  title = {{Error Analysis for Learning-based Coreference Resolution}},
  author = {Uryupina, Olga},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/38d86t8mhth5},
  url = {https://doi.org/10.63317/38d86t8mhth5}
}
```

- [← previous Enforcing Transitivity in Coreference Resolution](https://md.research.elamur.ai/free/papers/finkel-2008-enforcing-transitivity-coreference-resolution/)
- [next → FATE: a FrameNet-Annotated Corpus for Textual Entailment](https://md.research.elamur.ai/free/papers/burchardt-2008-fate-framenet-annotated-corpus-textual/)
- [↑ back to papers of 2008](https://md.research.elamur.ai/free/papers/2008/)

## References

13 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Breck Baldwin, Tom Morton, Amit Bagga, Jason Baldridge, Raman Chandraseker, Alexis Dimitriadis, Kieran Snyder, and Magdalena Wolska. 1997. Description of the UPenn camp system as used for coreference. In Message Understanding Conference Proceedings. [find](https://scholar.google.com/scholar?q=Breck+Baldwin%2C+Tom+Morton%2C+Amit+Bagga%2C+Jason+Baldridge%2C+Raman+Chandraseker%2C+Alexis+Dimitriadis%2C+Kieran+Snyder%2C+and+Magdalena+Wolska%2E+1997%2E+Description+of+the+UPenn+camp+system+as+used+for+coreference%2E) [our card](https://md.research.elamur.ai/free/papers/baldwin-1998-description-upenn-camp-system-as/)
2. Catalina Barbu, Richard Evans, and Ruslan Mitkov. 2002. A corpus based investigation of morphological disagreement in anaphoric relations. In Proceedings of the Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Catalina+Barbu%2C+Richard+Evans%2C+and+Ruslan+Mitkov%2E+2002%2E+A+corpus+based+investigation+of+morphological+disagreement+in+anaphoric+relations%2E+In+Proceedings+of+the+Language+Resources+and+Evaluation+Confe)
3. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Meeting of the North American Chapter of the Association for Computational Linguistics, pages 132–139. [find](https://scholar.google.com/scholar?q=Eugene+Charniak%2E+2000%2E+A+maximum%2Dentropy%2Dinspired+parser%2E+In+Proceedings+of+the+1st+Meeting+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%2C+pages+132%E2%80%93139%2E) [our card](https://md.research.elamur.ai/free/papers/charniak-2000-maximum-entropy-inspired-parser/)
4. Dan Cristea, Oana Postolache, and Ruslan Mitkov. 2002. Handling complex anaphora resolution cases. In Proceedings of the 4th Discourse Anaphora and Anaphor Resolution Colloquium. [find](https://scholar.google.com/scholar?q=Dan+Cristea%2C+Oana+Postolache%2C+and+Ruslan+Mitkov%2E+2002%2E+Handling+complex+anaphora+resolution+cases%2E+In+Proceedings+of+the+4th+Discourse+Anaphora+and+Anaphor+Resolution+Colloquium%2E)
5. James R. Curran and Stephen Clark. 2003. Language independent NER using a maximum entropy tagger. In Proceedings of the Seventh Conference on Natural Language Learning, pages 164–167. [find](https://scholar.google.com/scholar?q=James+R%2E+Curran+and+Stephen+Clark%2E+2003%2E+Language+independent+NER+using+a+maximum+entropy+tagger%2E+In+Proceedings+of+the+Seventh+Conference+on+Natural+Language+Learning%2C+pages+164%E2%80%93167%2E)
6. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiaoqiang+Luo%2C+Abe+Ittycheriah%2C+Hongyan+Jing%2C+Nanda+Kambhatla%2C+and+Salim+Roukos%2E+2004%2E+A+mention%2Dsynchronous+coreference+resolution+algorithm+based+on+the+bell+tree%2E+In+Proceedings+of+the+42nd+Annual+) [our card](https://md.research.elamur.ai/free/papers/luo-2004-mention-synchronous-coreference-resolution-algorithm/)
7. George Miller. 1990. Wordnet: An on-line lexical database. International Journal of Lexicography. [find](https://scholar.google.com/scholar?q=George+Miller%2E+1990%2E+Wordnet%3A+An+on%2Dline+lexical+database%2E+International+Journal+of+Lexicography%2E)
8. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 104–111. [find](https://scholar.google.com/scholar?q=Vincent+Ng+and+Claire+Cardie%2E+2002%2E+Improving+machine+learning+approaches+to+coreference+resolution%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1) [our card](https://md.research.elamur.ai/free/papers/ng-2002-improving-machine-learning-approaches-coreference/)
9. Massimo Poesio, Olga Uryupina, Renata Vieira, Mijail Alexandrov-Kabadjov, and Rodrigo Goulart. 2004. Discourse-new detectors for definite description resolution: a survey and preliminary proposal. In Proceedings of the Reference Resolution Workshop at ACL’04. [find](https://scholar.google.com/scholar?q=Massimo+Poesio%2C+Olga+Uryupina%2C+Renata+Vieira%2C+Mijail+Alexandrov%2DKabadjov%2C+and+Rodrigo+Goulart%2E+2004%2E+Discourse%2Dnew+detectors+for+definite+description+resolution%3A+a+survey+and+preliminary+proposal%2E+In+)
10. Jeffrey C. Reynar and Adwait Ratnaparkhi. 1997. A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth Conference on Applied Natural Language Processing. [find](https://scholar.google.com/scholar?q=Jeffrey+C%2E+Reynar+and+Adwait+Ratnaparkhi%2E+1997%2E+A+maximum+entropy+approach+to+identifying+sentence+boundaries%2E+In+Proceedings+of+the+Fifth+Conference+on+Applied+Natural+Language+Processing%2E)
11. Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics (Special Issue on Computational Anaphora Resolution), 27(4):521–544. [find](https://scholar.google.com/scholar?q=Wee+Meng+Soon%2C+Hwee+Tou+Ng%2C+and+Daniel+Chung+Yong+Lim%2E+2001%2E+A+machine+learning+approach+to+coreference+resolution+of+noun+phrases%2E+Computational+Linguistics+%28Special+Issue+on+Computational+Anaphora+R) [our card](https://md.research.elamur.ai/free/papers/soon-2001-machine-learning-approach-coreference-resolution/)
12. Olga Uryupina. 2006. Coreference resolution with and without linguistic knowledge. In Proceedings of the Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Olga+Uryupina%2E+2006%2E+Coreference+resolution+with+and+without+linguistic+knowledge%2E+In+Proceedings+of+the+Language+Resources+and+Evaluation+Conference%2E) [our card](https://md.research.elamur.ai/free/papers/uryupina-2006-coreference-resolution-without-linguistic-knowledge/)
13. Olga Uryupina. 2007. Knowledge Acquisition for Coreference Resolution. Ph.D. thesis, Saarland University. [find](https://scholar.google.com/scholar?q=Olga+Uryupina%2E+2007%2E+Knowledge+Acquisition+for+Coreference+Resolution%2E+Ph%2ED%2E+thesis%2C+Saarland+University%2E)