---
title: "Evaluation Tool for Rule-based Anaphora — Barbu 2001"
description: "Comparing anaphora resolution systems has been unreliable because each one was tested on its own data, with its own pre-processing, and often with some…"
authors: ["Catalina Barbu", "Ruslan Mitkov"]
year: 2001
venue: "Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P01-1006/"
doi: "10.3115/1073012.1073018"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/barbu-2001-evaluation-tool-rule-based-anaphora/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/barbu-2001-evaluation-tool-rule-based-anaphora/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2001](https://md.research.elamur.ai/free/papers/2001/) / Evaluation Tool for Rule-based Anaphora Resolution Methods

# Evaluation Tool for Rule-based Anaphora Resolution Methods

**Authors:** Catalina Barbu, [Ruslan Mitkov](https://md.research.elamur.ai/free/authors/ruslan-mitkov/) · **Year:** 2001 · **Venue:** Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P01-1006/) · [PDF](https://aclanthology.org/P01-1006.pdf) · [DOI 10.3115/1073012.1073018](https://doi.org/10.3115/1073012.1073018) · [OpenAlex](https://openalex.org/W2159767973)

**Citations:** 41 citations (openalex-snapshot · 2026-10-14) · [11](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Comparing anaphora resolution systems has been unreliable because each one was tested on its own data, with its own pre-processing, and often with some manual correction of the input. Barbu and Mitkov address this with an evaluation workbench, an extensible environment where algorithms are plugged in and run on identical texts through identical pre-processing, fully automatically. Pre-processing uses Conexor's FDG Parser plus a custom NP extractor. Its output is stored in an XML format, so any shallow parser could be swapped in. The authors re-implemented three knowledge-poor pronoun resolvers: Kennedy and Boguraev's parser-free method, Baldwin's CogNiac (in its resolve-all variant) and Mitkov's knowledge-poor approach. They tested these on a coreference-annotated corpus of technical documents. Kennedy and Boguraev scored highest, followed by Mitkov and then CogNiac. All three performed well below their originally published figures. This supports the view that fully automatic resolution is harder than earlier evaluations implied. The authors also suggest that evaluation reports describe how difficult the test data is, for example the number of candidate antecedents and the referential distance.

## Key points
- **Shared pipeline:** the workbench runs Conexor's FDG Parser followed by an NP extractor and stores the result in XML. It has a graphical interface, reports per-method and per-file results, and plans to add a full-parser mode and WordNet access.
- **Test corpus:** four technical texts (Beowulf HOW TO, Linux CD-Rom HOW TO, Access HOW TO and a Windows Help file).
  - It contains 28,272 words, 19,305 NPs and 422 pronouns, of which 362 are anaphoric.
  - It was annotated by two people with a slightly modified MUC scheme.
- **Overall success rate:**
  - Kennedy & Boguraev: 61.6%
  - Mitkov: 56.9%
  - CogNiac: 49.72%
- **Overall precision** (equal to recall, since all three methods are robust):
  - Kennedy & Boguraev: 52.84%
  - Mitkov: 48.81%
  - CogNiac: 42.65%
- **Proposed difficulty statistics for the corpus:**
  - about 10.85 candidate antecedents per pronoun on average;
  - an average referential distance of roughly 1.3 sentences and 4.3 NPs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/)

## Cited by

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 4 not yet.

- [Bootstrapping Path-Based Pronoun Resolution](https://md.research.elamur.ai/free/papers/bergsma-2006-bootstrapping-path-based-pronoun-resolution/)
- [Automatic Pronominal Anaphora Resolution in English Texts](https://md.research.elamur.ai/free/papers/liang-2003-automatic-pronominal-anaphora-resolution-english/)
- [Pronominal and Sortal Anaphora Resolution for Biomedical Literature](https://md.research.elamur.ai/free/papers/lin-2004-pronominal-sortal-anaphora-resolution-biomedical/)
- [Supervised Noun Phrase Coreference Research: The First Fifteen Years](https://md.research.elamur.ai/free/papers/ng-2010-supervised-noun-phrase-coreference-research/)
- [Anaphora Resolution Exercise: an Overview](https://md.research.elamur.ai/free/papers/san-2008-anaphora-resolution-exercise-overview/)
- [Identification des reprises et interprétation automatique des expressions pronominales dans des textes en français](https://md.research.elamur.ai/free/papers/trouilleux-2001-identification-reprises-interpretation-automatique/)
- [Stay Together: A System for Single and Split-antecedent Anaphora Resolution](https://md.research.elamur.ai/free/papers/yu-2021-stay-together-system-single-split/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{barbu2001evaluation,
  title = {{Evaluation Tool for Rule-based Anaphora Resolution Methods}},
  author = {Barbu, Catalina and Mitkov, Ruslan},
  booktitle = {Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics},
  year = {2001},
  doi = {10.3115/1073012.1073018},
  url = {https://doi.org/10.3115/1073012.1073018}
}
```

- [← previous DATE: A Dialogue Act Tagging Scheme for Evaluation of Spoken Dialogue Systems](https://md.research.elamur.ai/free/papers/walker-2001-date-dialogue-act-tagging-scheme/)
- [next → Extracting Paraphrases from a Parallel Corpus](https://md.research.elamur.ai/free/papers/barzilay-2001-extracting-paraphrases-parallel-corpus/)
- [↑ back to papers of 2001](https://md.research.elamur.ai/free/papers/2001/)

## References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chinatsu Aone and Scot W. Bennett. 1995. Evaluating automated and manual acquisition of anaphora resolution rules. In Proceedings of the 33th Annual Meeting of the Association for Computational Linguistics (ACL ’95), pages 122– 129. [find](https://scholar.google.com/scholar?q=Chinatsu+Aone+and+Scot+W%2E+Bennett%2E+1995%2E+Evaluating+automated+and+manual+acquisition+of+anaphora+resolution+rules%2E+In+Proceedings+of+the+33th+Annual+Meeting+of+the+Association+for+Computational+Lingui)
2. Breck Baldwin. 1997. Cogniac: High precision coreference with limited knowledge and linguistic resources. In R. Mitkov and B. Boguraev, editors, Operational factors in practical, robust anaphora resolution for unrestricted texts, pages 38 – 45. [find](https://scholar.google.com/scholar?q=Breck+Baldwin%2E+1997%2E+Cogniac%3A+High+precision+coreference+with+limited+knowledge+and+linguistic+resources%2E+In+R%2E+Mitkov+and+B%2E+Boguraev%2C+editors%2C+Operational+factors+in+practical%2C+robust+anaphora+resol) [our card](https://md.research.elamur.ai/free/papers/baldwin-1997-cogniac-high-precision-coreference-limited/)
3. Ido Dagan and Alon Itai. 1990. Automatic processing of large corpora for the resolution of anaphora references. In Proceedings of the 13th International Conference on Computational Linguistics (COLING’90), volume III, pages 1–3. [find](https://scholar.google.com/scholar?q=Ido+Dagan+and+Alon+Itai%2E+1990%2E+Automatic+processing+of+large+corpora+for+the+resolution+of+anaphora+references%2E+In+Proceedings+of+the+13th+International+Conference+on+Computational+Linguistics+%28COLING) [our card](https://md.research.elamur.ai/free/papers/dagan-1990-automatic-processing-large-corpora-resolution/)
4. Ido Dagan and Alon Itai. 1991. A statistical filter for resolving pronoun references. In Y.A. Feldman and A. Bruckstein, editors, Artificial Intelligence and Computer Vision, pages 125 – 135. Elsevier Science Publishers B.V. [find](https://scholar.google.com/scholar?q=Ido+Dagan+and+Alon+Itai%2E+1991%2E+A+statistical+filter+for+resolving+pronoun+references%2E+In+Y%2EA%2E+Feldman+and+A%2E+Bruckstein%2C+editors%2C+Artificial+Intelligence+and+Computer+Vision%2C+pages+125+%E2%80%93+135%2E+Elsevier)
5. Antonio Ferrandez, Manolo Palomar, and L. Moreno. 1997. Slot unification grammar and anaphora resolution. In Proceedings of the International Conference on Recent Advances in Natural Language Proceeding (RANLP’97), pages 294– 299. [find](https://scholar.google.com/scholar?q=Antonio+Ferrandez%2C+Manolo+Palomar%2C+and+L%2E+Moreno%2E+1997%2E+Slot+unification+grammar+and+anaphora+resolution%2E+In+Proceedings+of+the+International+Conference+on+Recent+Advances+in+Natural+Language+Proceedi)
6. Robert Gaizauskas and Kevin Humphreys. 2000. Quantitative evaluation of coreference algorithms in an information extraction system. In Simon Botley and Antony Mark McEnery, editors, Corpus-based and Computational Approaches to Discourse Anaphora, Studies in Corpus Linguistics, chapter 8, pages 145 –… [find](https://scholar.google.com/scholar?q=Robert+Gaizauskas+and+Kevin+Humphreys%2E+2000%2E+Quantitative+evaluation+of+coreference+algorithms+in+an+information+extraction+system%2E+In+Simon+Botley+and+Antony+Mark+McEnery%2C+editors%2C+Corpus%2Dbased+and+C)
7. Niyu Ge, J. Hale, and E. Charniak. 1998. A statistical approach to anaphora resolution. In Proceedings of the Sixth Workshop on Very Large Corpora, COLING-ACL ’98, pages 161 – 170, Montreal, Canada. [find](https://scholar.google.com/scholar?q=Niyu+Ge%2C+J%2E+Hale%2C+and+E%2E+Charniak%2E+1998%2E+A+statistical+approach+to+anaphora+resolution%2E+In+Proceedings+of+the+Sixth+Workshop+on+Very+Large+Corpora%2C+COLING%2DACL+%E2%80%9998%2C+pages+161+%E2%80%93+170%2C+Montreal%2C+Canada%2E) [our card](https://md.research.elamur.ai/free/papers/ge-1998-statistical-approach-anaphora-resolution/)
8. Jerry Hobbs. 1976. Pronoun resolution. Research report 76-1, City College, City University of New York. [find](https://scholar.google.com/scholar?q=Jerry+Hobbs%2E+1976%2E+Pronoun+resolution%2E+Research+report+76%2D1%2C+City+College%2C+City+University+of+New+York%2E)
9. Jerry Hobbs. 1978. Pronoun resolution. Lingua, 44:339–352. [find](https://scholar.google.com/scholar?q=Jerry+Hobbs%2E+1978%2E+Pronoun+resolution%2E+Lingua%2C+44%3A339%E2%80%93352%2E)
10. Christopher Kennedy and Branimir Boguraev. 1996. Anaphora for everyone: pronominal anaphora resolution without a parser. In Proceedings of the 16th International Conference on Computational Linguistics (COLING’96), pages 113–118, Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Christopher+Kennedy+and+Branimir+Boguraev%2E+1996%2E+Anaphora+for+everyone%3A+pronominal+anaphora+resolution+without+a+parser%2E+In+Proceedings+of+the+16th+International+Conference+on+Computational+Linguistic) [our card](https://md.research.elamur.ai/free/papers/kennedy-1996-anaphora-everyone-pronominal-anaphora-resolution/)
11. Shalom Lappin and H.J. Leass. 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4):535 – 562. [find](https://scholar.google.com/scholar?q=Shalom+Lappin+and+H%2EJ%2E+Leass%2E+1994%2E+An+algorithm+for+pronominal+anaphora+resolution%2E+Computational+Linguistics%2C+20%284%29%3A535+%E2%80%93+562%2E) [our card](https://md.research.elamur.ai/free/papers/lappin-1994-algorithm-pronominal-anaphora-resolution/)
12. Ruslan Mitkov, R. Evans, C. Orasan, C. Barbu, L. Jones, and V. Sotirova. 2000. Coreference and anaphora: developing annotating tools, annotated resources and annotation strategies. In Proceedings of the Discourse, Anaphora and Reference Resolution Conference (DAARC2000), pages 49–58, Lancaster, UK. [find](https://scholar.google.com/scholar?q=Ruslan+Mitkov%2C+R%2E+Evans%2C+C%2E+Orasan%2C+C%2E+Barbu%2C+L%2E+Jones%2C+and+V%2E+Sotirova%2E+2000%2E+Coreference+and+anaphora%3A+developing+annotating+tools%2C+annotated+resources+and+annotation+strategies%2E+In+Proceedings+of+t)
13. Ruslan Mitkov. 1998a. Evaluating anaphora resolution approaches. In Proceedings of the Discourse Anaphora and Anaphora Resolution Colloquium (DAARC’2), pages 164 – 172, Lancaster, UK. [find](https://scholar.google.com/scholar?q=Ruslan+Mitkov%2E+1998a%2E+Evaluating+anaphora+resolution+approaches%2E+In+Proceedings+of+the+Discourse+Anaphora+and+Anaphora+Resolution+Colloquium+%28DAARC%E2%80%992%29%2C+pages+164+%E2%80%93+172%2C+Lancaster%2C+UK%2E)
14. Ruslan Mitkov. 1998b. Robust pronoun resolution with limited knowledge. In Proceedings of the 18th International Conference on Computational Linguistics (COLING’98/ACL’98, pages 867 – 875. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Ruslan+Mitkov%2E+1998b%2E+Robust+pronoun+resolution+with+limited+knowledge%2E+In+Proceedings+of+the+18th+International+Conference+on+Computational+Linguistics+%28COLING%E2%80%9998%2FACL%E2%80%9998%2C+pages+867+%E2%80%93+875%2E+Morgan+Kauf) [our card](https://md.research.elamur.ai/free/papers/mitkov-1998-robust-pronoun-resolution-limited-knowledge/)
15. Ruslan Mitkov. 2000a. Towards a more consistent and comprehensive evaluation of anaphora resolution algorithms and systems. In Proceedings of the Discourse, Anaphora and Reference Resolution Conference (DAARC2000), pages 96 – 107, Lancaster, UK. [find](https://scholar.google.com/scholar?q=Ruslan+Mitkov%2E+2000a%2E+Towards+a+more+consistent+and+comprehensive+evaluation+of+anaphora+resolution+algorithms+and+systems%2E+In+Proceedings+of+the+Discourse%2C+Anaphora+and+Reference+Resolution+Conferenc)
16. Ruslan Mitkov. 2000b. Towards more comprehensive evaluation in anaphora resolution. In Proceedings of the Second International Conference on Language Resources and Evaluation, volume III, pages 1309 – 1314, Athens, Greece. [find](https://scholar.google.com/scholar?q=Ruslan+Mitkov%2E+2000b%2E+Towards+more+comprehensive+evaluation+in+anaphora+resolution%2E+In+Proceedings+of+the+Second+International+Conference+on+Language+Resources+and+Evaluation%2C+volume+III%2C+pages+1309+%E2%80%93)
17. Ruslan Mitkov. 2001. Outstanding issues in anaphora resolution. In Al. Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, pages 110–125. Springer. [find](https://scholar.google.com/scholar?q=Ruslan+Mitkov%2E+2001%2E+Outstanding+issues+in+anaphora+resolution%2E+In+Al%2E+Gelbukh%2C+editor%2C+Computational+Linguistics+and+Intelligent+Text+Processing%2C+pages+110%E2%80%93125%2E+Springer%2E)
18. Constantin Orăsan, Richard Evans, and Ruslan Mitkov. 2000. Enhancing preference-based anaphora resolution with genetic algorithms. In Proceedings of Natural Language Processing - NLP2000, pages 185 – 195. Springer. [find](https://scholar.google.com/scholar?q=Constantin+Ora%CC%86san%2C+Richard+Evans%2C+and+Ruslan+Mitkov%2E+2000%2E+Enhancing+preference%2Dbased+anaphora+resolution+with+genetic+algorithms%2E+In+Proceedings+of+Natural+Language+Processing+%2D+NLP2000%2C+pages+185+%E2%80%93)
19. P. Tapanainen and T. Järvinen. 1997. A non-projective dependency parser. In Proceedings of the 5th Conference of Applied Natural Language Processing, pages 64 – 71, Washington D.C., USA. [find](https://scholar.google.com/scholar?q=P%2E+Tapanainen+and+T%2E+Ja%CC%88rvinen%2E+1997%2E+A+non%2Dprojective+dependency+parser%2E+In+Proceedings+of+the+5th+Conference+of+Applied+Natural+Language+Processing%2C+pages+64+%E2%80%93+71%2C+Washington+D%2EC%2E%2C+USA%2E)
20. Joel R. Tetreault. 1999. Analysis of syntax-based pronoun resolution methods. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL ’99), pages 602 – 605, Maryland, USA. [find](https://scholar.google.com/scholar?q=Joel+R%2E+Tetreault%2E+1999%2E+Analysis+of+syntax%2Dbased+pronoun+resolution+methods%2E+In+Proceedings+of+the+37th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL+%E2%80%9999%29%2C+pages+602+%E2%80%93+605%2C+Mar) [our card](https://md.research.elamur.ai/free/papers/tetreault-1999-analysis-syntax-based-pronoun-resolution/)