---
title: "Combining Sample Selection and Error-Driven Pruning for Machine Learning of Coreference — Ng 2002"
description: "They build a baseline with RIPPER rule learning, 25 automatically computed features and best-first clustering. 18 references, cited by 16 papers here."
authors: ["Vincent Ng", "Claire Cardie"]
year: 2002
venue: "Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)"
source: "https://aclanthology.org/W02-1008/"
doi: "10.3115/1118693.1118701"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ng-2002-combining-sample-selection-error-driven/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ng-2002-combining-sample-selection-error-driven/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2002](https://md.research.elamur.ai/free/en/papers/2002/) / Combining Sample Selection and Error-Driven Pruning for Machine Learning of Coreference Rules

# Combining Sample Selection and Error-Driven Pruning for Machine Learning of Coreference Rules

**Authors:** [Vincent Ng](https://md.research.elamur.ai/free/en/authors/vincent-ng/) and [Claire Cardie](https://md.research.elamur.ai/free/en/authors/claire-cardie/) · **Year:** 2002 · **Venue:** [Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)](https://md.research.elamur.ai/free/en/venues/emnlp/) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W02-1008/) · [PDF](https://aclanthology.org/W02-1008.pdf) · [DOI 10.3115/1118693.1118701](https://doi.org/10.3115/1118693.1118701) · [OpenAlex](https://openalex.org/W2047885427)

**Citations:** 50 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/ng-2002-combining-sample-selection-error-driven/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper addresses noun phrase coreference resolution, which is usually treated as pairwise classification of NPs followed by clustering. Ng and Cardie identify three weaknesses of this setup. First, positive pairs are very rare. Second, many positive pairs drawn from a coreference chain are hard to learn from local cues. Third, pairwise decisions ignore transitivity, so better classifier accuracy need not improve clustering-level scores.

They build a baseline with RIPPER rule learning, 25 automatically computed features and best-first clustering. Each weakness gets its own fix:
- **NEG-SELECT** removes many negative training pairs.
- **Automatic positive-instance selection** keeps mainly the "easy" coreferent pairs.
- **RULE-SELECT** is a greedy error-driven method. It drops classifier rules whenever doing so raises the MUC F-measure on a separate pruning corpus.

Evaluated on MUC-6 and MUC-7, the final system reaches F-measures of 69.5 and 63.4. This beats the best official MUC systems and the strongest earlier learning-based approach (Soon et al., 2001). The work matters because it links training decisions directly to the evaluation metric, without changing the learner itself.

## Key points
- Only about 2% of training pairs in the MUC-6/MUC-7 coreference data are positive. Negative sample selection raised F-measure by roughly 3–5%, and positive sample selection added a further 12–14%.
- RULE-SELECT moved F-measure from 69.3 to 69.5 on MUC-6 and from 57.2 to 63.4 on MUC-7. On MUC-7, precision jumped from 55.1 to 73.6, while recall dipped from 59.5 to 54.2.
- Each dataset's formal evaluation corpus served as the pruning corpus for the other. As a control, the authors simply trained on the training and pruning data combined, without pruning. This gave 67.6 on MUC-6 and 57.8 on MUC-7, so the gains come from pruning rather than from extra data.
- The final system outperforms the best MUC entries by 4.6% (MUC-6) and 1.6% (MUC-7), and Soon et al. (2001) by 6.9% and 3.0%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 68 works · 52,012 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 7,349 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 7,183 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 7,418 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 7,357 works

🧭 Topics (1): [🔗 Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) · 537 works

## Cite

Cite as: Vincent Ng and Claire Cardie. 2002. Combining Sample Selection and Error-Driven Pruning for Machine Learning of Coreference Rules. Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ng2002combining,
  title = {{Combining Sample Selection and Error-Driven Pruning for Machine Learning of Coreference Rules}},
  author = {Vincent Ng and Claire Cardie},
  booktitle = {Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)},
  year = {2002},
  doi = {10.3115/1118693.1118701},
  url = {https://doi.org/10.3115/1118693.1118701}
}
```

- [← previous Multiword expressions: linguistic precision and reusability](https://md.research.elamur.ai/free/en/papers/copestake-2002-multiword-expressions-linguistic-precision-reusability/)
- [next → A Probabilistic Method for Analyzing Japanese Anaphora Integrating Zero Pronoun Detection and Resolution](https://md.research.elamur.ai/free/en/papers/seki-2002-probabilistic-method-analyzing-japanese-anaphora/)
- [↑ back to papers of 2002](https://md.research.elamur.ai/free/en/papers/2002/)

## References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. C. Aone and S. W. Bennett. 1995. Evaluating Automated and Manual Acquisition of Anaphora Resolution Strategies. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics, pages 122–129. [our card](https://md.research.elamur.ai/free/en/papers/aone-1995-evaluating-automated-manual-acquisition-anaphora/)
2. A. Blum and P. Langley. 1997. Selection of relevant features and examples in machine learning. Artificial Intelligence, pages 245–271. [find](https://scholar.google.com/scholar?q=A%2E+Blum+and+P%2E+Langley%2E+1997%2E+Selection+of+relevant+features+and+examples+in+machine+learning%2E+Artificial+Intelligence%2C+pages+245%E2%80%93271%2E)
3. C. Cardie and N. Howe. 1997. Improving minority class prediction using case-specific feature weights. In Proceedings of the Fourteenth International Conference on Machine Learning, pages 57–65. [find](https://scholar.google.com/scholar?q=C%2E+Cardie+and+N%2E+Howe%2E+1997%2E+Improving+minority+class+prediction+using+case%2Dspecific+feature+weights%2E+In+Proceedings+of+the+Fourteenth+International+Conference+on+Machine+Learning%2C+pages+57%E2%80%9365%2E)
4. C. Cardie and K. Wagstaff. 1999. Noun Phrase Coreference as Clustering. In Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC-99), pages 82– 89. [our card](https://md.research.elamur.ai/free/en/papers/cardie-1999-noun-phrase-coreference-as-clustering/)
5. W. Cohen. 1995. Fast Effective Rule Induction. In Proceedings of the Twelfth International Conference on Machine Learning, San Francisco, CA. [find](https://scholar.google.com/scholar?q=W%2E+Cohen%2E+1995%2E+Fast+Effective+Rule+Induction%2E+In+Proceedings+of+the+Twelfth+International+Conference+on+Machine+Learning%2C+San+Francisco%2C+CA%2E)
6. A. Dempster. 1968. A generalization of Bayesian inference. Journal of the Royal Statistical Society, 30:205– 247. [find](https://scholar.google.com/scholar?q=A%2E+Dempster%2E+1968%2E+A+generalization+of+Bayesian+inference%2E+Journal+of+the+Royal+Statistical+Society%2C+30%3A205%E2%80%93+247%2E)
7. T. Fawcett. 1996. Learning with skewed class distributions — summary of responses. Machine Learning List: Vol. 8, No. 20. [find](https://scholar.google.com/scholar?q=T%2E+Fawcett%2E+1996%2E+Learning+with+skewed+class+distributions+%E2%80%94+summary+of+responses%2E+Machine+Learning+List%3A+Vol%2E+8%2C+No%2E+20%2E)
8. D. F. Gordon and D. Perlis. 1989. Explicitly biased generalization. Computational Intelligence, 5:67–81. [find](https://scholar.google.com/scholar?q=D%2E+F%2E+Gordon+and+D%2E+Perlis%2E+1989%2E+Explicitly+biased+generalization%2E+Computational+Intelligence%2C+5%3A67%E2%80%9381%2E)
9. S. Harabagiu, R. Bunescu, and S. Maiorano. 2001. Text and Knowledge Mining for Coreference Resolution. In Proceedings of the Second Meeting of the North America Chapter of the Association for Computational Linguistics (NAACL-2001), pages 55–62. [find](https://scholar.google.com/scholar?q=S%2E+Harabagiu%2C+R%2E+Bunescu%2C+and+S%2E+Maiorano%2E+2001%2E+Text+and+Knowledge+Mining+for+Coreference+Resolution%2E+In+Proceedings+of+the+Second+Meeting+of+the+North+America+Chapter+of+the+Association+for+Computat)
10. A. Kehler. 1997. Probabilistic Coreference in Information Extraction. In Proceedings of the Second Conference on Empirical Methods in Natural Language Processing, pages 163–173. [our card](https://md.research.elamur.ai/free/en/papers/kehler-1997-probabilistic-coreference-information-extraction/)
11. M. Kubat and S. Matwin. 1997. Addressing the Curse of Imbalanced Training Sets: One-Sided Selection. In Proceedings of the 14th International Conference on Machine Learning (ICML-97), pages 179–186. [find](https://scholar.google.com/scholar?q=M%2E+Kubat+and+S%2E+Matwin%2E+1997%2E+Addressing+the+Curse+of+Imbalanced+Training+Sets%3A+One%2DSided+Selection%2E+In+Proceedings+of+the+14th+International+Conference+on+Machine+Learning+%28ICML%2D97%29%2C+pages+179%E2%80%93186%2E)
12. J. McCarthy and W. Lehnert. 1995. Using Decision Trees for Coreference Resolution. In Proceedings of the Fourteenth International Conference on Artificial Intelligence, pages 1050–1055. [find](https://scholar.google.com/scholar?q=J%2E+McCarthy+and+W%2E+Lehnert%2E+1995%2E+Using+Decision+Trees+for+Coreference+Resolution%2E+In+Proceedings+of+the+Fourteenth+International+Conference+on+Artificial+Intelligence%2C+pages+1050%E2%80%931055%2E)
13. MUC-6. 1995. Proceedings of the Sixth Message Understanding Conference (MUC-6). Morgan Kaufmann, San Francisco, CA. [find](https://scholar.google.com/scholar?q=MUC%2D6%2E+1995%2E+Proceedings+of+the+Sixth+Message+Understanding+Conference+%28MUC%2D6%29%2E+Morgan+Kaufmann%2C+San+Francisco%2C+CA%2E)
14. MUC-7. 1998. Proceedings of the Seventh Message Understanding Conference (MUC-7). Morgan Kaufmann, San Francisco, CA. [find](https://scholar.google.com/scholar?q=MUC%2D7%2E+1998%2E+Proceedings+of+the+Seventh+Message+Understanding+Conference+%28MUC%2D7%29%2E+Morgan+Kaufmann%2C+San+Francisco%2C+CA%2E)
15. V. Ng and C. Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/ng-2002-improving-machine-learning-approaches-coreference/)
16. M. Pazzani, C. Merz, P. Murphy, K. Ali, T. Hume, and C. Brunk. 1994. Reducing Misclassification Costs. In Proceedings of the Eleventh International Conference on Machine Learning, pages 217–225. [find](https://scholar.google.com/scholar?q=M%2E+Pazzani%2C+C%2E+Merz%2C+P%2E+Murphy%2C+K%2E+Ali%2C+T%2E+Hume%2C+and+C%2E+Brunk%2E+1994%2E+Reducing+Misclassification+Costs%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Machine+Learning%2C+pages+217%E2%80%93225%2E)
17. W. M. Soon, H. T. Ng, and D. C. Y. Lim. 2001. A Machine Learning Approach to Coreference Resolution of Noun Phrases. Computational Linguistics, 27(4):521–544. [our card](https://md.research.elamur.ai/free/en/papers/soon-2001-machine-learning-approach-coreference-resolution/)
18. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, and L. Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message Understanding Conference (MUC-6), pages 45– 52. Morgan Kaufmann. [our card](https://md.research.elamur.ai/free/en/papers/vilain-1995-model-theoretic-coreference-scoring-scheme/)

*Updated 23 October 2026*