---
title: "Bootstrapping Coreference Classifiers with — Ng 2003"
description: "Co-training normally needs two feature \"views\" that are each sufficient and conditionally independent."
authors: ["Vincent Ng", "Claire Cardie"]
year: 2003
venue: "Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/W03-1015/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/ng-2003-bootstrapping-coreference-classifiers-multiple-machine/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/ng-2003-bootstrapping-coreference-classifiers-multiple-machine/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2003](https://md.research.elamur.ai/free/papers/2003/) / Bootstrapping Coreference Classifiers with Multiple Machine Learning Algorithms

# Bootstrapping Coreference Classifiers with Multiple Machine Learning Algorithms

**Authors:** [Vincent Ng](https://md.research.elamur.ai/free/authors/vincent-ng/), [Claire Cardie](https://md.research.elamur.ai/free/authors/claire-cardie/) · **Year:** 2003 · **Venue:** Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W03-1015/) · [PDF](https://aclanthology.org/W03-1015.pdf) · [OpenAlex](https://openalex.org/W2135734357)

**Citations:** 40 citations (openalex-snapshot · 2026-10-14) · [7](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Co-training normally needs two feature "views" that are each sufficient and conditionally independent. Noun phrase coreference resolution has no natural split of this kind, so Ng and Cardie propose a single-view bootstrapping method. In place of two views, it uses two different learners: a generative naive Bayes classifier and a discriminative decision list. The authors argue that each learner offsets weaknesses of the other. For example, decision lists exploit single decisive feature-value pairs, and naive Bayes copes better with the skewed class distribution.

The method trains a pairwise coreference classifier from 1000 labeled NP-pair instances plus a large unlabeled pool. It is evaluated on MUC-6 and MUC-7. Standard Blum and Mitchell multi-view co-training fails to improve the system, while the two-learner approach raises F-measure substantially.

The authors also introduce a more conservative instance-ranking scheme. It prefers instances that both classifiers label the same way, especially when only one of them is confident. This scheme resists the performance decline that usually appears as bootstrapping goes on. That matters because a good stopping point is hard to pick when labeled data is scarce.

## Key points
- **Data:** MUC-6 and MUC-7, trained on 30 "dry run" texts and tested on the "formal evaluation" texts.
  - The dry-run texts yield 491659 (MUC-6) and 482125 (MUC-7) NP-pair instances.
  - Only 1000 randomly chosen instances are labeled.
- **Single-view bootstrapping, best F-measure vs. baseline:**
  - MUC-6: naive Bayes 51.6 → 63.9; decision list 28.7 → 54.1.
  - MUC-7: naive Bayes 40.1 → 54.6; decision list 45.8 → 50.8.
- **Comparison methods:**
  - Multi-view co-training gave no useful gains, e.g. MUC-6 naive Bayes fell to 48.7.
  - Self-training also helped little.
- **Ranking method:** three ordered preferences. First, the classifiers agree on the label. Second, among those, one is confident and the other is not. Third, if no instances get the same label, fall back to Blum and Mitchell's rank-by-confidence. The second preference draws on Steedman et al.'s S int-n selection.
- **Ranking results** (MUC-6, pool size 5000, growth size 50):
  - The new ranking learns more slowly than rank-by-confidence.
  - It avoids lasting deterioration and outperforms rank-by-confidence from about iteration 2800 onward.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/)

## Cited by

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 5 not yet.

- [Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle*](https://md.research.elamur.ai/free/papers/cao-2021-gender-inclusive-coreference-resolution-analysis/)
- [Bootstrapping coreference resolution using word associations](https://md.research.elamur.ai/free/papers/kobdani-2011-bootstrapping-coreference-resolution-word-associations/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ng2003bootstrapping,
  title = {{Bootstrapping Coreference Classifiers with Multiple Machine Learning Algorithms}},
  author = {Ng, Vincent and Cardie, Claire},
  booktitle = {Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing},
  year = {2003},
  doi = {10.3115/1119355.1119370},
  url = {https://doi.org/10.3115/1119355.1119370}
}
```

- [← previous Automatic Pronominal Anaphora Resolution in English Texts](https://md.research.elamur.ai/free/papers/liang-2003-automatic-pronominal-anaphora-resolution-english/)
- [next → CatVar: A Database of Categorial Variations for English](https://md.research.elamur.ai/free/papers/lex-2003-habash-catvar/)
- [↑ back to papers of 2003](https://md.research.elamur.ai/free/papers/2003/)

## References

19 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michele Banko and Eric Brill. 2001. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the ACL/EACL, pages 26–33. [find](https://scholar.google.com/scholar?q=Michele+Banko+and+Eric+Brill%2E+2001%2E+Scaling+to+very+very+large+corpora+for+natural+language+disambiguation%2E+In+Proceedings+of+the+ACL%2FEACL%2C+pages+26%E2%80%9333%2E)
2. Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of COLT, pages 92–100. [find](https://scholar.google.com/scholar?q=Avrim+Blum+and+Tom+Mitchell%2E+1998%2E+Combining+labeled+and+unlabeled+data+with+co%2Dtraining%2E+In+Proceedings+of+COLT%2C+pages+92%E2%80%93100%2E) [our card](https://md.research.elamur.ai/free/papers/blum-1998-combining-labeled-unlabeled-data-co-training/)
3. Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In Proceedings of EMNLP/VLC, pages 100–110. [find](https://scholar.google.com/scholar?q=Michael+Collins+and+Yoram+Singer%2E+1999%2E+Unsupervised+models+for+named+entity+classification%2E+In+Proceedings+of+EMNLP%2FVLC%2C+pages+100%E2%80%93110%2E)
4. Sally Goldman and Yan Zhou. 2000. Enhancing supervised learning with unlabeled data. In Proceedings of ICML, pages 327–334. [find](https://scholar.google.com/scholar?q=Sally+Goldman+and+Yan+Zhou%2E+2000%2E+Enhancing+supervised+learning+with+unlabeled+data%2E+In+Proceedings+of+ICML%2C+pages+327%E2%80%93334%2E)
5. Andrew McCallum and Kamal Nigam. 1998. Employing EM and pool-based active learning for text classification. In Proceedings of ICML, pages 359–367. [find](https://scholar.google.com/scholar?q=Andrew+McCallum+and+Kamal+Nigam%2E+1998%2E+Employing+EM+and+pool%2Dbased+active+learning+for+text+classification%2E+In+Proceedings+of+ICML%2C+pages+359%E2%80%93367%2E)
6. MUC-6. 1995. Proceedings of the Sixth Message Understanding Conference (MUC-6). [find](https://scholar.google.com/scholar?q=MUC%2D6%2E+1995%2E+Proceedings+of+the+Sixth+Message+Understanding+Conference+%28MUC%2D6%29%2E)
7. MUC-7. 1998. Proceedings of the Seventh Message Understanding Conference (MUC-7). [find](https://scholar.google.com/scholar?q=MUC%2D7%2E+1998%2E+Proceedings+of+the+Seventh+Message+Understanding+Conference+%28MUC%2D7%29%2E)
8. Christoph Mueller, Stefan Rapp, and Michael Strube. 2002. Applying co-training to reference resolution. In Proceedings of the ACL, pages 352–359. [find](https://scholar.google.com/scholar?q=Christoph+Mueller%2C+Stefan+Rapp%2C+and+Michael+Strube%2E+2002%2E+Applying+co%2Dtraining+to+reference+resolution%2E+In+Proceedings+of+the+ACL%2C+pages+352%E2%80%93359%2E)
9. Ion Muslea, Steven Minton, and Craig Knoblock. 2002. Active + Semi-Supervised Learning = Robust Multi-View Learning. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Ion+Muslea%2C+Steven+Minton%2C+and+Craig+Knoblock%2E+2002%2E+Active+%2B+Semi%2DSupervised+Learning+%3D+Robust+Multi%2DView+Learning%2E+In+Proceedings+of+ICML%2E)
10. Vincent Ng and Claire Cardie. 2002. Combining sample selection and error-driven pruning for machine learning of coreference rules. In Proceedings of EMNLP, pages 55–62. [find](https://scholar.google.com/scholar?q=Vincent+Ng+and+Claire+Cardie%2E+2002%2E+Combining+sample+selection+and+error%2Ddriven+pruning+for+machine+learning+of+coreference+rules%2E+In+Proceedings+of+EMNLP%2C+pages+55%E2%80%9362%2E) [our card](https://md.research.elamur.ai/free/papers/ng-2002-combining-sample-selection-error-driven/)
11. Vincent Ng and Claire Cardie. 2003. Weakly supervised natural language learning without redundant views. In Proceedings of HLT-NAACL. [find](https://scholar.google.com/scholar?q=Vincent+Ng+and+Claire+Cardie%2E+2003%2E+Weakly+supervised+natural+language+learning+without+redundant+views%2E+In+Proceedings+of+HLT%2DNAACL%2E)
12. Kamal Nigam and Rayid Ghani. 2000. Analyzing the effectiveness and applicability of co-training. In Proceedings of CIKM, pages 86–93. [find](https://scholar.google.com/scholar?q=Kamal+Nigam+and+Rayid+Ghani%2E+2000%2E+Analyzing+the+effectiveness+and+applicability+of+co%2Dtraining%2E+In+Proceedings+of+CIKM%2C+pages+86%E2%80%9393%2E)
13. Kamal Nigam, Andrew McCallum, Sabastian Thrun, and Tom Mitchell. 2000. Text classification from labeled and unlabeled documents using EM. Machine Learning, 39(2/3):103–134. [find](https://scholar.google.com/scholar?q=Kamal+Nigam%2C+Andrew+McCallum%2C+Sabastian+Thrun%2C+and+Tom+Mitchell%2E+2000%2E+Text+classification+from+labeled+and+unlabeled+documents+using+EM%2E+Machine+Learning%2C+39%282%2F3%29%3A103%E2%80%93134%2E)
14. David Pierce and Claire Cardie. 2001. Limitations of co-training for natural language learning from large datasets. In Proceedings of EMNLP, pages 1–9. [find](https://scholar.google.com/scholar?q=David+Pierce+and+Claire+Cardie%2E+2001%2E+Limitations+of+co%2Dtraining+for+natural+language+learning+from+large+datasets%2E+In+Proceedings+of+EMNLP%2C+pages+1%E2%80%939%2E)
15. Ellen Riloff and Rosie Jones. 1999. Learning dictionaries for information extraction by multi-level bootstrapping. In Proceedings of AAAI, pages 474–479. [find](https://scholar.google.com/scholar?q=Ellen+Riloff+and+Rosie+Jones%2E+1999%2E+Learning+dictionaries+for+information+extraction+by+multi%2Dlevel+bootstrapping%2E+In+Proceedings+of+AAAI%2C+pages+474%E2%80%93479%2E)
16. M. Steedman, R. Hwa, S. Clark, M. Osborne, A. Sarkar, J. Hockenmaier, P. Ruhlen, S. Baker, and J. Crim. 2003a. Example selection for bootstrapping statistical parsers. In Proceedings of HLT-NAACL. [find](https://scholar.google.com/scholar?q=M%2E+Steedman%2C+R%2E+Hwa%2C+S%2E+Clark%2C+M%2E+Osborne%2C+A%2E+Sarkar%2C+J%2E+Hockenmaier%2C+P%2E+Ruhlen%2C+S%2E+Baker%2C+and+J%2E+Crim%2E+2003a%2E+Example+selection+for+bootstrapping+statistical+parsers%2E+In+Proceedings+of+HLT%2DNAACL%2E)
17. M. Steedman, M. Osborne, A. Sarkar, S. Clark, R. Hwa, J. Hockenmaier, P. Ruhlen, S. Baker, and J. Crim. 2003b. Bootstrapping statistical parsers from small datasets. In Proceedings of the EACL. [find](https://scholar.google.com/scholar?q=M%2E+Steedman%2C+M%2E+Osborne%2C+A%2E+Sarkar%2C+S%2E+Clark%2C+R%2E+Hwa%2C+J%2E+Hockenmaier%2C+P%2E+Ruhlen%2C+S%2E+Baker%2C+and+J%2E+Crim%2E+2003b%2E+Bootstrapping+statistical+parsers+from+small+datasets%2E+In+Proceedings+of+the+EACL%2E)
18. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, and L. Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message Understanding Conference (MUC-6), pages 45–52. [find](https://scholar.google.com/scholar?q=M%2E+Vilain%2C+J%2E+Burger%2C+J%2E+Aberdeen%2C+D%2E+Connolly%2C+and+L%2E+Hirschman%2E+1995%2E+A+model%2Dtheoretic+coreference+scoring+scheme%2E+In+Proceedings+of+the+Sixth+Message+Understanding+Conference+%28MUC%2D6%29%2C+pages+45%E2%80%9352%2E) [our card](https://md.research.elamur.ai/free/papers/vilain-1995-model-theoretic-coreference-scoring-scheme/)
19. David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the ACL, pages 189–196. [find](https://scholar.google.com/scholar?q=David+Yarowsky%2E+1995%2E+Unsupervised+word+sense+disambiguation+rivaling+supervised+methods%2E+In+Proceedings+of+the+ACL%2C+pages+189%E2%80%93196%2E) [our card](https://md.research.elamur.ai/free/papers/yarowsky-1995-unsupervised-word-sense-disambiguation-rivaling/)