---
title: "Active Learning for Coreference Resolution — Laws 2012"
description: "Supervised coreference resolution (CR) relies on costly hand-labeled data."
authors: ["Florian Laws", "Florian Heimerl", "Hinrich Schütze"]
year: 2012
venue: "Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/N12-1055/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/laws-2012-active-learning-coreference-resolution/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/laws-2012-active-learning-coreference-resolution/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2012](https://md.research.elamur.ai/free/papers/2012/) / Active Learning for Coreference Resolution

# Active Learning for Coreference Resolution

**Authors:** Florian Laws, Florian Heimerl, [Hinrich Schütze](https://md.research.elamur.ai/free/authors/hinrich-schutze/) · **Year:** 2012 · **Venue:** Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N12-1055/) · [PDF](https://aclanthology.org/N12-1055.pdf) · [OpenAlex](https://openalex.org/W1780515473)

**Citations:** 9 citations (openalex-snapshot · 2026-10-14) · [10](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Supervised coreference resolution (CR) relies on costly hand-labeled data. Active learning (AL) can reduce that cost, but earlier attempts at AL for CR had failed. Uncertainty sampling performed no better than random selection, and the authors confirmed this in their own setup. They propose an AL scheme built on the mention-pair system SUCRE, which uses a decision-tree link classifier and best-first clustering. The scheme combines three ideas:
- **Query-by-committee selection:** links are chosen by committee disagreement rather than by single-model uncertainty.
- **Bootstrapped neighborhood pooling:** the committee's majority vote stands in for missing gold labels, which yields a class-balanced candidate pool in the style of Soon et al.
- **Neighborhood selection:** whole groups of links around a markable are labeled together, so each markable gets both positive and negative examples.

The method was tested on the English portion of the SemEval-2010 CR data, a subset of OntoNotes 2.0. Neighborhood-based AL beat a strong random baseline that was given gold neighborhoods. It did so on MUC, B3, CEAF and their mean, needing fewer labels and reaching a higher ceiling. This is the first positive result showing that AL can make CR annotation cheaper.

## Key points
- **Data and pool:** the training set has about 96,000 words and the test set about 24,000. The candidate pool holds 1.7 million links within a 100-markable window, and only 1.5% of these are coreferent.
- **Committee:** 10 copies of the SUCRE link classifier, each trained on a random subset of the links labeled so far. Links are ranked by vote entropy.
- **Results at 20,000 links:** neighborhood-selection AL reached MUC 63.07, B3 86.94, CEAF 84.42 and mean F1 78.14. The gold N-pool random baseline scored 61.60, 85.00, 82.85 and 76.48. The gains are significant under a sign test (p \< .05, 10 runs).
- **Link-level selection:** selecting individual links helped B3 and CEAF but hurt MUC. It produced too many singleton entities, which lowered recall.
- **Results at 50,000 links:** the advantage persisted, with a mean F1 of 78.42 versus 77.33 for the baseline.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/)

## Cited by

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 4 not yet.

- [Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle*](https://md.research.elamur.ai/free/papers/cao-2021-gender-inclusive-coreference-resolution-analysis/)
- [Annotating Mentions Alone Enables Efficient Domain Adaptation for Coreference Resolution](https://md.research.elamur.ai/free/papers/gandhi-2023-annotating-mentions-alone-enables-efficient/)
- [Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers](https://md.research.elamur.ai/free/papers/guha-2015-removing-training-wheels-coreference-dataset/)
- [Active Learning for Coreference Resolution using Discrete Annotation](https://md.research.elamur.ai/free/papers/li-2020-active-learning-coreference-resolution-discrete/)
- [Adapting Coreference Resolution Models through Active Learning](https://md.research.elamur.ai/free/papers/yuan-2022-adapting-coreference-resolution-models-through/)
- [Domain Adaptation with Active Learning for Coreference Resolution](https://md.research.elamur.ai/free/papers/zhao-2014-domain-adaptation-active-learning-coreference/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{laws2012active,
  title = {{Active Learning for Coreference Resolution}},
  author = {Laws, Florian and Heimerl, Florian and Schütze, Hinrich},
  booktitle = {Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2012},
  url = {https://aclanthology.org/N12-1055/}
}
```

- [← previous A Morphologically Annotated Hebrew CHILDES Corpus](https://md.research.elamur.ai/free/papers/aclbulk-2012-albert-morphologically-annotated-hebrew-childes-corpus/)
- [next → ADADELTA: An Adaptive Learning Rate Method](https://md.research.elamur.ai/free/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/papers/2012/)

## References

16 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Argamon-Engelson and I. Dagan. 1999. Committee-based sample selection for probabilistic classifiers. JAIR, 11:335–360. [find](https://scholar.google.com/scholar?q=S%2E+Argamon%2DEngelson+and+I%2E+Dagan%2E+1999%2E+Committee%2Dbased+sample+selection+for+probabilistic+classifiers%2E+JAIR%2C+11%3A335%E2%80%93360%2E)
2. A. Culotta, M. Wick, R. Hall, and A. McCallum. 2007. First-order probabilistic models for coreference resolution. In HLT-NAACL 2007. [find](https://scholar.google.com/scholar?q=A%2E+Culotta%2C+M%2E+Wick%2C+R%2E+Hall%2C+and+A%2E+McCallum%2E+2007%2E+First%2Dorder+probabilistic+models+for+coreference+resolution%2E+In+HLT%2DNAACL+2007%2E) [our card](https://md.research.elamur.ai/free/papers/culotta-2007-first-order-probabilistic-models-coreference/)
3. Pedro Domingos and Michael J. Pazzani. 1997. On the optimality of the simple bayesian classifier under zero-one loss. Mach. Learn., 29(2-3):103–130. [find](https://scholar.google.com/scholar?q=Pedro+Domingos+and+Michael+J%2E+Pazzani%2E+1997%2E+On+the+optimality+of+the+simple+bayesian+classifier+under+zero%2Done+loss%2E+Mach%2E+Learn%2E%2C+29%282%2D3%29%3A103%E2%80%93130%2E)
4. K. Dwyer and R. Holte. 2007. Decision tree instability and active learning. In ECML. [find](https://scholar.google.com/scholar?q=K%2E+Dwyer+and+R%2E+Holte%2E+2007%2E+Decision+tree+instability+and+active+learning%2E+In+ECML%2E)
5. C. Gasperin. 2009. Active learning for anaphora resolution. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing. [find](https://scholar.google.com/scholar?q=C%2E+Gasperin%2E+2009%2E+Active+learning+for+anaphora+resolution%2E+In+Proceedings+of+the+NAACL+HLT+2009+Workshop+on+Active+Learning+for+Natural+Language+Processing%2E) [our card](https://md.research.elamur.ai/free/papers/gasperin-2009-active-learning-anaphora-resolution/)
6. E. Hovy, M. Marcus, M. Palmer, L. Ramshaw, and R. Weischedel. 2006. Ontonotes: The 90% solution. In HLT-NAACL. [find](https://scholar.google.com/scholar?q=E%2E+Hovy%2C+M%2E+Marcus%2C+M%2E+Palmer%2C+L%2E+Ramshaw%2C+and+R%2E+Weischedel%2E+2006%2E+Ontonotes%3A+The+90%25+solution%2E+In+HLT%2DNAACL%2E)
7. H. Kobdani, H. Schütze, M. Schiehlen, and H. Kamp. 2011. Bootstrapping coreference resolution using word associations. In ACL. [find](https://scholar.google.com/scholar?q=H%2E+Kobdani%2C+H%2E+Schu%CC%88tze%2C+M%2E+Schiehlen%2C+and+H%2E+Kamp%2E+2011%2E+Bootstrapping+coreference+resolution+using+word+associations%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/kobdani-2011-bootstrapping-coreference-resolution-word-associations/)
8. D. Lewis and W. Gale. 1994. A sequential algorithm for training text classifiers. In SIGIR. [find](https://scholar.google.com/scholar?q=D%2E+Lewis+and+W%2E+Gale%2E+1994%2E+A+sequential+algorithm+for+training+text+classifiers%2E+In+SIGIR%2E)
9. V. Ng and C. Cardie. 2002. Improving machine learning approaches to coreference resolution. In ACL. [find](https://scholar.google.com/scholar?q=V%2E+Ng+and+C%2E+Cardie%2E+2002%2E+Improving+machine+learning+approaches+to+coreference+resolution%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/ng-2002-improving-machine-learning-approaches-coreference/)
10. M. Osborne and J. Baldridge. 2004. Ensemble-based active learning for parse selection. In HLT-NAACL. [find](https://scholar.google.com/scholar?q=M%2E+Osborne+and+J%2E+Baldridge%2E+2004%2E+Ensemble%2Dbased+active+learning+for+parse+selection%2E+In+HLT%2DNAACL%2E)
11. S. Pradhan, L. Ramshaw, M. Marcus, M. Palmer, R. Weischedel, and N. Xue. 2011. Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In CoNLL. [find](https://scholar.google.com/scholar?q=S%2E+Pradhan%2C+L%2E+Ramshaw%2C+M%2E+Marcus%2C+M%2E+Palmer%2C+R%2E+Weischedel%2C+and+N%2E+Xue%2E+2011%2E+Conll%2D2011+shared+task%3A+Modeling+unrestricted+coreference+in+ontonotes%2E+In+CoNLL%2E) [our card](https://md.research.elamur.ai/free/papers/pradhan-2011-conll-2011-shared-task-modeling/)
12. M. Recasens, L. Màrquez, E. Sapena, M. A. Martı́, M. Taulé, V. Hoste, M. Poesio, and Y. Versley. 2010. Semeval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop on Semantic Evaluation. [find](https://scholar.google.com/scholar?q=M%2E+Recasens%2C+L%2E+Ma%CC%80rquez%2C+E%2E+Sapena%2C+M%2E+A%2E+Mart%C4%B1%CC%81%2C+M%2E+Taule%CC%81%2C+V%2E+Hoste%2C+M%2E+Poesio%2C+and+Y%2E+Versley%2E+2010%2E+Semeval%2D2010+task+1%3A+Coreference+resolution+in+multiple+languages%2E+In+Proceedings+of+the+5th+In) [our card](https://md.research.elamur.ai/free/papers/recasens-2009-semeval-2010-task-1-coreference/)
13. E. Ringger, P. McClanahan, R. Haertel, G. Busby, M. Carmen, J. Carroll, K. Seppi, and D. Lonsdale. 2007. Active learning for part-of-speech tagging: Accelerating corpus annotation. In Linguistic Annotation Workshop at ACL-2007. [find](https://scholar.google.com/scholar?q=E%2E+Ringger%2C+P%2E+McClanahan%2C+R%2E+Haertel%2C+G%2E+Busby%2C+M%2E+Carmen%2C+J%2E+Carroll%2C+K%2E+Seppi%2C+and+D%2E+Lonsdale%2E+2007%2E+Active+learning+for+part%2Dof%2Dspeech+tagging%3A+Accelerating+corpus+annotation%2E+In+Linguistic+Annot)
14. W. M. Soon, D. Chung, D. Chung Yong Lim, Y. Lim, and H. T. Ng. 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4). [find](https://scholar.google.com/scholar?q=W%2E+M%2E+Soon%2C+D%2E+Chung%2C+D%2E+Chung+Yong+Lim%2C+Y%2E+Lim%2C+and+H%2E+T%2E+Ng%2E+2001%2E+A+machine+learning+approach+to+coreference+resolution+of+noun+phrases%2E+Computational+Linguistics%2C+27%284%29%2E) [our card](https://md.research.elamur.ai/free/papers/soon-2001-machine-learning-approach-coreference-resolution/)
15. K. Tomanek, J. Wermter, and U. Hahn. 2007. An approach to text corpus construction which cuts annotation costs and maintains reusability of annotated data. In EMNLP-CoNLL. [find](https://scholar.google.com/scholar?q=K%2E+Tomanek%2C+J%2E+Wermter%2C+and+U%2E+Hahn%2E+2007%2E+An+approach+to+text+corpus+construction+which+cuts+annotation+costs+and+maintains+reusability+of+annotated+data%2E+In+EMNLP%2DCoNLL%2E)
16. S. Tong and D. Koller. 2002. Support vector machine active learning with applications to text classification. JMLR, 2:45–66. [find](https://scholar.google.com/scholar?q=S%2E+Tong+and+D%2E+Koller%2E+2002%2E+Support+vector+machine+active+learning+with+applications+to+text+classification%2E+JMLR%2C+2%3A45%E2%80%9366%2E)