---
title: "Active Learning for Coreference Resolution using — Li 2020"
description: "Labeling coreference exhaustively is costly, because annotators must follow entity chains across long texts."
authors: ["Belinda Z. Li", "Gabriel Stanovsky", "Luke Zettlemoyer"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.738/"
doi: "10.18653/v1/2020.acl-main.738"
arxiv: "2004.13671"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/li-2020-active-learning-coreference-resolution-discrete/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/li-2020-active-learning-coreference-resolution-discrete/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2020](https://md.research.elamur.ai/free/papers/2020/) / Active Learning for Coreference Resolution using Discrete Annotation

# Active Learning for Coreference Resolution using Discrete Annotation

**Authors:** Belinda Z. Li, [Gabriel Stanovsky](https://md.research.elamur.ai/free/authors/gabriel-stanovsky/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/authors/luke-zettlemoyer/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.738/) · [PDF](https://aclanthology.org/2020.acl-main.738.pdf) · [DOI 10.18653/v1/2020.acl-main.738](https://doi.org/10.18653/v1/2020.acl-main.738) · [arXiv 2004.13671](https://arxiv.org/abs/2004.13671) · [OpenAlex](https://openalex.org/W3023185160)

**Citations:** 21 citations (openalex-snapshot · 2026-10-14) · [15](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Labeling coreference exhaustively is costly, because annotators must follow entity chains across long texts. Active learning can lower this cost. Earlier work (Sachan et al., 2015) relied on pairwise queries, which only ask whether two mentions corefer. This paper adds one step. When an annotator rejects a proposed pair, they also mark the first antecedent of the queried span. The authors call this "discrete annotation."

They also propose mention clustering for sample selection. Antecedent probabilities or committee votes are combined at the level of predicted clusters. Must-link and cannot-link constraints are kept transitively closed, and the closure is built up incrementally. Three selectors use these cluster-level scores:
- clustered entropy
- clustered query-by-committee
- LCC/MCU

Experiments use the span-ranking model of Lee et al. (2017) in AllenNLP on English CoNLL-2012. Annotator answers are simulated from gold labels, and costs come from real timing sessions. Every non-random discrete selector beats pairwise annotation per hour of annotation effort. This makes the protocol a practical option for building coreference data in new, low-resource domains.

## Key points
- **Timing study:** 7 in-house annotators produced 826 answers. The initial pairwise question took about 15.96s, and the follow-up question about 15.57s. Asking only the discrete question took 28.01s, so the yes/no question saves time when the answer is positive.
- **Overall gain:** With budgets above roughly 15 minutes per document, discrete annotation improves averaged MUC/B3/CEAFe F1 by about 4 points absolute over pairwise selection.
- **Ablation at about 315 annotation hours (entropy selector):**
  - full method: 57.08 F1
  - without clustered probabilities: 56.49
  - without incremental link closures: 56.98
  - pairwise annotation: 54.27
- **Cost of full labeling:** A typical development document would need about 15,050 pairwise queries but at most 201 discrete ones, which is about 2.6% of the pairwise effort.
- **Other results:** The active learning selectors outperform a fully-labelled random-document baseline. Gains also appear in mention detection and in pronoun resolution, where pronouns were identified with spaCy. Code and the annotation interface are released on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/)

## Cited by

Cited by

15 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 14 not yet.

- [Adapting Coreference Resolution Models through Active Learning](https://md.research.elamur.ai/free/papers/yuan-2022-adapting-coreference-resolution-models-through/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2020active,
  title = {{Active Learning for Coreference Resolution using Discrete Annotation}},
  author = {Li, Belinda Z. and Stanovsky, Gabriel and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.738},
  eprint = {2004.13671},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.738}
}
```

- [← previous Abstract Syntax as Interlingua: Scaling Up the Grammatical Framework from Controlled Languages to Robust Pipelines](https://md.research.elamur.ai/free/papers/ranta-2020-abstract-syntax-as-interlingua-scaling/)
- [next → AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://md.research.elamur.ai/free/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/papers/2020/)

## References

15 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer… [find](https://scholar.google.com/scholar?q=Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Ana+Marasovic%2C+Noah+A%2E+Smith%2C+and+Matt+Gardner%2E+2019%2E+Quoref%3A+A+reading+comprehension+dataset+with+questions+requiring+coreferential+reasoning%2E+Proceedings+of+the+2019+C) [our card](https://md.research.elamur.ai/free/papers/dasigi-2019-quoref-reading-comprehension-dataset-questions/)
2. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew E. Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. CoRR, abs/1803.07640. [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Joel+Grus%2C+Mark+Neumann%2C+Oyvind+Tafjord%2C+Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Matthew+E%2E+Peters%2C+Michael+Schmitz%2C+and+Luke+S%2E+Zettlemoyer%2E+2018%2E+Allennlp%3A+A+deep+semantic+natural+language+proc) [our card](https://md.research.elamur.ai/free/papers/gardner-2018-allennlp-deep-semantic-natural-language/)
3. Dan Garrette and Jason Baldridge. 2013. Learning a part-of-speech tagger from two hours of annotation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 138–147, Atlanta, Georgia. Association for C… [find](https://scholar.google.com/scholar?q=Dan+Garrette+and+Jason+Baldridge%2E+2013%2E+Learning+a+part%2Dof%2Dspeech+tagger+from+two+hours+of+annotation%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Association+for+Comput)
4. Caroline Gasperin. 2009. Active learning for anaphora resolution. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, HLT ’09, pages 1–8, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Caroline+Gasperin%2E+2009%2E+Active+learning+for+anaphora+resolution%2E+In+Proceedings+of+the+NAACL+HLT+2009+Workshop+on+Active+Learning+for+Natural+Language+Processing%2C+HLT+%E2%80%9909%2C+pages+1%E2%80%938%2C+Stroudsburg%2C+PA%2C) [our card](https://md.research.elamur.ai/free/papers/gasperin-2009-active-learning-anaphora-resolution/)
5. Burr Settles. 2010. Active learning literature survey. University of Wisconsin, Madison, 52(55-66):11. [find](https://scholar.google.com/scholar?q=Burr+Settles%2E+2010%2E+Active+learning+literature+survey%2E+University+of+Wisconsin%2C+Madison%2C+52%2855%2D66%29%3A11%2E) [our card](https://md.research.elamur.ai/free/papers/settles-2009-active-learning-literature-survey/)
6. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In ACL, page (to appear), Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gabriel+Stanovsky%2C+Noah+A%2E+Smith%2C+and+Luke+Zettlemoyer%2E+2019%2E+Evaluating+gender+bias+in+machine+translation%2E+In+ACL%2C+page+%28to+appear%29%2C+Florence%2C+Italy%2E+Association+for+Computational+Linguistics%2E) [our card](https://md.research.elamur.ai/free/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/)
7. A. R. Syed, A. Rosenberg, and E. Kislal. 2016. Supervised and unsupervised active learning for automatic speech recognition of low-resource languages. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5320–5324. [find](https://scholar.google.com/scholar?q=A%2E+R%2E+Syed%2C+A%2E+Rosenberg%2C+and+E%2E+Kislal%2E+2016%2E+Supervised+and+unsupervised+active+learning+for+automatic+speech+recognition+of+low%2Dresource+languages%2E+In+2016+IEEE+International+Conference+on+Acoustic)
8. A. R. Syed, A. Rosenberg, and M. Mandel. 2017. Active learning for low-resource speech recognition: Impact of selection size and language modeling data. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5315–5319. [find](https://scholar.google.com/scholar?q=A%2E+R%2E+Syed%2C+A%2E+Rosenberg%2C+and+M%2E+Mandel%2E+2017%2E+Active+learning+for+low%2Dresource+speech+recognition%3A+Impact+of+selection+size+and+language+modeling+data%2E+In+2017+IEEE+International+Conference+on+Acoust)
9. Shanheng Zhao and Hwee Tou Ng. 2014. Domain adaptation with active learning for coreference resolution. In Proceedings of the 5th International Workshop on Health Text Mining and Information Analysis (Louhi), pages 21–29, Gothenburg, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shanheng+Zhao+and+Hwee+Tou+Ng%2E+2014%2E+Domain+adaptation+with+active+learning+for+coreference+resolution%2E+In+Proceedings+of+the+5th+International+Workshop+on+Health+Text+Mining+and+Information+Analysis+) [our card](https://md.research.elamur.ai/free/papers/zhao-2014-domain-adaptation-active-learning-coreference/)
10. Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal+and+Ines+Montani%2E+2017%2E+spacy+2%3A+Natural+language+understanding+with+bloom+embeddings%2C+convolutional+neural+networks+and+incremental+parsing%2E+To+appear%2E)
11. Mahnoosh Kholghi, Laurianne Sitbon, Guido Zuccon, and Anthony Nguyen. 2015. Active learning: a step towards automating medical concept extraction. Journal of the American Medical Informatics Association, 23(2):289–296. [find](https://scholar.google.com/scholar?q=Mahnoosh+Kholghi%2C+Laurianne+Sitbon%2C+Guido+Zuccon%2C+and+Anthony+Nguyen%2E+2015%2E+Active+learning%3A+a+step+towards+automating+medical+concept+extraction%2E+Journal+of+the+American+Medical+Informatics+Associati)
12. Florian Laws, Florian Heimerl, and Hinrich Schütze. 2012. Active learning for coreference resolution. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 508–512, Montréal, Canada. Association for… [find](https://scholar.google.com/scholar?q=Florian+Laws%2C+Florian+Heimerl%2C+and+Hinrich+Schu%CC%88tze%2E+2012%2E+Active+learning+for+coreference+resolution%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapter+of+the+Association+for+Comput) [our card](https://md.research.elamur.ai/free/papers/laws-2012-active-learning-coreference-resolution/)
13. Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017. End-to-end neural coreference resolution. ArXiv, abs/1707.07045. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Luheng+He%2C+Mike+Lewis%2C+and+Luke+S%2E+Zettlemoyer%2E+2017%2E+End%2Dto%2Dend+neural+coreference+resolution%2E+ArXiv%2C+abs%2F1707%2E07045%2E) [our card](https://md.research.elamur.ai/free/papers/lee-2017-end-end-neural-coreference-resolution/)
14. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1– 40. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+Conll%2D2012+shared+task%3A+Modeling+multilingual+unrestricted+coreference+in+ontonotes%2E+In+Joint+Conference+on+EM) [our card](https://md.research.elamur.ai/free/papers/pradhan-2012-conll-2012-shared-task-modeling/)
15. Mrinmaya Sachan, Eduard Hovy, and Eric P. Xing. 2015. An active learning approach to coreference resolution. In Proceedings of the 24th International Conference on Artificial Intelligence, IJ-CAI’15, pages 1312–1318. AAAI Press. [find](https://scholar.google.com/scholar?q=Mrinmaya+Sachan%2C+Eduard+Hovy%2C+and+Eric+P%2E+Xing%2E+2015%2E+An+active+learning+approach+to+coreference+resolution%2E+In+Proceedings+of+the+24th+International+Conference+on+Artificial+Intelligence%2C+IJ%2DCAI%E2%80%9915%2C+)