--- title: "Evaluation of named entity coreference — Agarwal 2019" description: "The paper argues that the usual coreference scores can mislead when the real goal is to collect every reference to a named entity, whether by name…" authors: ["Oshin Agarwal", "Sanjay Subramanian", "Ani Nenkova", "Dan Roth"] year: 2019 venue: "Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference" source: "https://aclanthology.org/W19-2801/" doi: "10.18653/v1/w19-2801" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/agarwal-2019-evaluation-named-entity-coreference/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/agarwal-2019-evaluation-named-entity-coreference/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2019](https://md.research.elamur.ai/free/papers/2019/page/2/) / Evaluation of named entity coreference # Evaluation of named entity coreference **Authors:** [Oshin Agarwal](https://md.research.elamur.ai/free/authors/oshin-agarwal/), [Sanjay Subramanian](https://md.research.elamur.ai/free/authors/sanjay-subramanian/), [Ani Nenkova](https://md.research.elamur.ai/free/authors/ani-nenkova/), [Dan Roth](https://md.research.elamur.ai/free/authors/dan-roth/) · **Year:** 2019 · **Venue:** Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-2801/) · [PDF](https://aclanthology.org/W19-2801.pdf) · [DOI 10.18653/v1/w19-2801](https://doi.org/10.18653/v1/w19-2801) · [OpenAlex](https://openalex.org/W2964120642) **Citations:** 8 citations (openalex-snapshot · 2026-10-14) · [5](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper argues that the usual coreference scores can mislead when the real goal is to collect every reference to a named entity, whether by name, pronoun or nominal, as search or information extraction requires. Metrics such as MUC, B3, CEAFe and LEA ignore mention types. They give full credit to chains of pronouns that never connect to a name, and they barely penalise systems that miss whole entities. The authors propose Named Entity Coreference (NEC) metrics. For each gold chain, the best-matching predicted chain must contain a variant of the entity's name. Its F1 is averaged across gold chains, and unmatched chains score zero. Separate diagnostics report missed entities, accuracy per mention type, and wrongly split or merged entities. The authors apply these metrics to five off-the-shelf systems on OntoNotes PER, ORG and GPE entities. Gains in the standard score do not always carry over to NEC. Lee et al. (2018) is nonetheless clearly the strongest system. The metrics expose errors that matter for downstream use. ## Key points - In a toy example, two outputs receive identical MUC, B3, CEAF and LEA scores. NEC F1 gives the output with named mentions 0.83 and the pronoun-only output 0. - Named PER, ORG and GPE entities form 38% of OntoNotes coreference clusters but account for 54% of mentions that need resolution. Evaluation used the newswire, broadcast news and magazine portions. - The systems evaluated were the Stanford deterministic, statistical and neural versions, plus the end-to-end models of Lee et al. (2017, 2018). System chains were kept only if a name was the mention head, which was checked with spaCy dependency parses and gold NER tags. - Moving from the rule-based to the statistical Stanford system raises Coref F1 for PER (0.50→0.56) while NEC F1 falls (0.55→0.46). - Lee et al. (2018) reaches a PER NEC F1 of 0.80. It misses 7.5% of PER, 15% of ORG and 8% of GPE chains, while other systems often miss 20–40%. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) Topics: [Named entity recognition](https://md.research.elamur.ai/free/topics/named-entity-recognition/) · [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/) ## Cited by Cited by 5 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 3 not yet. - [Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle*](https://md.research.elamur.ai/free/papers/cao-2021-gender-inclusive-coreference-resolution-analysis/) - [NARC – Norwegian Anaphora Resolution Corpus](https://md.research.elamur.ai/free/papers/hlum-2022-narc-norwegian-anaphora-resolution-corpus/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{agarwal2019evaluation, title = {{Evaluation of named entity coreference}}, author = {Agarwal, Oshin and Subramanian, Sanjay and Nenkova, Ani and Roth, Dan}, booktitle = {Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference}, year = {2019}, doi = {10.18653/v1/w19-2801}, url = {https://doi.org/10.18653/v1/w19-2801} } ``` - [← previous Evaluating Style Transfer for Text](https://md.research.elamur.ai/free/papers/mir-2019-evaluating-style-transfer-text/) - [next → Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://md.research.elamur.ai/free/papers/rajani-2019-explain-yourself-leveraging-language-models/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/papers/2019/page/2/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Amit Bagga and Breck Baldwin. 1998. Entity– based cross–document coreferencing using the vector space model. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, COLING-ACL ’98, Université de Montréal, Montréal, Qu… [find](https://scholar.google.com/scholar?q=Amit+Bagga+and+Breck+Baldwin%2E+1998%2E+Entity%E2%80%93+based+cross%E2%80%93document+coreferencing+using+the+vector+space+model%2E+In+36th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+17th+Internatio) 2. Chen Chen and Vincent Ng. 2013. Linguistically aware coreference evaluation metrics. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1366–1374. [find](https://scholar.google.com/scholar?q=Chen+Chen+and+Vincent+Ng%2E+2013%2E+Linguistically+aware+coreference+evaluation+metrics%2E+In+Proceedings+of+the+Sixth+International+Joint+Conference+on+Natural+Language+Processing%2C+pages+1366%E2%80%931374%2E) [our card](https://md.research.elamur.ai/free/papers/chen-2013-linguistically-aware-coreference-evaluation-metrics/)