--- title: "Coreference Resolution with Reconcile — Stoyanov 2010" description: "Results for noun phrase coreference resolution are hard to compare across systems." authors: ["Veselin Stoyanov", "Claire Cardie", "Nathan Gilbert", "Ellen Riloff", "David Buttler", "David Hysom"] year: 2010 venue: "Proceedings of the ACL 2010 Conference Short Papers" source: "https://aclanthology.org/P10-2029/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/stoyanov-2010-coreference-resolution-reconcile/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/stoyanov-2010-coreference-resolution-reconcile/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2010](https://md.research.elamur.ai/free/papers/2010/) / Coreference Resolution with Reconcile # Coreference Resolution with Reconcile **Authors:** [Veselin Stoyanov](https://md.research.elamur.ai/free/authors/veselin-stoyanov/), [Claire Cardie](https://md.research.elamur.ai/free/authors/claire-cardie/), [Nathan Gilbert](https://md.research.elamur.ai/free/authors/nathan-gilbert/), [Ellen Riloff](https://md.research.elamur.ai/free/authors/ellen-riloff/), David Buttler, David Hysom · **Year:** 2010 · **Venue:** Proceedings of the ACL 2010 Conference Short Papers · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/P10-2029/) · [PDF](https://aclanthology.org/P10-2029.pdf) · [OpenAlex](https://openalex.org/W2124214517) **Citations:** 84 citations (openalex-snapshot · 2026-10-14) · [30](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Results for noun phrase coreference resolution are hard to compare across systems. Building a complete pipeline is costly, so many studies replace subtasks such as mention detection, anaphoricity determination, named entity recognition or semantic class identification with gold-standard annotations. Benchmarks also differ in task definitions, formats and scoring metrics. Together these factors make published numbers inconsistent and often unrealistic. The authors present Reconcile, a modular Java platform that follows the standard supervised, pairwise design of learning-based resolvers in the tradition of Soon et al. (2001). Its five stages are preprocessing, feature generation, classification, clustering and scoring. Every stage can be swapped or extended, and the platform handles the format and scoring quirks of the MUC and ACE benchmarks. As a demonstration, they build one configuration, Reconcile 2010, and test it on six data sets with the MUC and B3 metrics. Its scores are on par with comparable published supervised systems. This gives researchers a realistic, reproducible baseline and gives developers a ready-to-use resolver. ## Key points - **Architecture and components:** - Preprocessing wraps external tools: OpenNLP, Stanford parser/NER, Berkeley parser and UIUC. - In-house NP detectors follow either MUC or ACE definitions and recover about 95% of gold NPs. - There are over 80 pairwise features. - Learners come from Weka, libSVM and SVM light. - Clustering options are Single-link, Best-First and Most Recent First. - Scorers are MUC, B3 and CEAF. - The code base exceeds 38,000 lines of Java. - **Data:** MUC-6, MUC-7 and four ACE sets (ACE-2, ACE03, ACE04, ACE05, newswire only). Standard splits are used where they exist; otherwise the data is split randomly 70/30. - **Reconcile 2010 setup:** - OpenNLP for sentence splitting, tokenization and POS tagging. - Berkeley parser and Stanford NER. - 60 hand-selected features drawn from Soon et al., Ng and Cardie, and Bengtson and Roth. - Averaged Perceptron classifier. - Single-link clustering with a threshold tuned by cross-validation. - **Results:** - MUC scores: 68.50 (MUC6), 62.80 (MUC7) and 62.03–67.87 on the ACE sets. - B3 scores: 70.88 (MUC6), 65.86 (MUC7) and 73.71–79.39 on the ACE sets. - B3 is higher than MUC on every data set. - **Comparison and availability:** - On MUC6, Reconcile 2010 beats Soon et al. (62.6) but trails Ng and Cardie (70.4) and Yang et al. (71.3). - On MUC7, it exceeds Soon et al. (60.4) and Yang et al. (60.2) and is slightly below Ng and Cardie (63.4). - The release is available at cs.utah.edu/nlp/reconcile. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) Topics: [Named entity recognition](https://md.research.elamur.ai/free/topics/named-entity-recognition/) · [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/) ## Cited by Cited by 30 papers of our library cite it (of the 140673 whose bibliographies we read); 24 of them have a page here, 6 not yet. - [A Unified Event Coreference Resolution by Integrating Multiple Resolvers](https://md.research.elamur.ai/free/papers/chen-2011-unified-event-coreference-resolution-integrating/) - [Joint Inference and Disambiguation of Implicit Sentiments via Implicature Constraints](https://md.research.elamur.ai/free/papers/deng-2014-joint-inference-disambiguation-implicit-sentiments/) - [Easy Victories and Uphill Battles in Coreference Resolution](https://md.research.elamur.ai/free/papers/durrett-2013-easy-victories-uphill-battles-coreference/) - [A Joint Model for Entity Analysis: Coreference, Typing, and Linking](https://md.research.elamur.ai/free/papers/durrett-2014-joint-model-entity-analysis-coreference/) - [Latent Trees for Coreference Resolution](https://md.research.elamur.ai/free/papers/fernandes-2014-latent-trees-coreference-resolution/) - [Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers](https://md.research.elamur.ai/free/papers/guha-2015-removing-training-wheels-coreference-dataset/) - [The Taming of Reconcile as a Biomedical Coreference Resolver](https://md.research.elamur.ai/free/papers/kim-2011-taming-reconcile-as-biomedical-coreference/) - [Bootstrapping coreference resolution using word associations](https://md.research.elamur.ai/free/papers/kobdani-2011-bootstrapping-coreference-resolution-word-associations/) - [Creating a Coreference Resolution System for Polish](https://md.research.elamur.ai/free/papers/kopec-2012-creating-coreference-resolution-system-polish/) - [Error-Driven Analysis of Challenges in Coreference Resolution](https://md.research.elamur.ai/free/papers/kummerfeld-2013-error-driven-analysis-challenges-coreference/) - [Coreference Resolution for the Biomedical Domain: A Survey](https://md.research.elamur.ai/free/papers/lu-2021-coreference-resolution-biomedical-domain-survey/) - [Prune-and-Score: Learning for Greedy Coreference Resolution](https://md.research.elamur.ai/free/papers/ma-2014-prune-score-learning-greedy-coreference/) - [Transferring Coreference Resolvers with Posterior Regularization](https://md.research.elamur.ai/free/papers/martins-2015-transferring-coreference-resolvers-posterior-regularization/) - [Supervised Noun Phrase Coreference Research: The First Fifteen Years](https://md.research.elamur.ai/free/papers/ng-2010-supervised-noun-phrase-coreference-research/) - [Overview of BioNLP 2011 Protein Coreference Shared Task](https://md.research.elamur.ai/free/papers/nguyen-2011-overview-bionlp-2011-protein-coreference/) - [Solving Hard Coreference Problems](https://md.research.elamur.ai/free/papers/peng-2015-solving-hard-coreference-problems/) - [Event Detection and Co-reference with Minimal Supervision](https://md.research.elamur.ai/free/papers/peng-2016-event-detection-co-reference-minimal/) - [Coreference Resolution with World Knowledge](https://md.research.elamur.ai/free/papers/rahman-2011-coreference-resolution-world-knowledge/) - [Translation-Based Projection for Multilingual Coreference Resolution](https://md.research.elamur.ai/free/papers/rahman-2012-translation-based-projection-multilingual-coreference/) - [A Fully Coreference-annotated Corpus of Scholarly Papers from the ACL Anthology](https://md.research.elamur.ai/free/papers/schafer-2012-fully-coreference-annotated-corpus-scholarly/) - [Easy-first Coreference Resolution](https://md.research.elamur.ai/free/papers/stoyanov-2012-easy-first-coreference-resolution/) - [Limited memory incremental coreference resolution](https://md.research.elamur.ai/free/papers/webster-2014-limited-memory-incremental-coreference-resolution/) - [Domain Adaptation for Coreference Resolution: An Adaptive Ensemble Approach](https://md.research.elamur.ai/free/papers/yang-2012-domain-adaptation-coreference-resolution-adaptive/) - [Dynamic Knowledge-Base Alignment for Coreference Resolution](https://md.research.elamur.ai/free/papers/zheng-2013-dynamic-knowledge-base-alignment-coreference/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{stoyanov2010coreference, title = {{Coreference Resolution with Reconcile}}, author = {Stoyanov, Veselin and Cardie, Claire and Gilbert, Nathan and Riloff, Ellen and Buttler, David and Hysom, David}, booktitle = {Proceedings of the ACL 2010 Conference Short Papers}, year = {2010}, url = {https://aclanthology.org/P10-2029/} } ``` - [← previous Coreference Resolution in a Modular, Entity-Centered Model](https://md.research.elamur.ai/free/papers/haghighi-2010-coreference-resolution-modular-entity-centered/) - [next → Creating a Coreference Resolution System for Italian](https://md.research.elamur.ai/free/papers/poesio-2010-creating-coreference-resolution-system-italian/) - [↑ back to papers of 2010](https://md.research.elamur.ai/free/papers/2010/) ## References 27 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. A. Bagga and B. Baldwin. 1998. Algorithms for scoring coreference chains. In Linguistic Coreference Workshop at the Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=A%2E+Bagga+and+B%2E+Baldwin%2E+1998%2E+Algorithms+for+scoring+coreference+chains%2E+In+Linguistic+Coreference+Workshop+at+the+Language+Resources+and+Evaluation+Conference%2E) [our card](https://md.research.elamur.ai/free/papers/bagga-1998-algorithms-scoring-coreference-chains/) 2. Baldridge, J. 2005. The OpenNLP project. [http://opennlp.sourceforge.net/](http://opennlp.sourceforge.net/). [↗](http://opennlp.sourceforge.net/) 3. E. Bengtson and D. Roth. 2008. Understanding the value of features for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=E%2E+Bengtson+and+D%2E+Roth%2E+2008%2E+Understanding+the+value+of+features+for+coreference+resolution%2E+In+Proceedings+of+the+2008+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E) [our card](https://md.research.elamur.ai/free/papers/bengtson-2008-understanding-value-features-coreference-resolution/) 4. CC Group. 2009. Sentence Segmentation Tool. [http://l2r.cs.uiuc.edu/](http://l2r.cs.uiuc.edu/) cogcomp/atool.php?tkey=SS. [↗](http://l2r.cs.uiuc.edu/) 5. C. Chang and C. Lin. 2001. LIBSVM: a Library for Support Vector Machines. Available at [http://www.csie.ntu.edu.tw/cjlin/libsvm](http://www.csie.ntu.edu.tw/cjlin/libsvm). [↗](http://www.csie.ntu.edu.tw/cjlin/libsvm) 6. J. Finkel, T. Grenager, and C. Manning. 2005. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL. [find](https://scholar.google.com/scholar?q=J%2E+Finkel%2C+T%2E+Grenager%2C+and+C%2E+Manning%2E+2005%2E+Incorporating+Non%2Dlocal+Information+into+Information+Extraction+Systems+by+Gibbs+Sampling%2E+In+Proceedings+of+the+21st+International+Conference+on+Computat) 7. T. Finley and T. Joachims. 2005. Supervised clustering with support vector machines. In Proceedings of the Twenty-second International Conference on Machine Learning (ICML 2005). [find](https://scholar.google.com/scholar?q=T%2E+Finley+and+T%2E+Joachims%2E+2005%2E+Supervised+clustering+with+support+vector+machines%2E+In+Proceedings+of+the+Twenty%2Dsecond+International+Conference+on+Machine+Learning+%28ICML+2005%29%2E) 8. A. Haghighi and D. Klein. 2007. Unsupervised Coreference Resolution in a Nonparametric Bayesian Model. In Proceedings of the 45th Annual Meeting of the ACL. [find](https://scholar.google.com/scholar?q=A%2E+Haghighi+and+D%2E+Klein%2E+2007%2E+Unsupervised+Coreference+Resolution+in+a+Nonparametric+Bayesian+Model%2E+In+Proceedings+of+the+45th+Annual+Meeting+of+the+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/haghighi-2007-unsupervised-coreference-resolution-nonparametric-bayesian/) 9. T. Joachims. 2002. SVM Light , [http://svmlight.joachims.org](http://svmlight.joachims.org). [↗](http://svmlight.joachims.org) 10. D. Klein and C. Manning. 2003. Fast Exact Inference with a Factored Model for Natural Language Parsing. In Advances in Neural Information Processing (NIPS 2003). [find](https://scholar.google.com/scholar?q=D%2E+Klein+and+C%2E+Manning%2E+2003%2E+Fast+Exact+Inference+with+a+Factored+Model+for+Natural+Language+Parsing%2E+In+Advances+in+Neural+Information+Processing+%28NIPS+2003%29%2E) 11. Data sets ACE-2 ACE03 65.99 67.87 78.29 79.39 – – – – – – MUC7 62.80 65.86 60.4 63.4 60.2 ACE04 62.03 76.50 – – – ACE05 67.41 73.71 – – – S. Lappin and H. Leass. 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4):535–561. [find](https://scholar.google.com/scholar?q=Data+sets+ACE%2D2+ACE03+65%2E99+67%2E87+78%2E29+79%2E39+%E2%80%93+%E2%80%93+%E2%80%93+%E2%80%93+%E2%80%93+%E2%80%93+MUC7+62%2E80+65%2E86+60%2E4+63%2E4+60%2E2+ACE04+62%2E03+76%2E50+%E2%80%93+%E2%80%93+%E2%80%93+ACE05+67%2E41+73%2E71+%E2%80%93+%E2%80%93+%E2%80%93+S%2E+Lappin+and+H%2E+Leass%2E+1994%2E+An+algorithm+for+pronominal+anap) [our card](https://md.research.elamur.ai/free/papers/lappin-1994-algorithm-pronominal-anaphora-resolution/) 12. X. Luo, A. Ittycheriah, H. Jing, N. Kambhatla, and S. Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proceedings of the 42nd Annual Meeting of the ACL. [find](https://scholar.google.com/scholar?q=X%2E+Luo%2C+A%2E+Ittycheriah%2C+H%2E+Jing%2C+N%2E+Kambhatla%2C+and+S%2E+Roukos%2E+2004%2E+A+mention%2Dsynchronous+coreference+resolution+algorithm+based+on+the+bell+tree%2E+In+Proceedings+of+the+42nd+Annual+Meeting+of+the+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/luo-2004-mention-synchronous-coreference-resolution-algorithm/) 13. X. Luo. 2005. On Coreference Resolution Performance Metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP). [find](https://scholar.google.com/scholar?q=X%2E+Luo%2E+2005%2E+On+Coreference+Resolution+Performance+Metrics%2E+In+Proceedings+of+Human+Language+Technology+Conference+and+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28HLT%2FEMNLP%29%2E) [our card](https://md.research.elamur.ai/free/papers/luo-2005-coreference-resolution-performance-metrics/) 14. A. McCallum and B. Wellner. 2004. Conditional Models of Identity Uncertainty with Application to Noun Coreference. In Advances in Neural Information Processing (NIPS 2004). [find](https://scholar.google.com/scholar?q=A%2E+McCallum+and+B%2E+Wellner%2E+2004%2E+Conditional+Models+of+Identity+Uncertainty+with+Application+to+Noun+Coreference%2E+In+Advances+in+Neural+Information+Processing+%28NIPS+2004%29%2E) 15. MUC-6. 1995. Coreference Task Definition. In Proceedings of the Sixth Message Understanding Conference (MUC-6). [find](https://scholar.google.com/scholar?q=MUC%2D6%2E+1995%2E+Coreference+Task+Definition%2E+In+Proceedings+of+the+Sixth+Message+Understanding+Conference+%28MUC%2D6%29%2E) 16. MUC-7. 1997. Coreference Task Definition. In Proceedings of the Seventh Message Understanding Conference (MUC-7). [find](https://scholar.google.com/scholar?q=MUC%2D7%2E+1997%2E+Coreference+Task+Definition%2E+In+Proceedings+of+the+Seventh+Message+Understanding+Conference+%28MUC%2D7%29%2E) 17. V. Ng and C. Cardie. 2002. Improving Machine Learning Approaches to Coreference Resolution. In Proceedings of the 40th Annual Meeting of the ACL. [find](https://scholar.google.com/scholar?q=V%2E+Ng+and+C%2E+Cardie%2E+2002%2E+Improving+Machine+Learning+Approaches+to+Coreference+Resolution%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/ng-2002-improving-machine-learning-approaches-coreference/) 18. NIST. 2004. The ACE Evaluation Plan. NIST. [find](https://scholar.google.com/scholar?q=NIST%2E+2004%2E+The+ACE+Evaluation+Plan%2E+NIST%2E) 19. S. Petrov and D. Klein. 2007. Improved Inference for Unlexicalized Parsing. In Proceedings of the Joint Meeting of the Human Language Technology Conference and the North American Chapter of the Association for Computational Linguistics (HLT-NAACL 2007). [find](https://scholar.google.com/scholar?q=S%2E+Petrov+and+D%2E+Klein%2E+2007%2E+Improved+Inference+for+Unlexicalized+Parsing%2E+In+Proceedings+of+the+Joint+Meeting+of+the+Human+Language+Technology+Conference+and+the+North+American+Chapter+of+the+Associ) 20. M. Poesio and M. Kabadjov. 2004. A general-purpose, off-the-shelf anaphora resolution module: implementation and preliminary evaluation. In Proceedings of the Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=M%2E+Poesio+and+M%2E+Kabadjov%2E+2004%2E+A+general%2Dpurpose%2C+off%2Dthe%2Dshelf+anaphora+resolution+module%3A+implementation+and+preliminary+evaluation%2E+In+Proceedings+of+the+Language+Resources+and+Evaluation+Confere) [our card](https://md.research.elamur.ai/free/papers/poesio-2004-general-purpose-off-shelf-anaphora/) 21. L. Qiu, M.-Y. Kan, and T.-S. Chua. 2004. A public reference implementation of the rap anaphora resolution algorithm. In Proceedings of the Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=L%2E+Qiu%2C+M%2E%2DY%2E+Kan%2C+and+T%2E%2DS%2E+Chua%2E+2004%2E+A+public+reference+implementation+of+the+rap+anaphora+resolution+algorithm%2E+In+Proceedings+of+the+Language+Resources+and+Evaluation+Conference%2E) [our card](https://md.research.elamur.ai/free/papers/qiu-2004-public-reference-implementation-rap-anaphora/) 22. W. Soon, H. Ng, and D. Lim. 2001. A Machine Learning Approach to Coreference of Noun Phrases. Computational Linguistics, 27(4):521–541. [find](https://scholar.google.com/scholar?q=W%2E+Soon%2C+H%2E+Ng%2C+and+D%2E+Lim%2E+2001%2E+A+Machine+Learning+Approach+to+Coreference+of+Noun+Phrases%2E+Computational+Linguistics%2C+27%284%29%3A521%E2%80%93541%2E) 23. V. Stoyanov, N. Gilbert, C. Cardie, and E. Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL/IJCNLP. [find](https://scholar.google.com/scholar?q=V%2E+Stoyanov%2C+N%2E+Gilbert%2C+C%2E+Cardie%2C+and+E%2E+Riloff%2E+2009%2E+Conundrums+in+noun+phrase+coreference+resolution%3A+Making+sense+of+the+state%2Dof%2Dthe%2Dart%2E+In+Proceedings+of+ACL%2FIJCNLP%2E) [our card](https://md.research.elamur.ai/free/papers/stoyanov-2009-conundrums-noun-phrase-coreference-resolution/) 24. V. Stoyanov, C. Cardie, N. Gilbert, E. Riloff, D. Buttler, and D. Hysom. 2010. Reconcile: A coreference resolution research platform. Technical report, Cornell University. [find](https://scholar.google.com/scholar?q=V%2E+Stoyanov%2C+C%2E+Cardie%2C+N%2E+Gilbert%2C+E%2E+Riloff%2C+D%2E+Buttler%2C+and+D%2E+Hysom%2E+2010%2E+Reconcile%3A+A+coreference+resolution+research+platform%2E+Technical+report%2C+Cornell+University%2E) 25. Y. Versley, S. Ponzetto, M. Poesio, V. Eidelman, A. Jern, J. Smith, X. Yang, and A. Moschitti. 2008. BART: A modular toolkit for coreference resolution. In Proceedings of the Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Y%2E+Versley%2C+S%2E+Ponzetto%2C+M%2E+Poesio%2C+V%2E+Eidelman%2C+A%2E+Jern%2C+J%2E+Smith%2C+X%2E+Yang%2C+and+A%2E+Moschitti%2E+2008%2E+BART%3A+A+modular+toolkit+for+coreference+resolution%2E+In+Proceedings+of+the+Language+Resources+and+Ev) [our card](https://md.research.elamur.ai/free/papers/versley-2008-bart-modular-toolkit-coreference-resolution/) 26. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, and L. Hirschman. 1995. A Model-Theoretic Coreference Scoring Theme. In Proceedings of the Sixth Message Understanding Conference (MUC-6). [find](https://scholar.google.com/scholar?q=M%2E+Vilain%2C+J%2E+Burger%2C+J%2E+Aberdeen%2C+D%2E+Connolly%2C+and+L%2E+Hirschman%2E+1995%2E+A+Model%2DTheoretic+Coreference+Scoring+Theme%2E+In+Proceedings+of+the+Sixth+Message+Understanding+Conference+%28MUC%2D6%29%2E) 27. X. Yang, G. Zhou, J. Su, and C. Tan. 2003. Coreference resolution using competition learning approach. In Proceedings of the 41st Annual Meeting of the ACL. [find](https://scholar.google.com/scholar?q=X%2E+Yang%2C+G%2E+Zhou%2C+J%2E+Su%2C+and+C%2E+Tan%2E+2003%2E+Coreference+resolution+using+competition+learning+approach%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/yang-2003-coreference-resolution-competition-learning-approach/)