--- title: "When Annotation Schemes Change Rules Help — Zeldes 2016" description: "Coreference systems trained on OntoNotes inherit its annotation choices." authors: ["Amir Zeldes", "Shuo Zhang"] year: 2016 venue: "Proceedings of the Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2016)" source: "https://aclanthology.org/W16-0713/" doi: "10.18653/v1/w16-0713" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/zeldes-2016-when-annotation-schemes-change-rules/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/zeldes-2016-when-annotation-schemes-change-rules/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2016](https://md.research.elamur.ai/free/papers/2016/page/3/) / When Annotation Schemes Change Rules Help: A Configurable Approach to Coreference Resolution beyond OntoNotes # When Annotation Schemes Change Rules Help: A Configurable Approach to Coreference Resolution beyond OntoNotes **Authors:** [Amir Zeldes](https://md.research.elamur.ai/free/authors/amir-zeldes/), [Shuo Zhang](https://md.research.elamur.ai/free/authors/shuo-zhang/) · **Year:** 2016 · **Venue:** Proceedings of the Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2016) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W16-0713/) · [PDF](https://aclanthology.org/W16-0713.pdf) · [DOI 10.18653/v1/w16-0713](https://doi.org/10.18653/v1/w16-0713) · [OpenAlex](https://openalex.org/W2513312051) **Citations:** 20 citations (openalex-snapshot · 2026-10-14) · [11](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Coreference systems trained on OntoNotes inherit its annotation choices. Phenomena such as cataphora, predicative markables, generic indefinite anaphors, common-noun compound modifiers, metonymy and nested (i-within-i) mentions are therefore left out. Corpora that do annotate these are too small to train on. Zeldes and Zhang address this with xrenner, an open-source Python system for entity recognition and coreference resolution. It takes dependency parses as input and keeps almost all of its behaviour in editable configuration and lexicon files, so the same engine can follow different annotation schemes without retraining. Mention borders are defined by which dependency functions stop a chain. Fine-grained entity typing draws on lexicons built from DBPedia, WordNet, Freebase, PPDB and the OntoNotes and GUM training data. The authors tested two configurations: one on Wall Street Journal text from OntoNotes and one on GUM wiki-based data, which includes the non-standard phenomena. xrenner compared favourably with Stanford's rule-based dcoref and the stochastic Berkeley system, mainly because its recall was higher. The study shows that configurable rules can be a practical choice when target-scheme training data is scarce. The main cost is the need for rich lexical resources. ## Key points - Input is CoNLL-format dependency data from the Stanford Parser with Collapsed Typed Dependencies. Mention borders are set by a regex of non-linking functions, so adding `rcmod` makes the system follow GUM's exclusion of relative clauses. - The main configuration has more than 70 settings, covering definiteness, cardinality, agreement classes, apposition wrapping, coordination handling and postprocessing deletions. - For English, the model recognises 11 top-level entity types, which are further divided into 54 subclasses. These are taken from lexicon files rather than hard-coded. - On WSJ, xrenner had the highest recall and the highest overall F1, with precision falling between dcoref and Berkeley. Safeguards include antonym lists, cardinality mismatch checks and subclass-specific is-a matching. - Limitations: domain-specific lexical data is needed, and the rule order was set by linguistic intuition. Planned models include German (TüBa-D/Z scheme) and Coptic. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/) ## Cited by Cited by 11 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 5 not yet. - [Towards Named-Entity and Coreference Annotation of the Hebrew Bible](https://md.research.elamur.ai/free/papers/coref-2024-swanson-named-entity-coreference-annotation-hebrew/) - [Annotating Mentions Alone Enables Efficient Domain Adaptation for Coreference Resolution](https://md.research.elamur.ai/free/papers/gandhi-2023-annotating-mentions-alone-enables-efficient/) - [AMALGUM – A Free, Balanced, Multilayer English Web Corpus](https://md.research.elamur.ai/free/papers/lingbulk-2020-gessler-amalgum-free-balanced-multilayer-english/) - [Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective](https://md.research.elamur.ai/free/papers/porada-2024-challenges-evaluating-generalization-coreference-resolution/) - [A Predictive Model for Notional Anaphora in English](https://md.research.elamur.ai/free/papers/zeldes-2018-predictive-model-notional-anaphora-english/) - [Opinion Piece: Can we Fix the Scope for Coreference? Problems and Solutions for Benchmarks beyond OntoNotes](https://md.research.elamur.ai/free/papers/zeldes-2022-opinion-piece-can-we-fix/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{zeldes2016when, title = {{When Annotation Schemes Change Rules Help: A Configurable Approach to Coreference Resolution beyond OntoNotes}}, author = {Zeldes, Amir and Zhang, Shuo}, booktitle = {Proceedings of the Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2016)}, year = {2016}, doi = {10.18653/v1/w16-0713}, url = {https://doi.org/10.18653/v1/w16-0713} } ``` - [← previous WaveNet: A Generative Model for Raw Audio](https://md.research.elamur.ai/free/papers/oord-2016-wavenet-generative-model-raw-audio/) - [next → Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric](https://md.research.elamur.ai/free/papers/moosavi-2016-which-coreference-evaluation-metric-do/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/papers/2016/page/3/) ## References 20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Emilia Apostolova, Noriko Tomuro, Pattanasak Mongkolwat and Dina Demner-Fushman. 2012. Domain Adaptation of Coreference Resolution for Radiology Reports. In Proceedings of the 2012 Workshop on Biomedical Natural Language Processing (BioNLP 2012). Montreal, 118–121. [find](https://scholar.google.com/scholar?q=Emilia+Apostolova%2C+Noriko+Tomuro%2C+Pattanasak+Mongkolwat+and+Dina+Demner%2DFushman%2E+2012%2E+Domain+Adaptation+of+Coreference+Resolution+for+Radiology+Reports%2E+In+Proceedings+of+the+2012+Workshop+on+Biomedi) [our card](https://md.research.elamur.ai/free/papers/apostolova-2012-domain-adaptation-coreference-resolution-radiology/) 2. Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann and Zachary Ives. 2007. DBpedia: A Nucleus for a Web of Open Data. In 6th International Semantic Web Conference. Busan, South Korea, 11– 15. [find](https://scholar.google.com/scholar?q=S%C3%B6ren+Auer%2C+Christian+Bizer%2C+Georgi+Kobilarov%2C+Jens+Lehmann+and+Zachary+Ives%2E+2007%2E+DBpedia%3A+A+Nucleus+for+a+Web+of+Open+Data%2E+In+6th+International+Semantic+Web+Conference%2E+Busan%2C+South+Korea%2C+11%E2%80%93+15%2E) 3. BBN Technologies. 2007. Co-reference Guidelines for English OntoNotes. Version 6.0. Available online at: [http://www.ldc.upenn.edu/Catalog/docs/LDC200](http://www.ldc.upenn.edu/Catalog/docs/LDC200) 7T21/coreference/english-coref.pdf . [↗](http://www.ldc.upenn.edu/Catalog/docs/LDC200) 4. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge and Jamie Taylor. 2008. Freebase: A Collaboratively Created Graph Database for Structuring Human Knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. Vancouver, 1247–1250. [find](https://scholar.google.com/scholar?q=Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge+and+Jamie+Taylor%2E+2008%2E+Freebase%3A+A+Collaboratively+Created+Graph+Database+for+Structuring+Human+Knowledge%2E+In+Proceedings+of+the+2008+ACM+SIG) 5. Danqi Chen and Christopher D. Manning. 2014. A Fast and Accurate Dependency Parser using Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar, 740–750. [find](https://scholar.google.com/scholar?q=Danqi+Chen+and+Christopher+D%2E+Manning%2E+2014%2E+A+Fast+and+Accurate+Dependency+Parser+using+Neural+Networks%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMN) [our card](https://md.research.elamur.ai/free/papers/missing-2014-chen-fast-accurate-dependency-parser-neural/) 6. Greg Durrett and Dan Klein. 2013. Easy Victories and Uphill Battles in Coreference Resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2013). Seattle, 1971–1982. [find](https://scholar.google.com/scholar?q=Greg+Durrett+and+Dan+Klein%2E+2013%2E+Easy+Victories+and+Uphill+Battles+in+Coreference+Resolution%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP+2013%29%2E+Seattl) [our card](https://md.research.elamur.ai/free/papers/durrett-2013-easy-victories-uphill-battles-coreference/) 7. Greg Durrett and Dan Klein. 2014. A Joint Model for Entity Analysis: Coreference, Typing, and Linking. Transactions of the Association for Computational Linguistics 2:477–490. Christiane Fellbaum (ed.). 1998. [find](https://scholar.google.com/scholar?q=Greg+Durrett+and+Dan+Klein%2E+2014%2E+A+Joint+Model+for+Entity+Analysis%3A+Coreference%2C+Typing%2C+and+Linking%2E+Transactions+of+the+Association+for+Computational+Linguistics+2%3A477%E2%80%93490%2E+Christiane+Fellbaum+%28ed%2E) [our card](https://md.research.elamur.ai/free/papers/durrett-2014-joint-model-entity-analysis-coreference/) 8. WordNet: An Electronic Lexical Database. MIT Press: Cambridge, MA. Barbara A. Fox. 1993. [find](https://scholar.google.com/scholar?q=WordNet%3A+An+Electronic+Lexical+Database%2E+MIT+Press%3A+Cambridge%2C+MA%2E+Barbara+A%2E+Fox%2E+1993%2E) 9. Discourse Structure and Anaphora: Written and Conversational English. Cambridge: Cambridge University Press. Juri Ganitkevitch, Benjamin Van Durme and Chris Call-ison-Burch. 2013. PPDB: The Paraphrase Database. In Proceedings of NAACL-HLT. Atlanta, GA, 758– 764. [find](https://scholar.google.com/scholar?q=Discourse+Structure+and+Anaphora%3A+Written+and+Conversational+English%2E+Cambridge%3A+Cambridge+University+Press%2E+Juri+Ganitkevitch%2C+Benjamin+Van+Durme+and+Chris+Call%2Dison%2DBurch%2E+2013%2E+PPDB%3A+The+Paraphrase) [our card](https://md.research.elamur.ai/free/papers/ganitkevitch-2013-ppdb-paraphrase-database/) 10. Aria Haghighi and Dan Klein. 2009. Simple Coreference Resolution with Rich Syntactic and Semantic Features. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP 2009). Singapore, 1152–1161. [find](https://scholar.google.com/scholar?q=Aria+Haghighi+and+Dan+Klein%2E+2009%2E+Simple+Coreference+Resolution+with+Rich+Syntactic+and+Semantic+Features%2E+In+Proceedings+of+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP+2009%29%2E+Singapore%2C+) [our card](https://md.research.elamur.ai/free/papers/haghighi-2009-simple-coreference-resolution-rich-syntactic/) 11. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw and Ralph Weischedel. 2006. OntoNotes: The 90% Solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers. New York, 57–60. [find](https://scholar.google.com/scholar?q=Eduard+Hovy%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Lance+Ramshaw+and+Ralph+Weischedel%2E+2006%2E+OntoNotes%3A+The+90%25+Solution%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+NAACL%2C+Companion+V) 12. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu and Dan Jurafsky. 2013. Deterministic Coreference Resolution Based on Entity-Centric, Precision-Ranked Rules. Computational Linguistics 39(4):885–916. [find](https://scholar.google.com/scholar?q=Heeyoung+Lee%2C+Angel+Chang%2C+Yves+Peirsman%2C+Nathanael+Chambers%2C+Mihai+Surdeanu+and+Dan+Jurafsky%2E+2013%2E+Deterministic+Coreference+Resolution+Based+on+Entity%2DCentric%2C+Precision%2DRanked+Rules%2E+Computational) [our card](https://md.research.elamur.ai/free/papers/lee-2013-deterministic-coreference-resolution-based-entity/) 13. Marie-Catherine de Marneffe and Christopher D. Manning. 2013. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+de+Marneffe+and+Christopher+D%2E+Manning%2E+2013%2E) 14. Stanford Typed Dependencies Manual. Stanford University, Technical Report. Massimo Poesio and Ron Artstein. 2008. Anaphoric Annotation in the ARRAU Corpus. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis and Daniel Tapias (eds.), Proceedings of th… [find](https://scholar.google.com/scholar?q=Stanford+Typed+Dependencies+Manual%2E+Stanford+University%2C+Technical+Report%2E+Massimo+Poesio+and+Ron+Artstein%2E+2008%2E+Anaphoric+Annotation+in+the+ARRAU+Corpus%2E+In+Nicoletta+Calzolari%2C+Khalid+Choukri%2C+Bent) [our card](https://md.research.elamur.ai/free/papers/poesio-2008-anaphoric-annotation-arrau-corpus/) 15. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng and Michael Strube. 2014. Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. Baltimore, MD, 30–35. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Xiaoqiang+Luo%2C+Marta+Recasens%2C+Eduard+Hovy%2C+Vincent+Ng+and+Michael+Strube%2E+2014%2E+Scoring+Coreference+Partitions+of+Predicted+Mentions%3A+A+Reference+Implementation%2E+In+Proceedings+of+the) [our card](https://md.research.elamur.ai/free/papers/pradhan-2014-scoring-coreference-partitions-predicted-mentions/) 16. Altaf Rahman and Vincent Ng. 2011. Narrowing the Modeling Gap: A Cluster-Ranking Approach to Coreference Resolution. Journal of Artificial Intelligence Research 40(1):469–521. [find](https://scholar.google.com/scholar?q=Altaf+Rahman+and+Vincent+Ng%2E+2011%2E+Narrowing+the+Modeling+Gap%3A+A+Cluster%2DRanking+Approach+to+Coreference+Resolution%2E+Journal+of+Artificial+Intelligence+Research+40%281%29%3A469%E2%80%93521%2E) 17. Marta Recasens and Eduard Hovy. 2010. Coreference Resolution across Corpora: Languages, Coding Schemes, and Preprocessing Information. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Uppsala, 1423– 1432. [find](https://scholar.google.com/scholar?q=Marta+Recasens+and+Eduard+Hovy%2E+2010%2E+Coreference+Resolution+across+Corpora%3A+Languages%2C+Coding+Schemes%2C+and+Preprocessing+Information%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/papers/recasens-2010-coreference-resolution-across-corpora-languages/) 18. Utpal Kumar Sikdar, Asif Ekbal, Sriparna Saha, Olga Uryupina and Massimo Poesio. 2013. Anaphora Resolution for Bengali: An Experiment with Domain Adaptation. Computación y Sistemas 17(2):137–146. [find](https://scholar.google.com/scholar?q=Utpal+Kumar+Sikdar%2C+Asif+Ekbal%2C+Sriparna+Saha%2C+Olga+Uryupina+and+Massimo+Poesio%2E+2013%2E+Anaphora+Resolution+for+Bengali%3A+An+Experiment+with+Domain+Adaptation%2E+Computaci%C3%B3n+y+Sistemas+17%282%29%3A137%E2%80%93146%2E) 19. Heike Telljohann, Erhard W. Hinrichs, Sandra Kübler, Heike Zinsmeister and Kathrin Beck. 2015. Stylebook for the Tübingen Treebank of Written German (TüBa-D/Z). Seminar für Sprachwissenschaft, Universität Tübingen, Technical Report. [find](https://scholar.google.com/scholar?q=Heike+Telljohann%2C+Erhard+W%2E+Hinrichs%2C+Sandra+K%C3%BCbler%2C+Heike+Zinsmeister+and+Kathrin+Beck%2E+2015%2E+Stylebook+for+the+T%C3%BCbingen+Treebank+of+Written+German+%28T%C3%BCBa%2DD%2FZ%29%2E+Seminar+f%C3%BCr+Sprachwissenschaft%2C+Univers) 20. Amir Zeldes. 2016. The GUM Corpus: Creating Multilayer Resources in the Classroom. Language Resources and Evaluation. Available online at: http://dx.doi.org/10.1007/s10579-016-9343-x . Shanheng Zhao and Hwee Tou Ng. 2014. Domain Adaptation with Active Learning for Coreference Resolution. In Proceedi… [↗](http://dx.doi.org/10.1007/s10579-016-9343-x)