--- title: "Model-based Annotation of Coreference — Aralikatte 2020" description: "Standard coreference annotation asks people to connect pronouns with earlier text spans." authors: ["Rahul Aralikatte", "Anders Søgaard"] year: 2020 venue: "Proceedings of the Twelfth Language Resources and Evaluation Conference" source: "https://aclanthology.org/2020.lrec-1.9/" arxiv: "1906.10724" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/aralikatte-2020-model-based-annotation-coreference/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aralikatte-2020-model-based-annotation-coreference/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Model-based Annotation of Coreference # Model-based Annotation of Coreference **Authors:** [Rahul Aralikatte](https://md.research.elamur.ai/free/en/authors/rahul-aralikatte/), [Anders Søgaard](https://md.research.elamur.ai/free/en/authors/anders-sogaard/) · **Year:** 2020 · **Venue:** Proceedings of the Twelfth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.lrec-1.9/) · [PDF](https://aclanthology.org/2020.lrec-1.9.pdf) · [arXiv 1906.10724](https://arxiv.org/abs/1906.10724) · [OpenAlex](https://openalex.org/W2954843021) **Citations:** 1 citations (openalex-snapshot · 2026-10-14) · [7](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Standard coreference annotation asks people to connect pronouns with earlier text spans. The authors argue this is unnatural, because readers reason over a mental model of the entities a text describes rather than over strings. They propose model-based annotation instead. Documents are first linked to a knowledge base, and annotators attach each pronoun to one or more entries from a pre-extracted entity list. They can also add missing entities or mark a pronoun as No Reference. To test the idea, they collected 200 English documents: 100 Wikipedia summary paragraphs and 100 teacher-student dialogues from QuAC. They then ran two Amazon Mechanical Turk setups, a grounded task showing the entity list and a conventional free span-selection task. The grounded setup produced clearly higher inter-annotator agreement and, according to the authors, faster annotation. The data also forms two new benchmarks on which existing resolvers perform weakly. The approach matters because it reduces annotator effort and provides cluster representatives from the outset. It may also extend to ellipsis, implicit arguments or frame roles. ## Key points - **Corpus composition:** the 100 Wikipedia documents are summaries containing at least five pronouns, of which 30 come from WikiCoref titles and 70 were picked at random. The 100 QuAC items are the first datapoints of that dataset. Entities were harvested from Wikipedia and Wikidata links. - **Annotation design:** only pronouns were markable. In each source, 30 documents were doubly annotated and 70 singly. Workers had to be native English speakers with an approval rate above 90%. Every fifth document was a hidden test item, and workers needed over 90% agreement with gold to continue. - **Inter-annotator agreement (exact match / F1):** | Source | Grounded | Free span | |---|---|---| | Wiki | 0.70 / 0.74 | 0.50 / 0.65 | | QuAC | 0.65 / 0.67 | 0.52 / 0.64 | - **Baseline results (macro-averaged MUC, B³ and CEAFφ4 F1):** | System | Wiki | QuAC | |---|---|---| | Clark and Manning (2016) | 27.95 | 23.39 | | Lee et al. (2018) | 26.67 | 23.79 | | Aralikatte et al. (2019), using gold mention brackets | 45.96 | 46.14 | - **Availability:** the data and the baseline coreference links are released at github.com/rahular/model-based-coref. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) ## Cited by Cited by 7 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 5 not yet. - [Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning](https://md.research.elamur.ai/free/en/papers/dasigi-2019-quoref-reading-comprehension-dataset-questions/) - [Annotating Mentions Alone Enables Efficient Domain Adaptation for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/gandhi-2023-annotating-mentions-alone-enables-efficient/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{aralikatte2020model, title = {{Model-based Annotation of Coreference}}, author = {Aralikatte, Rahul and Søgaard, Anders}, booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference}, year = {2020}, eprint = {1906.10724}, archivePrefix = {arXiv}, url = {https://arxiv.org/abs/1906.10724} } ``` - [← previous MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/) - [next → Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/pan-2020-modeling-intra-inter-modality-incongruity/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) ## References 8 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Aralikatte, R., Lamm, M., Hardt, D., and Søgaard, A. (2019). Ellipsis and coreference resolution as question answering. CoRR, abs/1908.11141. — [arXiv:1908.11141](https://arxiv.org/abs/1908.11141) [find](https://scholar.google.com/scholar?q=Aralikatte%2C+R%2E%2C+Lamm%2C+M%2E%2C+Hardt%2C+D%2E%2C+and+S%C3%B8gaard%2C+A%2E+%282019%29%2E+Ellipsis+and+coreference+resolution+as+question+answering%2E+CoRR%2C+abs%2F1908%2E11141%2E+%E2%80%94+%5BarXiv%3A1908%2E11141%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1908%2E11141%29) 2. Blackburn, P. and Bos, J. (2005). Representation and Inference for Natural Language. [find](https://scholar.google.com/scholar?q=Blackburn%2C+P%2E+and+Bos%2C+J%2E+%282005%29%2E+Representation+and+Inference+for+Natural+Language%2E) 3. CSLI, Stanford, CA. Bower, G. and Morrow, D. (1990). Mental models in narrative comprehension. Science, 247(4938):44–48. [find](https://scholar.google.com/scholar?q=CSLI%2C+Stanford%2C+CA%2E+Bower%2C+G%2E+and+Morrow%2C+D%2E+%281990%29%2E+Mental+models+in+narrative+comprehension%2E+Science%2C+247%284938%29%3A44%E2%80%9348%2E) 4. Cai, J. and Strube, M. (2010). Evaluation metrics for end-to-end coreference resolution systems. In Proceedings of the SIGDIAL 2010 Conference, pages 28–36. Association for Computational Linguistics, Sep. [find](https://scholar.google.com/scholar?q=Cai%2C+J%2E+and+Strube%2C+M%2E+%282010%29%2E+Evaluation+metrics+for+end%2Dto%2Dend+coreference+resolution+systems%2E+In+Proceedings+of+the+SIGDIAL+2010+Conference%2C+pages+28%E2%80%9336%2E+Association+for+Computational+Linguistics%2C+) [our card](https://md.research.elamur.ai/free/en/papers/cai-2010-evaluation-metrics-end-end-coreference/) 5. Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-t., Choi, Y., Liang, P., and Zettlemoyer, L. (2018). QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium, October-November. Association for… [find](https://scholar.google.com/scholar?q=Choi%2C+E%2E%2C+He%2C+H%2E%2C+Iyyer%2C+M%2E%2C+Yatskar%2C+M%2E%2C+Yih%2C+W%2E%2Dt%2E%2C+Choi%2C+Y%2E%2C+Liang%2C+P%2E%2C+and+Zettlemoyer%2C+L%2E+%282018%29%2E+QuAC%3A+Question+answering+in+context%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+i) [our card](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/) 6. Clark, K. and Manning, C. D. (2016). Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256– 2262, Austin, Texas, November. Association for Computational Linguistics. nouns in picture n… [find](https://scholar.google.com/scholar?q=Clark%2C+K%2E+and+Manning%2C+C%2E+D%2E+%282016%29%2E+Deep+reinforcement+learning+for+mention%2Dranking+coreference+models%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/clark-2016-deep-reinforcement-learning-mention-ranking/) 7. Webster, K., Recasens, M., Axelrod, V., and Baldridge, J. (2018). Mind the gap: A balanced corpus of gendered ambiguous pronouns. In Transactions of the ACL, page to appear. [find](https://scholar.google.com/scholar?q=Webster%2C+K%2E%2C+Recasens%2C+M%2E%2C+Axelrod%2C+V%2E%2C+and+Baldridge%2C+J%2E+%282018%29%2E+Mind+the+gap%3A+A+balanced+corpus+of+gendered+ambiguous+pronouns%2E+In+Transactions+of+the+ACL%2C+page+to+appear%2E) [our card](https://md.research.elamur.ai/free/en/papers/webster-2018-mind-gap-balanced-corpus-gendered/) 8. Yimam, S. M., Gurevych, I., de Castilho, R. E., and Biemann, C. (2013). Webanno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Yimam%2C+S%2E+M%2E%2C+Gurevych%2C+I%2E%2C+de+Castilho%2C+R%2E+E%2E%2C+and+Biemann%2C+C%2E+%282013%29%2E+Webanno%3A+A+flexible%2C+web%2Dbased+and+visually+supported+system+for+distributed+annotations%2E+In+Proceedings+of+ACL%2E)