---
title: "The Reliability of Anaphoric Annotation — Poesio 2005"
description: "Poesio and Artstein revisit how reliable anaphoric annotation is."
authors: ["Massimo Poesio", "Ron Artstein"]
year: 2005
venue: "Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky"
source: "https://aclanthology.org/W05-0311/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/poesio-2005-reliability-anaphoric-annotation-reconsidered-taking/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/poesio-2005-reliability-anaphoric-annotation-reconsidered-taking/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2005](https://md.research.elamur.ai/free/en/papers/2005/) / The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account

# The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account

**Authors:** [Massimo Poesio](https://md.research.elamur.ai/free/en/authors/massimo-poesio/), [Ron Artstein](https://md.research.elamur.ai/free/en/authors/ron-artstein/) · **Year:** 2005 · **Venue:** Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W05-0311/) · [PDF](https://aclanthology.org/W05-0311.pdf) · [OpenAlex](https://openalex.org/W7061850635)

**Citations:** 12 citations (openalex-snapshot · 2026-10-14) · [63](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Poesio and Artstein revisit how reliable anaphoric annotation is. They argue that earlier agreement studies used too few coders or lacked a suitable chance-corrected statistic. They also argue that disagreement often reflects real ambiguity in the text rather than a flawed scheme. To test this, eighteen naive, paid students annotated a TRAINS 91 dialogue in MMAX 2. The coding manual was loosely derived from MATE and GNOME. It let coders flag an expression as ambiguous and mark simple discourse deixis, and it left out bridging references.

Agreement was measured with Krippendorff's α rather than K, following Passonneau, because α gives credit for partial overlap between anaphoric chains. The authors compared three chain distance metrics (Passonneau's, Jaccard, Dice) and several ways of building chains from the raw annotations. The chain-construction method affected α more than the choice of metric. Qualitative analysis showed that almost all disagreements, apart from occasional careless errors, came from genuine referential ambiguity. This matters for designing corpora and for future methods that detect ambiguous expressions automatically.

## Key points
- **Setup:** 18 University of Essex students, mostly undergraduates in Psychology and Language and Linguistics, annotated dialogue 3.2 of TRAINS 91 in MMAX 2. They trained on dialogue 3.1, and the work was split over two sessions five days apart.
- **Conditions compared:** six chain conditions (no chain, partial chain, and full chains in inclusive/exclusive and [+top]/[−top] variants), each crossed with three distance metrics.
- **Best result:** the Inclusive [−top] chain with the Dice distance gave the highest α. For the first half of the dialogue, α was about 0.69 with place markables and about 0.58 without them.
- **Careless errors:** obvious mistakes accounted for only 35 of 1296 judgments (about 2.7%).
- **Ambiguity example:** for the pronoun "it" in utterance 3.1, 9 coders flagged it as ambiguous, 6 chose engine E2 and 3 chose the boxcar. This shows that implicit ambiguity, revealed by disagreement between coders, matters as much as ambiguity coders mark explicitly.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/)

## Cited by

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 46 not yet.

- [We Need to Consider Disagreement in Evaluation](https://md.research.elamur.ai/free/en/papers/basile-2021-we-need-consider-disagreement-evaluation/)
- [Investigating Reasons for Disagreement in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/jiang-2022-investigating-reasons-disagreement-natural-language/)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://md.research.elamur.ai/free/en/papers/nie-2020-what-can-we-learn-collective/)
- [A Probabilistic Annotation Model for Crowdsourcing Coreference](https://md.research.elamur.ai/free/en/papers/paun-2018-probabilistic-annotation-model-crowdsourcing-coreference/)
- [Inherent Disagreements in Human Textual Inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/)
- [The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/)
- [Anaphora Resolution with the ARRAU Corpus](https://md.research.elamur.ai/free/en/papers/poesio-2018-anaphora-resolution-arrau-corpus/)
- [A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation](https://md.research.elamur.ai/free/en/papers/poesio-2019-crowdsourced-corpus-multiple-judgments-disagreement/)
- [CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes](https://md.research.elamur.ai/free/en/papers/pradhan-2011-conll-2011-shared-task-modeling/)
- [CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/)
- [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://md.research.elamur.ai/free/en/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/)
- [A Typology of Near-Identity Relations for Coreference (NIDENT)](https://md.research.elamur.ai/free/en/papers/recasens-2010-typology-near-identity-relations-coreference/)
- [Integrating Predictions from Neural-Network Relation Classifiers into Coreference and Bridging Resolution](https://md.research.elamur.ai/free/en/papers/roesiger-2018-integrating-predictions-neural-network-relation/)
- [Categories of coherence relations in discourse annotation](https://md.research.elamur.ai/free/en/papers/scholman-2016-categories-coherence-relations-discourse-annotation/)
- [The Universal Anaphora Scorer](https://md.research.elamur.ai/free/en/papers/yu-2022-universal-anaphora-scorer/)
- [AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference](https://md.research.elamur.ai/free/en/papers/yuan-2023-ambicoref-evaluating-human-model-sensitivity/)
- [Opinion Piece: Can we Fix the Scope for Coreference? Problems and Solutions for Benchmarks beyond OntoNotes](https://md.research.elamur.ai/free/en/papers/zeldes-2022-opinion-piece-can-we-fix/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{poesio2005reliability,
  title = {{The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account}},
  author = {Poesio, Massimo and Artstein, Ron},
  booktitle = {Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky},
  year = {2005},
  url = {https://aclanthology.org/W05-0311/}
}
```

- [← previous The Proposition Bank: An Annotated Corpus of Semantic Roles](https://md.research.elamur.ai/free/en/papers/palmer-2005-proposition-bank-annotated-corpus-semantic/)
- [next → The Types and Distributions of Errors in a Wide Coverage Surface Realizer Evaluation](https://md.research.elamur.ai/free/en/papers/callaway-2005-types-distributions-errors-wide-coverage/)
- [↑ back to papers of 2005](https://md.research.elamur.ai/free/en/papers/2005/)

## References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. P. Buitelaar. 1998. CoreLex : Systematic Polysemy and Underspecification. Ph.D. thesis, Brandeis University. [find](https://scholar.google.com/scholar?q=P%2E+Buitelaar%2E+1998%2E+CoreLex+%3A+Systematic+Polysemy+and+Underspecification%2E+Ph%2ED%2E+thesis%2C+Brandeis+University%2E)
2. D. Byron. 2003. Annotation of pronouns and their antecedents: A comparison of two domains. Technical Report 703, University of Rochester. [find](https://scholar.google.com/scholar?q=D%2E+Byron%2E+2003%2E+Annotation+of+pronouns+and+their+antecedents%3A+A+comparison+of+two+domains%2E+Technical+Report+703%2C+University+of+Rochester%2E)
3. M. Eckert and M. Strube. 2001. Dialogue acts, synchronising units and anaphora resolution. Journal of Semantics. [find](https://scholar.google.com/scholar?q=M%2E+Eckert+and+M%2E+Strube%2E+2001%2E+Dialogue+acts%2C+synchronising+units+and+anaphora+resolution%2E+Journal+of+Semantics%2E)
4. D. Gross, J. Allen, and D. Traum. 1993. The TRAINS 91 dialogues. TRAINS Technical Note 92-1, Computer Science Dept. University of Rochester, June. [find](https://scholar.google.com/scholar?q=D%2E+Gross%2C+J%2E+Allen%2C+and+D%2E+Traum%2E+1993%2E+The+TRAINS+91+dialogues%2E+TRAINS+Technical+Note+92%2D1%2C+Computer+Science+Dept%2E+University+of+Rochester%2C+June%2E)
5. L. Hirschman. 1998. MUC-7 coreference task definition, version 3.0. In N. Chinchor, editor, In Proc. of the 7th Message Understanding Conference. [find](https://scholar.google.com/scholar?q=L%2E+Hirschman%2E+1998%2E+MUC%2D7+coreference+task+definition%2C+version+3%2E0%2E+In+N%2E+Chinchor%2C+editor%2C+In+Proc%2E+of+the+7th+Message+Understanding+Conference%2E) [our card](https://md.research.elamur.ai/free/en/papers/hirschman-1998-appendix-f-muc-7-coreference/)
6. H. Kamp and U. Reyle. 1993. From Discourse to Logic. D. Reidel, Dordrecht. [find](https://scholar.google.com/scholar?q=H%2E+Kamp+and+U%2E+Reyle%2E+1993%2E+From+Discourse+to+Logic%2E+D%2E+Reidel%2C+Dordrecht%2E)
7. K. Krippendorff. 1980. Content Analysis: An introduction to its Methodology. Sage Publications. [find](https://scholar.google.com/scholar?q=K%2E+Krippendorff%2E+1980%2E+Content+Analysis%3A+An+introduction+to+its+Methodology%2E+Sage+Publications%2E)
8. C. D. Manning and H. Schuetze. 1999. Foundations of Statistical Natural Language Processing. MIT Press. [find](https://scholar.google.com/scholar?q=C%2E+D%2E+Manning+and+H%2E+Schuetze%2E+1999%2E+Foundations+of+Statistical+Natural+Language+Processing%2E+MIT+Press%2E)
9. C. Müller and M. Strube. 2003. Multi-level annotation in MMAX. In Proc. of the 4th SIGDIAL. [find](https://scholar.google.com/scholar?q=C%2E+Mu%CC%88ller+and+M%2E+Strube%2E+2003%2E+Multi%2Dlevel+annotation+in+MMAX%2E+In+Proc%2E+of+the+4th+SIGDIAL%2E)
10. M. Palmer, H. Dang, and C. Fellbaum. 2005. Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Journal of Natural Language Engineering. To appear. [find](https://scholar.google.com/scholar?q=M%2E+Palmer%2C+H%2E+Dang%2C+and+C%2E+Fellbaum%2E+2005%2E+Making+fine%2Dgrained+and+coarse%2Dgrained+sense+distinctions%2C+both+manually+and+automatically%2E+Journal+of+Natural+Language+Engineering%2E+To+appear%2E)
11. R. J. Passonneau. 1997. Instructions for applying discourse reference annotation for multiple applications (DRAMA). Unpublished manuscript., December. [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Passonneau%2E+1997%2E+Instructions+for+applying+discourse+reference+annotation+for+multiple+applications+%28DRAMA%29%2E+Unpublished+manuscript%2E%2C+December%2E)
12. R. J. Passonneau. 2004. Computing reliability for coreference annotation. In Proc. of LREC, Lisbon. [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Passonneau%2E+2004%2E+Computing+reliability+for+coreference+annotation%2E+In+Proc%2E+of+LREC%2C+Lisbon%2E)
13. M. Poesio and R. Vieira. 1998. A corpus-based investigation of definite description use. Computational Linguistics, 24(2):183–216, June. [find](https://scholar.google.com/scholar?q=M%2E+Poesio+and+R%2E+Vieira%2E+1998%2E+A+corpus%2Dbased+investigation+of+definite+description+use%2E+Computational+Linguistics%2C+24%282%29%3A183%E2%80%93216%2C+June%2E)
14. M. Poesio, F. Bruneseaux, and L. Romary. 1999. The MATE meta-scheme for coreference in dialogues in multiple languages. In M. Walker, editor, Proc. of the ACL Workshop on Standards and Tools for Discourse Tagging, pages 65–74. [find](https://scholar.google.com/scholar?q=M%2E+Poesio%2C+F%2E+Bruneseaux%2C+and+L%2E+Romary%2E+1999%2E+The+MATE+meta%2Dscheme+for+coreference+in+dialogues+in+multiple+languages%2E+In+M%2E+Walker%2C+editor%2C+Proc%2E+of+the+ACL+Workshop+on+Standards+and+Tools+for+Disco) [our card](https://md.research.elamur.ai/free/en/papers/poesio-1999-mate-meta-scheme-coreference-dialogues/)
15. M. Poesio. 1996. Semantic ambiguity and perceived ambiguity. In K. van Deemter and S. Peters, editors, Semantic Ambiguity and Underspecification, chapter 8, pages 159–201. CSLI, Stanford, CA. [find](https://scholar.google.com/scholar?q=M%2E+Poesio%2E+1996%2E+Semantic+ambiguity+and+perceived+ambiguity%2E+In+K%2E+van+Deemter+and+S%2E+Peters%2C+editors%2C+Semantic+Ambiguity+and+Underspecification%2C+chapter+8%2C+pages+159%E2%80%93201%2E+CSLI%2C+Stanford%2C+CA%2E)
16. M. Poesio. 2004. The MATE/GNOME scheme for anaphoric annotation, revisited. In Proc. of SIGDIAL, Boston, May. [find](https://scholar.google.com/scholar?q=M%2E+Poesio%2E+2004%2E+The+MATE%2FGNOME+scheme+for+anaphoric+annotation%2C+revisited%2E+In+Proc%2E+of+SIGDIAL%2C+Boston%2C+May%2E) [our card](https://md.research.elamur.ai/free/en/papers/poesio-2004-mate-gnome-proposals-anaphoric-annotation/)
17. E. F. Prince. 1992. The ZPG letter: subjects, definiteness, and information status. In S. Thompson and W. Mann, editors, Discourse description: diverse analyses of a fund-raising text, pages 295–325. John Benjamins. [find](https://scholar.google.com/scholar?q=E%2E+F%2E+Prince%2E+1992%2E+The+ZPG+letter%3A+subjects%2C+definiteness%2C+and+information+status%2E+In+S%2E+Thompson+and+W%2E+Mann%2C+editors%2C+Discourse+description%3A+diverse+analyses+of+a+fund%2Draising+text%2C+pages+295%E2%80%93325%2E+)
18. A. Rosenberg and E. Binkowski. 2004. Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=A%2E+Rosenberg+and+E%2E+Binkowski%2E+2004%2E+Augmenting+the+kappa+statistic+to+determine+interannotator+reliability+for+multiply+labeled+data+points%2E+In+Proc%2E+of+NAACL%2E)
19. C. L. Sidner. 1979. Towards a computational theory of definite anaphora comprehension in English discourse. Ph.D. thesis, MIT. [find](https://scholar.google.com/scholar?q=C%2E+L%2E+Sidner%2E+1979%2E+Towards+a+computational+theory+of+definite+anaphora+comprehension+in+English+discourse%2E+Ph%2ED%2E+thesis%2C+MIT%2E)
20. S. Siegel and N. J. Castellan. 1988. Nonparametric statistics for the Behavioral Sciences. McGraw-Hill. [find](https://scholar.google.com/scholar?q=S%2E+Siegel+and+N%2E+J%2E+Castellan%2E+1988%2E+Nonparametric+statistics+for+the+Behavioral+Sciences%2E+McGraw%2DHill%2E)
21. K. van Deemter and R. Kibble. 2000. On coreferring: Coreference in MUC and related annotation schemes. Computational Linguistics, 26(4):629–637. Squib. [find](https://scholar.google.com/scholar?q=K%2E+van+Deemter+and+R%2E+Kibble%2E+2000%2E+On+coreferring%3A+Coreference+in+MUC+and+related+annotation+schemes%2E+Computational+Linguistics%2C+26%284%29%3A629%E2%80%93637%2E+Squib%2E) [our card](https://md.research.elamur.ai/free/en/papers/deemter-2000-coreferring-coreference-muc-related-annotation/)
22. B. L. Webber. 1979. A Formal Approach to Discourse Anaphora. Garland, New York. [find](https://scholar.google.com/scholar?q=B%2E+L%2E+Webber%2E+1979%2E+A+Formal+Approach+to+Discourse+Anaphora%2E+Garland%2C+New+York%2E)
23. B. L. Webber. 1991. Structure and ostension in the interpretation of discourse deixis. Language and Cognitive Processes, 6(2):107–135. [find](https://scholar.google.com/scholar?q=B%2E+L%2E+Webber%2E+1991%2E+Structure+and+ostension+in+the+interpretation+of+discourse+deixis%2E+Language+and+Cognitive+Processes%2C+6%282%29%3A107%E2%80%93135%2E)