--- title: "Statistical Methods for Annotation Analysis — Wilkens 2023" description: "Part 2 presents models that infer labels from annotator behaviour, extending to a variational autoencoder, and… Cited by 16 papers here." authors: ["Rodrigo Wilkens"] year: 2023 venue: "Computational Linguistics" source: "https://aclanthology.org/2023.cl-3.7/" doi: "10.1162/coli_r_00483" licence: "CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)" licence_url: "https://creativecommons.org/licenses/by-nc-nd/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wilkens-2023-statistical-methods-annotation-analysis/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wilkens-2023-statistical-methods-annotation-analysis/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Statistical Methods for Annotation Analysis # Statistical Methods for Annotation Analysis **Authors:** [Rodrigo Wilkens](https://md.research.elamur.ai/free/en/authors/rodrigo-wilkens/) · **Year:** 2023 · **Venue:** [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/) ([Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/cl/2023/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)](https://creativecommons.org/licenses/by-nc-nd/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2023.cl-3.7/) · [PDF](https://aclanthology.org/2023.cl-3.7.pdf) · [DOI 10.1162/coli\_r\_00483](https://doi.org/10.1162/coli_r_00483) · [OpenAlex](https://openalex.org/W4378193734) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/wilkens-2023-statistical-methods-annotation-analysis/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This entry is a book review, not an original study. Rodrigo Wilkens assesses "Statistical Methods for Annotation Analysis" by Silviu Paun, Ron Artstein and Massimo Poesio (Springer, 2022). The book addresses a recurring problem in corpus building: when several coders label the same items, researchers must measure how reliably they agree and also derive one final label per item. It updates Artstein and Poesio's 2008 survey of agreement coefficients and adds topics that survey omitted, including latent models of agreement, probabilistic aggregation models and training directly on multi-coder data. Part 1 covers coefficients such as κ, π and α. It also covers design issues (missing labels, unit boundaries, skewed data) and how to interpret coefficient values. Part 2 presents models that infer labels from annotator behaviour, extending to a variational autoencoder, and methods for learning from disagreement with soft labels. The reviewer praises the book's critical and historical framing and its practical examples. He considers it valuable for researchers and for anyone validating an annotated dataset, although practitioners may find some task-specific sections less relevant. ## Key points - The book is volume 54 of Springer's Synthesis Lectures on Human Language Technologies series, edited by Graeme Hirst. It runs to xix+197 pages and is reviewed in Computational Linguistics 49(3), 2023. - It builds on the 2008 article "Inter-coder Agreement for Computational Linguistics." It discusses Krippendorff's α, Scott's π and Cohen's κ, as well as weighted coefficients and the difficulty of interpreting them. - Chapter 4 treats reliability as confidence estimation over latent item classes, introducing item difficulty. This moves beyond simple majority voting, which ignores coder bias. - Chapter 5 covers probabilistic annotation models. These include hierarchical priors for annotators, inter-dependent labels (NER, information extraction), labels without predefined classes such as anaphora, and VAE-based aggregation. - Chapter 6 covers learning from multi-annotated corpora through soft labels, individual coder labels and pooling of coder confusion. It closes with guidance on whether to aggregate annotations or keep them all, and on crowdsourced versus gold labels. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [UCLouvain](https://md.research.elamur.ai/free/en/universities/uclouvain/) — University, Louvain-la-Neuve, Belgium · 20 works · 3,741 citations 📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 209 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 227 works · [🎯 Statistics, Probability and Uncertainty](https://md.research.elamur.ai/free/en/shelves/decision-sciences/statistics-probability-and-uncertainty/) · 61 works ## Cite Cite as: Rodrigo Wilkens. 2023. Statistical Methods for Annotation Analysis. Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{wilkens2023statistical, title = {{Statistical Methods for Annotation Analysis}}, author = {Rodrigo Wilkens}, journal = {Computational Linguistics}, year = {2023}, doi = {10.1162/coli\_r\_00483}, url = {https://doi.org/10.1162/coli_r_00483} } ``` - [← previous Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for Tigrinya](https://md.research.elamur.ai/free/en/papers/gaim-2023-question-answering-low-resourced-language/) - [next → Susceptibility to Influence of Large Language Models](https://md.research.elamur.ai/free/en/papers/griffin-2023-susceptibility-influence-large-language-models/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) *Updated 30 October 2026*