--- title: "Comparing Bayesian Models of Annotation — Paun 2018" description: "In NLP, crowdsourced labels are usually merged by majority vote and checked with chance-corrected… 5 references, cited by 47 papers here." authors: ["Silviu Paun", "Bob Carpenter", "Jon Chamberlain", "Dirk Hovy", "Udo Kruschwitz", "Massimo Poesio"] year: 2018 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/Q18-1040/" doi: "10.1162/tacl_a_00040" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/paun-2018-comparing-bayesian-models-annotation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/paun-2018-comparing-bayesian-models-annotation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Comparing Bayesian Models of Annotation # Comparing Bayesian Models of Annotation **Authors:** [Silviu Paun](https://md.research.elamur.ai/free/en/authors/silviu-paun/), [Bob Carpenter](https://md.research.elamur.ai/free/en/authors/bob-carpenter/), [Jon Chamberlain](https://md.research.elamur.ai/free/en/authors/jon-chamberlain/), [Dirk Hovy](https://md.research.elamur.ai/free/en/authors/dirk-hovy/), [Udo Kruschwitz](https://md.research.elamur.ai/free/en/authors/udo-kruschwitz/) and [Massimo Poesio](https://md.research.elamur.ai/free/en/authors/massimo-poesio/) · **Year:** 2018 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2018](https://md.research.elamur.ai/free/en/venues/tacl/2018/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/Q18-1040/) · [PDF](https://aclanthology.org/Q18-1040.pdf) · [DOI 10.1162/tacl\_a\_00040](https://doi.org/10.1162/tacl_a_00040) · [OpenAlex](https://openalex.org/W2899689163) **Citations:** 83 citations (OpenAlex, 2026-10-14) · [47](https://md.research.elamur.ai/free/en/papers/paun-2018-comparing-bayesian-models-annotation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary In NLP, crowdsourced labels are usually merged by majority vote and checked with chance-corrected agreement coefficients. Both approaches treat annotators as equally competent and ignore bias and item effects. Probabilistic annotation models address these issues, but they had rarely been compared directly on real data. Paun et al. benchmark six Bayesian models that differ in how parameters are shared: - a fully pooled multinomial model; - unpooled models (D&S, MACE); - partially pooled hierarchical models (HIER D&S, ITEMDIFF, LOGRNDEFF). Several of these were generalized from binary to multiclass labels. All are implemented in Stan with full posterior inference. They are judged on four criteria: match with gold labels, prediction of new annotations, annotator characterization, and item difficulty. The test data are four datasets with different shares of spam-like annotators. Models that represent annotators explicitly generally perform best, though unpooled ones can overfit. HIER D&S is usually strongest, except on a game-collected dataset where annotators fall into two quality groups. The study gives practical guidance on choosing and implementing annotation models instead of defaulting to majority voting. ## Key points - The six models compared are MULTINOM (pooled), D&S and MACE (unpooled), and HIER D&S, ITEMDIFF and LOGRNDEFF (partially pooled via hierarchical priors). - All models were written in Stan using Hamiltonian Monte Carlo. A non-centered parameterization proved essential for fitting the hierarchical models reliably. - Label switching was handled by relabeling each chain after inference. The relabeling used a small gold sample of about five items per class to pick the permutation with the highest accuracy. - HIER D&S usually leads on the Snow et al. (2008) Amazon Mechanical Turk datasets. On the game-with-a-purpose PD dataset, MACE and D&S beat it, because a few strong annotators coexist with many weak ones there. - The authors suggest that a multi-modal hierarchical prior might suit two-population data like PD better. Supplementary material is published by the DALI project at QMUL. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 149 works · 48,852 citations - [University of Essex](https://md.research.elamur.ai/free/en/universities/university-of-essex/) — University, Colchester, United Kingdom · 25 works · 3,285 citations - [Queen Mary University of London](https://md.research.elamur.ai/free/en/universities/queen-mary-university-of-london/) — University, London, United Kingdom · 39 works · 3,307 citations - [Bocconi University](https://md.research.elamur.ai/free/en/universities/bocconi-university/) — University, Milan, Italy · 16 works · 2,850 citations 📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 85 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 87 works · [🎯 Statistics, Probability and Uncertainty](https://md.research.elamur.ai/free/en/shelves/decision-sciences/statistics-probability-and-uncertainty/) · 32 works ## Cite Cite as: Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz and Massimo Poesio. 2018. Comparing Bayesian Models of Annotation. Transactions of the Association for Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{paun2018comparing, title = {{Comparing Bayesian Models of Annotation}}, author = {Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz and Massimo Poesio}, journal = {Transactions of the Association for Computational Linguistics}, year = {2018}, doi = {10.1162/tacl\_a\_00040}, url = {https://doi.org/10.1162/tacl_a_00040} } ``` - [← previous Deep Enhanced Representation for Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/en/papers/bai-2018-deep-enhanced-representation-implicit-discourse/) - [next → Scheduled Multi-Task Learning: From Syntax to Translation](https://md.research.elamur.ai/free/en/papers/kiperwasser-2018-scheduled-multi-task-learning-syntax/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) ## References 5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Paul S. Albert and Lori E. Dodd. 2004. A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard. Biometrics, 60(2):427–435. [find](https://scholar.google.com/scholar?q=Paul+S%2E+Albert+and+Lori+E%2E+Dodd%2E+2004%2E+A+cautionary+note+on+the+robustness+of+latent+class+models+for+estimating+diagnostic+error+without+a+gold+standard%2E+Biometrics%2C+60%282%29%3A427%E2%80%93435%2E) 2. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. [find](https://scholar.google.com/scholar?q=Ron+Artstein+and+Massimo+Poesio%2E+2008%2E+Inter%2Dcoder+agreement+for+computational+linguistics%2E+Computational+Linguistics%2C+34%284%29%3A555%E2%80%93596%2E) 3. James O. Berger. 2013. Statistical Decision Theory and Bayesian Analysis. Springer. [find](https://scholar.google.com/scholar?q=James+O%2E+Berger%2E+2013%2E+Statistical+Decision+Theory+and+Bayesian+Analysis%2E+Springer%2E) 4. José M. Bernardo and Adrian F. M. Smith. 2001. Bayesian Theory. IOP Publishing. [find](https://scholar.google.com/scholar?q=Jos%C3%A9+M%2E+Bernardo+and+Adrian+F%2E+M%2E+Smith%2E+2001%2E+Bayesian+Theory%2E+IOP+Publishing%2E) 5. Michael Betancourt and Mark Girolami. 2015. Hamiltonian Monte Carlo for hierarchical models. Current trends in Bayesian methodology with applications, 79:30. \#\# Conclusions [find](https://scholar.google.com/scholar?q=Michael+Betancourt+and+Mark+Girolami%2E+2015%2E+Hamiltonian+Monte+Carlo+for+hierarchical+models%2E+Current+trends+in+Bayesian+methodology+with+applications%2C+79%3A30%2E+%5C%23%5C%23+Conclusions) *Updated 24 October 2026*