--- title: "On Releasing Annotator-Level Labels and — Prabhakaran 2021" description: "NLP datasets with multiple annotators per item usually collapse those judgements into one label, by majority vote, averaging, or expert adjudication." authors: ["Vinodkumar Prabhakaran", "Aida Mostafazadeh Davani", "Mark Diaz"] year: 2021 venue: "Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop" source: "https://aclanthology.org/2021.law-1.14/" doi: "10.18653/v1/2021.law-1.14" arxiv: "2110.05699" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/prabhakaran-2021-releasing-annotator-level-labels-information/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/prabhakaran-2021-releasing-annotator-level-labels-information/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2021](https://md.research.elamur.ai/free/papers/2021/page/3/) / On Releasing Annotator-Level Labels and Information in Datasets # On Releasing Annotator-Level Labels and Information in Datasets **Authors:** [Vinodkumar Prabhakaran](https://md.research.elamur.ai/free/authors/vinodkumar-prabhakaran/), [Aida Mostafazadeh Davani](https://md.research.elamur.ai/free/authors/aida-mostafazadeh-davani/), [Mark Diaz](https://md.research.elamur.ai/free/authors/mark-diaz/) · **Year:** 2021 · **Venue:** Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.law-1.14/) · [PDF](https://aclanthology.org/2021.law-1.14.pdf) · [DOI 10.18653/v1/2021.law-1.14](https://doi.org/10.18653/v1/2021.law-1.14) · [arXiv 2110.05699](https://arxiv.org/abs/2110.05699) · [OpenAlex](https://openalex.org/W3206621920) **Citations:** 75 citations (openalex-snapshot · 2026-10-14) · [93](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary NLP datasets with multiple annotators per item usually collapse those judgements into one label, by majority vote, averaging, or expert adjudication. For subjective tasks, this can hide systematic disagreement rooted in annotators' backgrounds and experiences. The authors test whether majority voting represents annotators fairly. They examine eight binary tasks from three datasets: hate speech, sentiment, and six Ekman emotions. For each annotator, they compute Cohen's Kappa between that person's labels and the majority label. They then compare these scores across socio-demographic groups in the sentiment data, which includes self-reported annotator demographics. In most tasks, a large share of annotators agree only weakly with the majority. In the sentiment data, White annotators match the aggregated labels significantly better than Black annotators. Aggregation can therefore make a dataset under-represent some groups' views. The authors recommend releasing individual annotator labels and, where it can be done responsibly, annotator demographics. They also point to modelling work that already benefits from annotator-level labels. ## Key points - Data analysed: - Hate-speech (Kennedy et al., 2020): 27,665 instances, 18 annotators, 86,529 annotations. - Sentiment (Díaz et al., 2018): 14,071 instances, 1,481 annotators, 59,240 annotations. Its 5-point scale was binarised. - Emotion (Demszky et al., 2020): 58,011 instances, 82 annotators, 211,225 annotations. Only the raw labels for anger, disgust, fear, joy, sadness and surprise were used. - The majority label was the one chosen by at least half of an item's annotators. Each annotator was scored with Cohen's Kappa against it. - Low agreement was common: - For disgust, 27 of 82 annotators had κ \< 0.3 with the majority. - For sentiment, roughly a third of annotators (450+) agreed poorly. - Joy was the main task where agreement was fairly uniform across annotators. - Group differences: - One-way ANOVA found a significant race effect (F(2, 2387)=3.77, p=0.02). - White annotators averaged κ 0.42, versus 0.37 for Black annotators (Tukey p=0.03). - Gender and political-affiliation differences were not significant. - The paper describes four levels of annotation release: aggregated label only, label distribution, anonymised per-annotator labels, and per-annotator labels with demographics. It argues for the richer levels. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/) · [Law](https://md.research.elamur.ai/free/shelves/law/) ## Cited by Cited by 93 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 83 not yet. - [MultiPICo: Multilingual Perspectivist Irony Corpus](https://md.research.elamur.ai/free/papers/casola-2024-multipico-multilingual-perspectivist-irony-corpus/) - [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://md.research.elamur.ai/free/papers/fernandes-2023-bridging-gap-survey-integrating-human/) - [EPIC: Multi-Perspective Annotation of a Corpus of Irony](https://md.research.elamur.ai/free/papers/frenda-2023-epic-multi-perspective-annotation-corpus/) - [Challenges and Strategies in Cross-Cultural NLP](https://md.research.elamur.ai/free/papers/hershcovich-2022-challenges-strategies-cross-cultural-nlp/) - [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/papers/kaddour-2023-challenges-applications-large-language-models/) - [A Holistic Approach to Undesired Content Detection in the Real World](https://md.research.elamur.ai/free/papers/markov-2022-holistic-approach-undesired-content-detection/) - [The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://md.research.elamur.ai/free/papers/plank-2022-problem-human-label-variation-ground/) - [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://md.research.elamur.ai/free/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/) - [Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks](https://md.research.elamur.ai/free/papers/rottger-2022-two-contrasting-data-annotation-paradigms/) - [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://md.research.elamur.ai/free/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{prabhakaran2021releasing, title = {{On Releasing Annotator-Level Labels and Information in Datasets}}, author = {Prabhakaran, Vinodkumar and Davani, Aida Mostafazadeh and Diaz, Mark}, booktitle = {Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop}, year = {2021}, doi = {10.18653/v1/2021.law-1.14}, eprint = {2110.05699}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/2021.law-1.14} } ``` - [← previous On Generalization in Coreference Resolution](https://md.research.elamur.ai/free/papers/toshniwal-2021-generalization-coreference-resolution/) - [next → On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/papers/bommasani-2021-opportunities-risks-foundation-models/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/papers/2021/page/3/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Cecilia Ovesdotter Alm. 2011. Subjective natural language problems: Motivations, applications, characterizations, and implications. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 107–112. [find](https://scholar.google.com/scholar?q=Cecilia+Ovesdotter+Alm%2E+2011%2E+Subjective+natural+language+problems%3A+Motivations%2C+applications%2C+characterizations%2C+and+implications%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Com)