---
title: "Beyond Black-Box Labels — paper by Rair et al. (2026)"
description: "The authors propose a schema-level diagnostic that audits an expert-designed annotation scheme before gold labels are fixed… 23 references."
authors: ["Nisrine Rair", "Alban Goupil", "Valeriu Vrabie", "Emmanuel Chochoy"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1281/"
doi: "10.18653/v1/2026.findings-acl.1281"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rair-2026-beyond-black-box-labels-interpretable/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rair-2026-beyond-black-box-labels-interpretable/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/) / Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks

# Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks

**Authors:** Nisrine Rair, Alban Goupil, Valeriu Vrabie and Emmanuel Chochoy · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1281/) · [PDF](https://aclanthology.org/2026.findings-acl.1281.pdf) · [DOI 10.18653/v1/2026.findings-acl.1281](https://doi.org/10.18653/v1/2026.findings-acl.1281) · [OpenAlex](https://openalex.org/W7166826477)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Subjective NLP datasets usually merge many annotators' judgments into a single gold label. This hides why annotators disagree: criteria may be vague, intended categories may be indistinguishable, or several readings may be legitimate. The authors propose a schema-level diagnostic that audits an expert-designed annotation scheme before gold labels are fixed, using only multi-annotator yes/no criterion judgments. A schema is formalized as categories, binary criteria, and a mapping that links each criterion to the category it supports. The diagnostic measures two failure modes. Instability means a criterion keeps producing split votes on the units where annotators engage with it. Overlap means criteria tied to mutually exclusive categories fire on the same unit. The authors test it on Persuasive Value Extraction from mostly French commercial documents, using an LLM annotator panel. Disagreement turns out to be structured: a few criteria drive the instability, and almost half of covered sentences trigger several categories. These signals match where domain experts disagree. Designers thus gain evidence for tightening guidelines, restructuring categories, or moving to multi-label annotation.

## Key points
- The schema is formalized as T = (C, Q, µ). C holds the categories, including a non-target category c0 ("no persuasive value"). Q holds the binary expert criteria, and µ maps each criterion to a category.
- Annotations form a binary tensor Y (units × annotators × criteria). Stability is computed only on units where at least t annotators marked a criterion. This prevents rarely triggered criteria from looking stable just because everyone agrees they are absent.
- On Persuasive Value Extraction (PVE), instability is concentrated in a small number of criteria. Nearly half of covered sentences activate more than one category, which clashes with the task's single-label framing.
- The diagnostic signals were checked against held-out expert judgments. The authors caution that results describe failure modes under an automated LLM judgment regime, not human population agreement.
- Code, anonymized sentences and the full annotation tensor are released at github.com/NisrineRair/annotation-schema-diagnostic. Raw client documents remain confidential.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [Université de Reims Champagne-Ardenne](https://ror.org/03hypw319) · [Centre de Recherche en Sciences et Technologies de l'Information et de la Communication](https://ror.org/016wdna72).

📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 209 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 227 works · [🎯 Statistics, Probability and Uncertainty](https://md.research.elamur.ai/free/en/shelves/decision-sciences/statistics-probability-and-uncertainty/) · 61 works

## Cite

Cite as: Nisrine Rair, Alban Goupil, Valeriu Vrabie and Emmanuel Chochoy. 2026. Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rair2026beyond,
  title = {{Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks}},
  author = {Nisrine Rair, Alban Goupil, Valeriu Vrabie and Emmanuel Chochoy},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1281},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1281}
}
```

- [← previous Beyond Bias Scores: Unmasking Vacuous Neutrality in Small Language Models](https://md.research.elamur.ai/free/en/papers/manduru-2026-beyond-bias-scores-unmasking-vacuous/)
- [next → Beyond BLEU: Linguistic Invisibility and Interactional Repair Sequence in End-to-End Sign Language Translation](https://md.research.elamur.ai/free/en/papers/wang-2026-beyond-bleu-linguistic-invisibility-interactional/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/)

## References

23 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lora Aroyo and Chris Welty. 2015. [Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation](https://md.research.elamur.ai/free/en/papers/aroyo-2015-truth-lie-crowd-truth-seven/). AI Magazine, 36(1):15–24.
2. Ron Artstein and Massimo Poesio. 2008. [Survey Article: Inter-Coder Agreement for Computational Linguistics](https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/). Computational Linguistics, 34(4):555–596.
3. Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2023. [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://md.research.elamur.ai/free/en/papers/basile-2021-toward-perspectivist-turn-ground-truthing/). Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):6860–6868. ArXiv:2109.04270 [cs].
4. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. [We Need to Consider Disagreement in Evaluation](https://md.research.elamur.ai/free/en/papers/basile-2021-we-need-consider-disagreement-evaluation/). In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computa…
5. Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, 6:587–604. Place: Cambridge, MA Publisher: MIT Press. [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender+and+Batya+Friedman%2E+2018%2E+Data+Statements+for+Natural+Language+Processing%3A+Toward+Mitigating+System+Bias+and+Enabling+Better+Science%2E+Transactions+of+the+Association+for+Computational+)
6. Emmanuel Chochoy. 2025. La méthode MSMKC : Réinventer la vente à l’ère du chaos numérique. Economica & L’Éditeur à part. [find](https://scholar.google.com/scholar?q=Emmanuel+Chochoy%2E+2025%2E+La+m%C3%A9thode+MSMKC+%3A+R%C3%A9inventer+la+vente+%C3%A0+l%E2%80%99%C3%A8re+du+chaos+num%C3%A9rique%2E+Economica+%26+L%E2%80%99%C3%89diteur+%C3%A0+part%2E)
7. A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20–28. Publisher: [Royal Statistical Society, Oxford University Press]. [find](https://scholar.google.com/scholar?q=A%2E+P%2E+Dawid+and+A%2E+M%2E+Skene%2E+1979%2E+Maximum+Likelihood+Estimation+of+Observer+Error%2DRates+Using+the+EM+Algorithm%2E+Journal+of+the+Royal+Statistical+Society%2E+Series+C+%28Applied+Statistics%29%2C+28%281%29%3A20%E2%80%9328%2E+P)
8. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé, and Kate Crawford. 2021. Datasheets for Datasets. arXiv preprint. ArXiv:1803.09010 [cs]. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daum%C3%A9%2C+and+Kate+Crawford%2E+2021%2E+Datasheets+for+Datasets%2E+arXiv+preprint%2E+ArXiv%3A1803%2E09010+%5Bcs%5D%2E)
9. Daniil Ignatev, Denis Paperno, and Massimo Poesio. 2025. Hypernetworks for Perspectivist Adaptation. arXiv preprint. ArXiv:2510.13259 [cs]. [find](https://scholar.google.com/scholar?q=Daniil+Ignatev%2C+Denis+Paperno%2C+and+Massimo+Poesio%2E+2025%2E+Hypernetworks+for+Perspectivist+Adaptation%2E+arXiv+preprint%2E+ArXiv%3A2510%2E13259+%5Bcs%5D%2E)
10. Gunther Jikeli, Sameer Karali, Daniel Miehling, and Katharina Soemer. 2023. Antisemitic Messages? A Guide to High-Quality Annotation and a Labeled Dataset of Tweets. arXiv preprint. ArXiv:2304.14599 [cs]. — [arXiv:2304.14599](https://arxiv.org/abs/2304.14599) [find](https://scholar.google.com/scholar?q=Gunther+Jikeli%2C+Sameer+Karali%2C+Daniel+Miehling%2C+and+Katharina+Soemer%2E+2023%2E+Antisemitic+Messages%3F+A+Guide+to+High%2DQuality+Annotation+and+a+Labeled+Dataset+of+Tweets%2E+arXiv+preprint%2E+ArXiv%3A2304%2E14599+%5B)
11. Katerina Korre, Arianna Muti, Federico Ruggeri, and Alberto Barrón-Cedeño. 2025. Untangling Hate Speech Definitions: A Semantic Componential Analysis Across Cultures and Domains. arXiv preprint. ArXiv:2411.07417 [cs]. [find](https://scholar.google.com/scholar?q=Katerina+Korre%2C+Arianna+Muti%2C+Federico+Ruggeri%2C+and+Alberto+Barr%C3%B3n%2DCede%C3%B1o%2E+2025%2E+Untangling+Hate+Speech+Definitions%3A+A+Semantic+Componential+Analysis+Across+Cultures+and+Domains%2E+arXiv+preprint%2E+ArXiv)
12. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://md.research.elamur.ai/free/en/papers/davani-2022-dealing-disagreements-looking-beyond-majority/). Transactions of the Association for Computational Linguistics, 10:92–110. Place: Cambridge, MA Publisher: MIT Press.
13. Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. 2021. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. arXiv preprint. ArXiv:2103.14749 [stat]. [find](https://scholar.google.com/scholar?q=Curtis+G%2E+Northcutt%2C+Anish+Athalye%2C+and+Jonas+Mueller%2E+2021%2E+Pervasive+Label+Errors+in+Test+Sets+Destabilize+Machine+Learning+Benchmarks%2E+arXiv+preprint%2E+ArXiv%3A2103%2E14749+%5Bstat%5D%2E)
14. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. [Comparing Bayesian Models of Annotation](https://md.research.elamur.ai/free/en/papers/paun-2018-comparing-bayesian-models-annotation/). Transactions of the Association for Computational Linguistics, 6:571–585. Place: Cambridge, MA Publisher: MIT Press.
15. Ellie Pavlick and Tom Kwiatkowski. 2019. [Inherent Disagreements in Human Textual Inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/). Transactions of the Association for Computational Linguistics, 7:677–694. Place: Cambridge, MA Publisher: MIT Press.
16. Barbara Plank. 2022. [The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
17. Joan Plepi, Béla Neuendorf, Lucie Flek, and Charles Welch. 2022. Unifying Data Perspectivism and Personalization: An Application to Social Norms. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7391–7402, Abu Dhabi, United Arab Emirates. Association f… [find](https://scholar.google.com/scholar?q=Joan+Plepi%2C+B%C3%A9la+Neuendorf%2C+Lucie+Flek%2C+and+Charles+Welch%2E+2022%2E+Unifying+Data+Perspectivism+and+Personalization%3A+An+Application+to+Social+Norms%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Met)
18. Federico Ruggeri, Eleonora Misino, Arianna Muti, Katerina Korre, Paolo Torroni, and Alberto Barrón-Cedeño. 2024. Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology. arXiv preprint. ArXiv:2406.14099 [cs]. — [arXiv:2406.14099](https://arxiv.org/abs/2406.14099) [find](https://scholar.google.com/scholar?q=Federico+Ruggeri%2C+Eleonora+Misino%2C+Arianna+Muti%2C+Katerina+Korre%2C+Paolo+Torroni%2C+and+Alberto+Barr%C3%B3n%2DCede%C3%B1o%2E+2024%2E+Let+Guidelines+Guide+You%3A+A+Prescriptive+Guideline%2DCentered+Data+Annotation+Methodology)
19. Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. [Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks](https://md.research.elamur.ai/free/en/papers/rottger-2022-two-contrasting-data-annotation-paradigms/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190,…
20. Joni Salminen, Hind Almerekhi, Milica Milenković, Soon-gyo Jung, Jisun An, Haewoon Kwak, and Bernard Jansen. 2018. Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for Identifying and Classifying Hate in Online News Media. Proceedings of the International AAAI Conference on… [find](https://scholar.google.com/scholar?q=Joni+Salminen%2C+Hind+Almerekhi%2C+Milica+Milenkovic%CC%81%2C+Soon%2Dgyo+Jung%2C+Jisun+An%2C+Haewoon+Kwak%2C+and+Bernard+Jansen%2E+2018%2E+Anatomy+of+Online+Hate%3A+Developing+a+Taxonomy+and+Machine+Learning+Models+for+Identi)
21. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://md.research.elamur.ai/free/en/papers/sap-2020-social-bias-frames-reasoning-about/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online. Association…
22. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p… [find](https://scholar.google.com/scholar?q=Swabha+Swayamdipta%2C+Roy+Schwartz%2C+Nicholas+Lourie%2C+Yizhong+Wang%2C+Hannaneh+Hajishirzi%2C+Noah+A%2E+Smith%2C+and+Yejin+Choi%2E+2020%2E+Dataset+Cartography%3A+Mapping+and+Diagnosing+Datasets+with+Training+Dynamics%2E+)
23. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72:1385– 1470. [find](https://scholar.google.com/scholar?q=Alexandra+N%2E+Uma%2C+Tommaso+Fornaciari%2C+Dirk+Hovy%2C+Silviu+Paun%2C+Barbara+Plank%2C+and+Massimo+Poesio%2E+2021%2E+Learning+from+Disagreement%3A+A+Survey%2E+Journal+of+Artificial+Intelligence+Research%2C+72%3A1385%E2%80%93+1470%2E)

*Updated 1 November 2026*