---
title: "More Labels or Cases? Assessing Label Variation in Natural"
description: "The paper argues that summarizing many crowd votes in NLI, either as a majority label or as raw class… 9 references, cited by 8 papers here."
authors: ["Cornelia Gruber", "Katharina Hechinger", "Matthias Assenmacher", "Göran Kauermann", "Barbara Plank"]
year: 2024
venue: "Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language"
source: "https://aclanthology.org/2024.unimplicit-1.2/"
doi: "10.18653/v1/2024.unimplicit-1.2"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gruber-2024-more-labels-or-cases-assessing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gruber-2024-more-labels-or-cases-assessing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) / More Labels or Cases? Assessing Label Variation in Natural Language Inference

# More Labels or Cases? Assessing Label Variation in Natural Language Inference

**Authors:** Cornelia Gruber, Katharina Hechinger, [Matthias Assenmacher](https://md.research.elamur.ai/free/en/authors/matthias-assenmacher/), Göran Kauermann and [Barbara Plank](https://md.research.elamur.ai/free/en/authors/barbara-plank/) · **Year:** 2024 · **Venue:** [Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language](https://md.research.elamur.ai/free/en/venues/unimplicit/) ([Workshop on Understanding Implicit and Underspecified Language 2024](https://md.research.elamur.ai/free/en/venues/unimplicit/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.unimplicit-1.2/) · [PDF](https://aclanthology.org/2024.unimplicit-1.2.pdf) · [DOI 10.18653/v1/2024.unimplicit-1.2](https://doi.org/10.18653/v1/2024.unimplicit-1.2) · [OpenAlex](https://openalex.org/W7126419655)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/gruber-2024-more-labels-or-cases-assessing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper argues that summarizing many crowd votes in NLI, either as a majority label or as raw class frequencies, gives an unreliable picture of how uncertain an instance's label really is. The authors analyze ChaosSNLI: 1,514 SNLI development pairs, each relabeled by 100 quality-controlled annotators. They fit a Bayesian multinomial mixture model, estimated with an EM procedure. The model treats the votes as generated by hidden "true" classes and returns a posterior probability over those classes for every instance. They plot each instance in a ternary diagram and compare the borders of the latent classes with the borders implied by majority voting. The latent borders move in ways that make sense semantically. A bootstrap study then shrinks either the number of instances or the number of votes per instance. Class borders stay fairly stable with few instances, as long as each has many votes. With many instances but few votes, estimation breaks down. The practical lesson is that, when label variation is expected, annotation budgets should go to more labels per item.

## Key points
- The data is ChaosSNLI, the SNLI part of ChaosNLI, with N = 1,514 instances and J = 100 votes each. The instances are cases where the original five SNLI annotators split 3–2.
- By majority vote, 53.7% of instances are neutral, 27.8% entailment and 18.5% contradiction. This suggests neutral is the class people find hardest to agree on.
- The model is a pooled one, meaning all annotators are assumed equally skilled. Pooling is unavoidable here because ChaosSNLI has no annotator IDs.
- The stability tests used J = 100 with N ∈ {50, 100, 250, 500}, and N = 1000 with J ∈ {5, 10, 25, 50}. Class borders were bootstrapped and compared with the borders from the full data.
- Code and data are released at github.com/corneliagru/label-variation-nli.

## Interesting ideas
- Instead of keeping a winner-takes-all label, treat each item's vote counts as a draw from a mixture over hidden classes, and let the whole dataset decide where class boundaries lie.
- For a three-way decision, place every item in a ternary plot by its vote shares and overlay both the majority borders and the learned borders. This makes it visible where the two kinds of borders disagree.
- Test how stable an estimate is by bootstrapping subsamples along two separate axes: fewer items versus fewer votes per item. This shows which resource actually limits reliability.
- Spend annotation effort on many judgments for contested cases, not on broad coverage with thin labeling, since the uncertainty lives mainly in the votes.
- When annotator identities are known, measure their effect by removing one annotator's votes and refitting the model.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Agreement counts from the witness parsers on each arc or deprel could feed such a mixture, so silver labels become posteriors with calibrated confidence rather than plain majority picks.
- **I2** (answer): The entailed / unknown / contradicted verdicts of the readers could be audited this way across several runs or teachers, exposing zones where the readers are systematically confused.
- **I3** (learn): Before growing the silver corpus, rerun this kind of experiment to decide whether adding witness parsers or LLM teacher calls beats adding more sentences.
- **I4** (learn): Route sentences where witnesses disagree to extra teacher tiers or repeated queries, and accept a single cheap label where witnesses already agree.
- **I5** (tree): Leave-one-witness-out refits could reveal which parser or UD dialect conversion biases the agreed silver trees, and by how much.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Ludwig-Maximilians-Universität München](https://md.research.elamur.ai/free/en/universities/ludwig-maximilians-universitat-munchen/) — University, Munich, Germany · 113 works · 56,817 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Cornelia Gruber, Katharina Hechinger, Matthias Assenmacher, Göran Kauermann and Barbara Plank. 2024. More Labels or Cases? Assessing Label Variation in Natural Language Inference. Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gruber2024more,
  title = {{More Labels or Cases? Assessing Label Variation in Natural Language Inference}},
  author = {Cornelia Gruber, Katharina Hechinger, Matthias Assenmacher, Göran Kauermann and Barbara Plank},
  booktitle = {Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language},
  year = {2024},
  doi = {10.18653/v1/2024.unimplicit-1.2},
  url = {https://doi.org/10.18653/v1/2024.unimplicit-1.2}
}
```

- [← previous Manipulating language models' training data to study syntactic constraint learning: the case of English passivization](https://md.research.elamur.ai/free/en/papers/leong-2024-manipulating-language-models-training-data/)
- [next → MRL Parsing Without Tears: The Case of Hebrew](https://md.research.elamur.ai/free/en/papers/shmidman-2024-mrl-parsing-without-tears-case/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/)

## References

9 references, 3 of them with a card of ours. ↗ leads to the work.

1. Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty in natural language generation: From theory to applications. [↗](https://doi.org/10.48550/arxiv.2307.15703)
2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d15-1075)
3. Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1):1–22. [↗](https://doi.org/10.1111/j.2517-6161.1977.tb01600.x)
4. B. Efron. 1979. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1 – 26. [↗](https://doi.org/10.1214/aos/1176344552)
5. Cornelia Gruber, Patrick Oliver Schenk, Malte Schierholz, Frauke Kreuter, and Göran Kauermann. 2023. Sources of Uncertainty in Machine Learning – A Statisticians’ View. ArXiv:2305.16703 [cs, stat]. [↗](https://arxiv.org/abs/2305.16703)
6. Katharina Hechinger, Xiao Xiang Zhu, and Göran Kauermann. 2024. Categorising the world into local climate zones: towards quantifying labelling uncertainty for machine learning models. Journal of the Royal Statistical Society Series C: Applied Statistics, 73:143–161. [↗](https://doi.org/10.1093/jrsssc/qlad089)
7. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. [Learning Whom to Trust with MACE](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, Georgia. Association for Computational Linguistics.
8. Eyke Hüllermeier and Willem Waegeman. 2021. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 110(3):457–506. [↗](https://doi.org/10.1007/s10994-021-05946-3)
9. A Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. [Investigating reasons for disagreement in natural language inference](https://md.research.elamur.ai/free/en/papers/jiang-2022-investigating-reasons-disagreement-natural-language/). Transactions of the Association for Computational Linguistics, 10:1357–1374. [↗](https://doi.org/10.1162/tacl_a_00523)

*Updated 1 November 2026*