---
title: "Ecologically Valid Explanations for Label Variation in NLI (2023)"
description: "NLI datasets usually assume one correct label per premise–hypothesis pair, yet annotators often… 28 references, cited by 19 papers here."
authors: ["Nan-Jiang Jiang", "Chenhao Tan", "Marie-Catherine de Marneffe"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: EMNLP 2023"
source: "https://aclanthology.org/2023.findings-emnlp.712/"
doi: "10.18653/v1/2023.findings-emnlp.712"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jiang-2023-ecologically-valid-explanations-label-variation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jiang-2023-ecologically-valid-explanations-label-variation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Ecologically Valid Explanations for Label Variation in NLI

# Ecologically Valid Explanations for Label Variation in NLI

**Authors:** [Nan-Jiang Jiang](https://md.research.elamur.ai/free/en/authors/nan-jiang-jiang/), [Chenhao Tan](https://md.research.elamur.ai/free/en/authors/chenhao-tan/) and [Marie-Catherine de Marneffe](https://md.research.elamur.ai/free/en/authors/marie-catherine-de-marneffe/) · **Year:** 2023 · **Venue:** [Findings of the Association for Computational Linguistics: EMNLP 2023](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/findings/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-emnlp.712/) · [PDF](https://aclanthology.org/2023.findings-emnlp.712.pdf) · [DOI 10.18653/v1/2023.findings-emnlp.712](https://doi.org/10.18653/v1/2023.findings-emnlp.712) · [OpenAlex](https://openalex.org/W4389520277)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [19](https://md.research.elamur.ai/free/en/papers/jiang-2023-ecologically-valid-explanations-label-variation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
NLI datasets usually assume one correct label per premise–hypothesis pair, yet annotators often disagree, and it has been unclear where that disagreement comes from. The authors built LiveNLI, a small English resource suited to few-shot evaluation. They took 122 MNLI items and had annotators choose one or more labels, write a free-text justification for each label, and highlight the words that mattered. Because each explanation comes from the person who picked the label, rather than being written to justify a fixed gold answer as in e-SNLI, the explanations are "ecologically valid." The data shows that disagreement is widespread and largely systematic. Many disagreements come from readers assuming different Questions Under Discussion (QUDs). Readers also sometimes reach the same label through quite different reasoning, which the authors call within-label variation. Few-shot prompting of text-davinci-003 to predict and explain multilabel judgments gave mixed results. Some explanations were fluent and useful, but many were implausible, uninformative, or did not support the predicted label. The work argues that explanations are needed to understand, and eventually model, how people interpret text differently.

## Key points
- **LiveNLI composition:** 122 MNLI dev items, each with at least 10 label explanations, for 1,415 explanations in total. These were written by 48 native English speakers recruited through Surge AI. The items fall into three groups:
  - 60 items where only two of the original five MNLI annotators agreed;
  - 50 items with three agreeing, also covered in ChaosNLI;
  - 12 items with full original agreement.
- **Variation in agreement items:** label variation appears even on the 12 originally unanimous items, and the dissenting labels came with legitimate explanations.
- **Annotator vs. linguist reasons:** re-coding the reasons using Jiang and de Marneffe's (2022) taxonomy gave low agreement with the linguists' earlier coding (Krippendorff's α = 0.30 with MASI distance). The new coding also assigned more categories per item (1.33 vs. 1.22).
- **New QUD category:** it was added to the taxonomy, appeared in 28 of 122 items, and ranked third in frequency. Within-label variation was found in 16 cases, about 13% of the data.
- **GPT-3 experiments (text-davinci-003):**
  - Setup: 11 label-balanced few-shot examples, 259 test items, and gold multilabels taken as labels with probability above 0.2.
  - Prompting with explanations (Predict-then-Explain, Explain-then-Predict) did not beat predict-only. Exact-match accuracy was about 25 / 25 / 20.
  - In a 35-item manual check, roughly 80% of explanations supported their label, but only about 60% were also plausible. Support was weakest for Contradiction (67.8% vs. 88.2% for other labels).

## Interesting ideas
- Collect the label and its justification from the same annotator, instead of asking people to rationalize a fixed gold answer, so the explanation reflects the reasoning that actually produced the label.
- Treat systematic disagreement as signal, and represent the gold answer as a multilabel set using a probability threshold (e.g., 0.2) rather than a single class.
- Explain disagreements by which Question Under Discussion a reader assumes the sentence addresses, since the same hypothesis can be judged on different at-issue parts.
- Track within-label variation: identical labels can rest on different assumptions, such as whether two mentions corefer.
- Evaluate generated explanations on separate binary criteria (plausible, supports the label, specific, adds new information), with the model identity hidden from the judge.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): When LLM teachers or witness parsers produce silver labels, require each to justify its own choice, and keep disagreements with their rationales instead of collapsing them to a majority vote.
- **I2** (answer): School-test and fable QA readers could return a set of defensible answers, each tied to an explicit reading, when the text truly supports more than one interpretation.
- **I3** (frames): The pragmatics layer could mark the at-issue content of a clause and keep alternative QUD readings, so entailment judgments can state which question they answer.
- **I4** (world): When building the text world, keep competing coreference or same-event hypotheses as separate branches and record which branch supports each conclusion, rather than only the final verdict.
- **I5** (learn): Mova's explainability traces could be scored on the same axes, catching explanations that merely restate the input or fail to justify the answer they accompany.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The Ohio State University](https://md.research.elamur.ai/free/en/universities/the-ohio-state-university/) — University, Columbus, United States · 82 works · 18,502 citations
- [University of Chicago](https://md.research.elamur.ai/free/en/universities/university-of-chicago/) — University, Chicago, United States · 45 works · 50,617 citations
- [UCLouvain](https://md.research.elamur.ai/free/en/universities/uclouvain/) — University, Louvain-la-Neuve, Belgium · 20 works · 3,741 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Nan-Jiang Jiang, Chenhao Tan and Marie-Catherine de Marneffe. 2023. Ecologically Valid Explanations for Label Variation in NLI. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jiang2023ecologically,
  title = {{Ecologically Valid Explanations for Label Variation in NLI}},
  author = {Nan-Jiang Jiang, Chenhao Tan and Marie-Catherine {de Marneffe}},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.712},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.712}
}
```

- [← previous Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://md.research.elamur.ai/free/en/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/)
- [next → Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model](https://md.research.elamur.ai/free/en/papers/chintam-2023-identifying-adapting-transformer-components-responsible/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

28 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lora Aroyo and Chris Welty. 2015. [Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation](https://md.research.elamur.ai/free/en/papers/aroyo-2015-truth-lie-crowd-truth-seven/). AI Magazine, 36(1):15. [↗](https://doi.org/10.1609/aimag.v36i1.2564)
2. BIG-Bench authors. 2023. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Transactions on Machine Learning Research.
3. Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892–1915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Joris+Baan%2C+Wilker+Aziz%2C+Barbara+Plank%2C+and+Raquel+Fernandez%2E+2022%2E+Stop+measuring+calibration+when+humans+disagree%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
4. Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. [Investigating Reasons for Disagreement in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/jiang-2022-investigating-reasons-disagreement-natural-language/). Transactions of the Association for Computational Linguistics, 10:1357– 1374. [↗](https://doi.org/10.1162/tacl_a_00523)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
6. Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. 2022. [Can language models learn from explanations in context?](https://md.research.elamur.ai/free/en/papers/lampinen-2022-can-language-models-learn-explanations/) In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537–563, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2204.02329)
7. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. [e-snli: Natural language inference with natural language explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.1812.01193)
8. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2020. Make up your mind! adversarial generation of inconsistent natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4157– 4165, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Oana%2DMaria+Camburu%2C+Brendan+Shillingford%2C+Pasquale+Minervini%2C+Thomas+Lukasiewicz%2C+and+Phil+Blunsom%2E+2020%2E+Make+up+your+mind%21+adversarial+generation+of+inconsistent+natural+language+explanations%2E+In+Pr)
9. Kordula De Kuthy, Nils Reiter, and Arndt Riester. 2018. [QUD-based annotation of discourse structure and information structure: Tool and evaluation](https://md.research.elamur.ai/free/en/papers/kuthy-2018-qud-based-annotation-discourse-structure/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [↗](https://doi.org/10.63317/368a5yfgwg9q)
10. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161–1166, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Yoav+Goldberg%2C+and+Jonathan+Berant%2E+2019%2E+Are+we+modeling+the+task+or+the+annotator%3F+an+investigation+of+annotator+bias+in+natural+language+understanding+datasets%2E+In+Proceedings+of+the+2019)
11. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. [Annotation artifacts in natural language inference data](https://md.research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2017)
12. David B. Leake. 1991. Goal-based explanation evaluation. Cognitive Science, 15(4):509–545. [↗](https://doi.org/10.1207/s15516709cog1504_2)
13. Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2023. [We’re afraid language models aren’t modeling ambiguity](https://md.research.elamur.ai/free/en/papers/liu-2023-we-re-afraid-language-models/). [↗](https://doi.org/10.48550/arxiv.2304.14399)
14. Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Dipanjan Das, and Mirella Lapata. 2022. [Conditional generation with a question-answering blueprint](https://md.research.elamur.ai/free/en/papers/narayan-2023-conditional-generation-question-answering-blueprint/). [↗](https://doi.org/10.1162/tacl_a_00583)
15. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. [What can we learn from collective human opinions on natural language inference data?](https://md.research.elamur.ai/free/en/papers/nie-2020-what-can-we-learn-collective/) In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics.
16. Ellie Pavlick and Tom Kwiatkowski. 2019. [Inherent disagreements in human textual inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/). Transactions of the Association for Computational Linguistics, 7:677–694. [↗](https://doi.org/10.1162/tacl_a_00293)
17. Barbara Plank. 2022. [The “problem” of human label variation: On ground truth in data, modeling and evaluation](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.emnlp-main.731)
18. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. [Learning part-of-speech taggers with inter-annotator agreement loss](https://md.research.elamur.ai/free/en/papers/plank-2014-learning-part-speech-taggers-inter/). In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751.
19. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. [Anaphora resolution with the ARRAU corpus](https://md.research.elamur.ai/free/en/papers/poesio-2018-anaphora-resolution-arrau-corpus/). In Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference, pages 11–22, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w18-0702)
20. Craige Roberts. 2012. [Information structure in discourse: Towards an integrated formal theory of pragmatics](https://md.research.elamur.ai/free/en/papers/roberts-2012-information-structure-discourse-towards-integrated/). Semantics and Pragmatics, 5. [↗](https://doi.org/10.3765/sp.5.6)
21. Chenhao Tan. 2022. On the diversity and limits of human explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2173–2188, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.48448/aphg-ya28)
22. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. [↗](https://doi.org/10.1613/jair.1.12752)
23. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems.
24. Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing human-AI collaboration for generating free-text explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 632–658, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2112.08674)
25. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-1101)
26. Xi Ye and Greg Durrett. 2022. [The unreliability of explanations in few-shot prompting for textual reasoning](https://md.research.elamur.ai/free/en/papers/ye-2022-unreliability-explanations-few-shot-prompting/). In Advances in Neural Information Processing Systems.
27. Yi Zhang, Sujay Kumar Jauhar, Julia Kiseleva, Ryen White, and Dan Roth. 2021. Learning to decompose and organize complex tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2726–2735, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yi+Zhang%2C+Sujay+Kumar+Jauhar%2C+Julia+Kiseleva%2C+Ryen+White%2C+and+Dan+Roth%2E+2021%2E+Learning+to+decompose+and+organize+complex+tasks%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+t)
28. Xiang Zhou, Yixin Nie, and Mohit Bansal. 2022. [Distributed NLI: Learning to predict human opinion distributions for language reasoning](https://md.research.elamur.ai/free/en/papers/zhou-2022-distributed-nli-learning-predict-human/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 972–987, Dublin, Ireland. Association for Computational Linguistics.

*Updated 1 November 2026*