---
title: "Understanding and Predicting Human Label Variation in Natural"
description: "Earlier work proposed a post-hoc taxonomy of linguistic reasons for it, but nobody had asked the… 36 references, cited by 7 papers here."
authors: ["Nan-Jiang Jiang", "Chenhao Tan", "Marie-Catherine de Marneffe"]
year: 2023
venue: "arXiv:2304.12443"
source: "https://arxiv.org/abs/2304.12443"
doi: "10.48550/arxiv.2304.12443"
arxiv: "2304.12443"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jiang-2023-understanding-predicting-human-label-variation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jiang-2023-understanding-predicting-human-label-variation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation

# Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation

**Authors:** [Nan-Jiang Jiang](https://md.research.elamur.ai/free/en/authors/nan-jiang-jiang/), [Chenhao Tan](https://md.research.elamur.ai/free/en/authors/chenhao-tan/) and [Marie-Catherine de Marneffe](https://md.research.elamur.ai/free/en/authors/marie-catherine-de-marneffe/) · **Year:** 2023 · **Venue:** arXiv:2304.12443 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.12443) · [PDF](https://arxiv.org/pdf/2304.12443) · [DOI 10.48550/arxiv.2304.12443](https://doi.org/10.48550/arxiv.2304.12443) · [arXiv 2304.12443](https://arxiv.org/abs/2304.12443) · [OpenAlex](https://openalex.org/W4367060592)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/jiang-2023-understanding-predicting-human-label-variation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Annotators often disagree on Natural Language Inference (NLI) labels, and this disagreement is systematic rather than random noise. Earlier work proposed a post-hoc taxonomy of linguistic reasons for it, but nobody had asked the annotators themselves. The authors built LiveNLI: 122 English items drawn from MNLI, each re-annotated by at least 10 people. Every annotator chose one or more labels, marked highlights, and wrote a free-text explanation of their own choice, which makes the explanations ecologically valid.

The explanations largely confirm the existing taxonomy, but they also show:
- reasons the taxonomy did not cover;
- variation within a single label, where people pick the same answer for different reasons;
- that highlights alone explain little.

The authors then prompted GPT-3 in chain-of-thought style with these explanations to predict label distributions. Explanations helped slightly over plain prediction, but both the predicted distributions and the generated explanations remain weak. The resource matters for building models that represent and justify disagreement instead of collapsing it to one label.

## Key points
- LiveNLI has 122 re-annotated MNLI items with ≥10 annotations each, including multi-label choices, highlights and free-text explanations. Comparable datasets such as e-SNLI offer at most about 5 explanations per item.
- The analysis is checked against the 10-category taxonomy of Jiang and de Marneffe (2022), including categories like "Lexical" and "Probabilistic Enrichment."
- The Predict-only GPT-3 setup is over-confident: on 40% of items it assigns zero probability to some label. Generating explanations first yields flatter, closer distributions under KL and TVD.
- On a 70-item test split with 218 generated explanations (Explain-then-Predict):
  - 32 items contained at least one bad explanation (29 implausible, 7 misdescribing the input);
  - the labels attached to bad explanations got a mean predicted probability of 0.19 versus a true probability of 0.10.
- The Uniform baseline wins on KL, because KL punishes zero-probability predictions extremely harshly. The authors argue KL differences are therefore uninformative when models fall below uniform, and JSD is gentler.

## Interesting ideas
- Have the same person who picks a label also justify it, instead of asking someone to rationalize a pre-existing gold label; this keeps the explanation tied to the actual decision.
- Treat annotator disagreement as a target distribution to predict, not noise to vote away, and keep it for downstream use.
- Sort explanations by the implicit question each reader was answering (a QUD), which reveals why one item yields several readings, e.g., whether "most" implies a large number.
- The same label can be reached by different reasoning, so checking only the final answer hides real differences in understanding.
- Always include a uniform baseline and pick metrics aware of zero-probability penalties, since KL can make a trivial guess look best.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): when LLM teacher tiers produce silver labels, ask each teacher to explain its own chosen analysis rather than defend the majority one, so disagreements carry usable reasons.
- **I2** (learn): keep the spread of votes among witness parsers as soft targets and confidence signals instead of discarding non-unanimous arcs.
- **I3** (frames): when an implicature or quantifier enrichment is uncertain, store alternative readings with their triggering question instead of committing to one inference.
- **I4** (answer): grade fable QA and word problems on the explained reasoning path as well as the answer, flagging right answers reached for wrong reasons.
- **I5** (learn): add uniform and other trivial predictors as negative controls in evaluation, and report JSD/TVD alongside KL for any distributional output.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The Ohio State University](https://md.research.elamur.ai/free/en/universities/the-ohio-state-university/) — University, Columbus, United States · 82 works · 18,502 citations
- [University of Chicago](https://md.research.elamur.ai/free/en/universities/university-of-chicago/) — University, Chicago, United States · 45 works · 50,617 citations
- [UCLouvain](https://md.research.elamur.ai/free/en/universities/uclouvain/) — University, Louvain-la-Neuve, Belgium · 20 works · 3,741 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Nan-Jiang Jiang, Chenhao Tan and Marie-Catherine de Marneffe. 2023. Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation. arXiv:2304.12443.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jiang2023understanding,
  title = {{Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation}},
  author = {Nan-Jiang Jiang, Chenhao Tan and Marie-Catherine {de Marneffe}},
  year = {2023},
  doi = {10.48550/arxiv.2304.12443},
  eprint = {2304.12443},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.12443}
}
```

- [← previous Towards Explainable AI Writing Assistants for Non-native English Speakers](https://md.research.elamur.ai/free/en/papers/kim-2023-towards-explainable-ai-writing-assistants/)
- [next → We Understand Elliptical Sentences, and Language Models should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit](https://md.research.elamur.ai/free/en/papers/testa-2023-we-understand-elliptical-sentences-language/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

36 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lora Aroyo and Chris Welty. 2015. [Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation](https://md.research.elamur.ai/free/en/papers/aroyo-2015-truth-lie-crowd-truth-seven/). AI Magazine, 36(1):15. [↗](https://doi.org/10.1609/aimag.v36i1.2564)
2. Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892– 1915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Joris+Baan%2C+Wilker+Aziz%2C+Barbara+Plank%2C+and+Raquel+Fernandez%2E+2022%2E+Stop+measuring+calibration+when+humans+disagree%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d15-1075)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, volume 33, pages 1877– 1901. Curran Associates, Inc.
5. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. [e-snli: Natural language inference with natural language explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.1812.01193)
6. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2020. Make up your mind! adversarial generation of inconsistent natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4157–4165, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Oana%2DMaria+Camburu%2C+Brendan+Shillingford%2C+Pasquale+Minervini%2C+Thomas+Lukasiewicz%2C+and+Phil+Blunsom%2E+2020%2E+Make+up+your+mind%21+adversarial+generation+of+inconsistent+natural+language+explanations%2E+In+Pr)
7. Kordula De Kuthy, Nils Reiter, and Arndt Riester. 2018. [QUD-based annotation of discourse structure and information structure: Tool and evaluation](https://md.research.elamur.ai/free/en/papers/kuthy-2018-qud-based-annotation-discourse-structure/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [↗](https://doi.org/10.63317/368a5yfgwg9q)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
9. Virginie Do, Oana-Maria Camburu, Zeynep Akata, and Thomas Lukasiewicz. 2020. esnli-ve-2.0: Corrected visual-textual entailment with natural language explanations. CoRR, abs/2004.03744. [↗](https://arxiv.org/abs/2004.03744)
10. Peter Hase and Mohit Bansal. 2022. When can models learn from explanations? a formal framework for understanding the roles of explanation data. In Proceedings of the First Workshop on Learning with Natural Language Supervision, pages 29–39, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.48448/06xz-jg93)
11. Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. [Investigating Reasons for Disagreement in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/jiang-2022-investigating-reasons-disagreement-natural-language/). Transactions of the Association for Computational Linguistics, 10:1357–1374. [↗](https://doi.org/10.1162/tacl_a_00523)
12. Sawan Kumar and Partha Talukdar. 2020. [NILE : Natural language inference with faithful natural language explanations](https://md.research.elamur.ai/free/en/papers/kumar-2020-nile-natural-language-inference-faithful/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8730–8742, Online. Association for Computational Linguistics.
13. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. [Did it happen? the pragmatic complexity of veridicality assessment](https://md.research.elamur.ai/free/en/papers/marneffe-2012-did-it-happen-pragmatic-complexity/). Computational Linguistics, 38(2):301–333.
14. Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. WT5?! Training Text-to-Text Models to Explain their Predictions. arXiv:2004.14546 [cs]. ArXiv: 2004.14546. [↗](https://arxiv.org/abs/2004.14546)
15. Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Dipanjan Das, and Mirella Lapata. 2022. [Conditional generation with a question-answering blueprint](https://md.research.elamur.ai/free/en/papers/narayan-2023-conditional-generation-question-answering-blueprint/). [↗](https://doi.org/10.1162/tacl_a_00583)
16. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. [What can we learn from collective human opinions on natural language inference data?](https://md.research.elamur.ai/free/en/papers/nie-2020-what-can-we-learn-collective/) In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics.
17. Ellie Pavlick and Tom Kwiatkowski. 2019. [Inherent disagreements in human textual inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/). Transactions of the Association for Computational Linguistics, 7:677–694. [↗](https://doi.org/10.1162/tacl_a_00293)
18. Barbara Plank. 2022. [The “problem” of human label variation: On ground truth in data, modeling and evaluation](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671– 10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.emnlp-main.731)
19. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. [Learning part-of-speech taggers with inter-annotator agreement loss](https://md.research.elamur.ai/free/en/papers/plank-2014-learning-part-speech-taggers-inter/). In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751.
20. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the AR-RAU corpus. In Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference, pages 11–22, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Massimo+Poesio%2C+Yulia+Grishina%2C+Varada+Kolhatkar%2C+Nafise+Moosavi%2C+Ina+Roesiger%2C+Adam+Roussel%2C+Fabian+Simonjetz%2C+Alexandra+Uma%2C+Olga+Uryupina%2C+Juntao+Yu%2C+and+Heike+Zinsmeister%2E+2018%2E+Anaphora+resolutio)
21. Craige Roberts. 2012. [Information structure in discourse: Towards an integrated formal theory of pragmatics](https://md.research.elamur.ai/free/en/papers/roberts-2012-information-structure-discourse-towards-integrated/). Semantics and Pragmatics, 5. [↗](https://doi.org/10.3765/sp.5.6)
22. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. [Social bias frames: Reasoning about social and power implications of language](https://md.research.elamur.ai/free/en/papers/sap-2020-social-bias-frames-reasoning-about/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online. Association for Computational Linguistics.
23. Jamar Sullivan Jr., Will Brackenbury, Andrew Mc-Nutt, Kevin Bryson, Kwam Byll, Yuxin Chen, Michael Littman, Chenhao Tan, and Blase Ur. 2022. Explaining why: How instructions and user interfaces impact annotator rationales when labeling text data. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 521–531, Seattle, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jamar+Sullivan+Jr%2E%2C+Will+Brackenbury%2C+Andrew+Mc%2DNutt%2C+Kevin+Bryson%2C+Kwam+Byll%2C+Yuxin+Chen%2C+Michael+Littman%2C+Chenhao+Tan%2C+and+Blase+Ur%2E+2022%2E+Explaining+why%3A+How+instructions+and+user+interfaces+impact)
24. Chenhao Tan. 2022. On the diversity and limits of human explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2173–2188, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.48448/aphg-ya28)
25. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. [↗](https://doi.org/10.1613/jair.1.12752)
26. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems.
27. Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing human-AI collaboration for generating free-text explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 632–658, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2112.08674)
28. Sarah Wiegreffe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). [↗](https://doi.org/10.48550/arxiv.2102.12060)
29. Sarah Wiegreffe, Ana Marasović, and Noah A. Smith. 2021. [Measuring association between labels and free-text rationales](https://md.research.elamur.ai/free/en/papers/wiegreffe-2021-measuring-association-between-labels-free/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10266– 10284, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
30. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-1101)
31. Xi Ye and Greg Durrett. 2022. [The unreliability of explanations in few-shot prompting for textual reasoning](https://md.research.elamur.ai/free/en/papers/ye-2022-unreliability-explanations-few-shot-prompting/). In Advances in Neural Information Processing Systems.
32. Yordan Yordanov, Vid Kocijan, Thomas Lukasiewicz, and Oana-Maria Camburu. 2021. Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations. arXiv:2112.06204 [cs]. ArXiv: 2112.06204. [↗](https://arxiv.org/abs/2112.06204)
33. Hongming Zhang, Xinran Zhao, and Yangqiu Song. 2020. WinoWhy: A deep diagnosis of essential commonsense knowledge for answering Winograd schema challenge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5736–5745, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hongming+Zhang%2C+Xinran+Zhao%2C+and+Yangqiu+Song%2E+2020%2E+WinoWhy%3A+A+deep+diagnosis+of+essential+commonsense+knowledge+for+answering+Winograd+schema+challenge%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+)
34. Yi Zhang, Sujay Kumar Jauhar, Julia Kiseleva, Ryen White, and Dan Roth. 2021. Learning to decompose and organize complex tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2726–2735, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yi+Zhang%2C+Sujay+Kumar+Jauhar%2C+Julia+Kiseleva%2C+Ryen+White%2C+and+Dan+Roth%2E+2021%2E+Learning+to+decompose+and+organize+complex+tasks%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+t)
35. Xinyan Zhao and V.G.Vinod Vydiswaran. 2021. Lirex: Augmenting language inference with relevant explanations. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14532–14539. [find](https://scholar.google.com/scholar?q=Xinyan+Zhao+and+V%2EG%2EVinod+Vydiswaran%2E+2021%2E+Lirex%3A+Augmenting+language+inference+with+relevant+explanations%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+35%2816%29%3A14532%E2%80%9314539%2E)
36. Xiang Zhou, Yixin Nie, and Mohit Bansal. 2022. [Distributed NLI: Learning to predict human opinion distributions for language reasoning](https://md.research.elamur.ai/free/en/papers/zhou-2022-distributed-nli-learning-predict-human/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 972–987, Dublin, Ireland. Association for Computational Linguistics.

*Updated 1 November 2026*