---
title: "Identifying inherent disagreement in natural language inference"
description: "They build this four-way version of the CommitmentBank (CB) from the raw spread of… 65.60 accuracy. 21 references, cited by 21 papers here."
authors: ["Xinliang Frederick Zhang", "Marie-Catherine de Marneffe"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.390/"
doi: "10.18653/v1/2021.naacl-main.390"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2021-identifying-inherent-disagreement-natural-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2021-identifying-inherent-disagreement-natural-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) / Identifying inherent disagreement in natural language inference

# Identifying inherent disagreement in natural language inference

**Authors:** [Xinliang Frederick Zhang](https://md.research.elamur.ai/free/en/authors/xinliang-frederick-zhang/) and [Marie-Catherine de Marneffe](https://md.research.elamur.ai/free/en/authors/marie-catherine-de-marneffe/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2021](https://md.research.elamur.ai/free/en/venues/naacl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.390/) · [PDF](https://aclanthology.org/2021.naacl-main.390.pdf) · [DOI 10.18653/v1/2021.naacl-main.390](https://doi.org/10.18653/v1/2021.naacl-main.390) · [OpenAlex](https://openalex.org/W3166002735)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · [21](https://md.research.elamur.ai/free/en/papers/zhang-2021-identifying-inherent-disagreement-natural-language/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most NLI work turns several human judgments into one label by averaging, or it drops items where people disagree. Zhang and de Marneffe argue that some items cause disagreement by nature, and that a model should recognise them as such. They add a fourth class, "Disagreement", next to Entailment, Neutral and Contradiction. They build this four-way version of the CommitmentBank (CB) from the raw spread of annotations, using the share of agreeing votes, the variance and the mean.

Their model, Artificial Annotators (AAs), has three BERT members. Each member learns from a separate slice of every item's sorted annotations, so each one imitates an annotator who leans towards entailment, neutral or contradiction. A small MLP combines the three outputs into the final label.

On the CB test set, AAs beat all baselines, including two BERT baselines, by a statistically significant margin. They also handle neg-raising and context-dependent cases better. Accuracy is still only about 66%, so the task remains largely unsolved. The work shows that disagreement can be modelled as a signal in its own right.

## Key points
- **Data.** CB has 1,200 discourses with at least 8 Likert ratings each, on a scale from −3 to +3. It is recast as NLI and split 7:1:2 into 840/120/240 items. Disagreement is the largest class.
- **Labeling rule.** An item gets a systematic label if 80% of ratings fall in that label's range. Otherwise it gets one if the variance is ≤ 1 and the mean passes a threshold: > 1, |mean| ≤ 0.5, or \< −1. All other items become Disagreement.
- **Model.** Each item's annotations are sorted from high to low and cut into three parts (3/2/3 for 8 ratings). The mean of each part gives an auxiliary label, with ±0.5 as the cut-off. Each BERT predicts one of these labels. The three [CLS] vectors are concatenated and passed to an MLP. Training uses a weighted sum of four cross-entropy losses.
- **Results.** On test, AAs reach 65.60 accuracy and 64.97 F1, against 62.61 and 62.07 for Joint BERT (t-test, p ≤ 0.01). Run-to-run variance is also lower. AAs are weaker on Neutral, conditional and question items, where training data is scarce.
- **Error analysis.**
  - About 60% of errors label a systematic item as Disagreement; in 91% of those, the three members disagree with each other.
  - When AAs miss a true Disagreement item, the members wrongly agree 78% of the time.
  - On "I don't know/believe/think" items, AAs score 77.26 F1 on the neg-raising contradictions versus 59.38 on the rest.

## Interesting ideas
- Treating "people genuinely split" as an explicit label, defined from the shape of the vote distribution, avoids collapsing a bimodal split into a misleading neutral average.
- Real annotator identities are not needed to simulate biased annotators: sorting each item's judgments and splitting them into ordered slices yields three viewpoints to train separate members on.
- Checking whether ensemble members agree gives an interpretable diagnostic: it shows whether the model saw one mode or several. Errors then split cleanly into "too many modes" and "too few modes".
- Speaker commitment is graded and shaped by context, as in embedding under negation, modals, questions and conditionals. Neg-raising with first-person *think/believe* is a key case, and context decides whether it fires.
- Test items can be split into those a simple linguistic heuristic already solves and those it misses. This shows whether a learned model adds context reasoning beyond the rules.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): When witness parsers produce silver data, arcs or labels with a split vote could be marked as contested rather than resolved by majority. Contested items would then be kept out of training or routed to the LLM teacher.
- **I2** (learn): Disagreeing teacher tiers or witnesses could be ordered and partitioned the same way, so that a combiner learns from the spread of opinions instead of a single target.
- **I3** (answer): Evaluation reports could record whether a reader's internal witnesses agreed, separating overconfident wrong answers from ambiguous ones.
- **I4** (frames): The prag crate could tag embedded clauses with a commitment value plus an "uncertain" option, and trigger neg-raising only when context supports it. This would avoid a fixed rule that always produces a contradiction.
- **I5** (learn): Evaluation could use the rule-only path as a baseline control and report gains separately on rule-solvable and rule-failing items.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The Ohio State University](https://md.research.elamur.ai/free/en/universities/the-ohio-state-university/) — University, Columbus, United States · 57 works · 17,793 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Xinliang Frederick Zhang and Marie-Catherine de Marneffe. 2021. Identifying inherent disagreement in natural language inference. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2021identifying,
  title = {{Identifying inherent disagreement in natural language inference}},
  author = {Xinliang Frederick Zhang and Marie-Catherine {de Marneffe}},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.390},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.390}
}
```

- [← previous DziriBERT: a Pre-trained Language Model for the Algerian Dialect](https://md.research.elamur.ai/free/en/papers/abdaoui-2021-dziribert-pre-trained-language-model/)
- [next → A unified approach to sentence segmentation of punctuated text in many languages](https://md.research.elamur.ai/free/en/papers/wicks-2021-unified-approach-sentence-segmentation-punctuated/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/)

## References

21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Belfer, Nirmal Kanagasabai, Roman Sarrazingendron, Rohit Verma, and Derek Ruths. 2018. Sentiment analysis: It’s complicated! In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, pages 1886–1895. [find](https://scholar.google.com/scholar?q=Belfer%2C+Nirmal+Kanagasabai%2C+Roman+Sarrazingendron%2C+Rohit+Verma%2C+and+Derek+Ruths%2E+2018%2E+Sentiment+analysis%3A+It%E2%80%99s+complicated%21+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+)
2. Hannah Youngeun An and Aaron Steven White. 2019. The lexical and grammatical sources of neg-raising inferences. In Proceedings of the Society for Computation in Linguistics (SCiL 2020), pages 220–233. [↗](https://doi.org/10.48550/arxiv.1908.05253)
3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pages 632–642. [↗](https://doi.org/10.48550/arxiv.1508.05326)
4. Adrian Calma and Bernhard Sick. 2017. Simulation of annotators for active learning: Uncertain Oracles. In Proceedings of the Workshop and Tutorial on Interactive Adaptive Learning co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2017), pages 49–58. [find](https://scholar.google.com/scholar?q=Adrian+Calma+and+Bernhard+Sick%2E+2017%2E+Simulation+of+annotators+for+active+learning%3A+Uncertain+Oracles%2E+In+Proceedings+of+the+Workshop+and+Tutorial+on+Interactive+Adaptive+Learning+co%2Dlocated+with+Euro)
5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, pages 177–190. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges%2C+Evaluating+Predictive+Uncertainty%2C+Visual+Object+Classificati)
6. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. [The CommitmentBank: Investigating projection in naturally occurring discourse](https://md.research.elamur.ai/free/en/papers/marneffe-2019-commitmentbank-investigating-projection-naturally/). In Sinn und Bedeutung 23. [↗](https://doi.org/10.18148/sub/2019.v23i2.601)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pages 4171–4186.
8. William Gantt, Benjamin Kane, and Aaron Steven White. 2020. [Natural language inference with mixed effects](https://md.research.elamur.ai/free/en/papers/gantt-2020-natural-language-inference-mixed-effects/). In The Ninth Joint Conference on Lexical and Computational Semantics (*SEM 2020). [↗](https://doi.org/10.48550/arxiv.2010.10501)
9. Nanjiang Jiang and Marie-Catherine de Marneffe. 2019a. [Do you know that Florence is packed with visitors? Evaluating state-of-the-art models of speaker commitment](https://md.research.elamur.ai/free/en/papers/jiang-2019-do-you-know-that-florence/). In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 4208–4213.
10. Nanjiang Jiang and Marie-Catherine de Marneffe. 2019b. [Evaluating BERT for natural language inference: A case study on the Commitmentbank](https://md.research.elamur.ai/free/en/papers/jiang-2019-evaluating-bert-natural-language-inference/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, EMNLP 2019, pages 6085–6090.
11. Kian Kenyon-Dean, Eisha Ahmed, Scott Fujimoto, Jeremy Georges-Filteau, Christopher Glasz, Barleen Kaur, Auguste Lalande, Shruti Bhanderi, Robert Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015. [find](https://scholar.google.com/scholar?q=Kian+Kenyon%2DDean%2C+Eisha+Ahmed%2C+Scott+Fujimoto%2C+Jeremy+Georges%2DFilteau%2C+Christopher+Glasz%2C+Barleen+Kaur%2C+Auguste+Lalande%2C+Shruti+Bhanderi%2C+Robert+Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+f)
12. Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceedings of the Eight International Conference on Computational Semantics, IWCS 2009, pages 140–156. [find](https://scholar.google.com/scholar?q=Bill+MacCartney+and+Christopher+D%2E+Manning%2E+2009%2E+An+extended+model+of+natural+logic%2E+In+Proceedings+of+the+Eight+International+Conference+on+Computational+Semantics%2C+IWCS+2009%2C+pages+140%E2%80%93156%2E)
13. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 3428– 3448.
14. Ellie Pavlick and Chris Callison-Burch. 2016. Most “babies” are “little” and most “problem” are “huge”: Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. [find](https://scholar.google.com/scholar?q=Ellie+Pavlick+and+Chris+Callison%2DBurch%2E+2016%2E+Most+%E2%80%9Cbabies%E2%80%9D+are+%E2%80%9Clittle%E2%80%9D+and+most+%E2%80%9Cproblem%E2%80%9D+are+%E2%80%9Chuge%E2%80%9D%3A+Compositional+entailment+in+adjective%2Dnouns%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+As)
15. Ellie Pavlick and Tom Kwiatkowski. 2019. [Inherent disagreements in human textual inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/). Transactions of the Association for Computational Linguistics, pages 677–694. [↗](https://doi.org/10.1162/tacl_a_00293)
16. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, pages 1532–1543.
17. Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? Adapting pre-trained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2019, pages 7–14. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Sebastian+Ruder%2C+and+Noah+A%2E+Smith%2E+2019%2E+To+tune+or+not+to+tune%3F+Adapting+pre%2Dtrained+representations+to+diverse+tasks%2E+In+Proceedings+of+the+4th+Workshop+on+Representation+Learnin)
18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). CoRR, abs/1910.10683. [↗](https://arxiv.org/abs/1910.10683)
19. H. Sebastian Seung, Manfred Opper, and Haim Sompolinsky. 1992. Query by committee. In Proceedings of the Fifth Annual ACM Conference on Computational Learning Theory, COLT 1992, Pittsburgh, PA, USA, July 27-29, 1992, pages 287–294. [find](https://scholar.google.com/scholar?q=H%2E+Sebastian+Seung%2C+Manfred+Opper%2C+and+Haim+Sompolinsky%2E+1992%2E+Query+by+committee%2E+In+Proceedings+of+the+Fifth+Annual+ACM+Conference+on+Computational+Learning+Theory%2C+COLT+1992%2C+Pittsburgh%2C+PA%2C+USA%2C+J)
20. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [SuperGLUE: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pages 3261–3275.
21. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, pages 1112–1122. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)

*Updated 24 October 2026*