--- title: "From Disagreement to Understanding: The Case for Ambiguity" description: "They then present a four-stage ambiguity-aware pipeline: decide whether the pair is ambiguous; classify… 5 references, cited by 3 papers here." authors: ["Chathuri Jayaweera", "Bonnie J. Dorr"] year: 2025 venue: "Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP" source: "https://aclanthology.org/2025.nlperspectives-1.4/" doi: "10.18653/v1/2025.nlperspectives-1.4" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/jayaweera-2025-disagreement-understanding-case-ambiguity-detection/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jayaweera-2025-disagreement-understanding-case-ambiguity-detection/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/) / From Disagreement to Understanding: The Case for Ambiguity Detection in NLI # From Disagreement to Understanding: The Case for Ambiguity Detection in NLI **Authors:** Chathuri Jayaweera and [Bonnie J. Dorr](https://md.research.elamur.ai/free/en/authors/bonnie-dorr/) · **Year:** 2025 · **Venue:** [Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP](https://md.research.elamur.ai/free/en/venues/nlperspectives/) ([edition of NLPerspectives 2025](https://md.research.elamur.ai/free/en/venues/nlperspectives/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.nlperspectives-1.4/) · [PDF](https://aclanthology.org/2025.nlperspectives-1.4.pdf) · [DOI 10.18653/v1/2025.nlperspectives-1.4](https://doi.org/10.18653/v1/2025.nlperspectives-1.4) · [OpenAlex](https://openalex.org/W4416033674) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/jayaweera-2025-disagreement-understanding-case-ambiguity-detection/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This position paper challenges the common practice in Natural Language Inference (NLI) of treating annotator disagreement as noise to filter out or settle by majority vote. The authors argue that much of this disagreement is meaningful. It often comes from real ambiguity in the premise or hypothesis, which, unlike unclear guidelines or annotator habits, does not depend on how the annotation was carried out. They review earlier work that models label distributions (soft labels, UNLI, BERT-based artificial annotators) and note that it describes disagreement without explaining its cause. They then present a four-stage ambiguity-aware pipeline: 1. decide whether the pair is ambiguous; 2. classify the ambiguity type; 3. generate disambiguated readings; 4. run inference on each reading. They also merge several prior taxonomies into one with four top-level classes: Lexical, Syntactic, Semantic and Pragmatic. No experiments are reported. The paper's main point is that no dataset is annotated for the presence and type of ambiguity. It calls for building such resources and for exploring unsupervised or weakly supervised detection, with the goal of more explainable, human-aligned NLI. ## Key points - Disagreement sources follow Aroyo and Welty's "Triangle of Reference": sentence meaning, guideline underspecification, and annotator behavior. Jiang and de Marneffe (2022) refine these for NLI and split content ambiguity into Lexical, Implicature, Presupposition, Probabilistic Enrichment and Imperfection. - When the RTE1 sets were re-annotated, the new labels differed substantially from the original ones. Even after problematic items were removed, human agreement reached only about 91%. - Prior distribution-modeling work includes the following, all of which stop at describing disagreement: - Meissner et al. (2021): training on soft labels. - UNLI (Chen et al., 2020): predicting probabilities of entailment. - Zhang and de Marneffe (2021): simulated annotators built with BERT. - Zhou et al. (2022): improved distribution modeling beyond softmax assumptions. - Figure 7 presents the unified taxonomy, which merges the schemes of Li et al. (2024), Liu et al. (2023) and Jiang and de Marneffe (2022) under four broad classes: Lexical, Syntactic, Semantic and Pragmatic. - To the authors' knowledge, no system automatically labels the ambiguity type of a premise–hypothesis pair, and no NLI dataset carries gold ambiguity-type annotations. ## Interesting ideas - Put an ambiguity gate before inference: first decide whether an input has more than one reading, and only then reason, once per reading, so the answer can depend on the interpretation instead of being forced into one label. - Expand an ambiguous input into explicit disambiguated variants and pass each one through the same downstream classifier, which makes it clear which reading produced which verdict. - Group ambiguity types by linguistic level (lexical, syntactic, semantic, pragmatic) so that each level gets its own detection method instead of one generic detector. - Separate disagreement caused by the content itself from disagreement caused by the process (unclear guidelines, annotator habits), and treat only the first as real signal. - Use annotator disagreement and model uncertainty as weak, unsupervised cues for finding ambiguity candidates when gold ambiguity labels do not exist. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (answer): Readers answering questions on fables and tales could check whether the question or the passage allows several readings and return conditional answers ("if X means A, then yes"), each explained. - **I2** (frames): When crate prag finds a polysemous verb such as "likes" (romantic interest vs. distant admiration), it could build parallel frame sets, one per sense, and carry both into the world of the text. - **I3** (tree): The parser could flag syntactic ambiguity (for example competing attachment sites) as its own category, kept separate from lexical sense ambiguity handled in frames. - **I4** (learn): When witness parsers disagree on silver data, Mova could test whether the sentence is genuinely ambiguous before discarding it as noise, and keep such items as a labelled ambiguity set. - **I5** (learn): Spread among witness parsers or LLM teacher tiers could act as a cheap ambiguity detector that queues sentences for closer review. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Florida](https://md.research.elamur.ai/free/en/universities/university-of-florida/) — University, Gainesville, United States · 36 works · 364 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works ## Cite Cite as: Chathuri Jayaweera and Bonnie J. Dorr. 2025. From Disagreement to Understanding: The Case for Ambiguity Detection in NLI. Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{jayaweera2025disagreement, title = {{From Disagreement to Understanding: The Case for Ambiguity Detection in NLI}}, author = {Chathuri Jayaweera and Bonnie J. Dorr}, booktitle = {Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP}, year = {2025}, doi = {10.18653/v1/2025.nlperspectives-1.4}, url = {https://doi.org/10.18653/v1/2025.nlperspectives-1.4} } ``` - [← previous From Arabic Text to Puzzles: LLM-Driven Development of Arabic Educational Crosswords](https://md.research.elamur.ai/free/en/papers/zeinalipour-2025-arabic-text-puzzles-llm-driven/) - [next → German Aspect-based Sentiment Analysis in the Wild: B2B Dataset Creation and Cross-Domain Evaluation](https://md.research.elamur.ai/free/en/papers/fehle-2025-german-aspect-based-sentiment-analysis/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/) ## References 5 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Lora Aroyo and Chris Welty. 2015. [Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation](https://md.research.elamur.ai/free/en/papers/aroyo-2015-truth-lie-crowd-truth-seven/). AI Magazine, 36(1):15–24. Number: 1. 2. Samuel Bayer, John Burger, Lisa Ferro, John Henderson, and Alexander Yeh. 2005. MITRE’s Submissions to the EU Pascal RTE Challenge. In Proceedings of the Pattern Analysis, Statistical Modelling, and Computational Learning (PASCAL) Challenges Workshop on Recognising Textual Entailment, pages 41–44. [find](https://scholar.google.com/scholar?q=Samuel+Bayer%2C+John+Burger%2C+Lisa+Ferro%2C+John+Henderson%2C+and+Alexander+Yeh%2E+2005%2E+MITRE%E2%80%99s+Submissions+to+the+EU+Pascal+RTE+Challenge%2E+In+Proceedings+of+the+Pattern+Analysis%2C+Statistical+Modelling%2C+and+C) 3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d15-1075) 4. Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020. [Uncertain Natural Language Inference](https://md.research.elamur.ai/free/en/papers/chen-2020-uncertain-natural-language-inference/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.774) 5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Proceedings of the First international conference on Machine Learning Challenges: evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW’05, pages 177–190, Berlin, Heidelberg. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Proceedings+of+the+First+international+conference+on+Machine+Learning+Challenges%3A+evaluati) *Updated 2 November 2026*