---
title: "Let's CONFER — paper by Azin et al. (2025)"
description: "The authors build CONFER, 18,000 premise–hypothesis pairs generated semi-automatically from… 28 references, cited by 4 papers here."
authors: ["Tara Azin", "Daniel Dumitrescu", "Diana Inkpen", "Raj Singh"]
year: 2025
venue: "arXiv:2506.06133"
source: "https://arxiv.org/abs/2506.06133"
doi: "10.48550/arxiv.2506.06133"
arxiv: "2506.06133"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/azin-2025-let-s-confer-dataset-evaluating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/azin-2025-let-s-confer-dataset-evaluating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / Let's CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition

# Let's CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition

**Authors:** Tara Azin, Daniel Dumitrescu, [Diana Inkpen](https://md.research.elamur.ai/free/en/authors/diana-inkpen/) and [Raj Singh](https://md.research.elamur.ai/free/en/authors/raj-singh/) · **Year:** 2025 · **Venue:** arXiv:2506.06133 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2506.06133) · [PDF](https://arxiv.org/pdf/2506.06133) · [DOI 10.48550/arxiv.2506.06133](https://doi.org/10.48550/arxiv.2506.06133) · [arXiv 2506.06133](https://arxiv.org/abs/2506.06133) · [OpenAlex](https://openalex.org/W7110016663)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 4 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard NLI benchmarks rarely test presupposition, the background assumption that survives negation and questions. None examine how it behaves inside conditional sentences. Take a conditional "If A, then B" where B carries a presupposition p. Listeners sometimes take away p outright and sometimes only the weaker "if A, then p". This puzzle is known as the Proviso Problem. The authors build CONFER, 18,000 premise–hypothesis pairs generated semi-automatically from linguist-written templates. It covers several conditional types and two triggers: the adverb "again" and possessives. They evaluate RoBERTa and DeBERTa, both as released and after fine-tuning on IMPPRES, NOPE or CONFER. They also prompt GPT-4o, Llama, Gemma and DeepSeek-R1 in zero-shot and few-shot modes. Every system handles these cases poorly. Training on existing presupposition data does not carry over to conditionals and can even lower scores. The best LLM, GPT-4o, reaches only about two thirds accuracy. The work matters because it isolates one pragmatic phenomenon inside a fixed syntactic frame. That control shows when models rely on surface cues rather than inference.

## Key points
- CONFER has 18,000 sentence pairs built from templates and split into five conditional types (Type 1–5) crossed with two triggers ("again", possessive). Type 4 contains only possessive items. Data, prompts and code are on GitHub (Conditional-NLI/CONFER).
- IMPPRES and NOPE serve as comparison datasets. The authors note that IMPPRES has few conditionals with repetitive consequents and NOPE does not cover complex conditionals.
- Fine-tuned transformers often mislabel Contradiction as Neutral, especially in Type 2. In Type 2 the antecedent does not necessarily entail the consequent's presupposition.
- Among LLMs, GPT-4o scores highest, with 64% in zero-shot and about 67% in few-shot. DeepSeek-R1 is weakest at roughly 0.32 zero-shot accuracy, even though it targets reasoning. Llama gets worse when given few-shot examples.
- Type 5, which human annotators label Entailment, is the hardest type. For example, Llama's zero-shot recall on type5_again is 0.03, while Types 1–3 are mostly recognised correctly as Neutral.

## Interesting ideas
- Pin a single pragmatic phenomenon to one fixed syntactic frame and vary only the lexicon through templates, so failures can be traced to the phenomenon rather than to noise.
- Separate presupposition from entailment with a projection test: check whether the inference survives under negation, questions or a conditional.
- Treat a conditional's presupposition as either unconditional (p) or conditional ("if A then p"), and decide between them by asking whether A plausibly bears on p (scuba diver vs. flying to Toronto).
- Report results broken down by type × trigger instead of one accuracy number, which revealed that the failures cluster in Type 5 and that few-shot examples can hurt.
- Fine-tune on neighbouring datasets and test on the target structure to check whether a skill transfers or is just pattern matching.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Build template-generated probe sets (e.g., "again"/possessive inside if-clauses) as targeted regression tests with known gold labels for the explainable pipeline.
- **I2** (frames): The prag crate could tag trigger-introduced content (again, possessives, factives, change-of-state verbs) as projective, so it is kept when the asserting clause is negated or embedded.
- **I3** (world): When building the text world, accommodate p as a global fact by default. Store it as conditional only when compiled knowledge links the antecedent to p, and record which rule fired.
- **I4** (answer): Score readers on school tests and fable QA per construction and per trigger, so weak spots stay visible even when the overall score looks fine.
- **I5** (learn): Train repair rules or the parser on one silver source and evaluate on a held-out structure family, as a negative control against overfitting to surface patterns.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Ottawa](https://md.research.elamur.ai/free/en/universities/university-of-ottawa/) — University, Ottawa, Canada · 13 works · 2,514 citations
- [Carleton University](https://md.research.elamur.ai/free/en/universities/carleton-university/) — University, Ottawa, Canada · 4 works · 151 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [😉 Implicature](https://md.research.elamur.ai/free/en/topics/implicature/) · 138 works

## Cite

Cite as: Tara Azin, Daniel Dumitrescu, Diana Inkpen and Raj Singh. 2025. Let's CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition. arXiv:2506.06133.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{azin2025let,
  title = {{Let's CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition}},
  author = {Tara Azin, Daniel Dumitrescu, Diana Inkpen and Raj Singh},
  year = {2025},
  doi = {10.48550/arxiv.2506.06133},
  eprint = {2506.06133},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.06133}
}
```

- [← previous Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models](https://md.research.elamur.ai/free/en/papers/liu-2025-less-more-selective-reflection-compatible/)
- [next → LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect](https://md.research.elamur.ai/free/en/papers/naouara-2025-linto-audio-textual-datasets-train/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/)

## References

28 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lauri Karttunen. Presuppositions of compound sentences. Linguistic Inquiry, 4(2):169–193, 1973. [find](https://scholar.google.com/scholar?q=Lauri+Karttunen%2E+Presuppositions+of+compound+sentences%2E+Linguistic+Inquiry%2C+4%282%29%3A169%E2%80%93193%2C+1973%2E)
2. Robert Stalnaker. Presuppositions. Journal of Philosophical Logic, 2(4):447–457, 1973. [find](https://scholar.google.com/scholar?q=Robert+Stalnaker%2E+Presuppositions%2E+Journal+of+Philosophical+Logic%2C+2%284%29%3A447%E2%80%93457%2C+1973%2E)
3. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. [Are natural language inference models imppressive? learning implicature and presupposition](https://md.research.elamur.ai/free/en/papers/jeretic-2020-are-natural-language-inference-models/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online, 2020. Association for Computational Linguistics.
4. Alyssa Parrish, Sebastian Schuster, Alex Warstadt, et al. [Nope: A corpus of naturally-occurring presuppositions in english](https://md.research.elamur.ai/free/en/papers/parrish-2021-nope-corpus-naturally-occurring-presuppositions/). In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 349–366, 2021.
5. David Ian Beaver. Presupposition. In Johan Van Benthem and Alice Ter Meulen, editors, Handbook of Logic and Language, chapter 17, pages 939–1008. North-Holland, 1997. [find](https://scholar.google.com/scholar?q=David+Ian+Beaver%2E+Presupposition%2E+In+Johan+Van+Benthem+and+Alice+Ter+Meulen%2C+editors%2C+Handbook+of+Logic+and+Language%2C+chapter+17%2C+pages+939%E2%80%931008%2E+North%2DHolland%2C+1997%2E)
6. Christopher Potts. Presupposition and implicature. In The Handbook of Contemporary Semantic Theory, pages 168–202. John Wiley & Sons, Ltd, 2015. [find](https://scholar.google.com/scholar?q=Christopher+Potts%2E+Presupposition+and+implicature%2E+In+The+Handbook+of+Contemporary+Semantic+Theory%2C+pages+168%E2%80%93202%2E+John+Wiley+%26+Sons%2C+Ltd%2C+2015%2E)
7. David Lewis. Scorekeeping in a language game. Journal of Philosophical Logic, 8(1):339–359, 1979. [find](https://scholar.google.com/scholar?q=David+Lewis%2E+Scorekeeping+in+a+language+game%2E+Journal+of+Philosophical+Logic%2C+8%281%29%3A339%E2%80%93359%2C+1979%2E)
8. Bart Geurts. Local satisfaction guaranteed: A presupposition theory and its problems. Linguistics and Philosophy, 19:259–294, 1996. [find](https://scholar.google.com/scholar?q=Bart+Geurts%2E+Local+satisfaction+guaranteed%3A+A+presupposition+theory+and+its+problems%2E+Linguistics+and+Philosophy%2C+19%3A259%E2%80%93294%2C+1996%2E)
9. R. Singh. Matrix and embedded presuppositions. In The Wiley Blackwell Companion to Semantics, pages 1–42. Wiley-Blackwell, 2020. [find](https://scholar.google.com/scholar?q=R%2E+Singh%2E+Matrix+and+embedded+presuppositions%2E+In+The+Wiley+Blackwell+Companion+to+Semantics%2C+pages+1%E2%80%9342%2E+Wiley%2DBlackwell%2C+2020%2E)
10. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. Association for Computational Linguistics, 2015.
11. Adina Williams, Nikita Nangia, and Samuel Bowman. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics, 2018.
12. Robin Cooper, Richard Crouch, Jan Van Eijck, Chris Fox, Josef Van Genabith, Jan Jaspers, Hans Kamp, Manfred Pinkal, Massimo Poesio, and Stephen Pulman et al. FraCaS: A framework for computational semantics. 1994. Deliverable D6. [find](https://scholar.google.com/scholar?q=Robin+Cooper%2C+Richard+Crouch%2C+Jan+Van+Eijck%2C+Chris+Fox%2C+Josef+Van+Genabith%2C+Jan+Jaspers%2C+Hans+Kamp%2C+Manfred+Pinkal%2C+Massimo+Poesio%2C+and+Stephen+Pulman+et+al%2E+FraCaS%3A+A+framework+for+computational+sema)
13. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104. Association for Computational Linguistics, 2018. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+Swag%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Nat)
14. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. [Abductive commonsense reasoning](https://md.research.elamur.ai/free/en/papers/bhagavatula-2019-abductive-commonsense-reasoning/). In Proceedings of the 2020 International Conference on Learning Representations (ICLR), 2020.
15. Aaron Steven White and Kyle Rawlins. The role of veridicality and factivity in clause selection. In Proceedings of the 48th Annual Meeting of the North East Linguistic Society, Amherst, MA, USA, 2018. GLSA Publications. [find](https://scholar.google.com/scholar?q=Aaron+Steven+White+and+Kyle+Rawlins%2E+The+role+of+veridicality+and+factivity+in+clause+selection%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+North+East+Linguistic+Society%2C+Amherst%2C+MA%2C+USA%2C+2018%2E)
16. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. [The commitmentbank: Investigating projection in naturally occurring discourse](https://md.research.elamur.ai/free/en/papers/marneffe-2019-commitmentbank-investigating-projection-naturally/). In Proceedings of Sinn und Bedeutung, volume 23, pages 107–124, 2019.
17. Judith Degen. [Investigating the distribution of some (but not all) implicatures using corpora and web-based methods](https://md.research.elamur.ai/free/en/papers/degen-2015-investigating-distribution-some-but-not/). Semantics and Pragmatics, 8:11–1, 2015.
18. Sebastian Schuster, Yuxing Chen, and Judith Degen. Harnessing the richness of the linguistic signal in predicting pragmatic inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. [find](https://scholar.google.com/scholar?q=Sebastian+Schuster%2C+Yuxing+Chen%2C+and+Judith+Degen%2E+Harnessing+the+richness+of+the+linguistic+signal+in+predicting+pragmatic+inferences%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for)
19. Andre Cianflone, Yulan Feng, Jad Kabbara, and Jackie Chi Kit Cheung. Let’s do it “again”: A first computational approach to detecting adverbial presupposition triggers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2747[–2755, Melbourne, Australia, 2018](https://md.research.elamur.ai/free/en/papers/cianflone-2018-let-s-do-it-again/). Association for Computational Linguistics.
20. Nanjiang Jiang and Marie-Catherine de Marneffe. [Do you know that florence is packed with visitors? evaluating state-of-the-art models of speaker commitment](https://md.research.elamur.ai/free/en/papers/jiang-2019-do-you-know-that-florence/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4208–4213, Florence, Italy, 2019. Association for Computational Linguistics.
21. Alexis Ross and Ellie Pavlick. [How well do nli models capture verb veridicality?](https://md.research.elamur.ai/free/en/papers/ross-2019-how-well-do-nli-models/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240, 2019.
22. Paloma Jeretič, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, and Samuel R. Bowman. Investigating bert’s knowledge of language: Five analysis methods with npis. In Proceedings of EMNLP-IJCNLP, pages 2870[–2880, 2019](https://md.research.elamur.ai/free/en/papers/warstadt-2019-investigating-bert-s-knowledge-language/).
23. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. [Blimp: The benchmark of linguistic minimal pairs for english](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8:377–392, 2020.
24. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, 2014.
25. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. [Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint, 2019. arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
27. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. [Deberta: Decoding-enhanced bert with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In International Conference on Learning Representations, 2021.
28. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint, 2017. arXiv:1701.06538. [↗](https://arxiv.org/abs/1701.06538)

*Updated 2 November 2026*