---
title: "ConjNLI: Natural Language Inference Over Conjunctive Sentences"
description: "ConjNLI is a stress-test for natural language inference (NLI) built around coordinating conjunctions… 47 references, cited by 26 papers here."
authors: ["Swarnadeep Saha", "Yixin Nie", "Mohit Bansal"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.661/"
doi: "10.18653/v1/2020.emnlp-main.661"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/saha-2020-conjnli-natural-language-inference-over/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/saha-2020-conjnli-natural-language-inference-over/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / ConjNLI: Natural Language Inference Over Conjunctive Sentences

# ConjNLI: Natural Language Inference Over Conjunctive Sentences

**Authors:** [Swarnadeep Saha](https://md.research.elamur.ai/free/en/authors/swarnadeep-saha/), [Yixin Nie](https://md.research.elamur.ai/free/en/authors/yixin-nie/) and [Mohit Bansal](https://md.research.elamur.ai/free/en/authors/mohit-bansal/) · **Year:** 2020 · **Venue:** [Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2020](https://md.research.elamur.ai/free/en/venues/emnlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.661/) · [PDF](https://aclanthology.org/2020.emnlp-main.661.pdf) · [DOI 10.18653/v1/2020.emnlp-main.661](https://doi.org/10.18653/v1/2020.emnlp-main.661) · [OpenAlex](https://openalex.org/W3094492284)

**Citations:** 24 citations (OpenAlex, 2026-10-14) · [26](https://md.research.elamur.ai/free/en/papers/saha-2020-conjnli-natural-language-inference-over/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
ConjNLI is a stress-test for natural language inference (NLI) built around coordinating conjunctions ("and", "or", "but", "nor"). Earlier NLI benchmarks either avoid reasoning over conjuncts or use templates that cover only boolean "and". The authors took conjunctive sentences from Wikipedia and automatically built premise–hypothesis pairs by deleting, inserting or swapping a conjunct. Humans then verified the pairs and experts labelled them. The resulting items mix negation, quantifiers and several conjunctions per sentence. They also include non-boolean readings, such as a collective sum that cannot be split or an exclusive "or".

BERT and RoBERTa trained on MNLI do poorly on ConjNLI and seem to rely on word overlap. The authors test two remedies:
- **IAFT:** iterative adversarial fine-tuning on synthetic pairs labelled by boolean and non-boolean heuristics.
- **PA:** a predicate-aware RoBERTa that adds semantic-role-based embeddings.

Both give modest gains while keeping MNLI performance intact, but the dataset remains far from solved. The work shows that conjunction semantics is a distinct and still weak area for pretrained models.

## Key points
- About 38.8% of Penn Treebank sentences contain "and", "or" or "but". Around 30% of SNLI pairs include conjunctions, yet in a sample of 100 such pairs, 72% left the conjuncts unchanged between premise and hypothesis.
- Accuracy across all conjunction types: RoBERTa 65.60, PA 66.30, IAFT 67.90.
- Performance varies by conjunction type. "Or" is hardest (around 60–62), "but" is easiest (around 80–81), and IAFT drops on multi-conjunction sentences (62.88 vs 65.93 for RoBERTa).
- Non-boolean items are roughly 34% of the dev set, and every model scores about 10 points lower on them than on boolean items. For example, RoBERTa gets 68% on boolean items and 58% on non-boolean ones; IAFT gets 72% and 61%.
- Results are stable across random seeds. Over 10 seeds, the mean dev accuracy is 64.48 with a standard deviation of 0.59. Adversarial gains level off after about 12k synthetic examples, out of 6k–18k tried.

## Interesting ideas
- Minimal pairs are built by editing exactly one conjunct (remove, add or replace), so any change in the label can be traced to that single edit.
- Separating boolean from non-boolean conjunction matters. A combined quantity like "9 minutes and 9 seconds" or an either-or date must not be split into independent simple facts.
- Negation interacts with disjunction: "did not produce A or B" means neither A nor B, which licenses dropping a disjunct under negation.
- Cheap adversarial training data comes from running the whole dataset pipeline except the costly human labelling step, with labels assigned by heuristics.
- A seed-instability check (many random seeds, with the variance split into components) shows whether a test set is diverse or just a few repeated patterns.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): Generate such conjunct-edited premise/question pairs from fable and school-test texts to check that the reader's answer changes exactly when the world should change.
- **I2** (world): Before splitting a coordination into separate facts, the world builder should decide whether the conjoined phrase is collective (one measure or one entity) or exclusive, and record that decision explainably.
- **I3** (frames): Frame construction can push negation through "or" into each conjunct, so that polarity on every frame is explicit for later inference.
- **I4** (learn): Reuse our pair or silver-data generators without the teacher tier to mass-produce heuristic-labelled conjunct cases. Then check on a small gold set, plus the original benchmarks as negative controls, that nothing regressed.
- **I5** (learn): Apply the same multi-seed variance check to our evaluation suites, so that a gain is not mistaken for progress when the test is merely pattern-narrow.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 85 works · 66,118 citations
- [University of North Carolina at Chapel Hill](https://md.research.elamur.ai/free/en/universities/university-of-north-carolina-at-chapel-hill/) — University, Chapel Hill, United States · 37 works · 14,332 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Swarnadeep Saha, Yixin Nie and Mohit Bansal. 2020. ConjNLI: Natural Language Inference Over Conjunctive Sentences. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{saha2020conjnli,
  title = {{ConjNLI: Natural Language Inference Over Conjunctive Sentences}},
  author = {Swarnadeep Saha, Yixin Nie and Mohit Bansal},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.661},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.661}
}
```

- [← previous DAICT: A Dialectal Arabic Irony Corpus Extracted from Twitter](https://md.research.elamur.ai/free/en/papers/abbes-2020-daict-dialectal-arabic-irony-corpus/)
- [next → Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data](https://md.research.elamur.ai/free/en/papers/huang-2020-counterfactually-augmented-snli-training-data/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

47 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rajeev Agarwal and Lois Boggess. 1992. A simple but useful approach to conjunct identification. In Proceedings of the 30th annual meeting on Association for Computational Linguistics, pages 15–21. [find](https://scholar.google.com/scholar?q=Rajeev+Agarwal+and+Lois+Boggess%2E+1992%2E+A+simple+but+useful+approach+to+conjunct+identification%2E+In+Proceedings+of+the+30th+annual+meeting+on+Association+for+Computational+Linguistics%2C+pages+15%E2%80%9321%2E)
2. Michele Banko, Michael J Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. [Open information extraction from the web](https://md.research.elamur.ai/free/en/papers/banko-2007-open-information-extraction-web/). In IJ-CAI, volume 7, pages 2670–2676.
3. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. [↗](https://doi.org/10.48550/arxiv.1508.05326)
4. Xavier Carreras and Lluı́s Màrquez. 2005. [Introduction to the CoNLL-2005 shared task: Semantic role labeling](https://md.research.elamur.ai/free/en/papers/carreras-2005-introduction-conll-2005-shared-task/). In Proceedings of the Ninth Conference on Computational Natural Language Learning, CONLL ’05.
5. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proceedings+of+the+2018+Conference+of)
6. Jack Hoeksema. 1988. The semantics of non-boolean “and”. Journal of Semantics, 6(1):19–40. [find](https://scholar.google.com/scholar?q=Jack+Hoeksema%2E+1988%2E+The+semantics+of+non%2Dboolean+%E2%80%9Cand%E2%80%9D%2E+Journal+of+Semantics%2C+6%281%29%3A19%E2%80%9340%2E)
7. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. [Are natural language inference models imppressive? learning implicature and presupposition](https://md.research.elamur.ai/free/en/papers/jeretic-2020-are-natural-language-inference-models/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
8. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031.
9. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+LSTM+for+natural+language+inference%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computa)
10. Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1909.12434)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. [↗](https://doi.org/10.18653/v1/n19-1423)
12. Manfred Krifka. 1990. Boolean and non-boolean ‘and’. In Papers from the second symposium on Logic and Language, pages 161–188. [find](https://scholar.google.com/scholar?q=Manfred+Krifka%2E+1990%2E+Boolean+and+non%2Dboolean+%E2%80%98and%E2%80%99%2E+In+Papers+from+the+second+symposium+on+Logic+and+Language%2C+pages+161%E2%80%93188%2E)
13. Jessica Ficler and Yoav Goldberg. 2016a. Coordination annotation extension in the Penn Tree Bank. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 834–842. [find](https://scholar.google.com/scholar?q=Jessica+Ficler+and+Yoav+Goldberg%2E+2016a%2E+Coordination+annotation+extension+in+the+Penn+Tree+Bank%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+)
14. Jessica Ficler and Yoav Goldberg. 2016b. A neural network for coordination boundary prediction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 23–32. [find](https://scholar.google.com/scholar?q=Jessica+Ficler+and+Yoav+Goldberg%2E+2016b%2E+A+neural+network+for+coordination+boundary+prediction%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+23%E2%80%9332%2E)
15. Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2018. [Stress-testing neural models of natural language inference with multiply-quantified sentences](https://md.research.elamur.ai/free/en/papers/geiger-2018-stress-testing-neural-models-natural/). arXiv preprint arXiv:1810.13033. [↗](https://arxiv.org/abs/1810.13033)
16. Lila R Gleitman. 1965. Coordinating conjunctions in English. Language, 41(2):260–293. [↗](https://doi.org/10.2307/411878)
17. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. [Breaking NLI systems with sentences that require simple lexical inferences](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655.
18. Edward L Keenan and Leonard M Faltz. 2012. Boolean semantics for natural language, volume 23. Springer Science & Business Media. [find](https://scholar.google.com/scholar?q=Edward+L+Keenan+and+Leonard+M+Faltz%2E+2012%2E+Boolean+semantics+for+natural+language%2C+volume+23%2E+Springer+Science+%26+Business+Media%2E)
19. Nelson F Liu, Roy Schwartz, and Noah A Smith. 2019a. [Inoculation by fine-tuning: A method for analyzing challenge datasets](https://md.research.elamur.ai/free/en/papers/liu-2019-inoculation-fine-tuning-method-analyzing/). In NAACL.
20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
21. Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. [Building a large annotated corpus of English: The Penn Treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational Linguistics. [↗](https://doi.org/10.21236/ada273556)
22. Gerald J Massey. 1976. Tom, Dick, and Harry, and all the king’s men. American Philosophical Quarterly, 13(2):89–107. [find](https://scholar.google.com/scholar?q=Gerald+J+Massey%2E+1976%2E+Tom%2C+Dick%2C+and+Harry%2C+and+all+the+king%E2%80%99s+men%2E+American+Philosophical+Quarterly%2C+13%282%29%3A89%E2%80%93107%2E)
23. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448.
24. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353. [↗](https://doi.org/10.48550/arxiv.1806.00692)
25. Yixin Nie, Yicheng Wang, and Mohit Bansal. 2019. [Analyzing compositionality-sensitivity of NLI models](https://md.research.elamur.ai/free/en/papers/nie-2018-analyzing-compositionality-sensitivity-nli-models/). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6867–6874.
26. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2020%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+In+Proceedings+of+the+58th+Annual+Meetin)
27. Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2018. A span selection model for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Hiroki+Ouchi%2C+Hiroyuki+Shindo%2C+and+Yuji+Matsumoto%2E+2018%2E+A+span+selection+model+for+semantic+role+labeling%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E)
28. Deric Pang, Lucy H Lin, and Noah A Smith. 2019. Improving natural language inference with a pretrained parser. arXiv preprint arXiv:1909.08217. [↗](https://arxiv.org/abs/1909.08217)
29. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018a. [Collecting diverse natural language inference problems for sentence representation evaluation](https://md.research.elamur.ai/free/en/papers/poliak-2018-collecting-diverse-natural-language-inference/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 67–81. [↗](https://doi.org/10.48550/arxiv.1804.08207)
30. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018b. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Jason+Naradowsky%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+and+Benjamin+Van+Durme%2E+2018b%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+In+Proceedings+of+the+Seventh+Joint+Conference)
31. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. [EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference](https://md.research.elamur.ai/free/en/papers/ravichander-2019-equate-benchmark-evaluation-framework-quantitative/). In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349–361.
32. Kyle Richardson, Hai Hu, Lawrence S Moss, and Ashish Sabharwal. 2020. [Probing natural language inference models through semantic fragments](https://md.research.elamur.ai/free/en/papers/richardson-2019-probing-natural-language-inference-models/). In Proceedings of the AAAI Conference on Artificial Intelligence. [↗](https://doi.org/10.1609/aaai.v34i05.6397)
33. Ivan Sanchez, Jeff Mitchell, and Sebastian Riedel. 2018. [Behavior analysis of NLI models: Uncovering the influence of three factors on robustness](https://md.research.elamur.ai/free/en/papers/sanchez-2018-behavior-analysis-nli-models-uncovering/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1975–1985, New Orleans, Louisiana.
34. Peng Shi and Jimmy Lin. 2019. [Simple bert models for relation extraction and semantic role labeling](https://md.research.elamur.ai/free/en/papers/shi-2019-simple-bert-models-relation-extraction/). arXiv preprint arXiv:1904.05255. [↗](https://arxiv.org/abs/1904.05255)
35. Hiroki Teranishi, Hiroyuki Shindo, and Yuji Matsumoto. 2019. Decomposed local models for coordinate structure parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3394–3403. [find](https://scholar.google.com/scholar?q=Hiroki+Teranishi%2C+Hiroyuki+Shindo%2C+and+Yuji+Matsumoto%2E+2019%2E+Decomposed+local+models+for+coordinate+structure+parsing%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Associ)
36. Yicheng Wang and Mohit Bansal. 2018. Robust machine comprehension models via adversarial training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 575–581. [↗](https://doi.org/10.48550/arxiv.1804.06473)
37. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. [Inference is everything: Recasting semantic resources into a unified evaluation framework](https://md.research.elamur.ai/free/en/papers/white-2017-inference-is-everything-recasting-semantic/). In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 996–1005.
38. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
39. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771. [↗](https://arxiv.org/abs/1910.03771)
40. Alexis Ross and Ellie Pavlick. 2019. [How well do NLI models capture verb veridicality?](https://md.research.elamur.ai/free/en/papers/ross-2019-how-well-do-nli-models/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
41. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. 2020. Do neural models learn systematicity of monotonicity inference in natural language? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [↗](https://doi.org/10.5715/jnlp.27.653)
42. Swarnadeep Saha and Mausam. 2018. Open information extraction from conjunctive sentences. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2288–2299. [find](https://scholar.google.com/scholar?q=Swarnadeep+Saha+and+Mausam%2E+2018%2E+Open+information+extraction+from+conjunctive+sentences%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+pages+2288%E2%80%932299%2E)
43. Xiang Zhou, Yixin Nie, Hao Tan, and Mohit Bansal. 2020. The curse of performance instability in analysis datasets: Consequences, source, and suggestions. In EMNLP. Explanation In 870 or 871 he led The Great Summer Army to England. In 871 he led The Great Summer Army to England. neutral The year can be either 870 or 871, hence we definitely cannot say if it was 871. Upon completion, it will rise 64 stories or 711 ft. Upon completion, it will rise 711 ft. entailment 64 stories and 711 feet mean the same. ”or” is used to establish equivalence between two same things. During the shootout, Willis Brooks was killed while a fourth man was seriously wounded. During the shootout, Willis Brooks and two others were killed while a fourth man was seriously wounded. neutral Whether two others were killed is unknown. [find](https://scholar.google.com/scholar?q=Xiang+Zhou%2C+Yixin+Nie%2C+Hao+Tan%2C+and+Mohit+Bansal%2E+2020%2E+The+curse+of+performance+instability+in+analysis+datasets%3A+Consequences%2C+source%2C+and+suggestions%2E+In+EMNLP%2E+Explanation+In+870+or+871+he+led+The)
44. Gilbert was the freshman football coach of Franklin and Marshall College in 1938. [find](https://scholar.google.com/scholar?q=Gilbert+was+the+freshman+football+coach+of+Franklin+and+Marshall+College+in+1938%2E)
45. Gilbert was the freshman football coach of Franklin College in 1938. neutral Gilbert can be the coach of two colleges with slightly different names “Franklin and Marshall College” and “Franklin College”. [find](https://scholar.google.com/scholar?q=Gilbert+was+the+freshman+football+coach+of+Franklin+College+in+1938%2E+neutral+Gilbert+can+be+the+coach+of+two+colleges+with+slightly+different+names+%E2%80%9CFranklin+and+Marshall+College%E2%80%9D+and+%E2%80%9CFranklin+Colleg)
46. It premiered on 27 June 2016 and airs Mon-Fri 10-11pm IST. [find](https://scholar.google.com/scholar?q=It+premiered+on+27+June+2016+and+airs+Mon%2DFri+10%2D11pm+IST%2E)
47. It premiered on 28 June 2016 and airs Mon-Fri 10-11pm IST. contradiction If it premiered on 27 June, it cannot premier on 28 June. *Table 10: Some examples from C ONJ NLI with gold labels and explanations, used for training the annotators.* [find](https://scholar.google.com/scholar?q=It+premiered+on+28+June+2016+and+airs+Mon%2DFri+10%2D11pm+IST%2E+contradiction+If+it+premiered+on+27+June%2C+it+cannot+premier+on+28+June%2E+%2ATable+10%3A+Some+examples+from+C+ONJ+NLI+with+gold+labels+and+explanat)

*Updated 24 October 2026*