---
title: "Probing the Natural Language Inference Task with Automated"
description: "The authors test whether a purely logic-based pipeline can handle Natural Language Inference: deciding if a premise entails… 17 references."
authors: ["Zaid Marji", "Animesh Nighojkar", "John Licato"]
year: 2020
venue: "arXiv:2005.02573"
source: "https://arxiv.org/abs/2005.02573"
doi: "10.48550/arxiv.2005.02573"
arxiv: "2005.02573"
licence: "arXiv non-exclusive distribution licence (read only, no redistribution)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/marji-2020-probing-natural-language-inference-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/marji-2020-probing-natural-language-inference-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/) / Probing the Natural Language Inference Task with Automated Reasoning Tools

# Probing the Natural Language Inference Task with Automated Reasoning Tools

**Authors:** Zaid Marji, Animesh Nighojkar and John Licato · **Year:** 2020 · **Venue:** arXiv:2005.02573 · **Licence:** [arXiv non-exclusive distribution licence (read only, no redistribution)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2005.02573) · [PDF](https://arxiv.org/pdf/2005.02573) · [DOI 10.48550/arxiv.2005.02573](https://doi.org/10.48550/arxiv.2005.02573) · [arXiv 2005.02573](https://arxiv.org/abs/2005.02573) · [OpenAlex](https://openalex.org/W3021891771)

**Citations:** 3 citations (OpenAlex, 2026-10-14)

## Summary
The authors test whether a purely logic-based pipeline can handle Natural Language Inference: deciding if a premise entails, contradicts or is neutral toward a hypothesis. They use SNLI because its human-written picture captions are fairly grammatical. Sentences go through the Attempto Parsing Engine (APE), which turns valid Attempto Controlled English (ACE) into TPTP first-order formulae. A resolution prover then checks whether the hypothesis or its negation follows. Most SNLI sentences are not valid ACE, so the authors write Python syntactic rewrite rules, adapted from Abdelaal's Simple English Wikipedia work, that push sentences into ACE form. They also add semantic rules that inject WordNet hypernym axioms and closed-world exclusion axioms. On the SNLI development set, the rewrites more than double parse coverage. Entailment guesses made without the semantic rules are always correct. Overall accuracy is still below chance, and contradiction detection stays weak. The work matters as an interpretable baseline: it shows where controlled-language logic breaks on real NLI data, and it exposes mislabeled gold pairs.

## Key points
- Coverage on the SNLI dev set (10,000 pairs) rises from 7.06% to 16.61% after the syntactic rewrites.
- There are eight syntactic rules (R1–R8):
  - POS-prefixed adjective/noun phrases joined with "and"
  - CoreNLP coreference replaced by names, plus extra "X is Y" sentences
  - past tense converted to present via pattern.en
  - numerals written out as words
  - predeterminers dropped
  - adverb order and conjunction fixes
  - continuous tenses turned into simple present
- There are two semantic rules, S1 (noun hypernyms) and S2 (verb hypernyms, built on APE's predicate1/predicate2), both using WordNet. Non-hypernym pairs receive mutual-exclusion axioms under a closed-world assumption.
- Without semantic rules, overall accuracy is about 28.7%, entailment precision is 100%, and no contradictions are predicted. With semantic rules, contradiction accuracy is only 37.5%.
- The prover stops after 1500 generated clauses. Code is released at AMHRLab/NLIwithACE.

## Interesting ideas
- Background knowledge is added as separate axioms rather than by rewriting the sentences, which avoids over-generalizing both sides (e.g., turning "woman" and "boy" into "person" and breaking the entailment).
- Knowledge is applied in staged tiers: plain logic first, then sound hypernym axioms, then risky closed-world exclusions, so each verdict records which assumption level it needed.
- Coreference chains are flattened into names, and nominal mentions become explicit identity sentences, so no information is lost when pronouns are removed.
- A high-precision symbolic system doubles as an annotation auditor, since its confident disagreements with gold labels point to mislabeled pairs or hidden commonsense assumptions (hugging implies happiness).
- Reporting coverage separately from accuracy on the covered subset reveals that a system can be very precise but narrow.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (world): Taxonomy facts from WordNet/Wikidata should stay as a side knowledge layer consulted during inference, never merged into the text's own entity descriptions.
- **I2** (answer): Readers could tag each answer with the weakest knowledge tier that produced it, which keeps explanations honest about default assumptions.
- **I3** (frames): When coreference resolves "his wife" to an entity, Mova can emit an explicit relation fact alongside the substitution, so the link stays explainable.
- **I4** (learn): Mova's confident-but-contradicting outputs can be routed into a review queue to clean silver/gold data and surface implicit inferences that teachers assume.
- **I5** (learn): Evaluations should always report how many inputs reached a full parse-to-world pipeline next to per-class precision, so gains in reach and in correctness are tracked separately.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (3): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 58 works

## Cite

Cite as: Zaid Marji, Animesh Nighojkar and John Licato. 2020. Probing the Natural Language Inference Task with Automated Reasoning Tools. arXiv:2005.02573.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{marji2020probing,
  title = {{Probing the Natural Language Inference Task with Automated Reasoning Tools}},
  author = {Zaid Marji, Animesh Nighojkar and John Licato},
  year = {2020},
  doi = {10.48550/arxiv.2005.02573},
  eprint = {2005.02573},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.02573}
}
```

- [← previous Lemmatising Verbs in Middle English Corpora: The Benefit of Enriching the Penn-Helsinki Parsed Corpus of Middle English 2 (PPCME2), the Parsed Corpus of Middle English Poetry (PCMEP), and A Parsed Linguistic Atlas of Early Middle English (PLAEME)](https://md.research.elamur.ai/free/en/papers/trips-2020-lemmatising-verbs-middle-english-corpora/)
- [next → Real-Valued Logics for Typological Universals: Framework and Application](https://md.research.elamur.ai/free/en/papers/donicke-2020-real-valued-logics-typological-universals/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/)

## References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdelaal, H. S. 2019. Knowledge Extraction from Simplified Natural Language Text. Ph.D. Dissertation, National University of Ireland, Galway. [find](https://scholar.google.com/scholar?q=Abdelaal%2C+H%2E+S%2E+2019%2E+Knowledge+Extraction+from+Simplified+Natural+Language+Text%2E+Ph%2ED%2E+Dissertation%2C+National+University+of+Ireland%2C+Galway%2E)
2. Abzianidze, L. 2017. LangPro: Natural language theorem prover. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 115–120. Copenhagen, Denmark: Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1708.09417)
3. Angeli, G., and Manning, C. D. 2014. [Naturalli: Natural logic inference for common sense reasoning](https://md.research.elamur.ai/free/en/papers/angeli-2014-naturalli-natural-logic-inference-common/). In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 534–545.
4. Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1508.05326)
5. Fuchs, N. E.; Kaljurand, K.; and Kuhn, T. 2008. Attempto Controlled English for Knowledge Representation. In Baroglio, C.; Bonatti, P. A.; Małuszyński, J.; Marchiori, M.; Polleres, A.; and Schaffert, S., eds., Reasoning Web: 4th International Summer School 2008, Venice, Italy, September 7-11, 2008, Tutorial Lectures. Berlin, Heidelberg: Springer Berlin Heidelberg. 104–124. [find](https://scholar.google.com/scholar?q=Fuchs%2C+N%2E+E%2E%3B+Kaljurand%2C+K%2E%3B+and+Kuhn%2C+T%2E+2008%2E+Attempto+Controlled+English+for+Knowledge+Representation%2E+In+Baroglio%2C+C%2E%3B+Bonatti%2C+P%2E+A%2E%3B+Ma%C5%82uszyn%CC%81ski%2C+J%2E%3B+Marchiori%2C+M%2E%3B+Polleres%2C+A%2E%3B+and+Schaffert%2C)
6. Fuchs, N. E.; Kaljurand, K.; and Kuhn, T. 2011. Discourse Representation Structures for ACE 6.6. Technical Report August. [find](https://scholar.google.com/scholar?q=Fuchs%2C+N%2E+E%2E%3B+Kaljurand%2C+K%2E%3B+and+Kuhn%2C+T%2E+2011%2E+Discourse+Representation+Structures+for+ACE+6%2E6%2E+Technical+Report+August%2E)
7. Fuchs, N. E.; Kaljurand, K.; and Schneider, G. 2006. Attempto controlled english meets the challenges of knowledge representation, reasoning, interoperability and user interfaces. In FLAIRS Conference, volume 12, 664–669. [find](https://scholar.google.com/scholar?q=Fuchs%2C+N%2E+E%2E%3B+Kaljurand%2C+K%2E%3B+and+Schneider%2C+G%2E+2006%2E+Attempto+controlled+english+meets+the+challenges+of+knowledge+representation%2C+reasoning%2C+interoperability+and+user+interfaces%2E+In+FLAIRS+Conference)
8. Hu, H.; Chen, Q.; Richardson, K.; Mukherjee, A.; Moss, L. S.; and Kuebler, S. 2019. Monalog: a lightweight system for natural language inference based on monotonicity. arXiv preprint arXiv:1910.08772. [↗](https://arxiv.org/abs/1910.08772)
9. Kuhn, T., and Hess, P. D. M. 2010. Controlled English for Knowledge Representation. Faculty of Economics, Business Administration and Information Technology Doctor(November):246. [↗](https://doi.org/10.5167/uzh-33191)
10. Kuhn, T. 2014. A Survey and Classification of Controlled Natural Languages. Computational Linguistics 40(1):121–170. [↗](https://doi.org/10.1162/coli_a_00168)
11. Liu, X.; He, P.; Chen, W.; and Gao, J. 2019. [Multi-Task Deep Neural Networks for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/). 4487–4496.
12. Miller, G. A.; Beckwith, R.; Fellbaum, C.; Gross, D.; and Miller, K. J. 1990. Introduction to wordnet: An on-line lexical database. International Journal of Lexicography 3(4):235–244. [↗](https://doi.org/10.1093/ijl/3.4.235)
13. Mueller, E. T. 2015. Commonsense Reasoning: An Event Calculus Based Approach. Morgan Kaufmann, second edition edition. [find](https://scholar.google.com/scholar?q=Mueller%2C+E%2E+T%2E+2015%2E+Commonsense+Reasoning%3A+An+Event+Calculus+Based+Approach%2E+Morgan+Kaufmann%2C+second+edition+edition%2E)
14. Sutcliffe, G. 2009. The TPTP problem library and associated infrastructure : ttthe FOF and CNF Parts, v3.5.0. Journal of Automated Reasoning 43(4):337– 362. [find](https://scholar.google.com/scholar?q=Sutcliffe%2C+G%2E+2009%2E+The+TPTP+problem+library+and+associated+infrastructure+%3A+ttthe+FOF+and+CNF+Parts%2C+v3%2E5%2E0%2E+Journal+of+Automated+Reasoning+43%284%29%3A337%E2%80%93+362%2E)
15. Williams, A.; Nangia, N.; and Bowman, S. R. 2017. A broad-coverage challenge corpus for sentence understanding through inference. CoRR abs/1704.05426. [↗](https://arxiv.org/abs/1704.05426)
16. Zhang, Z.; Wu, Y.; Li, Z.; He, S.; and Zhao, H. 2018. I Know What You Want: Semantic Learning for Text Comprehension. [find](https://scholar.google.com/scholar?q=Zhang%2C+Z%2E%3B+Wu%2C+Y%2E%3B+Li%2C+Z%2E%3B+He%2C+S%2E%3B+and+Zhao%2C+H%2E+2018%2E+I+Know+What+You+Want%3A+Semantic+Learning+for+Text+Comprehension%2E)
17. Zhang, Z.; Wu, Y.; Zhao, H.; Li, Z.; Zhang, S.; Zhou, X.; and Zhou, X. 2019. Semantics-aware BERT for Language Understanding. (2017). [find](https://scholar.google.com/scholar?q=Zhang%2C+Z%2E%3B+Wu%2C+Y%2E%3B+Zhao%2C+H%2E%3B+Li%2C+Z%2E%3B+Zhang%2C+S%2E%3B+Zhou%2C+X%2E%3B+and+Zhou%2C+X%2E+2019%2E+Semantics%2Daware+BERT+for+Language+Understanding%2E+%282017%29%2E)

*Updated 24 October 2026*