---
title: "Decomposing Natural Logic Inferences for Neural NLI (2022)"
description: "The paper asks whether transformer NLI models represent the two features that natural logic uses to… 29 references, cited by 3 papers here."
authors: ["Julia Rozanova", "Deborah Ferreira", "Mokanarangan Thayaparan", "Marco Valentino", "Andre Freitas"]
year: 2022
venue: "Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP"
source: "https://aclanthology.org/2022.blackboxnlp-1.33/"
doi: "10.18653/v1/2022.blackboxnlp-1.33"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rozanova-2022-decomposing-natural-logic-inferences-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rozanova-2022-decomposing-natural-logic-inferences-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Decomposing Natural Logic Inferences for Neural NLI

# Decomposing Natural Logic Inferences for Neural NLI

**Authors:** [Julia Rozanova](https://md.research.elamur.ai/free/en/authors/julia-rozanova/), [Deborah Ferreira](https://md.research.elamur.ai/free/en/authors/deborah-ferreira/), [Mokanarangan Thayaparan](https://md.research.elamur.ai/free/en/authors/mokanarangan-thayaparan/), [Marco Valentino](https://md.research.elamur.ai/free/en/authors/marco-valentino/) and [Andre Freitas](https://md.research.elamur.ai/free/en/authors/andre-freitas/) · **Year:** 2022 · **Venue:** [Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP](https://md.research.elamur.ai/free/en/venues/blackboxnlp/) ([BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP 2022](https://md.research.elamur.ai/free/en/venues/blackboxnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.blackboxnlp-1.33/) · [PDF](https://aclanthology.org/2022.blackboxnlp-1.33.pdf) · [DOI 10.18653/v1/2022.blackboxnlp-1.33](https://doi.org/10.18653/v1/2022.blackboxnlp-1.33) · [OpenAlex](https://openalex.org/W4385572900)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether transformer NLI models represent the two features that natural logic uses to decide entailment. The first is context monotonicity: whether a sentence frame keeps or flips the direction of inference. The second is the inclusion relation between the inserted concepts. The authors built NLI-XY, a probing dataset. They took sentence frames with one blank from MED and HELP and filled them with noun pairs from MoNLI plus a small hand-curated set of phrases. Each premise/hypothesis pair carries labels for monotonicity and word-pair relation, and these two labels fully determine the entailment label. They trained diagnostic probes with control tasks and selectivity on internal representations of RoBERTa, BART and BERT NLI models. They also produced error breakdowns and UMAP plots. Models fine-tuned only on MNLI encode concept inclusion well but barely encode monotonicity. In negated or downward contexts they behave as if the context were upward. Extra fine-tuning on HELP strengthens the monotonicity signal and improves challenge-set scores. This links a weakness visible in model behaviour to a gap in the model's internal representations.

## Key points
- NLI-XY has three parts: frames f with one blank labelled ↑/↓, insertion pairs (X, Y) labelled with a relation in {=, ⊑, ⊒}, and the derived pairs (f(X), f(Y)).
- Frames come from MED (manually curated) and the HELP training set. Insertion pairs come from MoNLI plus hand-made phrase pairs with modifiers and prepositional phrases. Each slot and noun is tagged as singular, plural or mass so that combinations stay grammatical.
- Determiners and quantifiers count as part of the context, because their monotonicity is the target of the study.
- For roberta-large-mnli, the monotonicity probe reaches about 59% (accuracy at maximum selectivity) versus about 84% for roberta-large-mnli-HELP. On challenge sets, HELP fine-tuning moves HELP-Test from 36.69% to 97.63%, MED from 46.10% to 78.22%, and NLI-XY from 59.01% to 80.68%.
- bert-base-uncased-snli stays weak on both probes and challenge sets, with or without HELP. facebook/bart-large-mnli gains strongly on challenge sets after HELP.

## Interesting ideas
- Build the dataset so that intermediate labels (frame polarity and word relation) jointly and deterministically fix the final label; every error then traces back to the component that failed.
- Treat a sentence with one hole as a function from ordered concepts to ordered sentences, and classify it as order-preserving or order-reversing.
- Fold determiners and quantifiers into the context rather than the inserted phrase, because closed-class operators decide whether inference flips.
- Plot errors as a heat map over the context-type × relation-type grid. Consistent failure in a whole cell reveals a systematic shortcut, such as ignoring polarity, rather than random noise.
- Deliberately include semantically odd but formally valid pairs, such as "swim in a person" versus "an Irishman", to test whether reasoning follows structure rather than plausibility.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): Entailment-style test items could be generated with stored polarity and relation labels, so a wrong answer points directly at the faulty reasoning step.
- **I2** (world): Each frame position in the world model could be marked ↑/↓, letting hypernym lookups from WordNet propagate entailment correctly under negation and quantifiers such as "no" or "any".
- **I3** (frames): When building frames, attach quantifier and negation scope to the slot as a polarity attribute, instead of leaving it as a loose dependent in the UD tree.
- **I4** (learn): Evaluation reports could break errors down by such grid cells, so that a shortcut shows up as a filled-in cell rather than a small drop in average accuracy.
- **I5** (learn): These items can serve as negative controls, confirming that the reasoner applies monotonicity rules even when world knowledge offers no help.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Manchester](https://md.research.elamur.ai/free/en/universities/university-of-manchester/) — University, Manchester, United Kingdom · 30 works · 43,942 citations
- [Idiap Research Institute](https://md.research.elamur.ai/free/en/labs/idiap-research-institute/) — Research institute, Martigny-Combe, Switzerland · 20 works · 1,757 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Julia Rozanova, Deborah Ferreira, Mokanarangan Thayaparan, Marco Valentino and Andre Freitas. 2022. Decomposing Natural Logic Inferences for Neural NLI. Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rozanova2022decomposing,
  title = {{Decomposing Natural Logic Inferences for Neural NLI}},
  author = {Julia Rozanova, Deborah Ferreira, Mokanarangan Thayaparan, Marco Valentino and Andre Freitas},
  booktitle = {Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP},
  year = {2022},
  doi = {10.18653/v1/2022.blackboxnlp-1.33},
  url = {https://doi.org/10.18653/v1/2022.blackboxnlp-1.33}
}
```

- [← previous Automatic Dialect Density Estimation for African American English](https://md.research.elamur.ai/free/en/papers/johnson-2022-automatic-dialect-density-estimation-african/)
- [next → Linguistically Motivated Evaluation of the 2022 State-of-the-art Machine Translation Systems for Three Language Directions](https://md.research.elamur.ai/free/en/papers/macketanz-2022-linguistically-motivated-evaluation-2022-state/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

29 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lasha Abzianidze. 2017. LangPro: Natural language theorem prover. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 115–120, Copenhagen, Denmark. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1708.09417)
2. Guillaume Alain and Yoshua Bengio. 2018. Understanding intermediate layers using linear classifier probes. [find](https://scholar.google.com/scholar?q=Guillaume+Alain+and+Yoshua+Bengio%2E+2018%2E+Understanding+intermediate+layers+using+linear+classifier+probes%2E)
3. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. [↗](https://doi.org/10.48550/arxiv.1508.05326)
4. Zeming Chen, Qiyue Gao, and Lawrence S. Moss. 2021. Neurallog: Natural language inference with joint neural and logical reasoning. [find](https://scholar.google.com/scholar?q=Zeming+Chen%2C+Qiyue+Gao%2C+and+Lawrence+S%2E+Moss%2E+2021%2E+Neurallog%3A+Natural+language+inference+with+joint+neural+and+logical+reasoning%2E)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
6. Deborah Ferreira, Julia Rozanova, Mokanarangan Thayaparan, Marco Valentino, and André Freitas. 2021. Does my representation capture X? probe-ably. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 194–201, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Deborah+Ferreira%2C+Julia+Rozanova%2C+Mokanarangan+Thayaparan%2C+Marco+Valentino%2C+and+Andr%C3%A9+Freitas%2E+2021%2E+Does+my+representation+capture+X%3F+probe%2Dably%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Asso)
7. Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. [Neural natural language inference models partially embed theories of lexical entailment and negation](https://md.research.elamur.ai/free/en/papers/geiger-2020-neural-natural-language-inference-models/). In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 163–173, Online. Association for Computational Linguistics.
8. Emily Goodwin, Koustuv Sinha, and Timothy J. O’Donnell. 2020. Probing linguistic systematicity. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1958– 1969, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emily+Goodwin%2C+Koustuv+Sinha%2C+and+Timothy+J%2E+O%E2%80%99Donnell%2E+2020%2E+Probing+linguistic+systematicity%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1958%E2%80%93+)
9. Christine Herlihy and Rachel Rudinger. 2021. [MedNLI is not immune: Natural language inference artifacts in the clinical domain](https://md.research.elamur.ai/free/en/papers/herlihy-2021-mednli-is-not-immune-natural/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1020–1027, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48448/cj47-vp53)
10. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d19-1275)
11. John Hewitt and Christopher D. Manning. 2019. [A structural probe for finding syntax in word representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1419)
12. Hai Hu, Qi Chen, Kyle Richardson, Atreyee Mukherjee, Lawrence S. Moss, and Sandra Kuebler. 2020. MonaLog: a lightweight system for natural language inference based on monotonicity. In Proceedings of the Society for Computation in Linguistics 2020, pages 334–344, New York, New York. Association for Computational Linguistics. [↗](https://doi.org/10.7275/te06-g113)
13. Hai Hu and Larry Moss. 2018. Polarity computations in flexible categorial grammar. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 124–129, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hai+Hu+and+Larry+Moss%2E+2018%2E+Polarity+computations+in+flexible+categorial+grammar%2E+In+Proceedings+of+the+Seventh+Joint+Conference+on+Lexical+and+Computational+Semantics%2C+pages+124%E2%80%93129%2C+New+Orleans%2C+Lo)
14. Aikaterini-Lida Kalouli, Richard Crouch, and Valeria de Paiva. 2020. Hy-NLI: a hybrid system for natural language inference. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5235–5249, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aikaterini%2DLida+Kalouli%2C+Richard+Crouch%2C+and+Valeria+de+Paiva%2E+2020%2E+Hy%2DNLI%3A+a+hybrid+system+for+natural+language+inference%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Lingui)
15. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.703)
16. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000–1008, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2021%2E+Question+and+answer+test%2Dtrain+overlap+in+open%2Ddomain+question+answering+datasets%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Ch)
17. Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
18. Bill MacCartney and Christopher D. Manning. 2007. Natural logic for textul inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 193–200, Prague. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bill+MacCartney+and+Christopher+D%2E+Manning%2E+2007%2E+Natural+logic+for+textul+inference%2E+In+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2C+pages+193%E2%80%93200%2C+Prague%2E+Associati)
19. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1334)
20. Tiago Pimentel, Naomi Saphra, Adina Williams, and Ryan Cotterell. 2020. Pareto probing: Trading off accuracy for complexity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3138–3153, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tiago+Pimentel%2C+Naomi+Saphra%2C+Adina+Williams%2C+and+Ryan+Cotterell%2E+2020%2E+Pareto+probing%3A+Trading+off+accuracy+for+complexity%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Langu)
21. Kyle Richardson, Hai Hu, Lawrence S. Moss, and Ashish Sabharwal. 2019. [Probing natural language inference models through semantic fragments](https://md.research.elamur.ai/free/en/papers/richardson-2019-probing-natural-language-inference-models/). CoRR, abs/1909.07521. Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. [↗](https://arxiv.org/abs/1909.07521)
22. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019a. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31–40, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w19-4804)
23. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019b. HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 250–255, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/s19-1027)
24. Kelly Zhang and Samuel Bowman. 2018. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 359–361, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kelly+Zhang+and+Samuel+Bowman%2E+2018%2E+Language+modeling+teaches+you+more+than+translation+does%3A+Lessons+learned+through+auxiliary+syntactic+task+analysis%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+Blac)
25. Julia Rozanova, Deborah Ferreira, Mokanarangan Thayaparan, Marco Valentino, and André Freitas. 2021. [Supporting context monotonicity abstractions in neural nli models](https://md.research.elamur.ai/free/en/papers/rozanova-2021-supporting-context-monotonicity-abstractions-neural/). [↗](https://doi.org/10.48550/arxiv.2105.08008)
26. V. Sanchez. 1991. Studies on natural logic and categorial grammar. [find](https://scholar.google.com/scholar?q=V%2E+Sanchez%2E+1991%2E+Studies+on+natural+logic+and+categorial+grammar%2E)
27. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. [Probing pretrained language models for lexical semantics](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computational Linguistics.
28. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-1101)
29. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 Conference on Empirical

*Updated 24 October 2026*