---
title: "Math Natural Language Inference: this should be easy! (2025)"
description: "The paper tests whether current LLMs can handle natural language inference on research-level mathematical prose, a task the… 17 references."
authors: ["Valeria de Paiva", "Qiyue Gao", "Hai Hu", "Pavel Kovalev", "Yikang Liu", "Lawrence S. Moss", "Zhiheng Qian"]
year: 2025
venue: "Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025)"
source: "https://aclanthology.org/2025.starsem-1.14/"
doi: "10.18653/v1/2025.starsem-1.14"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/paiva-2025-math-natural-language-inference-this/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/paiva-2025-math-natural-language-inference-this/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/17/) / Math Natural Language Inference: this should be easy!

# Math Natural Language Inference: this should be easy!

**Authors:** [Valeria de Paiva](https://md.research.elamur.ai/free/en/authors/valeria-de-paiva/), Qiyue Gao, [Hai Hu](https://md.research.elamur.ai/free/en/authors/hai-hu/), Pavel Kovalev, Yikang Liu, [Lawrence S. Moss](https://md.research.elamur.ai/free/en/authors/lawrence-s-moss/) and Zhiheng Qian · **Year:** 2025 · **Venue:** [Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (\*SEM 2025)](https://md.research.elamur.ai/free/en/venues/starsem/) ([\*SEM 2025](https://md.research.elamur.ai/free/en/venues/starsem/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.starsem-1.14/) · [PDF](https://aclanthology.org/2025.starsem-1.14.pdf) · [DOI 10.18653/v1/2025.starsem-1.14](https://doi.org/10.18653/v1/2025.starsem-1.14) · [OpenAlex](https://openalex.org/W4416033813)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper tests whether current LLMs can handle natural language inference on research-level mathematical prose, a task the authors call Math NLI, and whether LLMs can build and label such corpora themselves. Premises come from abstracts in the TAC (Theory and Applications of Categories) corpus. Researchers with both mathematical and NLI experience wrote entailed, contradictory and neutral hypotheses for 31 selected sentences, giving a seed set of 279 pairs. A second corpus has GPT-written hypotheses. Ten LLMs labelled both sets. The authors measured accuracy and also how often the models agreed with one another. Items where models clashed with the human labels were analysed by hand to find recurring error types. Accuracy on the human set is moderate to high. Neutral pairs are the main weak spot, and models still slip on simple inferences. In some settings, a majority vote of LLMs is roughly as good as human labels. Hypothesis-only shortcuts no longer seem to work for these models. The authors conclude that mathematical text does not make NLI easier, and they release both corpora.

## Key points
- Seed corpus: 31 premises drawn from the 432 "Goldilocks" sentences (moderate length, little LaTeX) in the TAC corpus of about 3K sentences. Three annotators each wrote one E, one C and one N hypothesis per premise, giving 279 balanced pairs.
- Ten LLMs were evaluated in two groups:
  - Group 1: GPT-4, Llama 2, Llama 3, Claude 3, Mistral-large.
  - Group 2: Llama-3.1-70B-Instruct, Qwen2-72B-Instruct, Mixtral-8x22B, deepseek-llm-67b-chat, gemma-2-27b-it.
- Accuracy on the seed corpus ranged from 71% to 91%. Most models labelled E and C pairs well but misjudged N pairs.
  - Llama 3 got only 48% of neutral pairs right and called 35% of them contradictions.
  - Claude 3 did best on N, at 84.9%.
- About 20 "red pairs" contradicted human labels. Their failure types were:
  - ignored context;
  - vague quantifiers;
  - lexical/pragmatic ambiguity, such as "resemble";
  - naming of math entities;
  - unknown math concepts.
- On a hypothesis-only baseline, the Group 2 models labelled nearly every item N (0.96–1.00), which is the correct behaviour. On the GPT-generated corpus, accuracy ranged from about 60% to 87%. Both corpora are released at github.com/MathNLI/MathNLI.

## Interesting ideas
- Pairs where 4–5 independent models disagree with the gold label are pulled aside as "red pairs" for expert review, and they are discarded if experts find them controversial; this turns model disagreement into a cheap way to find bad or ambiguous data.
- A hypothesis-only run, where the model sees no premise, should yield nearly all N, so it works as a negative control for shortcut learning.
- Annotators treated domain concepts as black boxes, writing inferences that need no knowledge of what a term means; this works because coined names can mislead (for instance, skew monoidal categories do not in general qualify as monoidal ones).
- A common model error was dropping a scoping condition such as "in the nilpotent case" and comparing only the main clauses, which turned a neutral pair into a contradiction.
- Some words carry a defeasible implicature: "resemble" suggests the two things are not identical, and "numerous examples" was judged by experts to entail "two examples".

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver-data agreement could apply the same rule. Sentences where most witness parsers contradict the current gold get routed to review rather than trained on.
- **I2** (learn): Mova's readers could be fed questions with the story removed. Any answer above chance would signal reliance on surface cues instead of the built world.
- **I3** (world): Unknown compound terms should become opaque entities with no assumed hypernym from head-word matching. Then "V-n-category" would not silently inherit from "V-category".
- **I4** (frames): Frames should keep conditional and restrictive adjuncts as part of the asserted proposition. Contradiction checks should then fire only when the scopes match.
- **I5** (frames): The implicature module could hold a small lexicon of such cancellable inferences, marked separately from strict entailments. That would let answers explain which reading they relied on.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 149 works · 16,161 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 73 works · 51,948 citations
- [Indiana University](https://md.research.elamur.ai/free/en/universities/indiana-university/) — University, Bloomington, United States · 41 works · 3,663 citations
- [Topos Institute](https://md.research.elamur.ai/free/en/ngos/topos-institute/) — Nonprofit, Berkeley, United States · 4 works · 19 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 252 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Valeria de Paiva, Qiyue Gao, Hai Hu, Pavel Kovalev, Yikang Liu, Lawrence S. Moss and Zhiheng Qian. 2025. Math Natural Language Inference: this should be easy!. Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (\*SEM 2025).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{paiva2025math,
  title = {{Math Natural Language Inference: this should be easy!}},
  author = {Valeria {de Paiva}, Qiyue Gao, Hai Hu, Pavel Kovalev, Yikang Liu, Lawrence S. Moss and Zhiheng Qian},
  booktitle = {Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.starsem-1.14},
  url = {https://doi.org/10.18653/v1/2025.starsem-1.14}
}
```

- [← previous Martín Arista, Javier, ed., Sara Domínguez Barragán, Luisa Fidalgo Allo, Laura García Fernández, Yosra Hamdoun Bghiyel, Miguel Lacalle Palacios, Raquel Mateo Mendaza, Carmen Novo Urraca, Ana Elvira Ojanguren López, Esaúl Ruiz Narbona, Roberto Torre Alonso, Marta Tío Sáenz, and Raquel Vea Escarza. 2024. “Nerthusv5. Interface of textual, lexicographical and secondary sources of Old English.” www.nerthusproject.com.](https://md.research.elamur.ai/free/en/papers/tichy-2025-martin-arista-javier-ed-sara/)
- [next → MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages](https://md.research.elamur.ai/free/en/papers/sailor-2025-meralion-ser-robust-speech-emotion/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/17/)

## References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609. [↗](https://arxiv.org/abs/2309.16609)
2. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, and 1 others. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. [↗](https://arxiv.org/abs/2401.02954)
3. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of EMNLP. [↗](https://doi.org/10.48550/arxiv.1508.05326)
4. Jacob Collard, Valeria de Paiva, Brendan Fong, and Eswaran Subrahmanian. 2022. Extracting mathematical concepts from text. Preprint, arXiv:2208.13830. [↗](https://arxiv.org/abs/2208.13830)
5. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783. [↗](https://arxiv.org/abs/2407.21783)
6. Wesley H. Holliday, Matthew Mandelkern, and Cedegao E. Zhang. 2024. Conditional and modal reasoning in large language models. Preprint, arXiv:2401.17169. [↗](https://arxiv.org/abs/2401.17169)
7. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [↗](https://arxiv.org/abs/2401.04088)
8. Aikaterini-Lida Kalouli, Valeria de Paiva, and Livy Real. 2017. Correcting contradictions. In Proceedings of the Computing Natural Language Inference Workshop. [find](https://scholar.google.com/scholar?q=Aikaterini%2DLida+Kalouli%2C+Valeria+de+Paiva%2C+and+Livy+Real%2E+2017%2E+Correcting+contradictions%2E+In+Proceedings+of+the+Computing+Natural+Language+Inference+Workshop%2E)
9. Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. 2024. Causal reasoning and large language models: Opening a new frontier for causality. Preprint, arXiv:2305.00050. [↗](https://arxiv.org/abs/2305.00050)
10. Lovish Madaan, David Esiobu, Pontus Stenetorp, Barbara Plank, and Dieuwke Hupkes. 2024. Lost in inference: Rediscovering the role of natural language inference for large language models. arXiv preprint arXiv:2411.14103. [↗](https://arxiv.org/abs/2411.14103)
11. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. [Hypothesis only baselines in natural language inference](https://md.research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/). In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/s18-2023)
12. Anastasiia Sedova, Robert Litschko, Diego Frassinelli, Benjamin Roth, and Barbara Plank. 2024. To know or not to know? analyzing self-consistency of large language models under ambiguity. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 17203–17217, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anastasiia+Sedova%2C+Robert+Litschko%2C+Diego+Frassinelli%2C+Benjamin+Roth%2C+and+Barbara+Plank%2E+2024%2E+To+know+or+not+to+know%3F+analyzing+self%2Dconsistency+of+large+language+models+under+ambiguity%2E+In+Findings+)
13. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. 4. P: We define eventually cyclic Boolean flows and the eventually cyclic spectrum of a Boolean flow. H: The definition of the eventually cyclic spectrum of a Boolean flow uses the definition of eventually cyclic Boolean flows. [↗](https://arxiv.org/abs/2408.00118)
14. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288. [↗](https://arxiv.org/abs/2307.09288)
15. Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. 2024. [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature, 625:476 – 482. 5. P: The axioms resemble those for monoidal Abelian categories with the addition of an involutive functor. H: The axioms are the ones of monoidal Abelian categories. [↗](https://doi.org/10.1038/s41586-023-06747-5)
16. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 6. P: The category of Set-valued presheaves on a small category B is a topos. H: The category of Set-valued presheaves on a small category C is a topos.
17. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. 2024. [Are large language models really good logical reasoners? a comprehensive evaluation and beyond](https://md.research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/). Preprint, arXiv:2306.09841. [↗](https://arxiv.org/abs/2306.09841)

*Updated 1 November 2026*