---
title: "Large Language Models and Mathematical Reasoning Failures"
description: "The paper asks whether LLMs genuinely reason through mathematical word problems or just reach… 14 references, cited by 7 papers here."
authors: ["Johan Boye", "Birger Moell"]
year: 2025
venue: "arXiv:2502.11574"
source: "https://arxiv.org/abs/2502.11574"
doi: "10.48550/arxiv.2502.11574"
arxiv: "2502.11574"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/boye-2025-large-language-models-mathematical-reasoning/"
---
Elamur: [Mova](https://md.mova.elamur.ai/free/en/) · [Dictionary](https://md.dictionary.elamur.ai/free/en/) · **Research** · [Groupants](https://md.groupants.elamur.ai/free/en/)

[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/boye-2025-large-language-models-mathematical-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / Large Language Models and Mathematical Reasoning Failures

# Large Language Models and Mathematical Reasoning Failures

**Authors:** [Johan Boye](https://md.research.elamur.ai/free/en/authors/johan-boye/) and Birger Moell · **Year:** 2025 · **Venue:** arXiv:2502.11574 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2502.11574) · [PDF](https://arxiv.org/pdf/2502.11574) · [DOI 10.48550/arxiv.2502.11574](https://doi.org/10.48550/arxiv.2502.11574) · [arXiv 2502.11574](https://arxiv.org/abs/2502.11574) · [OpenAlex](https://openalex.org/W4407719116)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/boye-2025-large-language-models-mathematical-reasoning/cited-by/) in our library, of the 150,927 papers whose bibliographies we read

## Summary
The paper asks whether LLMs genuinely reason through mathematical word problems or just reach plausible answers by shortcuts. Most benchmarks only check the final answer, so the authors built a small, mostly original set of 50 high-school-level word problems. Four come from a Swedish puzzle book (Vaderlind, 1996). The problems mix counting, divisibility, algebra, probability, geometry and everyday common sense. Each of eight models was queried once through its API. An experienced teacher graded all 400 responses by hand, judging the reasoning as well as the result.

Reasoning-oriented models clearly outperformed older ones, but none was reliable. Recurring failures were:
- spatial reasoning, such as a leash wrapping around lampposts,
- planning and strategy,
- arithmetic slips,
- hidden assumptions and ignored constraints,
- generalizing too quickly from a few numbers.

Some correct answers rested on invalid arguments. The work shows that answer-only scoring can overstate LLM mathematical skill. It argues that flawed solutions can look convincing to non-experts, so step-level evaluation matters.

## Key points
- Models evaluated (correct answer plus correct solution, out of 50): o3-mini 40, o1 37, deepseek-r1 36, o1-preview 30, gemini-2.0-pro-exp 23, gpt-4o 14, llama-3.3-70B 10, mixtral-8x7B 0.
- In 21 of 400 responses (about 5%), the final answer was right but the justification was wrong. In a few other cases the reasoning was sound but a small calculation error spoiled the result.
- No model solved the retractable-leash problem (Problem 11). On the cube-edge numbering problem (Problem 26), only o3-mini gave a fully correct solution. On the "progressive set" problem (Problem 47), only o1 and o3-mini succeeded.
- The dataset, code and graded outputs are released at github.com/jboye12/llm-probs.
- The authors note several limitations: a single attempt per model, no trigonometry or calculus, chain-of-thought hidden for several models, and possible overlap with training data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 233 works · [➗ Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) · 73 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Johan Boye and Birger Moell. 2025. Large Language Models and Mathematical Reasoning Failures. arXiv:2502.11574.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{boye2025large,
  title = {{Large Language Models and Mathematical Reasoning Failures}},
  author = {Johan Boye and Birger Moell},
  year = {2025},
  doi = {10.48550/arxiv.2502.11574},
  eprint = {2502.11574},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.11574}
}
```

- [← previous Language Models Fail to Introspect About Their Knowledge of Language](https://md.research.elamur.ai/free/en/papers/song-2025-language-models-fail-introspect-about/)
- [next → Liger: Linearizing Large Language Models to Gated Recurrent Structures](https://md.research.elamur.ai/free/en/papers/lan-2025-liger-linearizing-large-language-models/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/)

## References

14 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Antoine Roux Albert Q. Jiang, Alexandre Sablayrolles et al. 2024. Mixtral of experts. Preprint, arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Antoine+Roux+Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles+et+al%2E+2024%2E+Mixtral+of+experts%2E+Preprint%2C+arXiv%3A2401%2E04088%2E)
2. Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov, and Sergei Tilga. 2024. [U-math: A university-level benchmark for evaluating mathematical skills in llms](https://md.research.elamur.ai/free/en/papers/chernyshev-2024-u-math-university-level-benchmark/). arXiv preprint arXiv:2412.03205.
3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
4. Google. 2024. Introducing gemini 2.0: our new ai model for the agentic era. [find](https://scholar.google.com/scholar?q=Google%2E+2024%2E+Introducing+gemini+2%2E0%3A+our+new+ai+model+for+the+agentic+era%2E)
5. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
6. Gautier Izacard Hugo Touvron, Thibaut Lavril et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). Preprint, arXiv:2302.13971.
7. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024. [Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers](https://md.research.elamur.ai/free/en/papers/li-2024-gsm-plus-comprehensive-benchmark-evaluating/). arXiv preprint arXiv:2402.19255.
8. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. [Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models](https://md.research.elamur.ai/free/en/papers/mirzadeh-2024-gsm-symbolic-understanding-limitations-mathematical/). arXiv preprint arXiv:2410.05229.
9. OpenAI. 2024a. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Preprint, arXiv:2303.08774.
10. OpenAI. 2024b. Introducing openai o1-preview. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024b%2E+Introducing+openai+o1%2Dpreview%2E)
11. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744.
12. Akshara Prabhakar, Thomas L. Griffiths, and R. Thomas McCoy. 2024. Deciphering the factors influencing the efficacy of chain-of-thought: Probability, memorization, and noisy reasoning. Preprint, arXiv:2407.01687. [find](https://scholar.google.com/scholar?q=Akshara+Prabhakar%2C+Thomas+L%2E+Griffiths%2C+and+R%2E+Thomas+McCoy%2E+2024%2E+Deciphering+the+factors+influencing+the+efficacy+of+chain%2Dof%2Dthought%3A+Probability%2C+memorization%2C+and+noisy+reasoning%2E+Preprint%2C+arXiv)
13. P. Vaderlind. 1996. Fler matematiska tankenötter. Telegram bokförlag. [find](https://scholar.google.com/scholar?q=P%2E+Vaderlind%2E+1996%2E+Fler+matematiska+tanken%C3%B6tter%2E+Telegram+bokf%C3%B6rlag%2E)
14. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837.

*Updated 3 November 2026*