---
title: "Evaluating Mathematical Reasoning Beyond Accuracy — Xia 2024"
description: "The authors propose REASONEVAL, which scores every step of a solution on two dimensions… 25 references, cited by 66 papers here."
authors: ["Shijie Xia", "Xuefeng Li", "Yixin Liu", "Tongshuang Wu", "Pengfei Liu"]
year: 2024
venue: "arXiv:2404.05692"
source: "https://arxiv.org/abs/2404.05692"
doi: "10.48550/arxiv.2404.05692"
arxiv: "2404.05692"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xia-2024-evaluating-mathematical-reasoning-accuracy/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xia-2024-evaluating-mathematical-reasoning-accuracy/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Evaluating Mathematical Reasoning Beyond Accuracy

# Evaluating Mathematical Reasoning Beyond Accuracy

**Authors:** [Shijie Xia](https://md.research.elamur.ai/free/en/authors/shijie-xia/), Xuefeng Li, [Yixin Liu](https://md.research.elamur.ai/free/en/authors/yixin-liu/), [Tongshuang Wu](https://md.research.elamur.ai/free/en/authors/tongshuang-wu/) and [Pengfei Liu](https://md.research.elamur.ai/free/en/authors/pengfei-liu/) · **Year:** 2024 · **Venue:** [arXiv:2404.05692](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/) ([Proceedings of the AAAI Conference on Artificial Intelligence 2024](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/2024/)) · **Publisher:** [Association for the Advancement of Artificial Intelligence](https://md.research.elamur.ai/free/en/publishers/association-for-the-advancement-of-artificial-intelligence/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.05692) · [PDF](https://arxiv.org/pdf/2404.05692) · [DOI 10.48550/arxiv.2404.05692](https://doi.org/10.48550/arxiv.2404.05692) · [arXiv 2404.05692](https://arxiv.org/abs/2404.05692) · [OpenAlex](https://openalex.org/W4409362112)

**Citations:** 18 citations (OpenAlex, 2026-10-14) · [66](https://md.research.elamur.ai/free/en/papers/xia-2024-evaluating-mathematical-reasoning-accuracy/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most benchmarks for LLM math reasoning check only whether the final answer is right. This can hide flawed logic or wasted steps inside a solution. The authors propose REASONEVAL, which scores every step of a solution on two dimensions:
- **Validity:** the step contains no calculation or logical error.
- **Redundancy:** the step is correct but does not move the solution forward.

Each step is classified as positive, neutral or negative. Step scores are then combined into solution-level scores using min (for validity) and max (for redundancy). The evaluators are LLMs such as Llama2 and Mistral variants, fine-tuned with a classification head on the human step labels in PRM800K.

The authors test the evaluators on a new meta-evaluation set, MR-MATH. The strongest setups, which pair math-capable base models with high-quality labels, clearly beat baseline methods and generalize well. Applying REASONEVAL to math-specialized LLMs shows that higher answer accuracy on hard problems does not always mean better reasoning. Used as a data filter, it can also shrink SFT data while keeping accuracy and improving step quality. The best evaluator and the evaluation scripts are open-sourced.

## Key points
- The training data is PRM800K, with about 800K step-level human labels on roughly 75K GPT-4 solutions to MATH problems.
- MR-MATH contains Abel and WizardMath solutions labeled by undergraduates with strong math backgrounds: 83 have erroneous steps and 76 do not.
- Model scale matters for the evaluated math LLMs' false positive rate. LLaMA13B-PRM800K scores 40.6% versus 21.8% for LLaMA70B-PRM800K, despite identical training data. RLHF helps only for stronger models.
- For data selection, the authors filtered 10K MMIQC responses with REASONEVAL (WizardMath-7B-V1.1) and fine-tuned mistral-7b on the result:
  - The subset kept 56.7% of the data.
  - MATH accuracy was 22.0, versus 22.2 for the full set and 20.0 for a random subset of equal size.
  - Validity rose from 65.2 to 67.8.
  - Average solution length fell from about 723 to 701 tokens.
- The code and models are available at github.com/GAIR-NLP/ReasonEval.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 63 works · 15,047 citations
- [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) — University, New Haven, United States · 54 works · 11,964 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 44 works · 10,162 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 140 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 116 works · [➗ Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) · 21 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu and Pengfei Liu. 2024. Evaluating Mathematical Reasoning Beyond Accuracy. arXiv:2404.05692.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xia2024evaluating,
  title = {{Evaluating Mathematical Reasoning Beyond Accuracy}},
  author = {Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu and Pengfei Liu},
  year = {2024},
  doi = {10.48550/arxiv.2404.05692},
  eprint = {2404.05692},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.05692}
}
```

- [← previous Detoxifying Large Language Models via Knowledge Editing](https://md.research.elamur.ai/free/en/papers/wang-2024-detoxifying-large-language-models-knowledge/)
- [next → Machine Unlearning of Pre-trained Large Language Models](https://md.research.elamur.ai/free/en/papers/yao-2024-machine-unlearning-pre-trained-large/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Azerbayev, Z.; Schoelkopf, H.; Paster, K.; Dos Santos, M.; McAleer, S. M.; Jiang, A. Q.; Deng, J.; Biderman, S.; and Welleck, S. 2023. [Llemma: An Open Language Model for Mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/). In The Twelfth International Conference on Learning Representations.
2. Chern, E.; Zou, H.; Li, X.; Hu, J.; Feng, K.; Li, J.; and Liu, P. 2023. Generative AI for Math: Abel. [https://github.com/](https://github.com/) GAIR-NLP/abel. [↗](https://github.com/)
3. Clinciu, M.-A.; Eshghi, A.; and Hastie, H. 2021. A Study of Automatic Metrics for the Evaluation of Natural Language Explanations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2376–2387. Online: Association for Computati… [find](https://scholar.google.com/scholar?q=Clinciu%2C+M%2E%2DA%2E%3B+Eshghi%2C+A%2E%3B+and+Hastie%2C+H%2E+2021%2E+A+Study+of+Automatic+Metrics+for+the+Evaluation+of+Natural+Language+Explanations%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+)
4. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). ArXiv preprint, abs/2110.14168.
5. Dai, J.; Pan, X.; Sun, R.; Ji, J.; Xu, X.; Liu, M.; Wang, Y.; and Yang, Y. 2023. [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://md.research.elamur.ai/free/en/papers/dai-2023-safe-rlhf-safe-reinforcement-learning/). In The Twelfth International Conference on Learning Representations.
6. Golovneva, O.; Chen, M. P.; Poff, S.; Corredor, M.; Zettlemoyer, L.; Fazel-Zarandi, M.; and Celikyilmaz, A. 2022. [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://md.research.elamur.ai/free/en/papers/golovneva-2022-roscoe-suite-metrics-scoring-step/). In The Eleventh International Conference on Learning Representations.
7. Hao, S.; Gu, Y.; Luo, H.; Liu, T.; Shao, X.; Wang, X.; Xie, S.; Ma, H.; Samavedhi, A.; Gao, Q.; Wang, Z.; and Hu, Z. 2024. LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models. arXiv:2404.05221. — [arXiv:2404.05221](https://arxiv.org/abs/2404.05221) [find](https://scholar.google.com/scholar?q=Hao%2C+S%2E%3B+Gu%2C+Y%2E%3B+Luo%2C+H%2E%3B+Liu%2C+T%2E%3B+Shao%2C+X%2E%3B+Wang%2C+X%2E%3B+Xie%2C+S%2E%3B+Ma%2C+H%2E%3B+Samavedhi%2C+A%2E%3B+Gao%2C+Q%2E%3B+Wang%2C+Z%2E%3B+and+Hu%2C+Z%2E+2024%2E+LLM+Reasoners%3A+New+Evaluation%2C+Library%2C+and+Analysis+of+Step%2Dby%2DStep+Reasonin)
8. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. [Measuring Mathematical Problem Solving With the MATH Dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). Jiang, A. Q.; Sablayr…
9. Lewkowycz, A.; Andreassen, A.; Dohan, D.; Dyer, E.; Michalewski, H.; Ramasesh, V.; Slone, A.; Anil, C.; Schlag, I.; Gutman-Solo, T.; et al. 2022. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Information Processing Systems, 35: 3843–3857.
10. Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023. [Let’s Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). In The Twelfth International Conference on Learning Representations.
11. Liu, H.; and Yao, A. C.-C. 2024. Augmenting math word problems via iterative question composing. ArXiv preprint, abs/2401.09003. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%3B+and+Yao%2C+A%2E+C%2E%2DC%2E+2024%2E+Augmenting+math+word+problems+via+iterative+question+composing%2E+ArXiv+preprint%2C+abs%2F2401%2E09003%2E)
12. Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E%3B+and+Hutter%2C+F%2E+2019%2E+Decoupled+Weight+Decay+Regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E)
13. OpenReview.net. Lu, P.; Qiu, L.; Yu, W.; Welleck, S.; and Chang, K.-W. 2023. A Survey of Deep Learning for Mathematical Reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14605– 14631. [find](https://scholar.google.com/scholar?q=OpenReview%2Enet%2E+Lu%2C+P%2E%3B+Qiu%2C+L%2E%3B+Yu%2C+W%2E%3B+Welleck%2C+S%2E%3B+and+Chang%2C+K%2E%2DW%2E+2023%2E+A+Survey+of+Deep+Learning+for+Mathematical+Reasoning%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Comp)
14. Luo, H.; Sun, Q.; Xu, C.; Zhao, P.; Lou, J.; Tao, C.; Geng, X.; Lin, Q.; Chen, S.; and Zhang, D. 2023. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). ArXiv preprint, abs/2308.09583.
15. Ma, Q.; Zhou, H.; Liu, T.; Yuan, J.; Liu, P.; You, Y.; and Yang, H. 2023. Let’s reward step by step: Step-Level reward model as the Navigators for Reasoning. ArXiv preprint, abs/2310.10080. [find](https://scholar.google.com/scholar?q=Ma%2C+Q%2E%3B+Zhou%2C+H%2E%3B+Liu%2C+T%2E%3B+Yuan%2C+J%2E%3B+Liu%2C+P%2E%3B+You%2C+Y%2E%3B+and+Yang%2C+H%2E+2023%2E+Let%E2%80%99s+reward+step+by+step%3A+Step%2DLevel+reward+model+as+the+Navigators+for+Reasoning%2E+ArXiv+preprint%2C+abs%2F2310%2E10080%2E)
16. OpenAI, R. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, 2303– 08774.
17. Prasad, A.; Saha, S.; Zhou, X.; and Bansal, M. 2023. Re-CEval: Evaluating Reasoning Chains via Correctness and Informativeness. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 10066– 10086. [find](https://scholar.google.com/scholar?q=Prasad%2C+A%2E%3B+Saha%2C+S%2E%3B+Zhou%2C+X%2E%3B+and+Bansal%2C+M%2E+2023%2E+Re%2DCEval%3A+Evaluating+Reasoning+Chains+via+Correctness+and+Informativeness%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+La)
18. Saparov, A.; and He, H. 2022. [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://md.research.elamur.ai/free/en/papers/saparov-2022-language-models-greedy-reasoners-systematic/). In The Eleventh International Conference on Learning Representations.
19. Sawada, T.; Paleka, D.; Havrilla, A.; Tadepalli, P.; Vidas, P.; Kranias, A.; Nay, J. J.; Gupta, K.; and Komatsuzaki, A. 2023. Arb: Advanced reasoning benchmark for large language models. ArXiv preprint, abs/2307.13692. [find](https://scholar.google.com/scholar?q=Sawada%2C+T%2E%3B+Paleka%2C+D%2E%3B+Havrilla%2C+A%2E%3B+Tadepalli%2C+P%2E%3B+Vidas%2C+P%2E%3B+Kranias%2C+A%2E%3B+Nay%2C+J%2E+J%2E%3B+Gupta%2C+K%2E%3B+and+Komatsuzaki%2C+A%2E+2023%2E+Arb%3A+Advanced+reasoning+benchmark+for+large+language+models%2E+ArXiv+preprin)
20. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09288. Tyen, G.; Mansoor, H.; Chen, P.; Mak, T.; and Cărbune, V. 2023. LLMs cann… [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%3B+Martin%2C+L%2E%3B+Stone%2C+K%2E%3B+Albert%2C+P%2E%3B+Almahairi%2C+A%2E%3B+Babaei%2C+Y%2E%3B+Bashlykov%2C+N%2E%3B+Batra%2C+S%2E%3B+Bhargava%2C+P%2E%3B+Bhosale%2C+S%2E%3B+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+ArXiv+)
21. Uesato, J.; Kushman, N.; Kumar, R.; Song, F.; Siegel, N.; Wang, L.; Creswell, A.; Irving, G.; and Higgins, I. 2022. [Solving math word problems with process-and outcome-based feedback](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/). ArXiv preprint, abs/2211.14275.
22. Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2023. [Math-Shepherd: A Label-Free Step-by-Step Verifier for LLMs in Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). ArXiv preprint, abs/2312.08935.
23. Yu, L.; Jiang, W.; Shi, H.; Jincheng, Y.; Liu, Z.; Zhang, Y.; Kwok, J.; Li, Z.; Weller, A.; and Liu, W. 2023. [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). In The Twelfth International Conference on Learning Representations.
24. Zeng, Z.; Chen, P.; Liu, S.; Jiang, H.; and Jia, J. 2024. MR-GSM8K: A Meta-Reasoning Revolution in Large Language Model Evaluation. arXiv:2312.17080. [find](https://scholar.google.com/scholar?q=Zeng%2C+Z%2E%3B+Chen%2C+P%2E%3B+Liu%2C+S%2E%3B+Jiang%2C+H%2E%3B+and+Jia%2C+J%2E+2024%2E+MR%2DGSM8K%3A+A+Meta%2DReasoning+Revolution+in+Large+Language+Model+Evaluation%2E+arXiv%3A2312%2E17080%2E)
25. Zhang, M.; Wang, Z.; Yang, Z.; Feng, W.; and Lan, A. 2023. Interpretable Math Word Problem Solution Generation via Step-by-step Planning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 6858–6877. [find](https://scholar.google.com/scholar?q=Zhang%2C+M%2E%3B+Wang%2C+Z%2E%3B+Yang%2C+Z%2E%3B+Feng%2C+W%2E%3B+and+Lan%2C+A%2E+2023%2E+Interpretable+Math+Word+Problem+Solution+Generation+via+Step%2Dby%2Dstep+Planning%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+f)

*Updated 24 October 2026*