--- title: "An Independent Evaluation of ChatGPT on Mathematical Word" description: "The authors describe this as the first independent study of the model. 59% accuracy. Cited by 14 papers here." authors: ["Paulo Shakarian", "Abhinav Koyyalamudi", "Noel Ngu", "Lakshmivihari Mareedu"] year: 2023 venue: "arXiv:2302.13814 · AAAI Spring Symposium 2023 (MAKE)" source: "https://arxiv.org/abs/2302.13814" doi: "10.48550/arxiv.2302.13814" arxiv: "2302.13814" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/shakarian-2023-independent-evaluation-chatgpt-mathematical-word/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shakarian-2023-independent-evaluation-chatgpt-mathematical-word/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP) # An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP) **Authors:** Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu and Lakshmivihari Mareedu · **Year:** 2023 · **Venue:** arXiv:2302.13814 · AAAI Spring Symposium 2023 (MAKE) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2302.13814) · [PDF](https://arxiv.org/pdf/2302.13814) · [DOI 10.48550/arxiv.2302.13814](https://doi.org/10.48550/arxiv.2302.13814) · [arXiv 2302.13814](https://arxiv.org/abs/2302.13814) · [OpenAlex](https://openalex.org/W4322759267) **Citations:** 35 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/shakarian-2023-independent-evaluation-chatgpt-mathematical-word/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper investigates how ChatGPT handles math word problems (MWPs) and which properties of a problem predict that the model will get it wrong. The authors describe this as the first independent study of the model. They submitted all 1,000 problems from DRAW-1K, which pairs each problem with answers and template algebraic equations, through an unofficial CLI Python wrapper, since no API existed then. They ran three settings: January 2023 with an answer-only prompt, February 2023 with a revised answer-only prompt, and February 2023 with ChatGPT free to explain its reasoning. Error rates depended heavily on whether the model showed its work. It failed on about 84% of problems with answer-only output and about 20% when it wrote out its solution. Counts of unknowns and arithmetic operations, taken from the gold equations, correlated with failure. The released response dataset and baseline classifiers that predict failure support work on introspective front-ends that judge in advance whether an LLM can be trusted on a given input. ## Key points - **Answer-only runs:** in January, 831 answers were wholly incorrect and 9 claimed "No Solution", giving 84% failure. February results were almost identical. - **Runs with visible work:** 513 problems were fully solved (51%), and counting partially correct outputs raises this to about 80%. For comparison, the EPT model reports 59% accuracy. - **Addition/subtraction counts:** failure probability grows linearly with these in every setting, with R² of 0.821 in January, 0.870 in February without work, and 0.915 with work shown. - **Multiplication/division counts:** - These show a linear link to failure only when work is shown (R² = 0.802). - The number of unknowns stops mattering in that setting. - **Failure-prediction baselines:** - Random forest (scikit-learn) and XGBoost classifiers were evaluated with five-fold cross-validation. - Their features come from oracle ground-truth equations. - Precision on incorrect answers reaches 0.90–0.98 in the answer-only settings. - The response data is published at github.com/lab-v2/ChatGPT_MWP_eval. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Arizona State University](https://md.research.elamur.ai/free/en/universities/arizona-state-university/) — University, Tempe, United States · 54 works · 2,356 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 189 works · [➗ Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) · 52 works 🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works ## Cite Cite as: Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu and Lakshmivihari Mareedu. 2023. An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP). arXiv:2302.13814 · AAAI Spring Symposium 2023 (MAKE). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{shakarian2023independent, title = {{An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)}}, author = {Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu and Lakshmivihari Mareedu}, year = {2023}, doi = {10.48550/arxiv.2302.13814}, eprint = {2302.13814}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2302.13814} } ``` - [← previous TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks](https://md.research.elamur.ai/free/en/papers/jiang-2023-tigerscore-towards-building-explainable-metric/) - [next → From Word Types to Tokens and Back: A Survey of Approaches to Word Meaning Representation and Interpretation](https://md.research.elamur.ai/free/en/papers/apidianaki-2023-word-types-tokens-back-survey/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) *Updated 31 October 2026*