--- title: "GPT-4 Can't Reason — paper by Arkoudas (2023)" description: "The author argues that standard reasoning benchmarks are poor evidence. 11 references, cited by 22 papers here." authors: ["Konstantine Arkoudas"] year: 2023 venue: "arXiv:2308.03762" source: "https://arxiv.org/abs/2308.03762" doi: "10.48550/arxiv.2308.03762" arxiv: "2308.03762" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/arkoudas-2023-gpt-4-t-reason/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/arkoudas-2023-gpt-4-t-reason/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / GPT-4 Can't Reason # GPT-4 Can't Reason **Authors:** Konstantine Arkoudas · **Year:** 2023 · **Venue:** arXiv:2308.03762 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2308.03762) · [PDF](https://arxiv.org/pdf/2308.03762) · [DOI 10.48550/arxiv.2308.03762](https://doi.org/10.48550/arxiv.2308.03762) · [arXiv 2308.03762](https://arxiv.org/abs/2308.03762) · [OpenAlex](https://openalex.org/W4385714397) **Citations:** 9 citations (OpenAlex, 2026-10-14) · [22](https://md.research.elamur.ai/free/en/papers/arkoudas-2023-gpt-4-t-reason/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This position paper by Konstantine Arkoudas asks whether GPT-4, then considered the strongest LLM, can actually reason. The author argues that standard reasoning benchmarks are poor evidence. Training data may be contaminated in ways that substring checks miss. Most benchmarks are also multiple-choice or binary classification tasks, so a model can score well by exploiting spurious statistical cues. Instead of using a corpus, he builds a set of 21 elementary problems, most written from scratch and some, such as Wason's selection task, altered to hide their origin. He then examines GPT-4's answers, explanations and proof attempts by hand, including how it reacts in dialogue when its mistakes are pointed out. The author's conclusion is that GPT-4 solves none of the problems and shows deep confusion about basic logical concepts. He rejects both a weak claim (average-human reasoning competence) and a strong claim (reliability sufficient for science, medicine or engineering). The paper matters because it pushes the field toward generative, manually scrutinized evaluation of LLM reasoning. ## Key points - The evaluation uses 21 diverse, mostly original reasoning problems, framed around a liberal notion of deductive reasoning and analysed qualitatively rather than with numeric metrics. - The author reports that GPT-4 fails all 21 problems. Its errors include being unable to tell left from right (Section 1.3.12), and he considers its flawed explanations more worrying than its wrong final answers. - Automatic metrics such as ROUGE-L, BLEU and embedding-based similarity are judged unfit for assessing natural-language derivations. - The paper critiques Liu et al.'s GPT-4 study, which relied on LogiQA, ReClor, ConTRoL, MED, ConjNLI, TaxiNLI and AR-LSAT. All of these are multiple-choice or classification tasks and may resemble pre-2021 data. - On theoretical grounds, the author notes that general first-order reasoning is undecidable, so no LLM can solve arbitrary reasoning problems soundly and completely. He nonetheless treats the question for any specific system as an empirical one. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works ## Cite Cite as: Konstantine Arkoudas. 2023. GPT-4 Can't Reason. arXiv:2308.03762. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{arkoudas2023gpt, title = {{GPT-4 Can't Reason}}, author = {Konstantine Arkoudas}, year = {2023}, doi = {10.48550/arxiv.2308.03762}, eprint = {2308.03762}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2308.03762} } ``` - [← previous Chinese Idiom Paraphrasing](https://md.research.elamur.ai/free/en/papers/qiang-2023-chinese-idiom-paraphrasing/) - [next → Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generation](https://md.research.elamur.ai/free/en/papers/kang-2023-knowledge-graph-augmented-language-models/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) ## References 11 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Arkoudas, K. and Musser, D., Fundamental Proof Methods in Computer Science, MIT Press, 2017. [find](https://scholar.google.com/scholar?q=Arkoudas%2C+K%2E+and+Musser%2C+D%2E%2C+Fundamental+Proof+Methods+in+Computer+Science%2C+MIT+Press%2C+2017%2E) 2. Barwise, J. and Perry, J., Situations and Attitudes, MIT Press, 1983. [find](https://scholar.google.com/scholar?q=Barwise%2C+J%2E+and+Perry%2C+J%2E%2C+Situations+and+Attitudes%2C+MIT+Press%2C+1983%2E) 3. Karpas, E., Abend, O., Belinkov, Y., Lenz, B., Lieber, O., Ratner, N., . . ., Tenenholtz, M., MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning, 2022. [find](https://scholar.google.com/scholar?q=Karpas%2C+E%2E%2C+Abend%2C+O%2E%2C+Belinkov%2C+Y%2E%2C+Lenz%2C+B%2E%2C+Lieber%2C+O%2E%2C+Ratner%2C+N%2E%2C+%2E+%2E+%2E%2C+Tenenholtz%2C+M%2E%2C+MRKL+Systems%3A+A+modular%2C+neuro%2Dsymbolic+architecture+that+combines+large+language+models%2C+external+knowled) 4. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y., [ReAct: Synergizing Reasoning and Acting in Language Models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/), https://arxiv.org/abs/2210.03629 , 2023. [↗](https://arxiv.org/abs/2210.03629) 5. Planken, L., Temporal Reasoning Problems and Algorithms for Solving Them: Literature Survey, 2008. [find](https://scholar.google.com/scholar?q=Planken%2C+L%2E%2C+Temporal+Reasoning+Problems+and+Algorithms+for+Solving+Them%3A+Literature+Survey%2C+2008%2E) 6. McCoy, T., Pavlick, E., Linzen, T., [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. 7. Liu H., Ning R., Teng, Z., Liu, J., Zhou, Q., Zhang, Y., [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/), 2023. 8. OpenAI, [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023. 9. Wang, J., Hu, X., Hou, W., Chen, H., Zheng, R., Wang, Y., . . ., Xie, X., On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective, 2023. [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Hu%2C+X%2E%2C+Hou%2C+W%2E%2C+Chen%2C+H%2E%2C+Zheng%2C+R%2E%2C+Wang%2C+Y%2E%2C+%2E+%2E+%2E%2C+Xie%2C+X%2E%2C+On+the+Robustness+of+ChatGPT%3A+An+Adversarial+and+Out%2Dof%2Ddistribution+Perspective%2C+2023%2E) 10. Niven, T., Kao, H.-Y. Probing Neural Network Comprehension of Natural Language Arguments, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. [find](https://scholar.google.com/scholar?q=Niven%2C+T%2E%2C+Kao%2C+H%2E%2DY%2E+Probing+Neural+Network+Comprehension+of+Natural+Language+Arguments%2C+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+2019%2E) 11. Johnson-Laird, P.N., How We Reason, Oxford University Press, 2006. [find](https://scholar.google.com/scholar?q=Johnson%2DLaird%2C+P%2EN%2E%2C+How+We+Reason%2C+Oxford+University+Press%2C+2006%2E) *Updated 30 October 2026*