---
title: "Evaluating the Logical Reasoning Ability of — Liu 2023"
description: "Are ChatGPT and GPT-4 really strong at logic-heavy language tasks? The authors test both models on two task families."
authors: ["Hanmeng Liu", "Ruoxi Ning", "Zhiyang Teng", "Jian Liu", "Qiji Zhou", "Yue Zhang"]
year: 2023
venue: "arXiv:2304.03439"
source: "https://arxiv.org/abs/2304.03439"
doi: "10.48550/arxiv.2304.03439"
arxiv: "2304.03439"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

# Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

**Authors:** [Hanmeng Liu](https://md.research.elamur.ai/free/en/authors/hanmeng-liu/), Ruoxi Ning, [Zhiyang Teng](https://md.research.elamur.ai/free/en/authors/zhiyang-teng/), [Jian Liu](https://md.research.elamur.ai/free/en/authors/jian-liu/), Qiji Zhou, [Yue Zhang](https://md.research.elamur.ai/free/en/authors/yue-zhang/) · **Year:** 2023 · **Venue:** arXiv:2304.03439 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.03439) · [PDF](https://arxiv.org/pdf/2304.03439) · [DOI 10.48550/arxiv.2304.03439](https://doi.org/10.48550/arxiv.2304.03439) · [arXiv 2304.03439](https://arxiv.org/abs/2304.03439) · [OpenAlex](https://openalex.org/W4363671827)

**Citations:** 100 citations (openalex-snapshot · 2026-10-14) · [112](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Are ChatGPT and GPT-4 really strong at logic-heavy language tasks? The authors test both models on two task families. The first is multiple-choice reading comprehension: LogiQA 2.0 (English and Chinese), ReClor and AR-LSAT. The second is natural language inference: ConTRoL, ConjNLI, HELP, MED and TaxiNLI, with MNLI as a conventional reference point. To check robustness, they also build a new out-of-distribution set from 2022 Chinese Civil Servant Exam questions. The models are compared with a fine-tuned RoBERTa baseline and human scores, using the API and smaller manual samples in the Chat UI.

Both GPT models beat RoBERTa on most benchmarks, and GPT-4 is usually best, particularly on established datasets such as ReClor. Accuracy drops sharply on newer or out-of-distribution data and on NLI, where GPT-4 also fails to output labels consistently. The prompt-formatted datasets are released as the LogiEval suite, a reusable yardstick for reasoning in prompt-based LLMs.

## Key points
- GPT-4 (API) accuracy versus RoBERTa / ChatGPT:
  - ReClor dev: 87.20% vs 55.01 / 57.38
  - LogiQA 2.0 test: 72.25% vs 48.76 / 52.37
  - LogiQA 2.0 ood: 58.49% vs 33.22 / 38.44
  - AR-LSAT: only 33.48%, with ChatGPT at 20.42%
- On NLI, GPT-4 reaches 89.42% on MED and 72.71% on ConjNLI. It manages only 46.01% on HELP, 60.08% on TaxiNLI and 64.08% on MNLI dev.
- Context matters: when 20 questions were asked in one conversation rather than in separate sessions, GPT-4 answered more of them correctly.
  - LogiQA 2.0 ood: 9/20 vs 5/20
  - ConTRoL: 13/20 vs 7/20
- Of GPT-4's first 10 LogiQA mistakes, 4 were logical fallacies, 3 were scope errors and 3 came from unresolved semantic ambiguity.
- The resources are public: LogiEval at github.com/csitfun/LogiEval and the out-of-distribution data in the LogiQA 2.0 repository.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 103 not yet.

- [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://md.research.elamur.ai/free/en/papers/gu-2024-cruxeval-benchmark-code-reasoning-understanding/)
- [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/)
- [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-trustworthy-llms-survey-guideline-evaluating/)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://md.research.elamur.ai/free/en/papers/pan-2023-logic-lm-empowering-large-language/)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://md.research.elamur.ai/free/en/papers/zhu-2023-promptrobust-towards-evaluating-robustness-large/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2023evaluating,
  title = {{Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4}},
  author = {Liu, Hanmeng and Ning, Ruoxi and Teng, Zhiyang and Liu, Jian and Zhou, Qiji and Zhang, Yue},
  year = {2023},
  doi = {10.48550/arxiv.2304.03439},
  eprint = {2304.03439},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.03439}
}
```

- [← previous Evaluating statistical language models as pragmatic reasoners](https://md.research.elamur.ai/free/en/papers/lipkin-2023-evaluating-statistical-language-models-pragmatic/)
- [next → Evaluating Verifiability in Generative Search Engines](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-verifiability-generative-search-engines/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. [find](https://scholar.google.com/scholar?q=Yejin+Bang%2C+Samuel+Cahyawijaya%2C+Nayeon+Lee%2C+Wenliang+Dai%2C+Dan+Su%2C+Bryan+Wilie%2C+Holy+Lovenia%2C+Ziwei+Ji%2C+Tiezheng+Yu%2C+Willy+Chung%2C+Quyet+V%2E+Do%2C+Yan+Xu%2C+and+Pascale+Fung%2E+2023%2E+A+multitask%2C+multilingual%2C) [our card](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
4. Maxwell John Cresswell. 1973. Logics and languages (1st ed.). Routledge. [find](https://scholar.google.com/scholar?q=Maxwell+John+Cresswell%2E+1973%2E+Logics+and+languages+%281st+ed%2E%29%2E+Routledge%2E)
5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Proc. of ICML. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Proc%2E+of+ICML%2E)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+BERT%3A+pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+CoRR%2E)
7. Lucja Iwańska. 1993. Logical reasoning in natural language: It is all about knowledge. Minds and Machines. [find](https://scholar.google.com/scholar?q=Lucja+Iwan%CC%81ska%2E+1993%2E+Logical+reasoning+in+natural+language%3A+It+is+all+about+knowledge%2E+Minds+and+Machines%2E)
8. Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. 2020. Taxinli: Taking a ride up the NLU hill. CoRR. [find](https://scholar.google.com/scholar?q=Pratik+Joshi%2C+Somak+Aditya%2C+Aalok+Sathe%2C+and+Monojit+Choudhury%2E+2020%2E+Taxinli%3A+Taking+a+ride+up+the+NLU+hill%2E+CoRR%2E)
9. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners. [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+Shane+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+2023%2E+Large+language+models+are+zero%2Dshot+reasoners%2E) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
10. Robert Kowalski. 1979. Logic for problem solving, volume 7. Ediciones Díaz de Santos. [find](https://scholar.google.com/scholar?q=Robert+Kowalski%2E+1979%2E+Logic+for+problem+solving%2C+volume+7%2E+Ediciones+D%C3%ADaz+de+Santos%2E)
11. Tao Li and Vivek Srikumar. 2019. Augmenting neural networks with first-order logic. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 292–302, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tao+Li+and+Vivek+Srikumar%2E+2019%2E+Augmenting+neural+networks+with+first%2Dorder+logic%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+292%E2%80%93302%2C+Florence%2C)
12. Hanmeng Liu, Leyang Cui, Jian Liu, and Yue Zhang. 2020. Natural language inference in context - investigating contextual reasoning over long texts. CoRR. [find](https://scholar.google.com/scholar?q=Hanmeng+Liu%2C+Leyang+Cui%2C+Jian+Liu%2C+and+Yue+Zhang%2E+2020%2E+Natural+language+inference+in+context+%2D+investigating+contextual+reasoning+over+long+texts%2E+CoRR%2E)
13. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+M%2E+Ott%2C+N%2E+Goyal%2C+J%2E+Du%2C+M%2E+Joshi%2C+D%2E+Chen%2C+O%2E+Levy%2C+M%2E+Lewis%2C+L%2E+Zettlemoyer%2C+and+V%2E+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv%2E)
14. Bill MacCartney and Christopher D. Manning. 2007a. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. [find](https://scholar.google.com/scholar?q=Bill+MacCartney+and+Christopher+D%2E+Manning%2E+2007a%2E+Natural+logic+for+textual+inference%2E+In+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2E)
15. Bill MacCartney and Christopher D Manning. 2007b. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. [find](https://scholar.google.com/scholar?q=Bill+MacCartney+and+Christopher+D+Manning%2E+2007b%2E+Natural+logic+for+textual+inference%2E+In+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2E)
16. OpenAI. 2023. Gpt-4 technical report. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+Gpt%2D4+technical+report%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
17. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? [find](https://scholar.google.com/scholar?q=Chengwei+Qin%2C+Aston+Zhang%2C+Zhuosheng+Zhang%2C+Jiaao+Chen%2C+Michihiro+Yasunaga%2C+and+Diyi+Yang%2E+2023%2E+Is+chatgpt+a+general%2Dpurpose+natural+language+processing+task+solver%3F) [our card](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/)
18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2019%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
19. Swarnadeep Saha, Yixin Nie, and Mohit Bansal. 2020. Conjnli: Natural language inference over conjunctive sentences. CoRR. [find](https://scholar.google.com/scholar?q=Swarnadeep+Saha%2C+Yixin+Nie%2C+and+Mohit+Bansal%2E+2020%2E+Conjnli%3A+Natural+language+inference+over+conjunctive+sentences%2E+CoRR%2E)
20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+CoRR%2E)
21. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E)
22. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. [find](https://scholar.google.com/scholar?q=GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+and+Interpreting+Neural+Networks+for+NLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
23. Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. 2022. From lsat: The progress and challenges of complex reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing. [find](https://scholar.google.com/scholar?q=Siyuan+Wang%2C+Zhongkun+Liu%2C+Wanjun+Zhong%2C+Ming+Zhou%2C+Zhongyu+Wei%2C+Zhumin+Chen%2C+and+Nan+Duan%2E+2022%2E+From+lsat%3A+The+progress+and+challenges+of+complex+reasoning%2E+IEEE%2FACM+Transactions+on+Audio%2C+Speech%2C+a)
24. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018a. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018a%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proc%2E+of+NAACL%2E)
25. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018b. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long P… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018b%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Cha)
26. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, , and Johan Bos. 2019a. Help: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM2019). [find](https://scholar.google.com/scholar?q=Hitomi+Yanaka%2C+Koji+Mineshima%2C+Daisuke+Bekki%2C+Kentaro+Inui%2C+Satoshi+Sekine%2C+%2C+and+Johan+Bos%2E+2019a%2E+Help%3A+A+dataset+for+identifying+shortcomings+of+neural+models+in+monotonicity+reasoning%2E+In+Proceedi)
27. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019b. Can neural networks understand monotonicity reasoning? [find](https://scholar.google.com/scholar?q=Hitomi+Yanaka%2C+Koji+Mineshima%2C+Daisuke+Bekki%2C+Kentaro+Inui%2C+Satoshi+Sekine%2C+Lasha+Abzianidze%2C+and+Johan+Bos%2E+2019b%2E+Can+neural+networks+understand+monotonicity+reasoning%3F)