---
title: "Are Large Language Models Really Good Logical Reasoners? A"
description: "The authors argue that earlier evaluations were too narrow: they used few models, scored only… 50 references, cited by 69 papers here."
authors: ["Fangzhi Xu", "Qika Lin", "Jiawei Han", "Tianzhe Zhao", "Jun Liu", "Erik Cambria"]
year: 2023
venue: "arXiv:2306.09841"
source: "https://arxiv.org/abs/2306.09841"
doi: "10.48550/arxiv.2306.09841"
arxiv: "2306.09841"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond

# Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond

**Authors:** [Fangzhi Xu](https://md.research.elamur.ai/free/en/authors/fangzhi-xu/), [Qika Lin](https://md.research.elamur.ai/free/en/authors/qika-lin/), [Jiawei Han](https://md.research.elamur.ai/free/en/authors/jiawei-han/), Tianzhe Zhao, Jun Liu and [Erik Cambria](https://md.research.elamur.ai/free/en/authors/erik-cambria/) · **Year:** 2023 · **Venue:** [arXiv:2306.09841](https://md.research.elamur.ai/free/en/venues/ieee-transactions-on-knowledge-and-data-engineering/) ([IEEE Transactions on Knowledge and Data Engineering 2023](https://md.research.elamur.ai/free/en/venues/ieee-transactions-on-knowledge-and-data-engineering/2023/)) · **Publisher:** [IEEE Computer Society](https://md.research.elamur.ai/free/en/publishers/ieee-computer-society/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.09841) · [PDF](https://arxiv.org/pdf/2306.09841) · [DOI 10.48550/arxiv.2306.09841](https://doi.org/10.48550/arxiv.2306.09841) · [arXiv 2306.09841](https://arxiv.org/abs/2306.09841) · [OpenAlex](https://openalex.org/W4407097799)

**Citations:** 72 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether current LLMs reason logically or mostly rely on surface patterns and memorized benchmark content. The authors argue that earlier evaluations were too narrow: they used few models, scored only accuracy, skipped error analysis, and relied on test data mixed with world knowledge. To address this, they collect fifteen logical reasoning datasets, grouped into deductive, inductive, abductive and mixed-form settings. They test seven models under several prompting setups: three earlier systems (text-davinci-003, ChatGPT, BARD) and four newer ones (LLaMA3.1-Chat, Mistral-Instruct-v0.3, Claude-3, GPT-4). Beyond answer accuracy, they judge the quality of generated explanations and trace failures to specific error categories. To separate reasoning from background knowledge, they release NeuLR, a benchmark built from content-neutral items. The findings feed a six-dimension scheme and per-model ability maps for each reasoning setting, which show where LLMs succeed and fail. The work matters because it gives a reusable protocol, usable with human annotation or automatically, for checking how models reason rather than only their final answers.

## Key points
- **Coverage:** 15 datasets in 4 reasoning settings (deductive, inductive, abductive, mixed-form), with 7 LLMs compared across zero-shot and few-shot prompting.
- **Metrics:** answers are scored for correctness, and explanations are rated for correctness, completeness and redundancy, combining objective and subjective views.
- **Error analysis:** faulty cases are sorted into five error types along two axes, evidence selection and the reasoning process itself.
- **NeuLR:** a new benchmark of 3,000 content-neutral samples spanning deductive, inductive and abductive reasoning. It is designed to limit knowledge bias and training-data overlap.
- **Evaluation scheme:** six properties (Correct, Rigorous, Self-aware, Active, Oriented, No hallucination), visualized as ability maps per model and per setting to guide future research.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 73 works · 24,390 citations
- [Xi'an Jiaotong University](https://md.research.elamur.ai/free/en/universities/xi-an-jiaotong-university/) — University, Xi'an, China · 7 works · 35,747 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu and Erik Cambria. 2023. Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond. arXiv:2306.09841.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xu2023are,
  title = {{Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond}},
  author = {Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu and Erik Cambria},
  year = {2023},
  doi = {10.48550/arxiv.2306.09841},
  eprint = {2306.09841},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.09841}
}
```

- [← previous ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://md.research.elamur.ai/free/en/papers/hao-2023-toolkengpt-augmenting-frozen-language-models/)
- [next → Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization](https://md.research.elamur.ai/free/en/papers/yang-2023-exploring-limits-chatgpt-query-aspect/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

50 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. G. Antoniou and A. Bikakis, “Dr-prolog: A system for defeasible reasoning with rules and ontologies on the semantic web,” IEEE Transactions on Knowledge and Data Engineering (TKDE), vol. 19, no. 2, pp. 233–245, 2007. T. Lukasiewicz, “A novel combination of answer set programming with description log… [find](https://scholar.google.com/scholar?q=G%2E+Antoniou+and+A%2E+Bikakis%2C+%E2%80%9CDr%2Dprolog%3A+A+system+for+defeasible+reasoning+with+rules+and+ontologies+on+the+semantic+web%2C%E2%80%9D+IEEE+Transactions+on+Knowledge+and+Data+Engineering+%28TKDE%29%2C+vol%2E+19%2C+no%2E+2%2C+pp)
2. Z. Yang, X. Du, R. Mao, J. Ni, and E. Cambria, “Logical reasoning over natural language as knowledge representation: A survey,” CoRR, vol. abs/2303.12023, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Yang%2C+X%2E+Du%2C+R%2E+Mao%2C+J%2E+Ni%2C+and+E%2E+Cambria%2C+%E2%80%9CLogical+reasoning+over+natural+language+as+knowledge+representation%3A+A+survey%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F2303%2E12023%2C+2023%2E)
3. B. Peng, C. Li, P. He, M. Galley, and J. Gao, “[Instruction tuning with GPT-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/),” CoRR, vol. abs/2304.03277, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2304.03277 [↗](https://doi.org/10.48550/arXiv.2304.03277)
4. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J. Nie, and J. Wen, “[A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/),” CoRR, vol. abs/2303.18223, 2023.
5. M. M. Amin, E. Cambria, and B. W. Schuller, “Will affective computing emerge from foundation models and general artificial intelligence? A first evaluation of chatgpt,” IEEE Intelligent Systems, vol. 38, no. 2, pp. 15–23, 2023. [find](https://scholar.google.com/scholar?q=M%2E+M%2E+Amin%2C+E%2E+Cambria%2C+and+B%2E+W%2E+Schuller%2C+%E2%80%9CWill+affective+computing+emerge+from+foundation+models+and+general+artificial+intelligence%3F+A+first+evaluation+of+chatgpt%2C%E2%80%9D+IEEE+Intelligent+Systems%2C+vol%2E+)
6. F. Jiao, Y. Guo, X. Song, and L. Nie, “Merit: Meta-path guided contrastive learning for logical reasoning,” in Findings of the Association for Computational Linguistics (Findings of ACL), 2022, pp. 3496–3509. [find](https://scholar.google.com/scholar?q=F%2E+Jiao%2C+Y%2E+Guo%2C+X%2E+Song%2C+and+L%2E+Nie%2C+%E2%80%9CMerit%3A+Meta%2Dpath+guided+contrastive+learning+for+logical+reasoning%2C%E2%80%9D+in+Findings+of+the+Association+for+Computational+Linguistics+%28Findings+of+ACL%29%2C+2022%2C+pp%2E+34)
7. F. Xu, Q. Lin, T. Zhao, J. JiaweiHan, and J. Liu, “PathReasoner: Modeling reasoning path with equivalent extension for logical question answering,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for C… [find](https://scholar.google.com/scholar?q=F%2E+Xu%2C+Q%2E+Lin%2C+T%2E+Zhao%2C+J%2E+JiaweiHan%2C+and+J%2E+Liu%2C+%E2%80%9CPathReasoner%3A+Modeling+reasoning+path+with+equivalent+extension+for+logical+question+answering%2C%E2%80%9D+in+Proceedings+of+the+62nd+Annual+Meeting+of+the+Ass)
8. Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, Q. V. Do, Y. Xu, and P. Fung, “A multi-task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,” CoRR, vol. abs/2302.[04023, 2023](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/).
9. N. Bian, X. Han, L. Sun, H. Lin, Y. Lu, and B. He, “[Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models](https://md.research.elamur.ai/free/en/papers/bian-2024-chatgpt-is-knowledgeable-but-inexperienced/),” CoRR, vol. abs/2303.16421, 2023.
10. S. Imani, L. Du, and H. Shrivastava, “[Mathprompter: Mathematical reasoning using large language models](https://md.research.elamur.ai/free/en/papers/imani-2023-mathprompter-mathematical-reasoning-large-language/),” CoRR, vol. abs/2303.05398, 2023.
11. P. A. Flach and A. C. Kakas, “Abductive and inductive reasoning: background and issues,” Abduction and induction: Essays on their relation and integration, pp. 1–27, 2000. [find](https://scholar.google.com/scholar?q=P%2E+A%2E+Flach+and+A%2E+C%2E+Kakas%2C+%E2%80%9CAbductive+and+inductive+reasoning%3A+background+and+issues%2C%E2%80%9D+Abduction+and+induction%3A+Essays+on+their+relation+and+integration%2C+pp%2E+1%E2%80%9327%2C+2000%2E)
12. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “[Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)…
13. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “[Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/),” Advances in Neural Information Processing Systems, vol. 35, pp. 27 730–27 744, 2022.
14. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen et al., “[Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/),” arXiv preprint arXiv:2305.10403, 2023.
15. C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, “[Is chatgpt a general-purpose natural language processing task solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/)” CoRR, vol. abs/2302.06476, 2023.
16. S. Tu, C. Li, J. Yu, X. Wang, L. Hou, and J. Li, “Chatlog: Recording and analyzing chatgpt across time,” CoRR, vol. abs/2304.14106, 2023. — [arXiv:2304.14106](https://arxiv.org/abs/2304.14106) [find](https://scholar.google.com/scholar?q=S%2E+Tu%2C+C%2E+Li%2C+J%2E+Yu%2C+X%2E+Wang%2C+L%2E+Hou%2C+and+J%2E+Li%2C+%E2%80%9CChatlog%3A+Recording+and+analyzing+chatgpt+across+time%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F2304%2E14106%2C+2023%2E+%E2%80%94+%5BarXiv%3A2304%2E14106%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2304%2E14106%29)
17. Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, F. Qi, Y. Fu, M. Sun, and J. He, “Ceval: A multi-level multi-discipline chinese evaluation suite for foundation models,” CoRR, vol. abs/2305.[08322, 2023](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/).
18. T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. R. McKeown, and T. B. Hashimoto, “[Benchmarking large language models for news summarization](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/),” CoRR, vol. abs/2301.13848, 2023.
19. Y. Pan, J. Liu, L. Zhang, T. Zhao, Q. Lin, X. Hu, and Q. Wang, “Inductive relation prediction with logical reasoning using contrastive representations,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022, pp. 4261–4274. [find](https://scholar.google.com/scholar?q=Y%2E+Pan%2C+J%2E+Liu%2C+L%2E+Zhang%2C+T%2E+Zhao%2C+Q%2E+Lin%2C+X%2E+Hu%2C+and+Q%2E+Wang%2C+%E2%80%9CInductive+relation+prediction+with+logical+reasoning+using+contrastive+representations%2C%E2%80%9D+in+Proceedings+of+the+2022+Conference+on+Empiri)
20. Z. Huang, M. Chiang, and W. Lee, “Line: Logical query reasoning over hierarchical knowledge graphs,” in 28th SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 615–625. [find](https://scholar.google.com/scholar?q=Z%2E+Huang%2C+M%2E+Chiang%2C+and+W%2E+Lee%2C+%E2%80%9CLine%3A+Logical+query+reasoning+over+hierarchical+knowledge+graphs%2C%E2%80%9D+in+28th+SIGKDD+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+2022%2C+pp%2E+615%E2%80%93625%2E)
21. K. Cheng, J. Liu, W. Wang, and Y. Sun, “Rlogic: Recursive logical rule learning from knowledge graphs,” in 28th SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 179–189. [find](https://scholar.google.com/scholar?q=K%2E+Cheng%2C+J%2E+Liu%2C+W%2E+Wang%2C+and+Y%2E+Sun%2C+%E2%80%9CRlogic%3A+Recursive+logical+rule+learning+from+knowledge+graphs%2C%E2%80%9D+in+28th+SIGKDD+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+2022%2C+pp%2E+179%E2%80%93189%2E)
22. P. N. Johnson-Laird, “Deductive reasoning,” Annual review of psychology, vol. 50, no. 1, pp. 109–135, 1999. [find](https://scholar.google.com/scholar?q=P%2E+N%2E+Johnson%2DLaird%2C+%E2%80%9CDeductive+reasoning%2C%E2%80%9D+Annual+review+of+psychology%2C+vol%2E+50%2C+no%2E+1%2C+pp%2E+109%E2%80%93135%2C+1999%2E)
23. V. Goel, “Anatomy of deductive reasoning,” Trends in cognitive sciences, vol. 11, no. 10, pp. 435–441, 2007. [find](https://scholar.google.com/scholar?q=V%2E+Goel%2C+%E2%80%9CAnatomy+of+deductive+reasoning%2C%E2%80%9D+Trends+in+cognitive+sciences%2C+vol%2E+11%2C+no%2E+10%2C+pp%2E+435%E2%80%93441%2C+2007%2E)
24. E. Heit and C. M. Rotello, “Relations between inductive reasoning and deductive reasoning.” Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 36, no. 3, p. 805, 2010. [find](https://scholar.google.com/scholar?q=E%2E+Heit+and+C%2E+M%2E+Rotello%2C+%E2%80%9CRelations+between+inductive+reasoning+and+deductive+reasoning%2E%E2%80%9D+Journal+of+Experimental+Psychology%3A+Learning%2C+Memory%2C+and+Cognition%2C+vol%2E+36%2C+no%2E+3%2C+p%2E+805%2C+2010%2E)
25. F. Yu, H. Zhang, and B. Wang, “Nature language reasoning, a survey,” arXiv preprint arXiv:2303.14725, 2023. [find](https://scholar.google.com/scholar?q=F%2E+Yu%2C+H%2E+Zhang%2C+and+B%2E+Wang%2C+%E2%80%9CNature+language+reasoning%2C+a+survey%2C%E2%80%9D+arXiv+preprint+arXiv%3A2303%2E14725%2C+2023%2E)
26. J. R. Josephson and S. G. Josephson, Abductive inference: Computation, philosophy, technology. Cambridge University Press, 1996. [find](https://scholar.google.com/scholar?q=J%2E+R%2E+Josephson+and+S%2E+G%2E+Josephson%2C+Abductive+inference%3A+Computation%2C+philosophy%2C+technology%2E+Cambridge+University+Press%2C+1996%2E)
27. D. Walton, “Abductive, presumptive and plausible arguments,” Informal Logic, vol. 21, no. 2, 2001. [find](https://scholar.google.com/scholar?q=D%2E+Walton%2C+%E2%80%9CAbductive%2C+presumptive+and+plausible+arguments%2C%E2%80%9D+Informal+Logic%2C+vol%2E+21%2C+no%2E+2%2C+2001%2E)
28. B. Dalvi, P. Jansen, O. Tafjord, Z. Xie, H. Smith, L. Pipatanangkura, and P. Clark, “[Explaining answers with entailment trees](https://md.research.elamur.ai/free/en/papers/dalvi-2021-explaining-answers-entailment-trees/),” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021, pp. 7358–7370.
29. [S. Han, H. Schoelkopf, Y. Zhao, Z. Qi, M. Riddell, L. Benson, L. Sun, E. Zubova, Y. Qiao, M. Burtell, D. Peng, J. Fan, Y. Liu, B. Wong, M. Sailor, A. Ni, L. Nan, J. Kasai, T. Yu, R. Zhang, S. R. Joty, A. R. Fabbri, W. Kryscinski, X. V. Lin, C. Xiong, and D. Radev, “FOLIO: natural language reasoning…](https://md.research.elamur.ai/free/en/papers/han-2022-folio-natural-language-reasoning-first/)
30. A. Talmor, O. Tafjord, P. Clark, Y. Goldberg, and J. Berant, “Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge,” in Advances in Neural Information Processing Systems (NeurIPS), 2020. [find](https://scholar.google.com/scholar?q=A%2E+Talmor%2C+O%2E+Tafjord%2C+P%2E+Clark%2C+Y%2E+Goldberg%2C+and+J%2E+Berant%2C+%E2%80%9CLeap%2Dof%2Dthought%3A+Teaching+pre%2Dtrained+models+to+systematically+reason+over+implicit+knowledge%2C%E2%80%9D+in+Advances+in+Neural+Information+Processi)
31. K. Sinha, S. Sodhani, J. Dong, J. Pineau, and W. L. Hamilton, “CLUTRR: A diagnostic benchmark for inductive reasoning from text,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP… [find](https://scholar.google.com/scholar?q=K%2E+Sinha%2C+S%2E+Sodhani%2C+J%2E+Dong%2C+J%2E+Pineau%2C+and+W%2E+L%2E+Hamilton%2C+%E2%80%9CCLUTRR%3A+A+diagnostic+benchmark+for+inductive+reasoning+from+text%2C%E2%80%9D+in+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+)
32. W. Yu, Z. Jiang, Y. Dong, and J. Feng, “[Reclor: A reading comprehension dataset requiring logical reasoning](https://md.research.elamur.ai/free/en/papers/yu-2020-reclor-reading-comprehension-dataset-requiring/),” in 8th International Conference on Learning Representations (ICLR), 2020.
33. J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang, “[Logiqa: A challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/),” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI), 2020, pp. 3622–3628.
34. H. Liu, R. Ning, Z. Teng, J. Liu, Q. Zhou, and Y. Zhang, “[Evaluating the logical reasoning ability of chatgpt and GPT-4](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/),” CoRR, vol. abs/2304.03439, 2023.
35. J. Weston, A. Bordes, S. Chopra, and T. Mikolov, “[Towards ai-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/),” in 4th International Conference on Learning Representations (ICLR), 2016.
36. P. Clark, O. Tafjord, and K. Richardson, “[Transformers as soft reasoners over language](https://md.research.elamur.ai/free/en/papers/clark-2020-transformers-soft-reasoners-over-language/),” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI), 2020, pp. 3882–3890.
37. C. Bhagavatula, R. L. Bras, C. Malaviya, K. Sakaguchi, A. Holtzman, H. Rashkin, D. Downey, W. Yih, and Y. Choi, “[Abductive commonsense reasoning](https://md.research.elamur.ai/free/en/papers/bhagavatula-2019-abductive-commonsense-reasoning/),” in 8th International Conference on Learning Representations (ICLR), 2020.
38. N. Young, Q. Bao, J. Bensemann, and M. Witbrock, “Abductionrules: Training transformers to explain unexpected inputs,” in Findings of the Association for Computational Linguistics, 2022, pp. 218–227. [find](https://scholar.google.com/scholar?q=N%2E+Young%2C+Q%2E+Bao%2C+J%2E+Bensemann%2C+and+M%2E+Witbrock%2C+%E2%80%9CAbductionrules%3A+Training+transformers+to+explain+unexpected+inputs%2C%E2%80%9D+in+Findings+of+the+Association+for+Computational+Linguistics%2C+2022%2C+pp%2E+218%E2%80%93227%2E)
39. O. Tafjord, B. Dalvi, and P. Clark, “[Proofwriter: Generating implications, proofs, and abductive statements over natural language](https://md.research.elamur.ai/free/en/papers/tafjord-2021-proofwriter-generating-implications-proofs-abductive/),” in Findings of the Association for Computational Linguistics, 2021, pp. 3621–3634.
40. P. Minervini, S. Riedel, P. Stenetorp, E. Grefenstette, and T. Rocktäschel, “Learning reasoning strategies in end-to-end differentiable proving,” in Proceedings of the 37th International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 119, 2020, pp. 6938–6… [find](https://scholar.google.com/scholar?q=P%2E+Minervini%2C+S%2E+Riedel%2C+P%2E+Stenetorp%2C+E%2E+Grefenstette%2C+and+T%2E+Rockta%CC%88schel%2C+%E2%80%9CLearning+reasoning+strategies+in+end%2Dto%2Dend+differentiable+proving%2C%E2%80%9D+in+Proceedings+of+the+37th+International+Conference+o)
41. Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey for in-context learning,” arXiv preprint arXiv:2301.[00234, 2022](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/).
42. Z. Wu, Y. Wang, J. Ye, and L. Kong, “Self-adaptive in-context learning,” arXiv preprint arXiv:2212.10375, 2022. [find](https://scholar.google.com/scholar?q=Z%2E+Wu%2C+Y%2E+Wang%2C+J%2E+Ye%2C+and+L%2E+Kong%2C+%E2%80%9CSelf%2Dadaptive+in%2Dcontext+learning%2C%E2%80%9D+arXiv+preprint+arXiv%3A2212%2E10375%2C+2022%2E)
43. W. Chen, X. Ma, X. Wang, and W. W. Cohen, “Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks,” arXiv preprint arXiv:2211.12588, 2022. [find](https://scholar.google.com/scholar?q=W%2E+Chen%2C+X%2E+Ma%2C+X%2E+Wang%2C+and+W%2E+W%2E+Cohen%2C+%E2%80%9CProgram+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2C%E2%80%9D+arXiv+preprint+arXiv%3A2211%2E12588%2C+2022%2E)
44. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, “Pal: Program-aided language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 10 764–10 799. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+A%2E+Madaan%2C+S%2E+Zhou%2C+U%2E+Alon%2C+P%2E+Liu%2C+Y%2E+Yang%2C+J%2E+Callan%2C+and+G%2E+Neubig%2C+%E2%80%9CPal%3A+Program%2Daided+language+models%2C%E2%80%9D+in+International+Conference+on+Machine+Learning%2E+PMLR%2C+2023%2C+pp%2E+10+764%E2%80%9310+799%2E)
45. F. Xu, Z. Wu, Q. Sun, S. Ren, F. Yuan, S. Yuan, Q. Lin, Y. Qiao, and J. Liu, “Symbol-LLM: Towards foundational symbol-centric interface for large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: A… [find](https://scholar.google.com/scholar?q=F%2E+Xu%2C+Z%2E+Wu%2C+Q%2E+Sun%2C+S%2E+Ren%2C+F%2E+Yuan%2C+S%2E+Yuan%2C+Q%2E+Lin%2C+Y%2E+Qiao%2C+and+J%2E+Liu%2C+%E2%80%9CSymbol%2DLLM%3A+Towards+foundational+symbol%2Dcentric+interface+for+large+language+models%2C%E2%80%9D+in+Proceedings+of+the+62nd+Annual+Me)
46. L. Pan, A. Albalak, X. Wang, and W. Wang, “[Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning](https://md.research.elamur.ai/free/en/papers/pan-2023-logic-lm-empowering-large-language/),” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 3806–3824.
47. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, “[Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/),” in NeurIPS, 2022.
48. I. Drori, S. Zhang, R. Shuttleworth, L. Tang, A. Lu, E. Ke, K. Liu, L. Chen, S. Tran, N. Cheng et al., “A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level,” Proceedings of the National Academy of Sciences (PNAS), vol. 1… [find](https://scholar.google.com/scholar?q=I%2E+Drori%2C+S%2E+Zhang%2C+R%2E+Shuttleworth%2C+L%2E+Tang%2C+A%2E+Lu%2C+E%2E+Ke%2C+K%2E+Liu%2C+L%2E+Chen%2C+S%2E+Tran%2C+N%2E+Cheng+et+al%2E%2C+%E2%80%9CA+neural+network+solves%2C+explains%2C+and+generates+university+math+problems+by+program+synthesis+a)
49. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung, “[Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/),” ACM Computing Surveys, vol. 55, no. 12, pp. 248:1–248:38, 2023.
50. E. Cambria, Q. Liu, S. Decherchi, F. Xing, and K. Kwok, “Sentic-Net 7: A commonsense-based neurosymbolic AI framework for explainable sentiment analysis,” in LREC, 2022, pp. 3829–3839. A PPENDIX [find](https://scholar.google.com/scholar?q=E%2E+Cambria%2C+Q%2E+Liu%2C+S%2E+Decherchi%2C+F%2E+Xing%2C+and+K%2E+Kwok%2C+%E2%80%9CSentic%2DNet+7%3A+A+commonsense%2Dbased+neurosymbolic+AI+framework+for+explainable+sentiment+analysis%2C%E2%80%9D+in+LREC%2C+2022%2C+pp%2E+3829%E2%80%933839%2E+A+PPENDIX)

*Updated 24 October 2026*