---
title: "The Role of Deductive and Inductive Reasoning in Large"
description: "The authors propose DID (De-In-Ductive), a prompting framework that puts its effort… 70.3% accuracy. 33 references, cited by 13 papers here."
authors: ["Chengkun Cai", "Xu Zhao", "Haoliang Liu", "Zhongyu Jiang", "Tianfang Zhang", "Zongkai Wu", "Jenq-Neng Hwang", "Lei Li"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.820/"
doi: "10.18653/v1/2025.acl-long.820"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cai-2025-role-deductive-inductive-reasoning-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cai-2025-role-deductive-inductive-reasoning-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / The Role of Deductive and Inductive Reasoning in Large Language Models

# The Role of Deductive and Inductive Reasoning in Large Language Models

**Authors:** [Chengkun Cai](https://md.research.elamur.ai/free/en/authors/chengkun-cai/), [Xu Zhao](https://md.research.elamur.ai/free/en/authors/xu-zhao/), Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang and [Lei Li](https://md.research.elamur.ai/free/en/authors/lei-li/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.820/) · [PDF](https://aclanthology.org/2025.acl-long.820.pdf) · [DOI 10.18653/v1/2025.acl-long.820](https://doi.org/10.18653/v1/2025.acl-long.820) · [OpenAlex](https://openalex.org/W4412945039)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [13](https://md.research.elamur.ai/free/en/papers/cai-2025-role-deductive-inductive-reasoning-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Prompting methods such as Tree of Thought improve LLM reasoning by exploring many output paths, but this is expensive, and fixed prompt templates adapt poorly to harder instances. The authors propose DID (De-In-Ductive), a prompting framework that puts its effort into structuring the input. It draws on cognitive-science accounts of how people combine two modes of reasoning. First, the model works inductively on simplified versions of the task, gradually raising their difficulty until it arrives at a general rule. Then it applies that rule deductively to the original problem.

Problem difficulty is scored with two measures:
- **Littlestone dimension**, which captures structural complexity;
- **information entropy** over the relevant numeric variables, which captures how hard a particular instance is.

These scores determine how finely the problem is decomposed and how long the reasoning chain should be. DID was tested on AIW, MR-GSM8K and a new Holiday Puzzle set using GPT-3.5 Turbo, GPT-4o and Claude 3.5 Sonnet. It reached 70.3% on AIW versus 62.2% for ToT, at a lower cost per case. This suggests that structuring the input can replace costly output search and requires no fine-tuning.

## Key points
- **Instance entropy.** It is computed as log₂ of the product of (1 + |xᵢ|) over the relevant variables. This explains why "Alice" problems with the same Littlestone dimension but larger numbers are harder for LLMs.
- **AIW accuracy and cost.**
  - DID reaches 70.3% accuracy, compared with 62.2% for ToT.
  - Cost per case is $0.0031 for DID, $0.0038 for ToT and $0.0022 for CoT.
  - DID uses more input tokens than ToT (90 vs 56).
- **MR-GSM8K (meta-reasoning: finding errors in given solutions).** DID performs on par with the baselines rather than clearly ahead, for example 87.7% on GPT-4o vs 89.1% for ToT. It is cheaper than ToT ($0.0128 vs $0.0194).
- **Holiday Puzzle (new set of 20 date and holiday calculation problems).**
  - Absolute accuracy is low for all methods.
  - DID still scores highest on every model, for example 15.4% on GPT-4o vs 7.5% for ToT.
  - DID costs less than ToT ($0.0181 vs $0.0262).
- **Training and setup.** DID works with one model instance through prompting alone, with no fine-tuning. This sets it apart from multi-agent or RL-trained approaches such as DeepSeek-R1.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 353 works · 200,043 citations
- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 359 works · 77,352 citations
- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [University of Copenhagen](https://md.research.elamur.ai/free/en/universities/university-of-copenhagen/) — University, Copenhagen, Denmark · 115 works · 15,292 citations
- [University of Manchester](https://md.research.elamur.ai/free/en/universities/university-of-manchester/) — University, Manchester, United Kingdom · 87 works · 44,938 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang and Lei Li. 2025. The Role of Deductive and Inductive Reasoning in Large Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cai2025role,
  title = {{The Role of Deductive and Inductive Reasoning in Large Language Models}},
  author = {Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang and Lei Li},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.820},
  url = {https://doi.org/10.18653/v1/2025.acl-long.820}
}
```

- [← previous STaR-SQL: Self-Taught Reasoner for Text-to-SQL](https://md.research.elamur.ai/free/en/papers/he-2025-star-sql-self-taught-reasoner/)
- [next → Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs](https://md.research.elamur.ai/free/en/papers/li-2025-towards-agentic-rag-deep-reasoning/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

33 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. 2024. Medec: A benchmark for medical error detection and correction in clinical notes. arXiv preprint arXiv:2412.19260. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha%2C+Wen%2Dwai+Yim%2C+Yujuan+Fu%2C+Zhaoyi+Sun%2C+Meliha+Yetisgen%2C+Fei+Xia%2C+and+Thomas+Lin%2E+2024%2E+Medec%3A+A+benchmark+for+medical+error+detection+and+correction+in+clinical+notes%2E+arXiv+preprint+arX)
2. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artifici… [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Nils+Blach%2C+Ales+Kubicek%2C+Robert+Gerstenberger%2C+Michal+Podstawski%2C+Lukas+Gianinazzi%2C+Joanna+Gajda%2C+Tomasz+Lehmann%2C+Hubert+Niewiadomski%2C+Piotr+Nyczyk%2C+et+al%2E+2024%2E+Graph+of+thoughts%3A+Solv)
3. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712.
4. Chengkun Cai, Xu Zhao, Yucheng Du, Haoliang Liu, and Lei Li. 2024. T 2 of thoughts: Temperature tree elicits reasoning in large language models. arXiv preprint arXiv:2405.14075. — [arXiv:2405.14075](https://arxiv.org/abs/2405.14075) [find](https://scholar.google.com/scholar?q=Chengkun+Cai%2C+Xu+Zhao%2C+Yucheng+Du%2C+Haoliang+Liu%2C+and+Lei+Li%2E+2024%2E+T+2+of+thoughts%3A+Temperature+tree+elicits+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2405%2E14075%2E+%E2%80%94+%5BarXiv%3A2405%2E14075%5D%28h)
5. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2E)
6. Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. 2025. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807. [find](https://scholar.google.com/scholar?q=Ahmed+El%2DKishky%2C+Alexander+Wei%2C+Andre+Saraiva%2C+Borys+Minaev%2C+Daniel+Selsam%2C+David+Dohan%2C+Francis+Song%2C+Hunter+Lightman%2C+Ignasi+Clavera%2C+Jakub+Pachocki%2C+et+al%2E+2025%2E+Competitive+programming+with+large+)
7. Joachim Funke. 2013. Complex problem solving: A case for complex cognition? In Complex problem solving: Principles and mechanisms, pages 25–47. Psychology Press. [find](https://scholar.google.com/scholar?q=Joachim+Funke%2E+2013%2E+Complex+problem+solving%3A+A+case+for+complex+cognition%3F+In+Complex+problem+solving%3A+Principles+and+mechanisms%2C+pages+25%E2%80%9347%2E+Psychology+Press%2E)
8. Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. 2015. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science, 349(6245):273–278. [find](https://scholar.google.com/scholar?q=Samuel+J+Gershman%2C+Eric+J+Horvitz%2C+and+Joshua+B+Tenenbaum%2E+2015%2E+Computational+rationality%3A+A+converging+paradigm+for+intelligence+in+brains%2C+minds%2C+and+machines%2E+Science%2C+349%286245%29%3A273%E2%80%93278%2E)
9. Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, and Giorgos Stamou. 2024. [Puzzle solving using reasoning of large language models: A survey](https://md.research.elamur.ai/free/en/papers/giadikiaroglou-2024-puzzle-solving-reasoning-large-language/). arXiv preprint arXiv:2402.11291.
10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
12. Hyeonbin Hwang, Doyoung Kim, Seungone Kim, Seonghyeon Ye, and Minjoon Seo. 2024. Self-explore: Enhancing mathematical reasoning in language models with fine-grained rewards. arXiv preprint arXiv:2404.10346. [find](https://scholar.google.com/scholar?q=Hyeonbin+Hwang%2C+Doyoung+Kim%2C+Seungone+Kim%2C+Seonghyeon+Ye%2C+and+Minjoon+Seo%2E+2024%2E+Self%2Dexplore%3A+Enhancing+mathematical+reasoning+in+language+models+with+fine%2Dgrained+rewards%2E+arXiv+preprint+arXiv%3A2404%2E)
13. Can Jin, Tong Che, Hongwu Peng, Yiyuan Li, Dimitris Metaxas, and Marco Pavone. 2024. Learning from teaching regularization: Generalizable correlations should be easy to imitate. In Advances in Neural Information Processing Systems, volume 37, pages 966–994. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Can+Jin%2C+Tong+Che%2C+Hongwu+Peng%2C+Yiyuan+Li%2C+Dimitris+Metaxas%2C+and+Marco+Pavone%2E+2024%2E+Learning+from+teaching+regularization%3A+Generalizable+correlations+should+be+easy+to+imitate%2E+In+Advances+in+Neural+)
14. Can Jin, Hongwu Peng, Qixin Zhang, Yujin Tang, Dimitris N Metaxas, and Tong Che. 2025. Two heads are better than one: Test-time scaling of multi-agent collaborative reasoning. arXiv preprint arXiv:2504.09772. — [arXiv:2504.09772](https://arxiv.org/abs/2504.09772) [find](https://scholar.google.com/scholar?q=Can+Jin%2C+Hongwu+Peng%2C+Qixin+Zhang%2C+Yujin+Tang%2C+Dimitris+N+Metaxas%2C+and+Tong+Che%2E+2025%2E+Two+heads+are+better+than+one%3A+Test%2Dtime+scaling+of+multi%2Dagent+collaborative+reasoning%2E+arXiv+preprint+arXiv%3A250)
15. Philip Nicholas Johnson-Laird. 1983. Mental models: Towards a cognitive science of language, inference, and consciousness. 6. Harvard University Press. [find](https://scholar.google.com/scholar?q=Philip+Nicholas+Johnson%2DLaird%2E+1983%2E+Mental+models%3A+Towards+a+cognitive+science+of+language%2C+inference%2C+and+consciousness%2E+6%2E+Harvard+University+Press%2E)
16. Thomas L Griffiths, Charles Kemp, and Joshua B Tenenbaum. 2008. Bayesian models of cognition. [find](https://scholar.google.com/scholar?q=Thomas+L+Griffiths%2C+Charles+Kemp%2C+and+Joshua+B+Tenenbaum%2E+2008%2E+Bayesian+models+of+cognition%2E)
17. Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. 2024. Chain of thought empowers transformers to solve inherently serial problems. arXiv preprint arXiv:2402.12875. [find](https://scholar.google.com/scholar?q=Zhiyuan+Li%2C+Hong+Liu%2C+Denny+Zhou%2C+and+Tengyu+Ma%2E+2024%2E+Chain+of+thought+empowers+transformers+to+solve+inherently+serial+problems%2E+arXiv+preprint+arXiv%3A2402%2E12875%2E)
18. Wentao Liu, Hanglei Hu, Jie Zhou, Yuyang Ding, Junsong Li, Jiayi Zeng, Mengliang He, Qin Chen, Bo Jiang, Aimin Zhou, et al. 2023. Mathematical language models: A survey. arXiv preprint arXiv:2312.07622. — [arXiv:2312.07622](https://arxiv.org/abs/2312.07622) [find](https://scholar.google.com/scholar?q=Wentao+Liu%2C+Hanglei+Hu%2C+Jie+Zhou%2C+Yuyang+Ding%2C+Junsong+Li%2C+Jiayi+Zeng%2C+Mengliang+He%2C+Qin+Chen%2C+Bo+Jiang%2C+Aimin+Zhou%2C+et+al%2E+2023%2E+Mathematical+language+models%3A+A+survey%2E+arXiv+preprint+arXiv%3A2312%2E0762)
19. Zhou Lu. 2024. When is inductive inference possible? In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Zhou+Lu%2E+2024%2E+When+is+inductive+inference+possible%3F+In+The+Thirty%2Deighth+Annual+Conference+on+Neural+Information+Processing+Systems%2E)
20. Gary Marcus. 2020. [The next decade in ai: four steps towards robust artificial intelligence](https://md.research.elamur.ai/free/en/papers/marcus-2020-next-decade-ai-four/). arXiv preprint arXiv:2002.06177.
21. Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, and Jenia Jitsev. 2024. Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models. arXiv preprint arXiv:2406.02061. [find](https://scholar.google.com/scholar?q=Marianna+Nezhurina%2C+Lucia+Cipolina%2DKun%2C+Mehdi+Cherti%2C+and+Jenia+Jitsev%2E+2024%2E+Alice+in+wonderland%3A+Simple+tasks+showing+complete+reasoning+breakdown+in+state%2Dof%2Dthe%2Dart+large+language+models%2E+arXiv+pr)
22. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
23. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744.
24. S Sloman. 2009. Causal models: How people think about the world and its alternatives. Oxford University Press. [find](https://scholar.google.com/scholar?q=S+Sloman%2E+2009%2E+Causal+models%3A+How+people+think+about+the+world+and+its+alternatives%2E+Oxford+University+Press%2E)
25. Ray J Solomonoff. 1964. A formal theory of inductive inference. part i. Information and control, 7(1):1–22. [find](https://scholar.google.com/scholar?q=Ray+J+Solomonoff%2E+1964%2E+A+formal+theory+of+inductive+inference%2E+part+i%2E+Information+and+control%2C+7%281%29%3A1%E2%80%9322%2E)
26. Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. 2011. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1279–1285. [find](https://scholar.google.com/scholar?q=Joshua+B+Tenenbaum%2C+Charles+Kemp%2C+Thomas+L+Griffiths%2C+and+Noah+D+Goodman%2E+2011%2E+How+to+grow+a+mind%3A+Statistics%2C+structure%2C+and+abstraction%2E+science%2C+331%286022%29%3A1279%E2%80%931285%2E)
27. Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157):1124–1131. [find](https://scholar.google.com/scholar?q=Amos+Tversky+and+Daniel+Kahneman%2E+1974%2E+Judgment+under+uncertainty%3A+Heuristics+and+biases%3A+Biases+in+judgments+reveal+some+heuristics+of+thinking+under+uncertainty%2E+science%2C+185%284157%29%3A1124%E2%80%931131%2E)
28. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
29. Peter C. Wason. 1960. On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology, 12(3):129–140. [find](https://scholar.google.com/scholar?q=Peter+C%2E+Wason%2E+1960%2E+On+the+failure+to+eliminate+hypotheses+in+a+conceptual+task%2E+Quarterly+Journal+of+Experimental+Psychology%2C+12%283%29%3A129%E2%80%93140%2E)
30. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837.
31. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2024%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Advances+in+Neural+Inf)
32. Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. 2023. Mr-gsm8k: A meta-reasoning benchmark for large language model evaluation. arXiv preprint arXiv:2312.17080. [find](https://scholar.google.com/scholar?q=Zhongshen+Zeng%2C+Pengguang+Chen%2C+Shu+Liu%2C+Haiyun+Jiang%2C+and+Jiaya+Jia%2E+2023%2E+Mr%2Dgsm8k%3A+A+meta%2Dreasoning+benchmark+for+large+language+model+evaluation%2E+arXiv+preprint+arXiv%3A2312%2E17080%2E)
33. Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumed, Yael Moros-Daval, César Ferri, and José Hernández-Orallo. 2024. Larger and more instructable language models become less reliable. Nature, pages 1–8. [find](https://scholar.google.com/scholar?q=Lexin+Zhou%2C+Wout+Schellaert%2C+Fernando+Mart%C3%ADnez%2DPlumed%2C+Yael+Moros%2DDaval%2C+C%C3%A9sar+Ferri%2C+and+Jos%C3%A9+Hern%C3%A1ndez%2DOrallo%2E+2024%2E+Larger+and+more+instructable+language+models+become+less+reliable%2E+Nature%2C+pages+)

*Updated 31 October 2026*