---
title: "Beyond Accuracy — paper by Mondorf and Plank (2024)"
description: "Accuracy alone, however, shows little about how they reach their answers, and it remains disputed… 125 references, cited by 40 papers here."
authors: ["Philipp Mondorf", "Barbara Plank"]
year: 2024
venue: "arXiv:2404.01869"
source: "https://arxiv.org/abs/2404.01869"
doi: "10.48550/arxiv.2404.01869"
arxiv: "2404.01869"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mondorf-2024-accuracy-evaluating-reasoning-behavior-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mondorf-2024-accuracy-evaluating-reasoning-behavior-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models — A Survey

# Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models — A Survey

**Authors:** [Philipp Mondorf](https://md.research.elamur.ai/free/en/authors/philipp-mondorf/) and [Barbara Plank](https://md.research.elamur.ai/free/en/authors/barbara-plank/) · **Year:** 2024 · **Venue:** arXiv:2404.01869 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.01869) · [PDF](https://arxiv.org/pdf/2404.01869) · [DOI 10.48550/arxiv.2404.01869](https://doi.org/10.48550/arxiv.2404.01869) · [arXiv 2404.01869](https://arxiv.org/abs/2404.01869) · [OpenAlex](https://openalex.org/W4393934511)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [40](https://md.research.elamur.ai/free/en/papers/mondorf-2024-accuracy-evaluating-reasoning-behavior-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models score well on reasoning benchmarks. Accuracy alone, however, shows little about how they reach their answers, and it remains disputed whether they truly reason or mostly replay patterns memorized from training data. Mondorf and Plank survey studies that examine the reasoning process rather than the final score. They distinguish reasoning *performance*, which is outcome-based, from reasoning *behavior*, meaning the steps, outputs and internal mechanisms a system shows while solving a task. They also separate core tasks, which test one skill in isolation, from integrated tasks such as commonsense or scientific reasoning. The review covers only core logical, mathematical and causal tasks, and it proposes a taxonomy of evaluation approaches.

The literature points in one direction. Models cope with problems resembling their training data but make conceptual errors out of distribution, relying on surface statistics and shortcuts. The authors therefore compare them more to stochastic parrots than to systematic reasoners. Given widespread benchmark contamination, they argue that static accuracy metrics are insufficient. They call for behavior-focused evaluation on novel scenarios and for comparative studies of human and LLM reasoning.

## Key points
- The survey poses two research questions: how LLMs behave on logical, mathematical and causal tasks, and which methods are used to evaluate that behavior. The authors present it as the first review dedicated to evaluation beyond task accuracy.
- The proposed taxonomy has four evaluation families, compared for strengths and weaknesses:
  - **Conclusion-based:** scalable and reproducible, but shallow.
  - **Rationale-based:** more nuanced, but harder to automate.
  - **Interactive:** adaptive or dialectic, flexible but poorly standardized.
  - **Mechanistic:** probes attention and activations, but is compute-heavy and may not generalize.
- Saparov & He (2023) parsed GPT-3 chain-of-thought steps into first-order logic. Larger models produced more valid and atomic steps, but these steps were often of low utility and led the models astray.
- Dziri et al. (2023) analyzed solutions to Einstein's puzzle as computation graphs. GPT-3, ChatGPT and GPT-4 relied on shortcut pattern matching, and early mistakes compounded in later steps.
- Chen et al. (2024b) showed that changing the order of premises, without changing the logic, disrupted ChatGPT, GPT-4, PaLM 2-L and Gemini Pro. Conclusions drawn from accuracy scores are further weakened by test data leaking into training sets, as documented by Balloccu et al. (2024) and Xu et al. (2024).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Ludwig-Maximilians-Universität München](https://md.research.elamur.ai/free/en/universities/ludwig-maximilians-universitat-munchen/) — University, Munich, Germany · 107 works · 56,478 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Philipp Mondorf and Barbara Plank. 2024. Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models — A Survey. arXiv:2404.01869.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mondorf2024beyond,
  title = {{Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models — A Survey}},
  author = {Philipp Mondorf and Barbara Plank},
  year = {2024},
  doi = {10.48550/arxiv.2404.01869},
  eprint = {2404.01869},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.01869}
}
```

- [← previous SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages](https://md.research.elamur.ai/free/en/papers/lovenia-2024-seacrowd-multilingual-multimodal-data-hub/)
- [next → Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?](https://md.research.elamur.ai/free/en/papers/lee-2024-long-context-language-models-subsume/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

125 references, 39 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. [Large language models for mathematical reasoning: Progresses and challenges](https://md.research.elamur.ai/free/en/papers/ahn-2024-large-language-models-mathematical-reasoning/). In Neele Falk, Sara Papi, and Mike Zhang (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Lingui…
2. AlephAlpha. Luminous language model, 2022. [find](https://scholar.google.com/scholar?q=AlephAlpha%2E+Luminous+language+model%2C+2022%2E)
3. American Psychological Association. Behavior. In APA Dictionary of Psychology, n.d. Retrieved March 15, 2024, from [https://dictionary.apa.org/behavior](https://dictionary.apa.org/behavior). [↗](https://dictionary.apa.org/behavior)
4. Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. [Evaluating large language models with NeuBAROCO: Syllogistic reasoning ability and human-like biases](https://md.research.elamur.ai/free/en/papers/ando-2023-evaluating-large-language-models-neubaroco/). In Stergios Chatzikyriakidis and Valeria de Paiva (eds.), Proceedings of the 4th Natural Logic Meets Machine Learn…
5. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… — [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
6. Anthropic. Model card and evaluations for claude models, 2023. Technical Report. Available at: [https://www-cdn.anthropic.com/files/4zrzovbb/website/](https://www-cdn.anthropic.com/files/4zrzovbb/website/) bd2a28d2535bfb0494cc8e2a3bf135d2e7523226.pdf. [↗](https://www-cdn.anthropic.com/files/4zrzovbb/website/)
7. Simone Balloccu, Patrı́cia Schmidtová, Mateusz Lango, and Ondrej Dusek. [Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs](https://md.research.elamur.ai/free/en/papers/balloccu-2024-leak-cheat-repeat-data-contamination/). In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computatio…
8. Alejandro Barredo Arrieta, Natalia Dı́az-Rodrı́guez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities… [find](https://scholar.google.com/scholar?q=Alejandro+Barredo+Arrieta%2C+Natalia+D%C4%B1%CC%81az%2DRodr%C4%B1%CC%81guez%2C+Javier+Del+Ser%2C+Adrien+Bennetot%2C+Siham+Tabik%2C+Alberto+Barbado%2C+Salvador+Garcia%2C+Sergio+Gil%2DLopez%2C+Daniel+Molina%2C+Richard+Benjamins%2C+Raja+Chatila%2C+a)
9. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pp. 610–623, New York, NY, USA, 2021. Associati… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference+on+Fa)
10. Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.14082, 2024. [find](https://scholar.google.com/scholar?q=Leonard+Bereska+and+Efstratios+Gavves%2E+Mechanistic+interpretability+for+ai+safety%E2%80%93a+review%2E+arXiv+preprint+arXiv%3A2404%2E14082%2C+2024%2E)
11. Leonardo Bertolazzi, Davide Mazzaccara, Filippo Merlo, and Raffaella Bernardi. Chat-GPT’s information seeking strategy: Insights from the 20-questions game. In C. Maria Keet, Hung-Yi Lee, and Sina Zarrieß (eds.), Proceedings of the 16th International Natural Language Generation Conference, pp. 153–1… [find](https://scholar.google.com/scholar?q=Leonardo+Bertolazzi%2C+Davide+Mazzaccara%2C+Filippo+Merlo%2C+and+Raffaella+Bernardi%2E+Chat%2DGPT%E2%80%99s+information+seeking+strategy%3A+Insights+from+the+20%2Dquestions+game%2E+In+C%2E+Maria+Keet%2C+Hung%2DYi+Lee%2C+and+Sina+Zar)
12. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demsz… — [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
13. Ali Borji. Stochastic parrots or intelligent systems? a perspective on true depth of understanding in llms, July 2023. Available at SSRN: [https://ssrn.com/abstract=4507038](https://ssrn.com/abstract=4507038) or http://dx.doi.org/10.2139/ssrn.4507038. — [doi:10.2139/ssrn.4507038](https://doi.org/10.2139/ssrn.4507038) [↗](https://ssrn.com/abstract=4507038)
14. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
15. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper](https://proceedings.neurips.cc/paper) files/paper/2020/ file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. [↗](https://proceedings.neurips.cc/paper)
16. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
17. R. M. J. Byrne, J. S. B. T. Evans, and S. E. Newstead. Human Reasoning: The Psychology Of Deduction. Psychology Press, 1st edition, 1993. doi: 10.4324/9781315785028. URL https://doi.org/10.4324/9781315785028. — [doi:10.4324/9781315785028](https://doi.org/10.4324/9781315785028) [↗](https://doi.org/10.4324/9781315785028)
18. Nuo Chen, Ning Wu, Shining Liang, Ming Gong, Linjun Shou, Dongmei Zhang, and Jia Li. Is bigger and deeper always better? probing llama across scales and layers. arXiv preprint arXiv:2312.04333, 2024a. — [arXiv:2312.04333](https://arxiv.org/abs/2312.04333) [find](https://scholar.google.com/scholar?q=Nuo+Chen%2C+Ning+Wu%2C+Shining+Liang%2C+Ming+Gong%2C+Linjun+Shou%2C+Dongmei+Zhang%2C+and+Jia+Li%2E+Is+bigger+and+deeper+always+better%3F+probing+llama+across+scales+and+layers%2E+arXiv+preprint+arXiv%3A2312%2E04333%2C+2024a%2E)
19. Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models. arXiv preprint arXiv:2402.08939, 2024b. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Ryan+A+Chi%2C+Xuezhi+Wang%2C+and+Denny+Zhou%2E+Premise+order+matters+in+reasoning+with+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E08939%2C+2024b%2E)
20. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402, 2023. [find](https://scholar.google.com/scholar?q=Zheng+Chu%2C+Jingchang+Chen%2C+Qianglong+Chen%2C+Weijiang+Yu%2C+Tao+He%2C+Haotian+Wang%2C+Weihua+Peng%2C+Ming+Liu%2C+Bing+Qin%2C+and+Ting+Liu%2E+A+survey+of+chain+of+thought+reasoning%3A+Advances%2C+frontiers+and+future%2E+arX)
21. Kelsey M Collins, Catherine Wong, Jessica Feng, Michelle Wei, and Joshua Tenenbaum. Structured, flexible, and robust: Benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks. In Proceedings of the Annual Meeting of the Cognitive Scienc… [find](https://scholar.google.com/scholar?q=Kelsey+M+Collins%2C+Catherine+Wong%2C+Jessica+Feng%2C+Michelle+Wei%2C+and+Joshua+Tenenbaum%2E+Structured%2C+flexible%2C+and+robust%3A+Benchmarking+and+improving+large+language+models+towards+more+human%2Dlike+behavior+)
22. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051, 2022. [find](https://scholar.google.com/scholar?q=Ishita+Dasgupta%2C+Andrew+K+Lampinen%2C+Stephanie+CY+Chan%2C+Antonia+Creswell%2C+Dharshan+Kumaran%2C+James+L+McClelland%2C+and+Felix+Hill%2E+Language+models+show+human%2Dlike+content+effects+on+reasoning%2E+arXiv+prepr)
23. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient fine-tuning of quantized llms. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 10088– 10115. Curran Associates, Inc.,… [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+Qlora%3A+Efficient+fine%2Dtuning+of+quantized+llms%2E+In+A%2E+Oh%2C+T%2E+Neumann%2C+A%2E+Globerson%2C+K%2E+Saenko%2C+M%2E+Hardt%2C+and+S%2E+Levine+%28eds%2E%29%2C+Advan)
24. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Comp…
25. Aliya R. Dewey. Arbitrating norms for reasoning tasks. Synthese, 200(6):502, Nov 2022. ISSN 1573-0964. doi: 10.1007/s11229-022-03981-8. URL https://doi.org/10.1007/ s11229-022-03981-8. — [doi:10.1007/s11229-022-03981-8](https://doi.org/10.1007/s11229-022-03981-8) [find](https://scholar.google.com/scholar?q=Aliya+R%2E+Dewey%2E+Arbitrating+norms+for+reasoning+tasks%2E+Synthese%2C+200%286%29%3A502%2C+Nov+2022%2E+ISSN+1573%2D0964%2E+doi%3A+10%2E1007%2Fs11229%2D022%2D03981%2D8%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F10%2E1007%2F+s11229%2D022%2D03981%2D8%2E+%E2%80%94+%5Bdoi%3A10%2E1007%2F)
26. Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning. arXiv preprint arXiv:2402.18312, 2024. [find](https://scholar.google.com/scholar?q=Subhabrata+Dutta%2C+Joykirat+Singh%2C+Soumen+Chakrabarti%2C+and+Tanmoy+Chakraborty%2E+How+to+think+step%2Dby%2Dstep%3A+A+mechanistic+understanding+of+chain%2Dof%2Dthought+reasoning%2E+arXiv+preprint+arXiv%3A2402%2E18312%2C+202)
27. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang (Lorraine) Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. [Faith and fate: Limits of transformers on compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/).…
28. Tiwalayo Eisape, Michael Tessler, Ishita Dasgupta, Fei Sha, Sjoerd Steenkiste, and Tal Linzen. A systematic comparison of syllogistic reasoning in humans and language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of th… [find](https://scholar.google.com/scholar?q=Tiwalayo+Eisape%2C+Michael+Tessler%2C+Ishita+Dasgupta%2C+Fei+Sha%2C+Sjoerd+Steenkiste%2C+and+Tal+Linzen%2E+A+systematic+comparison+of+syllogistic+reasoning+in+humans+and+language+models%2E+In+Kevin+Duh%2C+Helena+Gome)
29. Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. A primer on the inner workings of transformer-based language models. arXiv preprint arXiv:2405.00208, 2024. [find](https://scholar.google.com/scholar?q=Javier+Ferrando%2C+Gabriele+Sarti%2C+Arianna+Bisazza%2C+and+Marta+R+Costa%2Djussa%CC%80%2E+A+primer+on+the+inner+workings+of+transformer%2Dbased+language+models%2E+arXiv+preprint+arXiv%3A2405%2E00208%2C+2024%2E)
30. Peter A. Flach and Antonis C. Kakas. Abductive and Inductive Reasoning: Background and Issues, pp. 1–27. Springer Netherlands, Dordrecht, 2000. ISBN 978-94-017-0606-3. doi: 10.1007/978-94-017-0606-3 1. URL https://doi.org/10.1007/978-94-017-0606-3 1. — [doi:10.1007/978-94-017-0606-3](https://doi.org/10.1007/978-94-017-0606-3) [↗](https://doi.org/10.1007/978-94-017-0606-3)
31. Shane Frederick. Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4):25–42, December 2005. doi: 10.1257/089533005775196732. URL [https://www.aeaweb.org/articles?id=10.1257/089533005775196732](https://www.aeaweb.org/articles?id=10.1257/089533005775196732). — [doi:10.1257/089533005775196732](https://doi.org/10.1257/089533005775196732) [↗](https://www.aeaweb.org/articles?id=10.1257/089533005775196732)
32. Jörg Frohberg and Frank Binder. CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mari… [find](https://scholar.google.com/scholar?q=Jo%CC%88rg+Frohberg+and+Frank+Binder%2E+CRASS%3A+A+novel+data+set+and+benchmark+to+test+counterfactual+reasoning+of+large+language+models%2E+In+Nicoletta+Calzolari%2C+Fre%CC%81de%CC%81ric+Be%CC%81chet%2C+Philippe+Blache%2C+Khalid+Ch)
33. Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance. arXiv preprint arXiv:2305.[17306, 2023](https://md.research.elamur.ai/free/en/papers/fu-2023-chain-thought-hub-continuous-effort/).
34. Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. [ROSCOE: A suite of metrics for scoring step-by-step reasoning](https://md.research.elamur.ai/free/en/papers/golovneva-2022-roscoe-suite-metrics-scoring-step/). In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=…](https://openreview.net/forum?id=…) [↗](https://openreview.net/forum?id=…)
35. Alison Gopnik and David M. Sobel. Detecting blickets: How young children use information about novel causal powers in categorization and induction. Child Development, 71(5):1205– 1222, 2000. ISSN 00093920, 14678624. URL [http://www.jstor.org/stable/1131970](http://www.jstor.org/stable/1131970). [↗](http://www.jstor.org/stable/1131970)
36. Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Computational Science, 3(10):833–838, Oct 2023. ISSN 2662-8457. doi: 10.1038/s43588-023-00527-x. URL https://doi.org/10.1038/s4358… — [doi:10.1038/s43588-023-00527-x](https://doi.org/10.1038/s43588-023-00527-x) [↗](https://doi.org/10.1038/s4358…)
37. Simon Jerome Han, Keith J. Ransom, Andrew Perfors, and Charles Kemp. Inductive reasoning in humans and large language models. Cognitive Systems Research, 83:101155, 2024. ISSN 1389-0417. doi: https://doi.org/10.1016/j.cogsys.2023.101155. URL https://www.sciencedirect.com/science/article/pii/S138904… — [doi:10.1016/j.cogsys.2023.101155](https://doi.org/10.1016/j.cogsys.2023.101155) [↗](https://doi.org/10.1016/j.cogsys.2023.101155)
38. H. Carl Haywood and David Tzuriel. Applications and challenges in dynamic assessment. Peabody Journal of Education, 77(2):40–63, 2002. doi: 10.1207/S15327930PJE7702 \\ 5. URL https://doi.org/10.1207/S15327930PJE7702 5. — [doi:10.1207/s15327930pje7702](https://doi.org/10.1207/s15327930pje7702) [↗](https://doi.org/10.1207/S15327930PJE7702)
39. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL ht…
40. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
41. K.J. Holyoak and R.G. Morrison. The Cambridge Handbook of Thinking and Reasoning. Cambridge Handbooks in Psychology. Cambridge University Press, 2005. ISBN 9780521824170. URL [https://books.google.de/books?id=znbkHaC8QeMC](https://books.google.de/books?id=znbkHaC8QeMC). [↗](https://books.google.de/books?id=znbkHaC8QeMC)
42. Leon Horsten. Philosophy of Mathematics. In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2023 edition, 2023. [find](https://scholar.google.com/scholar?q=Leon+Horsten%2E+Philosophy+of+Mathematics%2E+In+Edward+N%2E+Zalta+and+Uri+Nodelman+%28eds%2E%29%2C+The+Stanford+Encyclopedia+of+Philosophy%2E+Metaphysics+Research+Lab%2C+Stanford+University%2C+Winter+2023+edition%2C+2023%2E)
43. Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, and Mrinmaya Sachan. Towards a mechanistic interpretation of multi-step reasoning capabilities of language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conferen… [find](https://scholar.google.com/scholar?q=Yifan+Hou%2C+Jiaoda+Li%2C+Yu+Fei%2C+Alessandro+Stolfo%2C+Wangchunshu+Zhou%2C+Guangtao+Zeng%2C+Antoine+Bosselut%2C+and+Mrinmaya+Sachan%2E+Towards+a+mechanistic+interpretation+of+multi%2Dstep+reasoning+capabilities+of+la)
44. Jie Huang and Kevin Chen-Chuan Chang. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/). In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 1049–1065, Toronto, Canada, July 2023. Association for Computational Lin…
45. Yinya Huang, Ruixin Hong, Hongming Zhang, Wei Shao, Zhicheng Yang, Dong Yu, Changshui Zhang, Xiaodan Liang, and Linqi Song. Clomo: Counterfactual logical modification with large language models. arXiv preprint arXiv:2311.17438, 2024. [find](https://scholar.google.com/scholar?q=Yinya+Huang%2C+Ruixin+Hong%2C+Hongming+Zhang%2C+Wei+Shao%2C+Zhicheng+Yang%2C+Dong+Yu%2C+Changshui+Zhang%2C+Xiaodan+Liang%2C+and+Linqi+Song%2E+Clomo%3A+Counterfactual+logical+modification+with+large+language+models%2E+arXiv)
46. Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, and Yonatan Belinkov. Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias. Transactions of the Association for Computational Linguistics, 12:771–785, 06 2024. ISSN 2307-387X. doi: 10.1162/tacl a 00673. URL [https://doi.o…](https://doi.o…) [↗](https://doi.o…)
47. Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng LYU, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, and Bernhard Schölkopf. CLadder: A benchmark to assess causal reasoning capabilities of language models. In Thirty-seventh Conference on Neural Inf… [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Yuen+Chen%2C+Felix+Leeb%2C+Luigi+Gresele%2C+Ojasv+Kamal%2C+Zhiheng+LYU%2C+Kevin+Blin%2C+Fernando+Gonzalez+Adauto%2C+Max+Kleiman%2DWeiner%2C+Mrinmaya+Sachan%2C+and+Bernhard+Scho%CC%88lkopf%2E+CLadder%3A+A+benchmark+to)
48. Zhijing Jin, Jiarui Liu, Zhiheng LYU, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona T. Diab, and Bernhard Schölkopf. Can large language models infer causation from correlation? In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=vqIH0O…](https://openreview.net/forum?id=vqIH0O…) [↗](https://openreview.net/forum?id=vqIH0O…)
49. P. N. Johnson-Laird. Mental models and human reasoning. Proc Natl Acad Sci U S A, 107 (43):18243–18250, 2010. doi: 10.1073/pnas.1012933107. — [doi:10.1073/pnas.1012933107](https://doi.org/10.1073/pnas.1012933107) [find](https://scholar.google.com/scholar?q=P%2E+N%2E+Johnson%2DLaird%2E+Mental+models+and+human+reasoning%2E+Proc+Natl+Acad+Sci+U+S+A%2C+107+%2843%29%3A18243%E2%80%9318250%2C+2010%2E+doi%3A+10%2E1073%2Fpnas%2E1012933107%2E+%E2%80%94+%5Bdoi%3A10%2E1073%2Fpnas%2E1012933107%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1073%2Fpnas)
50. Philip Johnson-Laird. How We Reason. Oxford University Press, 10 2008. ISBN 9780199551330. doi: 10.1093/acprof:oso/9780199551330.001.0001. URL https://doi.org/10.1093/acprof:oso/9780199551330.001.0001. — [doi:10.1093/acprof:oso/9780199551330.001.0001](https://doi.org/10.1093/acprof:oso/9780199551330.001.0001) [↗](https://doi.org/10.1093/acprof:oso/9780199551330.001.0001)
51. P.N. Johnson-Laird. How We Reason. Oxford University Press, 2006. ISBN 9780198569763. URL [https://books.google.de/books?id=UjYsJN0krNYC](https://books.google.de/books?id=UjYsJN0krNYC). [↗](https://books.google.de/books?id=UjYsJN0krNYC)
52. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
53. Karl Christoph Klauer, Jochen Musch, and Barbara Naumer. On belief bias in syllogistic reasoning. Psychological Review, 107(4):852–884, 2000. doi: 10.1037/0033-295X.107.4.852. URL https://doi.org/10.1037/0033-295X.107.4.852. — [doi:10.1037/0033-295x.107.4.852](https://doi.org/10.1037/0033-295x.107.4.852) [↗](https://doi.org/10.1037/0033-295X.107.4.852)
54. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 22199–22213. Curran Assoc…
55. Robert Koons. Defeasible Reasoning. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2022 edition, 2022. [find](https://scholar.google.com/scholar?q=Robert+Koons%2E+Defeasible+Reasoning%2E+In+Edward+N%2E+Zalta+%28ed%2E%29%2C+The+Stanford+Encyclopedia+of+Philosophy%2E+Metaphysics+Research+Lab%2C+Stanford+University%2C+Summer+2022+edition%2C+2022%2E)
56. E. Kosoy, D. M. Chan, A. Liu, J. Collins, J. Hamrick, S. H. Huang, et al. Towards understanding how machines can learn causal overhypotheses. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45. Cognitive Science Society, 2023. URL [https://escholarship.org/uc/item/9q29w1…](https://escholarship.org/uc/item/9q29w1…) [↗](https://escholarship.org/uc/item/9q29w1…)
57. Jacqueline P. Leighton. Defining and Describing Reason, pp. 3–11. Cambridge University Press, 2003. [find](https://scholar.google.com/scholar?q=Jacqueline+P%2E+Leighton%2E+Defining+and+Describing+Reason%2C+pp%2E+3%E2%80%9311%2E+Cambridge+University+Press%2C+2003%2E)
58. Alessandro Lenci. Understanding natural language understanding systems. a critical analysis. arXiv preprint arXiv:2303.04229, 2023. [find](https://scholar.google.com/scholar?q=Alessandro+Lenci%2E+Understanding+natural+language+understanding+systems%2E+a+critical+analysis%2E+arXiv+preprint+arXiv%3A2303%2E04229%2C+2023%2E)
59. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joe…
60. Jiaxuan Li, Lang Yu, and Allyson Ettinger. Counterfactual reasoning: Testing language models’ understanding of hypothetical scenarios. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Sho… [find](https://scholar.google.com/scholar?q=Jiaxuan+Li%2C+Lang+Yu%2C+and+Allyson+Ettinger%2E+Counterfactual+reasoning%3A+Testing+language+models%E2%80%99+understanding+of+hypothetical+scenarios%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoaki+Okazaki+%28eds%2E%29%2C+Pr)
61. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. [Evaluating the logical reasoning ability of chatgpt and gpt-4](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/). arXiv preprint arXiv:2304.03439, 2023.
62. Wentao Liu, Hanglei Hu, Jie Zhou, Yuyang Ding, Junsong Li, Jiayi Zeng, Mengliang He, Qin Chen, Bo Jiang, Aimin Zhou, and Liang He. Mathematical language models: A survey. arXiv preprint arXiv:2312.07622, 2024. — [arXiv:2312.07622](https://arxiv.org/abs/2312.07622) [find](https://scholar.google.com/scholar?q=Wentao+Liu%2C+Hanglei+Hu%2C+Jie+Zhou%2C+Yuyang+Ding%2C+Junsong+Li%2C+Jiayi+Zeng%2C+Mengliang+He%2C+Qin+Chen%2C+Bo+Jiang%2C+Aimin+Zhou%2C+and+Liang+He%2E+Mathematical+language+models%3A+A+survey%2E+arXiv+preprint+arXiv%3A2312%2E076)
63. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692, 2019.
64. David F. Lohman and Joni M. Lakin. Intelligence and Reasoning, pp. 419–441. Cambridge Handbooks in Psychology. Cambridge University Press, 2011. [find](https://scholar.google.com/scholar?q=David+F%2E+Lohman+and+Joni+M%2E+Lakin%2E+Intelligence+and+Reasoning%2C+pp%2E+419%E2%80%93441%2E+Cambridge+Handbooks+in+Psychology%2E+Cambridge+University+Press%2C+2011%2E)
65. Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 2024. [find](https://scholar.google.com/scholar?q=Kyle+Mahowald%2C+Anna+A+Ivanova%2C+Idan+A+Blank%2C+Nancy+Kanwisher%2C+Joshua+B+Tenenbaum%2C+and+Evelina+Fedorenko%2E+Dissociating+language+and+thought+in+large+language+models%2E+Trends+in+Cognitive+Sciences%2C+2024%2E)
66. Jitendra Malik. Workshop on foundation models. [https://www.youtube.com/watch?v=](https://www.youtube.com/watch?v=) dG628PEN1fY&t=15830s, August 2021. YouTube video. [↗](https://www.youtube.com/watch?v=)
67. R Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638, 2023. [find](https://scholar.google.com/scholar?q=R+Thomas+McCoy%2C+Shunyu+Yao%2C+Dan+Friedman%2C+Matthew+Hardy%2C+and+Thomas+L+Griffiths%2E+Embers+of+autoregression%3A+Understanding+large+language+models+through+the+problem+they+are+trained+to+solve%2E+arXiv+prep)
68. Ian R. McKenzie, Alexander Lyzhov, Michael Martin Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Xudong Shen, Joe Cavanagh, Andrew George Gritsevskiy, Derik Kauffman, Aaron T. Kirtland, Zhengping Zhou, Yuhui Zhang, Sicong Huang, Daniel Wurgaft, Max Weiss, Alexis Ross, Gabriel Recc… — [Inverse Scaling: When Bigger Isn't Better](https://md.research.elamur.ai/free/en/papers/mckenzie-2023-inverse-scaling-bigger-isn-t/)
69. Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. [Augmented language models: a survey](https://md.research.elamur.ai/free/en/papers/mialon-2023-augmented-language-models-survey/). Transactions on Machine Learning Researc…
70. Melanie Mitchell. Can large language models reason? AI: A Guide for Thinking Humans, September 2023. URL [https://aiguide.substack.com/p/](https://aiguide.substack.com/p/) can-large-language-models-reason. Accessed on: 13th of March 2024. [↗](https://aiguide.substack.com/p/)
71. Melanie Mitchell and David C. Krakauer. The debate over understanding in ai’s large language models. Proceedings of the National Academy of Sciences, 120(13):e2215907120, 2023. doi: 10.1073/pnas.2215907120. URL [https://www.pnas.org/doi/abs/10.1073/pnas](https://www.pnas.org/doi/abs/10.1073/pnas). 2215907120. — [doi:10.1073/pnas.2215907120](https://doi.org/10.1073/pnas.2215907120) [↗](https://www.pnas.org/doi/abs/10.1073/pnas)
72. Philipp Mondorf and Barbara Plank. Comparing inferential strategies of humans and large language models in deductive reasoning. arXiv preprint arXiv:2402.14856, 2024. [find](https://scholar.google.com/scholar?q=Philipp+Mondorf+and+Barbara+Plank%2E+Comparing+inferential+strategies+of+humans+and+large+language+models+in+deductive+reasoning%2E+arXiv+preprint+arXiv%3A2402%2E14856%2C+2024%2E)
73. Carey K. Morewedge and Daniel Kahneman. Associative processes in intuitive judgment. Trends in Cognitive Sciences, 14(10):435–440, 2010. doi: 10.1016/j.tics.2010.07.004. — [doi:10.1016/j.tics.2010.07.004](https://doi.org/10.1016/j.tics.2010.07.004) [find](https://scholar.google.com/scholar?q=Carey+K%2E+Morewedge+and+Daniel+Kahneman%2E+Associative+processes+in+intuitive+judgment%2E+Trends+in+Cognitive+Sciences%2C+14%2810%29%3A435%E2%80%93440%2C+2010%2E+doi%3A+10%2E1016%2Fj%2Etics%2E2010%2E07%2E004%2E+%E2%80%94+%5Bdoi%3A10%2E1016%2Fj%2Etics%2E2010%2E07%2E)
74. OpenAI. Chatgpt: Optimizing language models for dialogue, 2022. Available at: [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
75. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
76. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
77. David Papo. How can we study reasoning in the brain? Frontiers in Human Neuroscience, 9: 222, 2015. doi: 10.3389/fnhum.2015.00222. — [doi:10.3389/fnhum.2015.00222](https://doi.org/10.3389/fnhum.2015.00222) [find](https://scholar.google.com/scholar?q=David+Papo%2E+How+can+we+study+reasoning+in+the+brain%3F+Frontiers+in+Human+Neuroscience%2C+9%3A+222%2C+2015%2E+doi%3A+10%2E3389%2Ffnhum%2E2015%2E00222%2E+%E2%80%94+%5Bdoi%3A10%2E3389%2Ffnhum%2E2015%2E00222%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E3389%2Ffnhum%2E2015%2E0)
78. Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009. [find](https://scholar.google.com/scholar?q=Judea+Pearl%2E+Causality%2E+Cambridge+University+Press%2C+2+edition%2C+2009%2E)
79. Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018. [find](https://scholar.google.com/scholar?q=Judea+Pearl+and+Dana+Mackenzie%2E+The+book+of+why%3A+the+new+science+of+cause+and+effect%2E+Basic+books%2C+2018%2E)
80. Paulo Pirozelli, Marcos M José, Anarosa AF Brandão, Fabio G Cozman, et al. Assessing logical reasoning capabilities of encoder-only transformer models. arXiv preprint arXiv:2312.11720, 2023. — [arXiv:2312.11720](https://arxiv.org/abs/2312.11720) [find](https://scholar.google.com/scholar?q=Paulo+Pirozelli%2C+Marcos+M+Jose%CC%81%2C+Anarosa+AF+Branda%CC%83o%2C+Fabio+G+Cozman%2C+et+al%2E+Assessing+logical+reasoning+capabilities+of+encoder%2Donly+transformer+models%2E+arXiv+preprint+arXiv%3A2312%2E11720%2C+2023%2E+%E2%80%94+%5BarXi)
81. Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. [ReCEval: Evaluating reasoning chains via correctness and informativeness](https://md.research.elamur.ai/free/en/papers/prasad-2023-receval-evaluating-reasoning-chains-correctness/). In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10066–10086, Singa…
82. Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. [Reasoning with language model prompting: A survey](https://md.research.elamur.ai/free/en/papers/qiao-2022-reasoning-language-model-prompting-survey/). In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computat…
83. Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In The Twelfth International Confe… [find](https://scholar.google.com/scholar?q=Linlu+Qiu%2C+Liwei+Jiang%2C+Ximing+Lu%2C+Melanie+Sclar%2C+Valentina+Pyatkin%2C+Chandra+Bhagavatula%2C+Bailin+Wang%2C+Yoon+Kim%2C+Yejin+Choi%2C+Nouha+Dziri%2C+and+Xiang+Ren%2E+Phenomenal+yet+puzzling%3A+Testing+inductive+reas)
84. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL [https://api.semanticscholar.org/CorpusID:160025533](https://api.semanticscholar.org/CorpusID:160025533). [↗](https://api.semanticscholar.org/CorpusID:160025533)
85. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67, 2020. URL [http://jmlr.org/papers/v21/20…](http://jmlr.org/papers/v21/20…) [↗](http://jmlr.org/papers/v21/20…)
86. Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pre-training term frequencies on few-shot numerical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 840–854, Abu Dhabi, Unite… — [Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning](https://md.research.elamur.ai/free/en/papers/razeghi-2022-impact-pretraining-term-frequencies-few/)
87. Lance J. Rips. The psychology of knights and knaves. Cognition, 31(2):85–116, 1989. ISSN 0010-0277. doi: https://doi.org/10.1016/0010-0277(89)90019-X. URL https://www.sciencedirect.com/science/article/pii/001002778990019X. — [doi:10.1016/0010-0277(89)90019-x](https://doi.org/10.1016/0010-0277%2889%2990019-x) [↗](https://doi.org/10.1016/0010-0277(89)90019-X)
88. John Alan Robinson and Andrei Voronkov (eds.). Handbook of Automated Reasoning (in 2 volumes). Elsevier and MIT Press, 2001. ISBN 0-444-50813-9. URL [https://www.sciencedirect.com/book/9780444508133/handbook-of-automated-reasoning](https://www.sciencedirect.com/book/9780444508133/handbook-of-automated-reasoning). [↗](https://www.sciencedirect.com/book/9780444508133/handbook-of-automated-reasoning)
89. Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. Thinking like a skeptic: Defeasible inference in natural language. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Vered+Shwartz%2C+Jena+D%2E+Hwang%2C+Chandra+Bhagavatula%2C+Maxwell+Forbes%2C+Ronan+Le+Bras%2C+Noah+A%2E+Smith%2C+and+Yejin+Choi%2E+Thinking+like+a+skeptic%3A+Defeasible+inference+in+natural+language%2E+In+)
90. Soumya Sanyal, Zeyi Liao, and Xiang Ren. RobustLR: A diagnostic benchmark for evaluating logical robustness of deductive reasoners. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9614–9631, Abu Dh… [find](https://scholar.google.com/scholar?q=Soumya+Sanyal%2C+Zeyi+Liao%2C+and+Xiang+Ren%2E+RobustLR%3A+A+diagnostic+benchmark+for+evaluating+logical+robustness+of+deductive+reasoners%2E+In+Yoav+Goldberg%2C+Zornitsa+Kozareva%2C+and+Yue+Zhang+%28eds%2E%29%2C+Proceedin)
91. Abulhair Saparov and He He. [Language models are greedy reasoners: A systematic formal analysis of chain-of-thought](https://md.research.elamur.ai/free/en/papers/saparov-2022-language-models-greedy-reasoners-systematic/). In International Conference on Learning Representations, 2023.
92. Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Mehran Kazemi, Najoung Kim, and He He. [Testing the general deductive reasoning capacity of large language models using ood examples](https://md.research.elamur.ai/free/en/papers/saparov-2023-testing-general-deductive-reasoning-capacity/). Advances in Neural Information Processing Systems, 36, 2024.
93. Murray Shanahan. Talking about large language models. Commun. ACM, 67(2):68–79, jan 2024. ISSN 0001-0782. doi: 10.1145/3624724. URL https://doi.org/10.1145/3624724. — [doi:10.1145/3624724](https://doi.org/10.1145/3624724) [↗](https://doi.org/10.1145/3624724)
94. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Huai hsin Chi, Nathanael Scharli, and Denny Zhou. [Large language models can be easily distracted by irrelevant context](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/). In International Conference on Machine Learning, 2023. URL [https://api.semanticscholar.org/CorpusID:25645977…](https://api.semanticscholar.org/CorpusID:25645977…) [↗](https://api.semanticscholar.org/CorpusID:25645977…)
95. Steven Sloman. Causal Models: How People Think about the World and Its Alternatives. Oxford University Press, 08 2005. ISBN 9780195183115. doi: 10.1093/acprof:oso/9780195183115. 001.0001. URL https://doi.org/10.1093/acprof:oso/9780195183115.001.0001. — [doi:10.1093/acprof:oso/9780195183115](https://doi.org/10.1093/acprof:oso/9780195183115) [↗](https://doi.org/10.1093/acprof:oso/9780195183115.001.0001)
96. Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402.19450, 2024. [find](https://scholar.google.com/scholar?q=Saurabh+Srivastava%2C+Anto+PV%2C+Shashank+Menon%2C+Ajay+Sukumar%2C+Alan+Philipose%2C+Stevin+Prince%2C+Sooraj+Thomas%2C+et+al%2E+Functional+benchmarks+for+robust+evaluation+of+reasoning+performance%2C+and+the+reasoning+)
97. Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, and Mrinmaya Sachan. [A causal framework to quantify the robustness of mathematical reasoning with language models](https://md.research.elamur.ai/free/en/papers/stolfo-2023-causal-framework-quantify-robustness-mathematical/). In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Associa…
98. Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, Yue Wu, Wenhai Wang, Junsong Chen, Zhangyue Yin, Xiaozhe Ren, Jie Fu, Junxian He, Wu Yuan, Qi Liu, Xihui Liu, Yu Li, Hao Dong, Yu Cheng, Ming Zhang, Pheng Ann Heng, Jife… — [arXiv:2312.11562](https://arxiv.org/abs/2312.11562) [find](https://scholar.google.com/scholar?q=Jiankai+Sun%2C+Chuanyang+Zheng%2C+Enze+Xie%2C+Zhengying+Liu%2C+Ruihang+Chu%2C+Jianing+Qiu%2C+Jiaqi+Xu%2C+Mingyu+Ding%2C+Hongyang+Li%2C+Mengzhe+Geng%2C+Yue+Wu%2C+Wenhai+Wang%2C+Junsong+Chen%2C+Zhangyue+Yin%2C+Xiaozhe+Ren%2C+Jie+Fu%2C)
99. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy L… — [Gemini: A Family of Highly Capable Multimodal Models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/)
100. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv prepri…
101. Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. [Language models are not naysayers: an analysis of language models on negation benchmarks](https://md.research.elamur.ai/free/en/papers/truong-2023-language-models-naysayers-analysis-language/). In Alexis Palmer and Jose Camacho-collados (eds.), Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023…
102. Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124–1131, 1974. doi: 10.1126/science.185.4157.1124. URL [https://www.science.org/doi/abs/10.1126/science.185.4157.1124](https://www.science.org/doi/abs/10.1126/science.185.4157.1124). — [doi:10.1126/science.185.4157.1124](https://doi.org/10.1126/science.185.4157.1124) [↗](https://www.science.org/doi/abs/10.1126/science.185.4157.1124)
103. Jean-Baptiste Van der Henst, Yingrui Yang, and P.N. Johnson-Laird. Strategies in sentential reasoning. Cognitive Science, 26(4):425–468, 2002. ISSN 0364-0213. URL [https://www.sciencedirect.com/science/article/pii/S0364021302000745](https://www.sciencedirect.com/science/article/pii/S0364021302000745). [↗](https://www.sciencedirect.com/science/article/pii/S0364021302000745)
104. Dylan Vassberg and John Vassberg. Is einstein’s puzzle over-specified? In AJ75th Symposium: 21st Century Challenges in Computational Engineering & Science, Princeton University, Princeton, NJ, November 2009. [find](https://scholar.google.com/scholar?q=Dylan+Vassberg+and+John+Vassberg%2E+Is+einstein%E2%80%99s+puzzle+over%2Dspecified%3F+In+AJ75th+Symposium%3A+21st+Century+Challenges+in+Computational+Engineering+%26+Science%2C+Princeton+University%2C+Princeton%2C+NJ%2C+Novembe)
105. Vicuna. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. [https://vicuna.lmsys.org/](https://vicuna.lmsys.org/), 2023. Accessed: 2024-03-15. [↗](https://vicuna.lmsys.org/)
106. Yuxuan Wan, Wenxuan Wang, Yiliu Yang, Youliang Yuan, Jen tse Huang, Pinjia He, Wenxiang Jiao, and Michael R. Lyu. A & b == b & a: Triggering logical reasoning failures in large language models. arXiv preprint arXiv:2401.00757, 2024. — [arXiv:2401.00757](https://arxiv.org/abs/2401.00757) [find](https://scholar.google.com/scholar?q=Yuxuan+Wan%2C+Wenxuan+Wang%2C+Yiliu+Yang%2C+Youliang+Yuan%2C+Jen+tse+Huang%2C+Pinjia+He%2C+Wenxiang+Jiao%2C+and+Michael+R%2E+Lyu%2E+A+%26+b+%3D%3D+b+%26+a%3A+Triggering+logical+reasoning+failures+in+large+language+models%2E+arXiv+)
107. Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), May 2021. URL https://github.com/kingoflolz/mesh-transformer-jax. [↗](https://github.com/kingoflolz/mesh-transformer-jax)
108. Boshi Wang, Xiang Yue, and Huan Sun. Can ChatGPT defend its belief in truth? evaluating LLM reasoning via debate. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11865–11881, Singapore, December 2023. Association for Com… [find](https://scholar.google.com/scholar?q=Boshi+Wang%2C+Xiang+Yue%2C+and+Huan+Sun%2E+Can+ChatGPT+defend+its+belief+in+truth%3F+evaluating+LLM+reasoning+via+debate%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+Bali+%28eds%2E%29%2C+Findings+of+the+Association+for+C)
109. Xiaoqiang Wang, Bang Liu, and Lingfei Wu. Fac 2 e: Better understanding large language model capabilities by dissociating language and cognition. arXiv preprint arXiv:2403.00126, 2024a. [find](https://scholar.google.com/scholar?q=Xiaoqiang+Wang%2C+Bang+Liu%2C+and+Lingfei+Wu%2E+Fac+2+e%3A+Better+understanding+large+language+model+capabilities+by+dissociating+language+and+cognition%2E+arXiv+preprint+arXiv%3A2403%2E00126%2C+2024a%2E)
110. Xinpeng Wang, Chengzhi Hu, Bolei Ma, Paul Röttger, and Barbara Plank. Look at the text: Instruction-tuned language models are more robust multiple choice selectors than you think. arXiv preprint arXiv:2404.08382, 2024b. — [arXiv:2404.08382](https://arxiv.org/abs/2404.08382) [find](https://scholar.google.com/scholar?q=Xinpeng+Wang%2C+Chengzhi+Hu%2C+Bolei+Ma%2C+Paul+Ro%CC%88ttger%2C+and+Barbara+Plank%2E+Look+at+the+text%3A+Instruction%2Dtuned+language+models+are+more+robust+multiple+choice+selectors+than+you+think%2E+arXiv+preprint+arXi)
111. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Process…
112. Moritz Willig, Matej Zečević, Devendra Singh Dhami, and Kristian Kersting. Can foundation models talk causality? In UAI 2022 Workshop on Causal Representation Learning, 2022. URL [https://openreview.net/forum?id=DbJXEqU0kaM](https://openreview.net/forum?id=DbJXEqU0kaM). [↗](https://openreview.net/forum?id=DbJXEqU0kaM)
113. Michael D. Wolcott and Nikki G. Lobczowski. Using cognitive interviews and think-aloud protocols to understand thought processes. Currents in Pharmacy Teaching and Learning, 13 (2):181–188, 2021. ISSN 1877-1297. doi: https://doi.org/10.1016/j.cptl.2020.09.005. URL https://www.sciencedirect.com/scien… — [doi:10.1016/j.cptl.2020.09.005](https://doi.org/10.1016/j.cptl.2020.09.005) [↗](https://doi.org/10.1016/j.cptl.2020.09.005)
114. D. J. WOOD. Approach to the study of human reasoning. Nature, 223(5201):101–102, Jul 1969. ISSN 1476-4687. doi: 10.1038/223101a0. URL https://doi.org/10.1038/223101a0. — [doi:10.1038/223101a0](https://doi.org/10.1038/223101a0) [↗](https://doi.org/10.1038/223101a0)
115. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceeding… [find](https://scholar.google.com/scholar?q=Zhaofeng+Wu%2C+Linlu+Qiu%2C+Alexis+Ross%2C+Ekin+Akyu%CC%88rek%2C+Boyuan+Chen%2C+Bailin+Wang%2C+Najoung+Kim%2C+Jacob+Andreas%2C+and+Yoon+Kim%2E+Reasoning+or+reciting%3F+exploring+the+capabilities+and+limitations+of+language+mo)
116. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? a comprehensive evaluation from deductive, inductive and abductive views. arXiv preprint arXiv:2306.[09841, 2023](https://md.research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/).
117. Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824, 2024. [find](https://scholar.google.com/scholar?q=Ruijie+Xu%2C+Zengzhi+Wang%2C+Run%2DZe+Fan%2C+and+Pengfei+Liu%2E+Benchmarking+benchmark+leakage+in+large+language+models%2E+arXiv+preprint+arXiv%3A2404%2E18824%2C+2024%2E)
118. Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. [Logical reasoning over natural language as knowledge representation: A survey](https://md.research.elamur.ai/free/en/papers/yang-2023-logical-reasoning-over-natural-language/). arXiv preprint arXiv:2303.12023, 2023.
119. Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. [Language models as inductive reasoners](https://md.research.elamur.ai/free/en/papers/yang-2024-language-models-as-inductive-reasoners/). In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume…
120. Xi Ye and Greg Durrett. [The unreliability of explanations in few-shot prompting for textual reasoning](https://md.research.elamur.ai/free/en/papers/ye-2022-unreliability-explanations-few-shot-prompting/). In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30378–30392. Curran Associates, Inc., 2022. URL https:…
121. Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. Natural language reasoning, a survey. ACM Comput. Surv., may 2024. ISSN 0360-0300. doi: 10.1145/3664194. URL https://doi.org/10.1145/3664194. Just Accepted. [↗](https://doi.org/10.1145/3664194)
122. Wenhao Yu, Meng Jiang, Peter Clark, and Ashish Sabharwal. [IfQA: A dataset for open-domain question answering under counterfactual presuppositions](https://md.research.elamur.ai/free/en/papers/yu-2023-ifqa-dataset-open-domain-question/). In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL [https://openreview.net/](https://openreview.net/) forum?id=V49Jx2Lj04. [↗](https://openreview.net/)
123. Matej Zečević, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting. Causal parrots: Large language models may talk causality but are not causal. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=tv46tCzs83. [↗](https://openreview.net/forum?)
124. Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van Den Broeck. On the paradox of learning to reason from data. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI ’23, 2023. ISBN 978-1-956792-03-4. doi: 10.24963/ijcai.2023/375. URL… — [doi:10.24963/ijcai.2023/375](https://doi.org/10.24963/ijcai.2023/375) [find](https://scholar.google.com/scholar?q=Honghua+Zhang%2C+Liunian+Harold+Li%2C+Tao+Meng%2C+Kai%2DWei+Chang%2C+and+Guy+Van+Den+Broeck%2E+On+the+paradox+of+learning+to+reason+from+data%2E+In+Proceedings+of+the+Thirty%2DSecond+International+Joint+Conference+on)
125. Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, et al. Efficiently measuring the cognitive ability of llms: An adaptive testing perspective. arXiv preprint arXiv:2306.10512, 2023. [find](https://scholar.google.com/scholar?q=Yan+Zhuang%2C+Qi+Liu%2C+Yuting+Ning%2C+Weizhe+Huang%2C+Rui+Lv%2C+Zhenya+Huang%2C+Guanhao+Zhao%2C+Zheng+Zhang%2C+Qingyang+Mao%2C+Shijin+Wang%2C+et+al%2E+Efficiently+measuring+the+cognitive+ability+of+llms%3A+An+adaptive+testi)

*Updated 30 October 2026*