---
title: "Functional Benchmarks for Robust Evaluation of Reasoning"
description: "The authors propose \"functional\" benchmarks. 98 references, cited by 32 papers here. Among them: 3."
authors: ["Saurabh Srivastava", "Annarose M B", "Anto P", "Shashank Menon", "Ajay Sukumar", "Adwaith Samod T", "Alan Philipose", "Stevin Prince", "Sooraj Thomas"]
year: 2024
venue: "arXiv:2402.19450"
source: "https://arxiv.org/abs/2402.19450"
doi: "10.48550/arxiv.2402.19450"
arxiv: "2402.19450"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/srivastava-2024-functional-benchmarks-robust-evaluation-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/srivastava-2024-functional-benchmarks-robust-evaluation-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap

# Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap

**Authors:** [Saurabh Srivastava](https://md.research.elamur.ai/free/en/authors/saurabh-srivastava/), Annarose M B, Anto P, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince and Sooraj Thomas · **Year:** 2024 · **Venue:** arXiv:2402.19450 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.19450) · [PDF](https://arxiv.org/pdf/2402.19450) · [DOI 10.48550/arxiv.2402.19450](https://doi.org/10.48550/arxiv.2402.19450) · [arXiv 2402.19450](https://arxiv.org/abs/2402.19450) · [OpenAlex](https://openalex.org/W6948005944)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [32](https://md.research.elamur.ai/free/en/papers/srivastava-2024-functional-benchmarks-robust-evaluation-reasoning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Static reasoning benchmarks can overstate what LLMs can do. Their fixed question–answer pairs may leak into training data through paraphrases, synthetic data or fine-tuning, so a correct answer may come from recall rather than derivation. The authors propose "functional" benchmarks. Each test's solution logic is hand-written as code driven by seeded, typed random generators. Running the code produces fresh "snapshots": new problem instances that need the same reasoning steps and slot directly into existing evaluation harnesses. They apply this to part of MATH, producing MATH(). They then define the reasoning gap as the relative drop from static accuracy to accuracy across functional snapshots. Under simple few-shot prompting, strong closed and open-weight models show large gaps. Some problems, such as graduate-level ones, are answered correctly in static form but never on new instances. The framework offers a contamination-resistant way to separate memorization from genuine reasoning. It also suggests "gap-0" models as a training goal.

## Key points

- **Coverage:** 41.2% of MATH was functionalized. This subset covers every static problem solved by 4 closed-weight and 9 open-weight SOTA models, so full-MATH metrics can be reported for them.
- **Results:** reasoning gaps ranged from 58.35% to 80.31%. The authors note that chain-of-thought or similar prompting might shrink these gaps.
- **Released resources:** evaluation code and three MATH() snapshots (Oct, Nov and Dec 2023) are at github.com/consequentai/fneval. The functionalization code stays private, and new snapshots are promised every quarter.
- **Benchmark errors:** the analysis found flawed ground-truth answers in MATH, and some models matched those incorrect answers.
- **Future work:** functionalize all 5000 MATH problems, the 1000 GSM8K test items, HumanEval and MBPP, then study how prompting and tool use affect the gap.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works

## Cite

Cite as: Saurabh Srivastava, Annarose M B, Anto P, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince and Sooraj Thomas. 2024. Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap. arXiv:2402.19450.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{srivastava2024functional,
  title = {{Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap}},
  author = {Saurabh Srivastava, Annarose M B, Anto P, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince and Sooraj Thomas},
  year = {2024},
  doi = {10.48550/arxiv.2402.19450},
  eprint = {2402.19450},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.19450}
}
```

- [← previous Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review](https://md.research.elamur.ai/free/en/papers/ye-2024-there-yet-revealing-risks/)
- [next → Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding](https://md.research.elamur.ai/free/en/papers/li-2024-hunyuan-dit-powerful-multi/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

98 references, 58 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Long context prompting for claude 2.1, 2023. [find](https://scholar.google.com/scholar?q=Long+context+prompting+for+claude+2%2E1%2C+2023%2E)
2. 01.AI. Building the next generation of open-source and bilingual llms, 2023. [find](https://scholar.google.com/scholar?q=01%2EAI%2E+Building+the+next+generation+of+open%2Dsource+and+bilingual+llms%2C+2023%2E)
3. Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, Manzil Zaheer, Felix Yu, and Sanjiv Kumar. Rest meets react: Self-improvement for multi-step reasoning llm agent, 2023. [find](https://scholar.google.com/scholar?q=Renat+Aksitov%2C+Sobhan+Miryoosefi%2C+Zonglin+Li%2C+Daliang+Li%2C+Sheila+Babayan%2C+Kavya+Kopparapu%2C+Zachary+Fisher%2C+Ruiqi+Guo%2C+Sushant+Prakash%2C+Pranesh+Srinivasan%2C+Manzil+Zaheer%2C+Felix+Yu%2C+and+Sanjiv+Kumar%2E+Re)
4. Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, Manzil Zaheer, Felix Yu, and Sanjiv Kumar. Rest meets react: Self-improvement for multi-step reasoning llm agent, 2023. [find](https://scholar.google.com/scholar?q=Renat+Aksitov%2C+Sobhan+Miryoosefi%2C+Zonglin+Li%2C+Daliang+Li%2C+Sheila+Babayan%2C+Kavya+Kopparapu%2C+Zachary+Fisher%2C+Ruiqi+Guo%2C+Sushant+Prakash%2C+Pranesh+Srinivasan%2C+Manzil+Zaheer%2C+Felix+Yu%2C+and+Sanjiv+Kumar%2E+Re)
5. Anthropic. Introducing claude 2.1, 2023. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Introducing+claude+2%2E1%2C+2023%2E)
6. Daman Arora, Himanshu Gaurav Singh, and Mausam. [Have llms advanced enough? a challenging problem solving benchmark for large language models](https://md.research.elamur.ai/free/en/papers/arora-2023-have-llms-advanced-enough-challenging/), 2023.
7. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). CoRR, abs/2108.07732, 2021.
8. Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, and Jeremy Avigad. [Proofnet: Autoformalizing and formally proving undergraduate-level mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-proofnet-autoformalizing-formally-proving-undergraduate/), 2023.
9. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
10. Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Grégoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes, and Pascal Vincent. [Worldsense: A synthetic benchmark for grounded reasoning in large language models](https://md.research.elamur.ai/free/en/papers/benchekroun-2023-worldsense-synthetic-benchmark-grounded-reasoning/), 2023.
11. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. [PIQA: reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). CoRR, abs/1911.11641, 2019.
12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mi… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
13. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2023. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W%2E+Cohen%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2C+2023%2E)
14. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, 2023. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Scha%CC%88rli%2C+and+Denny+Zhou%2E+Teaching+large+language+models+to+self%2Ddebug%2C+2023%2E)
15. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024. [find](https://scholar.google.com/scholar?q=Zixiang+Chen%2C+Yihe+Deng%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+and+Quanquan+Gu%2E+Self%2Dplay+fine%2Dtuning+converts+weak+language+models+to+strong+language+models%2C+2024%2E)
16. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac : Question answering in context. CoRR, abs/1808.[07036, 2018](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/).
17. François Chollet. On the measure of intelligence, 2019. [find](https://scholar.google.com/scholar?q=Franc%CC%A7ois+Chollet%2E+On+the+measure+of+intelligence%2C+2019%2E)
18. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chap…
19. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). CoRR, abs/1803.05457, 2018.
20. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168, 2021.
21. Marius-Constantin Dinu, Claudiu Leoveanu-Condrei, Markus Holzleitner, Werner Zellinger, and Sepp Hochreiter. Symbolicai: A framework for logic-based approaches combining generative models and solvers, 2024. [find](https://scholar.google.com/scholar?q=Marius%2DConstantin+Dinu%2C+Claudiu+Leoveanu%2DCondrei%2C+Markus+Holzleitner%2C+Werner+Zellinger%2C+and+Sepp+Hochreiter%2E+Symbolicai%3A+A+framework+for+logic%2Dbased+approaches+combining+generative+models+and+solvers%2C)
22. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). CoRR, abs/1903.00161, 2019.
23. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina N… — [The Capacity for Moral Self-Correction in Large Language Models](https://md.research.elamur.ai/free/en/papers/ganguli-2023-capacity-moral-self-correction-large/)
24. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models, 2023. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+Pal%3A+Program%2Daided+language+models%2C+2023%2E)
25. Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. [Roscoe: A suite of metrics for scoring step-by-step reasoning](https://md.research.elamur.ai/free/en/papers/golovneva-2022-roscoe-suite-metrics-scoring-step/), 2023.
26. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. [Tora: A tool-integrated reasoning agent for mathematical problem solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/), 2024.
27. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. [Cruxeval: A benchmark for code reasoning, understanding and execution](https://md.research.elamur.ai/free/en/papers/gu-2024-cruxeval-benchmark-code-reasoning-understanding/), 2024.
28. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). CoRR, abs/2009.03300, 2020.
29. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). CoRR, abs/2103.03874, 2021.
30. Namgyu Ho, Laura Schmid, and Se-Young Yun. [Large language models are reasoning teachers](https://md.research.elamur.ai/free/en/papers/ho-2023-large-language-models-are-reasoning/), 2023.
31. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/), 2023.
32. Shima Imani, Liang Du, and Harsh Shrivastava. [Mathprompter: Mathematical reasoning using large language models](https://md.research.elamur.ai/free/en/papers/imani-2023-mathprompter-mathematical-reasoning-large-language/), 2023.
33. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
34. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/), 2023.
35. Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. [The impact of reasoning step length on large language models](https://md.research.elamur.ai/free/en/papers/jin-2024-impact-reasoning-step-length-large/), 2024.
36. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
37. Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. [xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval](https://md.research.elamur.ai/free/en/papers/khan-2023-xcodeeval-large-scale-multilingual-multitask/), 2023.
38. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questi… — [Natural Questions: A Benchmark for Question Answering Research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
39. Mosh Levy, Alon Jacoby, and Yoav Goldberg. [Same task, more tokens: the impact of input length on the reasoning performance of large language models](https://md.research.elamur.ai/free/en/papers/levy-2024-same-task-more-tokens-impact/), 2024.
40. Changmao Li and Jeffrey Flanigan. [Task contamination: Language models may not be few-shot anymore](https://md.research.elamur.ai/free/en/papers/li-2023-task-contamination-language-models-may/), 2023.
41. Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator, 2023. [find](https://scholar.google.com/scholar?q=Chengshu+Li%2C+Jacky+Liang%2C+Andy+Zeng%2C+Xinyun+Chen%2C+Karol+Hausman%2C+Dorsa+Sadigh%2C+Sergey+Levine%2C+Li+Fei%2DFei%2C+Fei+Xia%2C+and+Brian+Ichter%2E+Chain+of+code%3A+Reasoning+with+a+language+model%2Daugmented+code+emula)
42. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/), 2023.
43. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
44. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. — [Let's Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
45. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. [Llm+p: Empowering large language models with optimal planning proficiency](https://md.research.elamur.ai/free/en/papers/liu-2023-llm-p-empowering-large-language/), 2023.
46. Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. [Chain of hindsight aligns language models with feedback](https://md.research.elamur.ai/free/en/papers/liu-2023-chain-hindsight-aligns-language-models/), 2023.
47. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2C+2023%2E)
48. Zijie Meng, Yan Zhang, Zhaopeng Feng, Yang Feng, Gaoang Wang, Joey Tianyi Zhou, Jian Wu, and Zuozhu Liu. Divide and conquer for large language models reasoning, 2024. [find](https://scholar.google.com/scholar?q=Zijie+Meng%2C+Yan+Zhang%2C+Zhaopeng+Feng%2C+Yang+Feng%2C+Gaoang+Wang%2C+Joey+Tianyi+Zhou%2C+Jian+Wu%2C+and+Zuozhu+Liu%2E+Divide+and+conquer+for+large+language+models+reasoning%2C+2024%2E)
49. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. [Gaia: a benchmark for general ai assistants](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/), 2023.
50. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. CoRR, abs/1809.02789, 2018. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+Can+a+suit+of+armor+conduct+electricity%3F+A+new+dataset+for+open+book+question+answering%2E+CoRR%2C+abs%2F1809%2E02789%2C+2018%2E)
51. Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. [Orca 2: Teaching small language models how to reason](https://md.research.elamur.ai/free/en/papers/mitra-2023-orca-2-teaching-small-language/), 2023.
52. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/), 2023.
53. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. [Scalable extraction of training data from (production) language models](https://md.research.elamur.ai/free/en/papers/nasr-2023-scalable-extraction-training-data-production/), 2023.
54. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
55. Aaron Parisi, Yao Zhao, and Noah Fiedel. [Talm: Tool augmented language models](https://md.research.elamur.ai/free/en/papers/parisi-2022-talm-tool-augmented-language-models/), 2022.
56. Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah D. Goodman. [Certified deductive reasoning with language models](https://md.research.elamur.ai/free/en/papers/poesia-2023-certified-deductive-reasoning-language-models/), 2023.
57. Michael Poli, Jue Wang, Stefano Massaroli, Jeffrey Quesnelle, Ryan Carlow, Eric Nguyen, and Armin Thomas. StripedHyena: Moving Beyond Transformers with Hybrid Signal Processing Models, 12 2023. [find](https://scholar.google.com/scholar?q=Michael+Poli%2C+Jue+Wang%2C+Stefano+Massaroli%2C+Jeffrey+Quesnelle%2C+Ryan+Carlow%2C+Eric+Nguyen%2C+and+Armin+Thomas%2E+StripedHyena%3A+Moving+Beyond+Transformers+with+Hybrid+Signal+Processing+Models%2C+12+2023%2E)
58. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q and a benchmark, 2023. [find](https://scholar.google.com/scholar?q=David+Rein%2C+Betty+Li+Hou%2C+Asa+Cooper+Stickland%2C+Jackson+Petty%2C+Richard+Yuanzhe+Pang%2C+Julien+Dirani%2C+Julian+Michael%2C+and+Samuel+R%2E+Bowman%2E+Gpqa%3A+A+graduate%2Dlevel+google%2Dproof+q+and+a+benchmark%2C+2023%2E)
59. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WINOGRANDE: an adversarial winograd schema challenge at scale. CoRR, abs/1907.10641, 2019. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+WINOGRANDE%3A+an+adversarial+winograd+schema+challenge+at+scale%2E+CoRR%2C+abs%2F1907%2E10641%2C+2019%2E)
60. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). CoRR, abs/1904.09728, 2019.
61. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. [Arb: Advanced reasoning benchmark for large language models](https://md.research.elamur.ai/free/en/papers/sawada-2023-arb-advanced-reasoning-benchmark-large/), 2023.
62. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. [Are emergent abilities of large language models a mirage?](https://md.research.elamur.ai/free/en/papers/schaeffer-2023-emergent-abilities-large-language-models/), 2023.
63. Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction, 2023. [find](https://scholar.google.com/scholar?q=Pratyusha+Sharma%2C+Jordan+T%2E+Ash%2C+and+Dipendra+Misra%2E+The+truth+is+in+there%3A+Improving+reasoning+in+language+models+with+layer%2Dselective+rank+reduction%2C+2023%2E)
64. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. [The curse of recursion: Training on generated data makes models forget](https://md.research.elamur.ai/free/en/papers/shumailov-2023-curse-recursion-training-generated-data/), 2023.
65. Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Ad-lam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Sim… — [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/)
66. Charlie Snell, Dan Klein, and Ruiqi Zhong. [Learning by distilling context](https://md.research.elamur.ai/free/en/papers/snell-2022-learning-distilling-context/), 2022.
67. Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. Gpt-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems, 2023. [find](https://scholar.google.com/scholar?q=Kaya+Stechly%2C+Matthew+Marquez%2C+and+Subbarao+Kambhampati%2E+Gpt%2D4+doesn%E2%80%99t+know+it%E2%80%99s+wrong%3A+An+analysis+of+iterative+prompting+for+reasoning+problems%2C+2023%2E)
68. Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, Yue Wu, Wenhai Wang, Junsong Chen, Zhangyue Yin, Xiaozhe Ren, Jie Fu, Junxian He, Wu Yuan, Qi Liu, Xihui Liu, Yu Li, Hao Dong, Yu Cheng, Ming Zhang, Pheng Ann Heng, Jife… [find](https://scholar.google.com/scholar?q=Jiankai+Sun%2C+Chuanyang+Zheng%2C+Enze+Xie%2C+Zhengying+Liu%2C+Ruihang+Chu%2C+Jianing+Qiu%2C+Jiaqi+Xu%2C+Mingyu+Ding%2C+Hongyang+Li%2C+Mengzhe+Geng%2C+Yue+Wu%2C+Wenhai+Wang%2C+Junsong+Chen%2C+Zhangyue+Yin%2C+Xiaozhe+Ren%2C+Jie+Fu%2C)
69. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/), 2022.
70. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). CoRR, abs/1811.00937, 2018.
71. Gemini Team. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/), 2023.
72. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
73. Gladys Tyen, Hassan Mansoor, Victor Cărbune, Peter Chen, and Tony Mak. Llms cannot find reasoning errors, but can correct them!, 2024. [find](https://scholar.google.com/scholar?q=Gladys+Tyen%2C+Hassan+Mansoor%2C+Victor+Ca%CC%86rbune%2C+Peter+Chen%2C+and+Tony+Mak%2E+Llms+cannot+find+reasoning+errors%2C+but+can+correct+them%21%2C+2024%2E)
74. Petar Veličković, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. The clrs algorithmic reasoning benchmark, 2022. [find](https://scholar.google.com/scholar?q=Petar+Velic%CC%8Ckovic%CC%81%2C+Adria%CC%80+Puigdome%CC%80nech+Badia%2C+David+Budden%2C+Razvan+Pascanu%2C+Andrea+Banino%2C+Misha+Dashevskiy%2C+Raia+Hadsell%2C+and+Charles+Blundell%2E+The+clrs+algorithmic+reasoning+benchmark%2C+2022%2E)
75. Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. [Math-shepherd: Verify and reinforce llms step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/), 2024.
76. Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan, Wenhu Chen, and William Yang Wang. Understanding the reasoning ability of language models from the perspective of reasoning paths aggregation, 2024. [find](https://scholar.google.com/scholar?q=Xinyi+Wang%2C+Alfonso+Amayuelas%2C+Kexun+Zhang%2C+Liangming+Pan%2C+Wenhu+Chen%2C+and+William+Yang+Wang%2E+Understanding+the+reasoning+ability+of+language+models+from+the+perspective+of+reasoning+paths+aggregation)
77. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/), 2023.
78. Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting, 2024. [find](https://scholar.google.com/scholar?q=Xuezhi+Wang+and+Denny+Zhou%2E+Chain%2Dof%2Dthought+reasoning+without+prompting%2C+2024%2E)
79. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2C+2023%2E)
80. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.[11903, 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/).
81. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks, 2023. [find](https://scholar.google.com/scholar?q=Zhaofeng+Wu%2C+Linlu+Qiu%2C+Alexis+Ross%2C+Ekin+Akyu%CC%88rek%2C+Boyuan+Chen%2C+Bailin+Wang%2C+Najoung+Kim%2C+Jacob+Andreas%2C+and+Yoon+Kim%2E+Reasoning+or+reciting%3F+exploring+the+capabilities+and+limitations+of+language+mo)
82. Steve Yadlowsky, Lyric Doshi, and Nilesh Tripuraneni. Pretraining data mixtures enable narrow model selection capabilities in transformer models, 2023. [find](https://scholar.google.com/scholar?q=Steve+Yadlowsky%2C+Lyric+Doshi%2C+and+Nilesh+Tripuraneni%2E+Pretraining+data+mixtures+enable+narrow+model+selection+capabilities+in+transformer+models%2C+2023%2E)
83. Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi R. Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Yiquan Wang, Heng Ji, and Chengxiang Zhai. If llm is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents, 2024. [find](https://scholar.google.com/scholar?q=Ke+Yang%2C+Jiateng+Liu%2C+John+Wu%2C+Chaoqi+Yang%2C+Yi+R%2E+Fung%2C+Sha+Li%2C+Zixuan+Huang%2C+Xu+Cao%2C+Xingyao+Wang%2C+Yiquan+Wang%2C+Heng+Ji%2C+and+Chengxiang+Zhai%2E+If+llm+is+the+wizard%2C+then+code+is+the+wand%3A+A+survey+on+)
84. Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning?, 2024. [find](https://scholar.google.com/scholar?q=Sohee+Yang%2C+Elena+Gribovskaya%2C+Nora+Kassner%2C+Mor+Geva%2C+and+Sebastian+Riedel%2E+Do+large+language+models+latently+perform+multi%2Dhop+reasoning%3F%2C+2024%2E)
85. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2C+2023%2E)
86. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/), 2023.
87. Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F. Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking llms via uncertainty quantification, 2024. [find](https://scholar.google.com/scholar?q=Fanghua+Ye%2C+Mingming+Yang%2C+Jianhui+Pang%2C+Longyue+Wang%2C+Derek+F%2E+Wong%2C+Emine+Yilmaz%2C+Shuming+Shi%2C+and+Zhaopeng+Tu%2E+Benchmarking+llms+via+uncertainty+quantification%2C+2024%2E)
88. Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (stop): Recursively self-improving code generation, 2023. [find](https://scholar.google.com/scholar?q=Eric+Zelikman%2C+Eliana+Lorch%2C+Lester+Mackey%2C+and+Adam+Tauman+Kalai%2E+Self%2Dtaught+optimizer+%28stop%29%3A+Recursively+self%2Dimproving+code+generation%2C+2023%2E)
89. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? CoRR, abs/1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+CoRR%2C+abs%2F1905%2E07830%2C+2019%2E)
90. Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Teaching large language models to refuse unknown questions, 2023. [find](https://scholar.google.com/scholar?q=Hanning+Zhang%2C+Shizhe+Diao%2C+Yong+Lin%2C+Yi+R%2E+Fung%2C+Qing+Lian%2C+Xingyao+Wang%2C+Yangyi+Chen%2C+Heng+Ji%2C+and+Tong+Zhang%2E+R%2Dtuning%3A+Teaching+large+language+models+to+refuse+unknown+questions%2C+2023%2E)
91. Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. [Planning with large language models for code generation](https://md.research.elamur.ai/free/en/papers/zhang-2023-planning-large-language-models-code/), 2023.
92. Yanfang Zhang, Yiliu Sun, Yibing Zhan, Dapeng Tao, Dacheng Tao, and Chen Gong. Large language models as an indirect reasoner: Contrapositive and contradiction for automated reasoning, 2024. [find](https://scholar.google.com/scholar?q=Yanfang+Zhang%2C+Yiliu+Sun%2C+Yibing+Zhan%2C+Dapeng+Tao%2C+Dacheng+Tao%2C+and+Chen+Gong%2E+Large+language+models+as+an+indirect+reasoner%3A+Contrapositive+and+contradiction+for+automated+reasoning%2C+2024%2E)
93. Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. [Progressive-hint prompting improves reasoning in large language models](https://md.research.elamur.ai/free/en/papers/zheng-2023-progressive-hint-prompting-improves-reasoning/), 2023.
94. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/), 2023.
95. Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. Ar-lsat: Investigating analytical reasoning of text, 2021. [find](https://scholar.google.com/scholar?q=Wanjun+Zhong%2C+Siyuan+Wang%2C+Duyu+Tang%2C+Zenan+Xu%2C+Daya+Guo%2C+Jiahai+Wang%2C+Jian+Yin%2C+Ming+Zhou%2C+and+Nan+Duan%2E+Ar%2Dlsat%3A+Investigating+analytical+reasoning+of+text%2C+2021%2E)
96. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/), 2023.
97. Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. [Teaching algorithmic reasoning via in-context learning](https://md.research.elamur.ai/free/en/papers/zhou-2022-teaching-algorithmic-reasoning-context-learning/), 2022.
98. Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V. Le, Ed H. Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. Self-discover: Large language models self-compose reasoning structures, 2024. [find](https://scholar.google.com/scholar?q=Pei+Zhou%2C+Jay+Pujara%2C+Xiang+Ren%2C+Xinyun+Chen%2C+Heng%2DTze+Cheng%2C+Quoc+V%2E+Le%2C+Ed+H%2E+Chi%2C+Denny+Zhou%2C+Swaroop+Mishra%2C+and+Huaixiu+Steven+Zheng%2E+Self%2Ddiscover%3A+Large+language+models+self%2Dcompose+reasoning+s)

*Updated 3 November 2026*