---
title: "ARB: Advanced Reasoning Benchmark for Large Language Models"
description: "ARB (Advanced Reasoning Benchmark) offers a harder test built from graduate-level and… 72 references, cited by 52 papers here."
authors: ["Tomohiro Sawada", "Daniel Paleka", "Alexander Havrilla", "Pranav Tadepalli", "Paula Vidas", "Alexander Kranias", "John J. Nay", "Kshitij Gupta", "Aran Komatsuzaki"]
year: 2023
venue: "arXiv:2307.13692"
source: "https://arxiv.org/abs/2307.13692"
doi: "10.48550/arxiv.2307.13692"
arxiv: "2307.13692"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sawada-2023-arb-advanced-reasoning-benchmark-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sawada-2023-arb-advanced-reasoning-benchmark-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / ARB: Advanced Reasoning Benchmark for Large Language Models

# ARB: Advanced Reasoning Benchmark for Large Language Models

**Authors:** Tomohiro Sawada, [Daniel Paleka](https://md.research.elamur.ai/free/en/authors/daniel-paleka/), Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, [John J. Nay](https://md.research.elamur.ai/free/en/authors/john-j-nay/), Kshitij Gupta and [Aran Komatsuzaki](https://md.research.elamur.ai/free/en/authors/aran-komatsuzaki/) · **Year:** 2023 · **Venue:** arXiv:2307.13692 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.13692) · [PDF](https://arxiv.org/pdf/2307.13692) · [DOI 10.48550/arxiv.2307.13692](https://doi.org/10.48550/arxiv.2307.13692) · [arXiv 2307.13692](https://arxiv.org/abs/2307.13692) · [OpenAlex](https://openalex.org/W4385292963)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/sawada-2023-arb-advanced-reasoning-benchmark-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Many popular LLM benchmarks are saturating. Models like GPT-4 now score highly on them even though they still lack expert-level competence. ARB (Advanced Reasoning Benchmark) offers a harder test built from graduate-level and professional problems in mathematics, physics, chemistry, biology and law. The sources were written for specialists: Harvard PhD comprehensive exams in mathematics, university and contest problem books, PhD qualifying question collections in physics, McGraw-Hill MCAT practice tests and an older bar exam practice set. Several were chosen because they rarely appear online, which lowers the risk that models saw them during training. Questions are multiple choice, short answer or open response. Recent models, including GPT-4 and Claude, solve well under half of the more demanding quantitative tasks. Grading symbolic answers and multi-step reasoning is difficult, so the authors also propose a rubric method. GPT-4 drafts a scoring rubric from a reference solution and then grades solutions against it. Human checks show these scores align fairly well with manual grading. ARB thus provides a tougher target for tracking progress and a step toward cheaper automated evaluation.

## Key points
- The mathematics portion contains 52 numerical, 34 symbolic and 19 proof-like problems. The physics portion mixes numerical and symbolic items, some of which include figures.
- High-school contest math was left out on purpose because benchmarks like MATH already cover it. The Putnam and Brayman sources come with official solutions, which are included in the dataset.
- Human raters scored the GPT-4 rubrics on a 1–5 scale. Coverage of key steps was high (about 3.9–4.4), but point allocation across steps was weaker. GPT-3.5-turbo tended to concentrate points on one or two steps.
- On a 10-point scale, Pearson correlation between human and GPT-4 rubric scores was 0.91 for symbolic physics, 0.78 for symbolic math and 0.82 for proof-like problems. The main failure was GPT-4 giving partial credit to attempts that fell entirely outside the rubric.
- Rubric grading suggests GPT-4 recalls physics concepts well but makes mistakes in algebraic manipulation. It is weakest at planning solutions for proof-style math.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 152 works · 27,763 citations
- [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) — University, Zurich, Switzerland · 84 works · 9,181 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 65 works · 14,425 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta and Aran Komatsuzaki. 2023. ARB: Advanced Reasoning Benchmark for Large Language Models. arXiv:2307.13692.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sawada2023arb,
  title = {{ARB: Advanced Reasoning Benchmark for Large Language Models}},
  author = {Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta and Aran Komatsuzaki},
  year = {2023},
  doi = {10.48550/arxiv.2307.13692},
  eprint = {2307.13692},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.13692}
}
```

- [← previous What Algorithms can Transformers Learn? A Study in Length Generalization](https://md.research.elamur.ai/free/en/papers/zhou-2023-algorithms-transformers-learn-study-length/)
- [next → CREPE: Open-Domain Question Answering with False Presuppositions](https://md.research.elamur.ai/free/en/papers/yu-2023-crepe-open-domain-question-answering/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

72 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300–22312, 2022. [find](https://scholar.google.com/scholar?q=Ibrahim+M+Alabdulmohsin%2C+Behnam+Neyshabur%2C+and+Xiaohua+Zhai%2E+Revisiting+neural+scaling+laws+in+language+and+vision%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A22300%E2%80%9322312%2C+2022%2E)
2. Daman Arora, Himanshu Gaurav Singh, and Mausam. [Have LLMs advanced enough? A challenging problem solving benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/arora-2023-have-llms-advanced-enough-challenging/), 2023.
3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
4. Barbri. Barbri Practice Questions: Multistate Testing Practice Questions. Thomson/Bar/Bri, 2007. ISBN 9780314174017. [find](https://scholar.google.com/scholar?q=Barbri%2E+Barbri+Practice+Questions%3A+Multistate+Testing+Practice+Questions%2E+Thomson%2FBar%2FBri%2C+2007%2E+ISBN+9780314174017%2E)
5. Michael Bommarito II and Daniel Martin Katz. GPT takes the bar exam. arXiv preprint arXiv:2212.14402, 2022. [find](https://scholar.google.com/scholar?q=Michael+Bommarito+II+and+Daniel+Martin+Katz%2E+GPT+takes+the+bar+exam%2E+arXiv+preprint+arXiv%3A2212%2E14402%2C+2022%2E)
6. Samuel R. Bowman. The dangers of underclaiming: Reasons for caution when reporting how NLP systems fail, 2021. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2E+The+dangers+of+underclaiming%3A+Reasons+for+caution+when+reporting+how+NLP+systems+fail%2C+2021%2E)
7. Volodymyr Brayman and A. G. Kukush. Undergraduate Mathematics Competitions (1995-2016): Taras Shevchenko National University of Kyiv. [find](https://scholar.google.com/scholar?q=Volodymyr+Brayman+and+A%2E+G%2E+Kukush%2E+Undergraduate+Mathematics+Competitions+%281995%2D2016%29%3A+Taras+Shevchenko+National+University+of+Kyiv%2E)
8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
9. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
10. Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws, 2023. [find](https://scholar.google.com/scholar?q=Ethan+Caballero%2C+Kshitij+Gupta%2C+Irina+Rish%2C+and+David+Krueger%2E+Broken+neural+scaling+laws%2C+2023%2E)
11. Candice McCloskey Campbell, Shaun Murphree, Jennifer M. Warner, Amy B. Wachholz, Kathy A. Zahler, and George J. Hademenos. McGraw-Hill Education 3 MCAT Practice Tests, Third Edition. McGraw-Hill Education, Jan 2017. ISBN 1259859622. [find](https://scholar.google.com/scholar?q=Candice+McCloskey+Campbell%2C+Shaun+Murphree%2C+Jennifer+M%2E+Warner%2C+Amy+B%2E+Wachholz%2C+Kathy+A%2E+Zahler%2C+and+George+J%2E+Hademenos%2E+McGraw%2DHill+Education+3+MCAT+Practice+Tests%2C+Third+Edition%2E+McGraw%2DHill+Educa)
12. Bryan Caplan. GPT retakes my midterm and gets an A, 2023. URL [https://betonit.substack.com/](https://betonit.substack.com/) p/gpt-retakes-my-midterm-and-gets-an. [↗](https://betonit.substack.com/)
13. Cheng-Han Chiang and Hung-yi Lee. Can Large Language Models be an alternative to human evaluations? arXiv e-prints, art. arXiv:2305.01937, may 2023. doi: 10.48550/arXiv.2305.01937. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+Can+Large+Language+Models+be+an+alternative+to+human+evaluations%3F+arXiv+e%2Dprints%2C+art%2E+arXiv%3A2305%2E01937%2C+may+2023%2E+doi%3A+10%2E48550%2FarXiv%2E2305%2E01937%2E)
14. François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. [find](https://scholar.google.com/scholar?q=Fran%C3%A7ois+Chollet%2E+On+the+measure+of+intelligence%2E+arXiv+preprint+arXiv%3A1911%2E01547%2C+2019%2E)
15. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
16. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168)
17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
18. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. GPTScore: Evaluate as you desire. arXiv e-prints, art. arXiv:2302.04166, feb 2023. doi: 10.48550/arXiv.2302.04166. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+GPTScore%3A+Evaluate+as+you+desire%2E+arXiv+e%2Dprints%2C+art%2E+arXiv%3A2302%2E04166%2C+feb+2023%2E+doi%3A+10%2E48550%2FarXiv%2E2302%2E04166%2E)
19. Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance, 2023. — [Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance](https://md.research.elamur.ai/free/en/papers/fu-2023-chain-thought-hub-continuous-effort/)
20. Răzvan Gelca and Titu Andreescu. Putnam and beyond. Springer, 2017. [find](https://scholar.google.com/scholar?q=Ra%CC%86zvan+Gelca+and+Titu+Andreescu%2E+Putnam+and+beyond%2E+Springer%2C+2017%2E)
21. Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. Large language models are not abstract reasoners, 2023. [find](https://scholar.google.com/scholar?q=Ga%C3%ABl+Gendron%2C+Qiming+Bao%2C+Michael+Witbrock%2C+and+Gillian+Dobbie%2E+Large+language+models+are+not+abstract+reasoners%2C+2023%2E)
22. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies, 2021. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2C+2021%2E)
23. Zoubin Ghahramani. Introducing PaLM 2, 2023. URL [https://blog.google/technology/ai/](https://blog.google/technology/ai/) google-palm-2-ai-large-language-model. [↗](https://blog.google/technology/ai/)
24. Department of Mathematics Harvard University. Qualifying examination for fall 2021, Aug 2021. URL [https://www.math.harvard.edu/media/quals-F21_with_solutions.pdf](https://www.math.harvard.edu/media/quals-F21_with_solutions.pdf). [↗](https://www.math.harvard.edu/media/quals-F21_with_solutions.pdf)
25. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/), 2020. URL https://arxiv.org/abs/2009. 03300. [↗](https://arxiv.org/abs/2009)
26. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). CoRR, abs/2103.03874, 2021. URL https://arxiv.org/abs/2103.03874. [↗](https://arxiv.org/abs/2103.03874)
27. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
28. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/), 2020.
29. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. GPT-4 passes the bar exam. Available at SSRN 4389233, 2023. [find](https://scholar.google.com/scholar?q=Daniel+Martin+Katz%2C+Michael+James+Bommarito%2C+Shang+Gao%2C+and+Pablo+Arredondo%2E+GPT%2D4+passes+the+bar+exam%2E+Available+at+SSRN+4389233%2C+2023%2E)
30. Tom Kocmi and Christian Federmann. Large language models are state-of-the-art evaluators of translation quality. arXiv e-prints, art. arXiv:2302.14520, feb 2023. doi: 10.48550/arXiv.2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+arXiv+e%2Dprints%2C+art%2E+arXiv%3A2302%2E14520%2C+feb+2023%2E+doi%3A+10%2E48550%2FarXiv%2E2302%2E14520%2E)
31. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/), 2023.
32. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152–1157, San Diego,… [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+Aida+Amini%2C+Nate+Kushman%2C+and+Hannaneh+Hajishirzi%2E+MAWPS%3A+A+math+word+problem+repository%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+)
33. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. [Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models](https://md.research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/). PLOS Digital Health,…
34. Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. [A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets](https://md.research.elamur.ai/free/en/papers/laskar-2023-systematic-study-comprehensive-evaluation-chatgpt/), 2023.
35. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/), 2022. URL https://arxiv…
36. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making large language models better reasoners with step-aware verifier, 2023. [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Zeqi+Lin%2C+Shizhuo+Zhang%2C+Qiang+Fu%2C+Bei+Chen%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+Making+large+language+models+better+reasoners+with+step%2Daware+verifier%2C+2023%2E)
37. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
38. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. — [Let's Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
39. Swee Cheng Lim, Choy Heng Lai, and Leong Chuan Kwek. Problems and solutions on optics. World Scientific, 2019. [find](https://scholar.google.com/scholar?q=Swee+Cheng+Lim%2C+Choy+Heng+Lai%2C+and+Leong+Chuan+Kwek%2E+Problems+and+solutions+on+optics%2E+World+Scientific%2C+2019%2E)
40. Yung-kuo Lim. Problems and solutions on thermodynamics and Statistical Mechanics. World Scientific, 1996. [find](https://scholar.google.com/scholar?q=Yung%2Dkuo+Lim%2E+Problems+and+solutions+on+thermodynamics+and+Statistical+Mechanics%2E+World+Scientific%2C+1996%2E)
41. Yung-kuo Lim. Problems and solutions in quantum mechanics: Major, American universities ph. D. qualifying questions and, solutions. World Scientific, 1998. [find](https://scholar.google.com/scholar?q=Yung%2Dkuo+Lim%2E+Problems+and+solutions+in+quantum+mechanics%3A+Major%2C+American+universities+ph%2E+D%2E+qualifying+questions+and%2C+solutions%2E+World+Scientific%2C+1998%2E)
42. Yung-kuo Lim. Problems and solutions on electromagnetism. World Scientific Pub. Co, 2007. [find](https://scholar.google.com/scholar?q=Yung%2Dkuo+Lim%2E+Problems+and+solutions+on+electromagnetism%2E+World+Scientific+Pub%2E+Co%2C+2007%2E)
43. Yung-kuo Lim and Yuan-qi Qiang. Problems and solutions on Mechanics. World Scientif., 2001. [find](https://scholar.google.com/scholar?q=Yung%2Dkuo+Lim+and+Yuan%2Dqi+Qiang%2E+Problems+and+solutions+on+Mechanics%2E+World+Scientif%2E%2C+2001%2E)
44. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning, to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, Vancouver, Can… [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+Program+induction+by+rationale+generation%3A+Learning%2C+to+solve+and+explain+algebraic+word+problems%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+)
45. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. [G-eval: NLG evaluation using GPT-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv e-prints, art. arXiv:2303.16634, mar 2023. doi: 10.48550/ arXiv.2303.16634.
46. Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fab… — [doi:10.7717/peerj-cs.103](https://doi.org/10.7717/peerj-cs.103) [find](https://scholar.google.com/scholar?q=Aaron+Meurer%2C+Christopher+P%2E+Smith%2C+Mateusz+Paprocki%2C+Ondr%CC%8Cej+C%CC%8Cert%C3%ADk%2C+Sergey+B%2E+Kirpichev%2C+Matthew+Rocklin%2C+AMiT+Kumar%2C+Sergiu+Ivanov%2C+Jason+K%2E+Moore%2C+Sartaj+Singh%2C+Thilina+Rathnayake%2C+Sean+Vig%2C+Bria)
47. Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984, Online, jul 2020. Association for Computational Linguistics. do… [find](https://scholar.google.com/scholar?q=Shen%2Dyun+Miao%2C+Chao%2DChun+Liang%2C+and+Keh%2DYih+Su%2E+A+diverse+corpus+for+evaluating+and+developing+English+math+word+problem+solvers%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Compu)
48. Shakked Noy and Whitney Zhang. Experimental evidence on the productivity effects of generative artificial intelligence. Available at SSRN 4375283, 2023. [find](https://scholar.google.com/scholar?q=Shakked+Noy+and+Whitney+Zhang%2E+Experimental+evidence+on+the+productivity+effects+of+generative+artificial+intelligence%2E+Available+at+SSRN+4375283%2C+2023%2E)
49. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/), 2021.
50. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
51. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/), 2021.
52. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. [Discovering language model behaviors with model-written evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/). arXiv preprint arXiv:2212.09251, 2022.
53. Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. arXiv e-prints, art. arXiv:2204.01075, April 2022. doi: 10.48550/arXiv. 2204.01075. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) [find](https://scholar.google.com/scholar?q=Mahima+Pushkarna%2C+Andrew+Zaldivar%2C+and+Oddur+Kjartansson%2E+Data+Cards%3A+Purposeful+and+Transparent+Dataset+Documentation+for+Responsible+AI%2E+arXiv+e%2Dprints%2C+art%2E+arXiv%3A2204%2E01075%2C+April+2022%2E+doi%3A+10%2E48)
54. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
55. Subhro Roy and Dan Roth. [Solving general arithmetic word problems](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/), 2016.
56. Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu, and Lakshmivihari Mareedu. [An independent evaluation of ChatGPT on mathematical word problems (MWP)](https://md.research.elamur.ai/free/en/papers/shakarian-2023-independent-evaluation-chatgpt-mathematical-word/). arXiv preprint arXiv:2302.13814, 2023.
57. Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever Hans or Neural Theory of Mind? Stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763, 2023. [find](https://scholar.google.com/scholar?q=Natalie+Shapira%2C+Mosh+Levy%2C+Seyed+Hossein+Alavi%2C+Xuhui+Zhou%2C+Yejin+Choi%2C+Yoav+Goldberg%2C+Maarten+Sap%2C+and+Vered+Shwartz%2E+Clever+Hans+or+Neural+Theory+of+Mind%3F+Stress+testing+social+reasoning+in+large+l)
58. Paulo N de Souza and Jorge N. Silva. Berkeley problems in Mathematics. Springer New York, 2008. [find](https://scholar.google.com/scholar?q=Paulo+N+de+Souza+and+Jorge+N%2E+Silva%2E+Berkeley+problems+in+Mathematics%2E+Springer+New+York%2C+2008%2E)
59. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615, 2022.…
60. Jacob Steinhardt. AI forecasting: One year in, 2022. URL [https://bounded-regret.ghost.io/](https://bounded-regret.ghost.io/) ai-forecasting-one-year-in/. [↗](https://bounded-regret.ghost.io/)
61. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). CoRR, abs/1811.00937, 2018. URL [http://arxiv.org/](http://arxiv.org/) abs/1811.00937. [↗](http://arxiv.org/)
62. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change), 2023. [find](https://scholar.google.com/scholar?q=Karthik+Valmeekam%2C+Alberto+Olmo%2C+Sarath+Sreedharan%2C+and+Subbarao+Kambhampati%2E+Large+language+models+still+can%E2%80%99t+plan+%28a+benchmark+for+LLMs+on+planning+and+reasoning+about+change%29%2C+2023%2E)
63. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. [SuperGLUE: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/), 2019a.
64. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). 2019b. In the Proceedings of ICLR.
65. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/), 2023.
66. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/), 2022a.
67. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models, 2022b. [URL https://arxiv.org/abs/2201.11903](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). [↗](https://arxiv.org/abs/2201.11903)
68. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/), 2018.
69. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2C+2023%2E)
70. Ping Yu, Tianlu Wang, Olga Golovneva, Badr Alkhamissy, Gargi Ghosh, Mona Diab, and Asli Celikyilmaz. ALERT: Adapting language models to reasoning tasks, 2022. [find](https://scholar.google.com/scholar?q=Ping+Yu%2C+Tianlu+Wang%2C+Olga+Golovneva%2C+Badr+Alkhamissy%2C+Gargi+Ghosh%2C+Mona+Diab%2C+and+Asli+Celikyilmaz%2E+ALERT%3A+Adapting+language+models+to+reasoning+tasks%2C+2022%2E)
71. Sarah Zhang, Reece Shuttleworth, Derek Austin, Yann Hicke, Leonard Tang, Sathwik Karnik, Darnell Granberry, and Iddo Drori. A dataset and benchmark for automatically answering and generating machine learning final exams. arXiv preprint arXiv:2206.05442, 2022. — [arXiv:2206.05442](https://arxiv.org/abs/2206.05442) [find](https://scholar.google.com/scholar?q=Sarah+Zhang%2C+Reece+Shuttleworth%2C+Derek+Austin%2C+Yann+Hicke%2C+Leonard+Tang%2C+Sathwik+Karnik%2C+Darnell+Granberry%2C+and+Iddo+Drori%2E+A+dataset+and+benchmark+for+automatically+answering+and+generating+machine+l)
72. Hefei Zhongguo-Kexue-Jishu-Daxue. Major American universities Ph. D. qualifying questions and solutions. 5. Problems and solutions on thermodynamics and statistical mechanics. World Scientific, 1990. [find](https://scholar.google.com/scholar?q=Hefei+Zhongguo%2DKexue%2DJishu%2DDaxue%2E+Major+American+universities+Ph%2E+D%2E+qualifying+questions+and+solutions%2E+5%2E+Problems+and+solutions+on+thermodynamics+and+statistical+mechanics%2E+World+Scientific%2C+1990%2E)

*Updated 2 November 2026*