---
title: "Have LLMs Advanced Enough? A Challenging Problem Solving"
description: "As LLMs approach saturation on reasoning benchmarks such as GSM-8K, MATH and MMLU, the… 24 references, cited by 56 papers here."
authors: ["Daman Arora", "Himanshu Singh", "Mausam"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.468/"
doi: "10.18653/v1/2023.emnlp-main.468"
arxiv: "2305.15074"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/arora-2023-have-llms-advanced-enough-challenging/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/arora-2023-have-llms-advanced-enough-challenging/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models

# Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models

**Authors:** [Daman Arora](https://md.research.elamur.ai/free/en/authors/daman-arora/), [Himanshu Singh](https://md.research.elamur.ai/free/en/authors/himanshu-singh/) and [Mausam](https://md.research.elamur.ai/free/en/authors/mausam/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.468/) · [PDF](https://aclanthology.org/2023.emnlp-main.468.pdf) · [DOI 10.18653/v1/2023.emnlp-main.468](https://doi.org/10.18653/v1/2023.emnlp-main.468) · [arXiv 2305.15074](https://arxiv.org/abs/2305.15074) · [OpenAlex](https://openalex.org/W4389519108)

**Citations:** 51 citations (OpenAlex, 2026-10-14) · [56](https://md.research.elamur.ai/free/en/papers/arora-2023-have-llms-advanced-enough-challenging/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
As LLMs approach saturation on reasoning benchmarks such as GSM-8K, MATH and MMLU, the authors propose a harder test. JEEBench contains 515 Physics, Chemistry and Mathematics problems taken from the 2016–2023 editions of India's JEE-Advanced engineering entrance exam. Diagram-based items were removed, and the remaining problems were converted to LaTeX and manually checked. Solving them requires recalling domain concepts, turning them into equations, and carrying out long chains of algebra. The authors evaluated Falcon7B-Instruct, Alpaca-LoRA, GPT-3, GPT-3.5, GPT-4 and PaLM-2. Even GPT-4 with chain-of-thought and self-consistency stays below 40%. Self-critique does not help, and error analysis points to flawed algebra, poor translation of concepts into equations, and failure to retrieve the right concepts. When told about negative marking, GPT-4 scores lower. A post-hoc confidence threshold on self-consistency votes recovers some of this loss. The benchmark exposes clear gaps in scientific problem solving and model calibration, and it is publicly released.

## Key points
- **Composition:** 515 questions (Math 236, Chemistry 156, Physics 123) in four answer formats: single-correct MCQ, multi-correct MCQ, Integer and Numeric. About 40% of the original items were dropped because they contained figures.
- **Difficulty:** On 50-question zero-shot samples, GPT-4 solves over 80% of MMLU high-school science/math, roughly 60% of MATH, but only about 20% of JEEBench-Math.
- **Confidence thresholding:** Tuned thresholds were τ_single = 0.125 and τ_multiple = 0.75. On the test set, total marks rose from 66 (CoT) to 69 (CoT+SC) to 72 (with thresholding).
- **Human comparison:** On the uncontaminated 2023 paper, GPT-4 scored 49/229 on text-only questions. Projected to the full exam, this is about 77/360, roughly the 80–90th percentile of candidates.
- **Contamination and access:** Estimated contamination is about 6%. GPT-4+CoT+SC scores 0.338 on 2023 questions versus 0.396 on the rest. Code and data are available at github.com/dair-iitd/jeebench.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 209 works · 244,782 citations
- [Indian Institute of Technology Delhi](https://md.research.elamur.ai/free/en/universities/indian-institute-of-technology-delhi/) — University, New Delhi, India · 24 works · 6,933 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Daman Arora, Himanshu Singh and Mausam. 2023. Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{arora2023have,
  title = {{Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models}},
  author = {Daman Arora, Himanshu Singh and Mausam},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.468},
  eprint = {2305.15074},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.468}
}
```

- [← previous AutoAgents: A Framework for Automatic Agent Generation](https://md.research.elamur.ai/free/en/papers/chen-2023-autoagents-framework-automatic-agent-generation/)
- [next → M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://md.research.elamur.ai/free/en/papers/wang-2023-m4-multi-generator-multi-domain/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

24 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam. 2023. Let’s sample step by step: Adaptive-consistency for efficient reasoning and coding with llms. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Pranjal+Aggarwal%2C+Aman+Madaan%2C+Yiming+Yang%2C+and+Mausam%2E+2023%2E+Let%E2%80%99s+sample+step+by+step%3A+Adaptive%2Dconsistency+for+efficient+reasoning+and+coding+with+llms%2E+In+Conference+on+Empirical+Methods+in+Natura)
2. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al-shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-… [find](https://scholar.google.com/scholar?q=Ebtesam+Almazrouei%2C+Hamza+Alobeidli%2C+Abdulaziz+Al%2Dshamsi%2C+Alessandro+Cappelli%2C+Ruxandra+Cojocaru%2C+Merouane+Debbah%2C+Etienne+Goffinet%2C+Daniel+Heslow%2C+Julien+Launay%2C+Quentin+Malartic%2C+Badreddine+Noune%2C+B)
3. Daman Arora and Subbarao Kambhampati. 2023. [Learning and leveraging verifiers to improve planning capabilities of pre-trained language models](https://md.research.elamur.ai/free/en/papers/arora-2023-learning-leveraging-verifiers-improve-planning/).
4. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. [C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). arXiv preprint arXiv:2305.08322.
5. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.
6. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
7. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/).
8. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, Vancouver… [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+In+Proceedings+of+the+55th+Annual+Meetin)
9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
10. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tony+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+2022%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+q)
11. Jesse Dodge, Ana Marasovic, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Ana+Marasovic%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+2021%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+clean+crawled+corpus%2E+In+Confer)
12. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Sean+Welleck%2C+Bodhisattwa+Prasad+Majumder%2C+Shashank+Gupta%2C)
13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
14. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS.
15. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/).
16. Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887–89… [find](https://scholar.google.com/scholar?q=Danqing+Huang%2C+Shuming+Shi%2C+Chin%2DYew+Lin%2C+Jian+Yin%2C+and+Wei%2DYing+Ma%2E+2016%2E+How+well+do+computers+solve+math+word+problems%3F+large%2Dscale+dataset+construction+and+evaluation%2E+In+Proceedings+of+the+54th+A)
17. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/).
18. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
19. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop. [find](https://scholar.google.com/scholar?q=Karthik+Valmeekam%2C+Alberto+Olmo%2C+Sarath+Sreedharan%2C+and+Subbarao+Kambhampati%2E+2022%2E+Large+language+models+still+can%E2%80%99t+plan+%28a+benchmark+for+LLMs+on+planning+and+reasoning+about+change%29%2E+In+NeurIPS+202)
20. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2023a. [Scibench: Evaluating college-level scientific problem-solving abilities of large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/).
21. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations.
22. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). ArXiv, abs/1707.06209.
23. Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020. Ape210k: A large-scale and template-rich dataset of math word problems. [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Mingyue+Shang%2C+Yang+Liu%2C+Liang+Wang%2C+and+Jingming+Liu%2E+2020%2E+Ape210k%3A+A+large%2Dscale+and+template%2Drich+dataset+of+math+word+problems%2E)
24. Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. 2022. [minif2f: a cross-system benchmark for formal olympiad-level mathematics](https://md.research.elamur.ai/free/en/papers/zheng-2021-minif2f-cross-system-benchmark-formal/). In International Conference on Learning Representations.

*Updated 2 November 2026*