---
title: "NPHardEval — paper by Fan et al. (2024) — summary and ideas"
description: "The work matters because it offers a contamination-resistant, fully automatic and quantitatively… 52 references, cited by 35 papers here."
authors: ["Lizhou Fan", "Wenyue Hua", "Lingyao Li", "Haoyang Ling", "Yongfeng Zhang"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.225/"
doi: "10.18653/v1/2024.acl-long.225"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fan-2024-nphardeval-dynamic-benchmark-reasoning-ability/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fan-2024-nphardeval-dynamic-benchmark-reasoning-ability/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes

# NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes

**Authors:** Lizhou Fan, [Wenyue Hua](https://md.research.elamur.ai/free/en/authors/wenyue-hua/), Lingyao Li, Haoyang Ling and [Yongfeng Zhang](https://md.research.elamur.ai/free/en/authors/yongfeng-zhang/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.225/) · [PDF](https://aclanthology.org/2024.acl-long.225.pdf) · [DOI 10.18653/v1/2024.acl-long.225](https://doi.org/10.18653/v1/2024.acl-long.225) · [OpenAlex](https://openalex.org/W4402672103)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · [35](https://md.research.elamur.ai/free/en/papers/fan-2024-nphardeval-dynamic-benchmark-reasoning-ability/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most reasoning benchmarks for LLMs are fixed and publicly available, so models can memorize them or be tuned toward them, which inflates scores. These benchmarks also lack a principled scale of difficulty and sometimes rely on manual grading. NPHardEval grounds reasoning evaluation in computational complexity theory instead. It contains 900 algorithmic problems from nine tasks across the P, NP-complete and NP-hard classes, and each task is graded into ten difficulty levels. Instances are generated synthetically and checked automatically with well-known algorithms. This lets the authors regenerate the data monthly and reduce contamination risk. They test 12 closed- and open-source LLMs zero-shot to see how accuracy falls as complexity and difficulty rise. They also check robustness by finetuning small open models on earlier benchmark versions and testing whether this helps them "hack" new versions. The provided excerpt omits the detailed numerical results. The work matters because it offers a contamination-resistant, fully automatic and quantitatively structured measure of logical reasoning that goes beyond recalled knowledge.

## Key points
- **Tasks and classes:**
  - P: SPP, EDP and SAS.
  - NP-complete: TSP-D, GCP-D and KSP.
  - NP-hard: GCP, TSP and MSP.
  - NP-intermediate problems such as integer factorization are left out because they depend heavily on calculation.
- **Difficulty scaling:** levels grow systematically. For GCP-D, graphs grow from 6 nodes and 6 edges at level 1 to 15 nodes and 24 edges at level 10.
- **Models evaluated (12):**
  - Closed-source: PaLM 2, two Anthropic models (Claude 2 and Claude Instant) and two OpenAI models (GPT 4 Turbo and GPT 3.5 Turbo).
  - Open-source: Yi-34b, Qwen-14b, Mistral-7b, Phi-2, MPT-30b, Vicuna-13b and Phi-1.5.
- **Overfitting test:** Phi-2, Mistral-7b and Qwen-14b were finetuned on five benchmark versions. Each checkpoint was then scored on two versions of differing difficulty.
- **Release:** data is refreshed monthly, and only the latest three months stay available. Code and data are at github.com/casmlab/NPHardEval.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling and Yongfeng Zhang. 2024. NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{fan2024nphardeval,
  title = {{NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes}},
  author = {Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling and Yongfeng Zhang},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.225},
  url = {https://doi.org/10.18653/v1/2024.acl-long.225}
}
```

- [← previous MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures](https://md.research.elamur.ai/free/en/papers/ni-2024-mixeval-deriving-wisdom-crowd-llm/)
- [next → A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts](https://md.research.elamur.ai/free/en/papers/lee-2024-human-inspired-reading-agent-gist/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

52 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. T.H. Cormen, C.E. Leiserson, R.L. Rivest, and C. Stein. 2022. Introduction to Algorithms, fourth edition. MIT Press. [find](https://scholar.google.com/scholar?q=T%2EH%2E+Cormen%2C+C%2EE%2E+Leiserson%2C+R%2EL%2E+Rivest%2C+and+C%2E+Stein%2E+2022%2E+Introduction+to+Algorithms%2C+fourth+edition%2E+MIT+Press%2E)
2. Google DeepMind. 2023. Gemini. [find](https://scholar.google.com/scholar?q=Google+DeepMind%2E+2023%2E+Gemini%2E)
3. Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. 2023. War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv preprint arXiv:2311.17227. [find](https://scholar.google.com/scholar?q=Wenyue+Hua%2C+Lizhou+Fan%2C+Lingyao+Li%2C+Kai+Mei%2C+Jianchao+Ji%2C+Yingqiang+Ge%2C+Libby+Hemphill%2C+and+Yongfeng+Zhang%2E+2023%2E+War+and+peace+%28waragent%29%3A+Large+language+model%2Dbased+multi%2Dagent+simulation+of+world+w)
4. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. [Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). arXiv preprint arXiv:1903.00161.
5. Wenyue Hua and Yongfeng Zhang. 2022. System 1+ system 2= better world: Neural-symbolic chain of logic reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 601–612. [find](https://scholar.google.com/scholar?q=Wenyue+Hua+and+Yongfeng+Zhang%2E+2022%2E+System+1%2B+system+2%3D+better+world%3A+Neural%2Dsymbolic+chain+of+logic+reasoning%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2022%2C+pages+601%E2%80%93612)
6. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). arXiv preprint arXiv:2305.14387.
7. Lizhou Fan, Sara Lafia, Lingyao Li, Fangyuan Yang, and Libby Hemphill. 2023a. Datachat: Prototyping a conversational agent for dataset search and visualization. arXiv preprint arXiv:2305.18358. — [arXiv:2305.18358](https://arxiv.org/abs/2305.18358) [find](https://scholar.google.com/scholar?q=Lizhou+Fan%2C+Sara+Lafia%2C+Lingyao+Li%2C+Fangyuan+Yang%2C+and+Libby+Hemphill%2E+2023a%2E+Datachat%3A+Prototyping+a+conversational+agent+for+dataset+search+and+visualization%2E+arXiv+preprint+arXiv%3A2305%2E18358%2E+%E2%80%94+%5BarX)
8. Lizhou Fan, Lingyao Li, Zihui Ma, Sanggyu Lee, Huizi Yu, and Libby Hemphill. 2023b. A bibliometric review of large language models research from 2017 to 2023. arXiv preprint arXiv:2304.02020. [find](https://scholar.google.com/scholar?q=Lizhou+Fan%2C+Lingyao+Li%2C+Zihui+Ma%2C+Sanggyu+Lee%2C+Huizi+Yu%2C+and+Libby+Hemphill%2E+2023b%2E+A+bibliometric+review+of+large+language+models+research+from+2017+to+2023%2E+arXiv+preprint+arXiv%3A2304%2E02020%2E)
9. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. [Mathematical capabilities of chatgpt](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/). arXiv preprint arXiv:2301.13867.
10. Zhenxiang Gao, Lingyao Li, Siyuan Ma, Qinyong Wang, Libby Hemphill, and Rong Xu. 2023. Examining the potential of chatgpt on biomedical information retrieval: Fact-checking drug-disease associations. Annals of Biomedical Engineering, pages 1–9. [find](https://scholar.google.com/scholar?q=Zhenxiang+Gao%2C+Lingyao+Li%2C+Siyuan+Ma%2C+Qinyong+Wang%2C+Libby+Hemphill%2C+and+Rong+Xu%2E+2023%2E+Examining+the+potential+of+chatgpt+on+biomedical+information+retrieval%3A+Fact%2Dchecking+drug%2Ddisease+associations%2E+)
11. Yingqiang Ge, Wenyue Hua, Jianchao Ji, Juntao Tan, Shuyuan Xu, and Yongfeng Zhang. 2023a. Openagi: When llm meets domain experts. arXiv preprint arXiv:2304.04370. — [arXiv:2304.04370](https://arxiv.org/abs/2304.04370) [find](https://scholar.google.com/scholar?q=Yingqiang+Ge%2C+Wenyue+Hua%2C+Jianchao+Ji%2C+Juntao+Tan%2C+Shuyuan+Xu%2C+and+Yongfeng+Zhang%2E+2023a%2E+Openagi%3A+When+llm+meets+domain+experts%2E+arXiv+preprint+arXiv%3A2304%2E04370%2E+%E2%80%94+%5BarXiv%3A2304%2E04370%5D%28https%3A%2F%2Farxiv%2Eor)
12. Yingqiang Ge, Yujie Ren, Wenyue Hua, Shuyuan Xu, Juntao Tan, and Yongfeng Zhang. 2023b. Llm as os (llmao), agents as apps: Envisioning aios, agents and the aios-agent ecosystem. arXiv preprint arXiv:2312.03815. — [arXiv:2312.03815](https://arxiv.org/abs/2312.03815) [find](https://scholar.google.com/scholar?q=Yingqiang+Ge%2C+Yujie+Ren%2C+Wenyue+Hua%2C+Shuyuan+Xu%2C+Juntao+Tan%2C+and+Yongfeng+Zhang%2E+2023b%2E+Llm+as+os+%28llmao%29%2C+agents+as+apps%3A+Envisioning+aios%2C+agents+and+the+aios%2Dagent+ecosystem%2E+arXiv+preprint+arXiv%3A2)
13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
14. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS.
15. Jie Huang and Kevin Chen-Chuan Chang. 2022. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/). arXiv preprint arXiv:2212.10403.
16. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). arXiv preprint arXiv:2310.01798.
17. Glenn D Israel et al. 1992. Determining sample size. [find](https://scholar.google.com/scholar?q=Glenn+D+Israel+et+al%2E+1992%2E+Determining+sample+size%2E)
18. Mojan Javaheripi and Sébastien Bubeck. 2023. Phi-2. [find](https://scholar.google.com/scholar?q=Mojan+Javaheripi+and+S%C3%A9bastien+Bubeck%2E+2023%2E+Phi%2D2%2E)
19. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
20. David S Johnson. 1990. A catalog of complexity classes. In Algorithms and complexity, pages 67–161. Elsevier. [find](https://scholar.google.com/scholar?q=David+S+Johnson%2E+1990%2E+A+catalog+of+complexity+classes%2E+In+Algorithms+and+complexity%2C+pages+67%E2%80%93161%2E+Elsevier%2E)
21. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. [Language models can solve computer tasks](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/). arXiv preprint arXiv:2303.17491.
22. Andreas Kipf, Ryan Marcus, Alexander van Renen, Mihail Stoian, Alfons Kemper, Tim Kraska, and Thomas Neumann. 2019. Sosd: A benchmark for learned indexes. [find](https://scholar.google.com/scholar?q=Andreas+Kipf%2C+Ryan+Marcus%2C+Alexander+van+Renen%2C+Mihail+Stoian%2C+Alfons+Kemper%2C+Tim+Kraska%2C+and+Thomas+Neumann%2E+2019%2E+Sosd%3A+A+benchmark+for+learned+indexes%2E)
23. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199– 22213.
24. Lingyao Li, Lizhou Fan, Shubham Atreja, and Libby Hemphill. 2023a. "hot" chatgpt: The promise of chatgpt in detecting and discriminating hateful, offensive, and toxic comments on social media. arXiv preprint arXiv:2304.10619. — [arXiv:2304.10619](https://arxiv.org/abs/2304.10619) [find](https://scholar.google.com/scholar?q=Lingyao+Li%2C+Lizhou+Fan%2C+Shubham+Atreja%2C+and+Libby+Hemphill%2E+2023a%2E+%22hot%22+chatgpt%3A+The+promise+of+chatgpt+in+detecting+and+discriminating+hateful%2C+offensive%2C+and+toxic+comments+on+social+media%2E+arXiv+p)
25. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023b. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/). arXiv preprint arXiv:2309.05463.
26. Carla Negri Lintzmayer, Mauro Henrique Mulati, and Anderson Faustino da Silva. 2011. Register allocation with graph coloring by ant colony optimization. In 2011 30th International Conference of the Chilean Computer Science Society, pages 247–255. IEEE. [find](https://scholar.google.com/scholar?q=Carla+Negri+Lintzmayer%2C+Mauro+Henrique+Mulati%2C+and+Anderson+Faustino+da+Silva%2E+2011%2E+Register+allocation+with+graph+coloring+by+ant+colony+optimization%2E+In+2011+30th+International+Conference+of+the+Ch)
27. LMSYS. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality. [find](https://scholar.google.com/scholar?q=LMSYS%2E+2023%2E+Vicuna%3A+An+open%2Dsource+chatbot+impressing+gpt%2D4+with+90%25+chatgpt+quality%2E)
28. Robert C MacCallum, Keith F Widaman, Shaobo Zhang, and Sehee Hong. 1999. Sample size in factor analysis. Psychological methods, 4(1):84. [find](https://scholar.google.com/scholar?q=Robert+C+MacCallum%2C+Keith+F+Widaman%2C+Shaobo+Zhang%2C+and+Sehee+Hong%2E+1999%2E+Sample+size+in+factor+analysis%2E+Psychological+methods%2C+4%281%29%3A84%2E)
29. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) arXiv preprint arXiv:2202.12837.
30. Janice M Morse. 1992. Qualitative health research, volume 22. Sage Newbury Park, CA. [find](https://scholar.google.com/scholar?q=Janice+M+Morse%2E+1992%2E+Qualitative+health+research%2C+volume+22%2E+Sage+Newbury+Park%2C+CA%2E)
31. MosaicML. 2023. Mpt-30b: Raising the bar for open-source foundation models. [find](https://scholar.google.com/scholar?q=MosaicML%2E+2023%2E+Mpt%2D30b%3A+Raising+the+bar+for+open%2Dsource+foundation+models%2E)
32. Roberto Roberti and Mario Ruthmair. 2021. Exact methods for the traveling salesman problem with drone. Transportation Science, 55(2):315–335. [find](https://scholar.google.com/scholar?q=Roberto+Roberti+and+Mario+Ruthmair%2E+2021%2E+Exact+methods+for+the+traveling+salesman+problem+with+drone%2E+Transportation+Science%2C+55%282%29%3A315%E2%80%93335%2E)
33. Rylan Schaeffer. 2023. [Pretraining on the test set is all you need](https://md.research.elamur.ai/free/en/papers/schaeffer-2023-pretraining-test-set-all-need/). arXiv preprint arXiv:2309.08632.
34. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. [Are emergent abilities of large language models a mirage?](https://md.research.elamur.ai/free/en/papers/schaeffer-2023-emergent-abilities-large-language-models/) arXiv preprint arXiv:2304.15004.
35. Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. 2023. Gpt-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems. arXiv preprint arXiv:2310.12397. — [arXiv:2310.12397](https://arxiv.org/abs/2310.12397) [find](https://scholar.google.com/scholar?q=Kaya+Stechly%2C+Matthew+Marquez%2C+and+Subbarao+Kambhampati%2E+2023%2E+Gpt%2D4+doesn%E2%80%99t+know+it%E2%80%99s+wrong%3A+An+analysis+of+iterative+prompting+for+reasoning+problems%2E+arXiv+preprint+arXiv%3A2310%2E12397%2E+%E2%80%94+%5BarXiv%3A2310%2E)
36. Bo Sun, Ali Zeynali, Tongxin Li, Mohammad Hajies-maili, Adam Wierman, and Danny HK Tsang. 2020. Competitive algorithms for the online multiple knapsack problem with application to electric vehicle charging. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 4(3):1–32. [find](https://scholar.google.com/scholar?q=Bo+Sun%2C+Ali+Zeynali%2C+Tongxin+Li%2C+Mohammad+Hajies%2Dmaili%2C+Adam+Wierman%2C+and+Danny+HK+Tsang%2E+2020%2E+Competitive+algorithms+for+the+online+multiple+knapsack+problem+with+application+to+electric+vehicle+cha)
37. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261.
38. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. [Large language models still can’t plan (a benchmark for llms on planning and reasoning about change)](https://md.research.elamur.ai/free/en/papers/valmeekam-2022-planbench-extensible-benchmark-evaluating-large/). arXiv preprint arXiv:2206.10498.
39. Boshi Wang, Xiang Deng, and Huan Sun. 2022. Iteratively prompt pre-trained language models for chain of thought. arXiv preprint arXiv:2203.08383. [find](https://scholar.google.com/scholar?q=Boshi+Wang%2C+Xiang+Deng%2C+and+Huan+Sun%2E+2022%2E+Iteratively+prompt+pre%2Dtrained+language+models+for+chain+of+thought%2E+arXiv+preprint+arXiv%3A2203%2E08383%2E)
40. Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Xiaojiang Huang, Yanbin Lu, and Yingzhen Yang. 2023. Recmind: Large language model powered agent for recommendation. arXiv preprint arXiv:2308.14296. [find](https://scholar.google.com/scholar?q=Yancheng+Wang%2C+Ziyan+Jiang%2C+Zheng+Chen%2C+Fan+Yang%2C+Yingxue+Zhou%2C+Eunah+Cho%2C+Xing+Fan%2C+Xiaojiang+Huang%2C+Yanbin+Lu%2C+and+Yingzhen+Yang%2E+2023%2E+Recmind%3A+Large+language+model+powered+agent+for+recommendation)
41. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682.
42. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
43. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/). arXiv preprint arXiv:2303.03846.
44. Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. 2022. [Large language models are reasoners with self-verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/). arXiv preprint arXiv:2212.09561.
45. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. [Autogen: Enabling next-gen llm applications via multi-agent conversation](https://md.research.elamur.ai/free/en/papers/wu-2023-autogen-enabling-next-gen-llm/) framework. arXiv preprint arXiv:2308.08155.
46. Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan. 2023. [Superclue: A comprehensive chinese large language model benchmark](https://md.research.elamur.ai/free/en/papers/xu-2023-superclue-comprehensive-chinese-large/). arXiv preprint arXiv:2307.15020.
47. Taro Yamane. 1973. Statistics: An introductory analysis. [find](https://scholar.google.com/scholar?q=Taro+Yamane%2E+1973%2E+Statistics%3A+An+introductory+analysis%2E)
48. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+arXiv+preprint+ar)
49. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
50. Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023. [Evaluating the performance of large language models on gaokao benchmark](https://md.research.elamur.ai/free/en/papers/zhang-2023-evaluating-performance-large-language/). arXiv preprint arXiv:2305.12474.
51. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223.
52. Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2023. Dyval: Graph-informed dynamic evaluation of large language models. arXiv preprint arXiv:2309.17167. [find](https://scholar.google.com/scholar?q=Kaijie+Zhu%2C+Jiaao+Chen%2C+Jindong+Wang%2C+Neil+Zhenqiang+Gong%2C+Diyi+Yang%2C+and+Xing+Xie%2E+2023%2E+Dyval%3A+Graph%2Dinformed+dynamic+evaluation+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E17167%2E)

*Updated 31 October 2026*