---
title: "Puzzle Solving using Reasoning of Large Language Models: A"
description: "The authors propose a taxonomy based on the skills a puzzle requires rather than… 61.6% accuracy. 106 references, cited by 38 papers here."
authors: ["Panagiotis Giadikiaroglou", "Maria Lymperaiou", "Giorgos Filandrianos", "Giorgos Stamou"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.646/"
doi: "10.18653/v1/2024.emnlp-main.646"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/giadikiaroglou-2024-puzzle-solving-reasoning-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/giadikiaroglou-2024-puzzle-solving-reasoning-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Puzzle Solving using Reasoning of Large Language Models: A Survey

# Puzzle Solving using Reasoning of Large Language Models: A Survey

**Authors:** Panagiotis Giadikiaroglou, [Maria Lymperaiou](https://md.research.elamur.ai/free/en/authors/maria-lymperaiou/), [Giorgos Filandrianos](https://md.research.elamur.ai/free/en/authors/giorgos-filandrianos/) and [Giorgos Stamou](https://md.research.elamur.ai/free/en/authors/giorgos-stamou/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.646/) · [PDF](https://aclanthology.org/2024.emnlp-main.646.pdf) · [DOI 10.18653/v1/2024.emnlp-main.646](https://doi.org/10.18653/v1/2024.emnlp-main.646) · [OpenAlex](https://openalex.org/W4404782713)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [38](https://md.research.elamur.ai/free/en/papers/giadikiaroglou-2024-puzzle-solving-reasoning-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This EMNLP 2024 survey examines how well large language models handle puzzles and which techniques help them. The authors propose a taxonomy based on the skills a puzzle requires rather than on question format or reasoning type. Rule-based puzzles have explicit rules and are either deterministic (Sudoku, Chess, crosswords, Game of 24) or stochastic (Minesweeper, card games, social deduction games). Rule-less puzzles include riddles, programming puzzles and commonsense reasoning puzzles. Text-free, multimodal and mathematical puzzles are excluded.

The survey groups solving methods into three families: prompting (CoT, Self-Consistency, ToT, XoT and related thought structures), neuro-symbolic translation of puzzles into formal code, and task-specific fine-tuning. It also catalogues the available datasets and benchmarks.

The main finding is that LLMs still fall well short of human performance, especially on multi-step logical inference. Richer prompting structures help, but they do not close the gap. The authors identify missing benchmarks and methods, which makes the survey a useful map for future work on LLM reasoning.

## Key points
- **Benchmarks catalogued** include:
  - riddles: RiddleSense, BrainTeaser, BiRdQA, CC-Riddle, PUZZLEQA, MARB
  - programming: P3
  - commonsense: LatEval, True Detective, DetectBench
  - rule-based: BoardgameQA
- **Reported results:**
  - On Game of 24, Self-Refine achieved a success rate 13% higher than CoT.
  - On DetectBench, GPT-4 was the strongest model at 61.6% accuracy, and a new Detective Thinking Prompt gave the best prompting results.
- **Fine-tuning transfers poorly.** Models tuned on CSQA scored lower on RiddleSense than models tuned on RiddleSense itself.
- **Gaps and format effects:**
  - Few datasets exist for stochastic rule-based puzzles and programming puzzles.
  - Neuro-symbolic approaches are rarely applied.
  - Multiple-choice formats are noticeably easier for models than free-text answers.
- **Scope and tracking:** the review covers about four years of work, mainly from ACL, EMNLP, NAACL, NeurIPS, ICLR and arXiv. Ongoing progress is tracked at https://puzzlellms.github.io/.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Technical University of Athens](https://md.research.elamur.ai/free/en/universities/national-technical-university-of-athens/) — University, Athens, Greece · 8 works · 461 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos and Giorgos Stamou. 2024. Puzzle Solving using Reasoning of Large Language Models: A Survey. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{giadikiaroglou2024puzzle,
  title = {{Puzzle Solving using Reasoning of Large Language Models: A Survey}},
  author = {Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos and Giorgos Stamou},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.646},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.646}
}
```

- [← previous IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages](https://md.research.elamur.ai/free/en/papers/khan-2024-indicllmsuite-blueprint-creating-pre-training/)
- [next → Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities](https://md.research.elamur.ai/free/en/papers/wilf-2024-think-twice-perspective-taking-improves/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

106 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Forest Agostinelli, Stephen Marcus McAleer, Alexander Shmakov, and Pierre Baldi. 2019. Solving the rubik’s cube with deep reinforcement learning and search. Nature Machine Intelligence, 1:356 – 363. [find](https://scholar.google.com/scholar?q=Forest+Agostinelli%2C+Stephen+Marcus+McAleer%2C+Alexander+Shmakov%2C+and+Pierre+Baldi%2E+2019%2E+Solving+the+rubik%E2%80%99s+cube+with+deep+reinforcement+learning+and+search%2E+Nature+Machine+Intelligence%2C+1%3A356+%E2%80%93+363%2E)
2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Tachard Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Z. Chen, Eric Chu, J. Clark, Laurent El Shafey, Yanping Huang, Kathleen S. Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… — [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
3. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. [A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). ArXiv, abs/2302.04…
4. Qiming Bao, Gaël Gendron, Alex Yuxuan Peng, Wanjun Zhong, Neşet Özkan Tan, Yang Chen, Michael Witbrock, and Jiamou Liu. 2023. A systematic evaluation of large language models on out-of-distribution logical reasoning tasks. ArXiv, abs/2310.09430. — [arXiv:2310.09430](https://arxiv.org/abs/2310.09430) [find](https://scholar.google.com/scholar?q=Qiming+Bao%2C+Ga%C3%ABl+Gendron%2C+Alex+Yuxuan+Peng%2C+Wanjun+Zhong%2C+Nes%CC%A7et+%C3%96zkan+Tan%2C+Yang+Chen%2C+Michael+Witbrock%2C+and+Jiamou+Liu%2E+2023%2E+A+systematic+evaluation+of+large+language+models+on+out%2Dof%2Ddistribution+l)
5. Houda Nait El Barj and Theophile Sautory. 2024. Reinforcement learning from llm feedback to counteract goal misgeneralization. [find](https://scholar.google.com/scholar?q=Houda+Nait+El+Barj+and+Theophile+Sautory%2E+2024%2E+Reinforcement+learning+from+llm+feedback+to+counteract+goal+misgeneralization%2E)
6. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2023. Graph of thoughts: Solving elaborate problems with large language models. [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Nils+Blach%2C+Ales+Kubicek%2C+Robert+Gerstenberger%2C+Lukas+Gianinazzi%2C+Joanna+Gajda%2C+Tomasz+Lehmann%2C+Michal+Podstawski%2C+Hubert+Niewiadomski%2C+Piotr+Nyczyk%2C+and+Torsten+Hoefler%2E+2023%2E+Graph+of+)
7. Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Aidan O’Mahony, Onur Mutlu, and Torsten Hoefler. 2024. Demystifying chains, trees, and g… [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Florim+Memedi%2C+Zhenyu+Zhang%2C+Robert+Gerstenberger%2C+Guangyuan+Piao%2C+Nils+Blach%2C+Piotr+Nyczyk%2C+Marcin+Copik%2C+Grzegorz+Kwas%CC%81niewski%2C+J%C3%BCrgen+M%C3%BCller%2C+Lukas+Gianinazzi%2C+Ales+Kubicek%2C+Hubert+Ni)
8. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). ArXiv, abs/1911.11641.
9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, C… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
10. Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Ya… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+S%2E+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+Xinyun+C)
11. Murray Campbell, A.Joseph Hoane, and Feng hsiung Hsu. 2002. Deep blue. Artificial Intelligence, 134(1):57–83. [find](https://scholar.google.com/scholar?q=Murray+Campbell%2C+A%2EJoseph+Hoane%2C+and+Feng+hsiung+Hsu%2E+2002%2E+Deep+blue%2E+Artificial+Intelligence%2C+134%281%29%3A57%E2%80%9383%2E)
12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Unsupervised+cross%2Dlingu)
13. Banghao Chen, Zhaofeng Zhang, Nicolas Langren’e, and Shengxin Zhu. 2023. [Unleashing the potential of prompt engineering in large language models: a comprehensive review](https://md.research.elamur.ai/free/en/papers/chen-2023-unleashing-potential-prompt-engineering-large/). ArXiv, abs/2310.14735.
14. Juntao Chen. 2022. Different algorithms to solve a rubik’s cube. Journal of Physics: Conference Series, 2386(1):012018. [find](https://scholar.google.com/scholar?q=Juntao+Chen%2E+2022%2E+Different+algorithms+to+solve+a+rubik%E2%80%99s+cube%2E+Journal+of+Physics%3A+Conference+Series%2C+2386%281%29%3A012018%2E)
15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Aleth… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
16. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv, abs/2211.12588. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W%2E+Cohen%2E+2022%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+ArXiv%2C+abs%2F2211%2E12588%2E)
17. Eric C. Chi and Kenneth Lange. 2013. Techniques for solving sudoku puzzles. [find](https://scholar.google.com/scholar?q=Eric+C%2E+Chi+and+Kenneth+Lange%2E+2013%2E+Techniques+for+solving+sudoku+puzzles%2E)
18. Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. 2024. Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. [find](https://scholar.google.com/scholar?q=Yew+Ken+Chia%2C+Vernon+Toh+Yan+Han%2C+Deepanway+Ghosal%2C+Lidong+Bing%2C+and+Soujanya+Poria%2E+2024%2E+Puzzlevqa%3A+Diagnosing+multimodal+reasoning+challenges+of+language+models+with+abstract+visual+patterns%2E)
19. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2023. A survey of chain of thought reasoning: Advances, frontiers and future. ArXiv, abs/2309.15402. [find](https://scholar.google.com/scholar?q=Zheng+Chu%2C+Jingchang+Chen%2C+Qianglong+Chen%2C+Weijiang+Yu%2C+Tao+He%2C+Haotian+Wang%2C+Weihua+Peng%2C+Ming+Liu%2C+Bing+Qin%2C+and+Ting+Liu%2E+2023%2E+A+survey+of+chain+of+thought+reasoning%3A+Advances%2C+frontiers+and+futur)
20. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. ArXiv, abs/2205.09712. [find](https://scholar.google.com/scholar?q=Antonia+Creswell%2C+Murray+Shanahan%2C+and+Irina+Higgins%2E+2022%2E+Selection%2Dinference%3A+Exploiting+large+language+models+for+interpretable+logical+reasoning%2E+ArXiv%2C+abs%2F2205%2E09712%2E)
21. Maksym Del and Mark Fishel. 2022. True detective: A deep abductive reasoning benchmark undoable for gpt-3 and challenging for gpt-4. In STARSEM. [find](https://scholar.google.com/scholar?q=Maksym+Del+and+Mark+Fishel%2E+2022%2E+True+detective%3A+A+deep+abductive+reasoning+benchmark+undoable+for+gpt%2D3+and+challenging+for+gpt%2D4%2E+In+STARSEM%2E)
22. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/).
23. Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Ming-Jie Ma, Wei Zhang, Si Qin, S. Rajmohan, Qingwei Lin, and Dongmei Zhang. 2023. Everything of thoughts: Defying the law of penrose triangle for thought generation. ArXiv, abs/2311.04254. [find](https://scholar.google.com/scholar?q=Ruomeng+Ding%2C+Chaoyun+Zhang%2C+Lu+Wang%2C+Yong+Xu%2C+Ming%2DJie+Ma%2C+Wei+Zhang%2C+Si+Qin%2C+S%2E+Rajmohan%2C+Qingwei+Lin%2C+and+Dongmei+Zhang%2E+2023%2E+Everything+of+thoughts%3A+Defying+the+law+of+penrose+triangle+for+though)
24. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. [A survey on in-context learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/).
25. Avia Efrat, Uri Shaham, Dan Kilman, and Omer Levy. 2021. Cryptonite: A cryptic crossword benchmark for extreme ambiguity in language. ArXiv, abs/2103.01242. [find](https://scholar.google.com/scholar?q=Avia+Efrat%2C+Uri+Shaham%2C+Dan+Kilman%2C+and+Omer+Levy%2E+2021%2E+Cryptonite%3A+A+cryptic+crossword+benchmark+for+extreme+ambiguity+in+language%2E+ArXiv%2C+abs%2F2103%2E01242%2E)
26. Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. 2023a. Language models can be logical solvers. ArXiv, abs/2311.06158. — [arXiv:2311.06158](https://arxiv.org/abs/2311.06158) [find](https://scholar.google.com/scholar?q=Jiazhan+Feng%2C+Ruochen+Xu%2C+Junheng+Hao%2C+Hiteshi+Sharma%2C+Yelong+Shen%2C+Dongyan+Zhao%2C+and+Weizhu+Chen%2E+2023a%2E+Language+models+can+be+logical+solvers%2E+ArXiv%2C+abs%2F2311%2E06158%2E+%E2%80%94+%5BarXiv%3A2311%2E06158%5D%28https%3A%2F%2Far)
27. Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Henry Mguni, Yali Du, and Jun Wang. 2023b. Chessgpt: Bridging policy learning and language modeling. ArXiv, abs/2306.09200. [find](https://scholar.google.com/scholar?q=Xidong+Feng%2C+Yicheng+Luo%2C+Ziyan+Wang%2C+Hongrui+Tang%2C+Mengyue+Yang%2C+Kun+Shao%2C+David+Henry+Mguni%2C+Yali+Du%2C+and+Jun+Wang%2E+2023b%2E+Chessgpt%3A+Bridging+policy+learning+and+language+modeling%2E+ArXiv%2C+abs%2F2306%2E0)
28. Peter A. Flach and Antonis C. Kakas. 2000. Abductive and inductive reasoning: background and issues. [find](https://scholar.google.com/scholar?q=Peter+A%2E+Flach+and+Antonis+C%2E+Kakas%2E+2000%2E+Abductive+and+inductive+reasoning%3A+background+and+issues%2E)
29. Yao Fu, Hao-Chun Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). ArXiv, abs/2210.00720.
30. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models. ArXiv, abs/2211.10435. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+2022%2E+Pal%3A+Program%2Daided+language+models%2E+ArXiv%2C+abs%2F2211%2E10435%2E)
31. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+2021%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+)
32. Deepanway Ghosal, Vernon Toh Yan Han, Chia Yew Ken, and Soujanya Poria. 2024. Are language models puzzle prodigies? algorithmic puzzles unveil serious challenges in multimodal reasoning. [find](https://scholar.google.com/scholar?q=Deepanway+Ghosal%2C+Vernon+Toh+Yan+Han%2C+Chia+Yew+Ken%2C+and+Soujanya+Poria%2E+2024%2E+Are+language+models+puzzle+prodigies%3F+algorithmic+puzzles+unveil+serious+challenges+in+multimodal+reasoning%2E)
33. Zhouhong Gu, Zihan Li, Lin Zhang, Zhuozhi Xiong, Sihang Jiang, Xiaoxuan Zhu, Shusen Wang, Zili Wang, Jianchen Wang, Haoning Ye, Wenhao Huang, Yikai Zhang, Hongwei Feng, and Yanghua Xiao. 2023. Go beyond the obvious: Probing the gap of informal reasoning ability between humanity and llms by detective… [find](https://scholar.google.com/scholar?q=Zhouhong+Gu%2C+Zihan+Li%2C+Lin+Zhang%2C+Zhuozhi+Xiong%2C+Sihang+Jiang%2C+Xiaoxuan+Zhu%2C+Shusen+Wang%2C+Zili+Wang%2C+Jianchen+Wang%2C+Haoning+Ye%2C+Wenhao+Huang%2C+Yikai+Zhang%2C+Hongwei+Feng%2C+and+Yanghua+Xiao%2E+2023%2E+Go+beyo)
34. Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. 2023. Suspicion-agent: Playing imperfect information games with theory of mind aware gpt-4. ArXiv, abs/2309.17277. — [arXiv:2309.17277](https://arxiv.org/abs/2309.17277) [find](https://scholar.google.com/scholar?q=Jiaxian+Guo%2C+Bo+Yang%2C+Paul+Yoo%2C+Bill+Yuchen+Lin%2C+Yusuke+Iwasawa%2C+and+Yutaka+Matsuo%2E+2023%2E+Suspicion%2Dagent%3A+Playing+imperfect+information+games+with+theory+of+mind+aware+gpt%2D4%2E+ArXiv%2C+abs%2F2309%2E17277%2E+%E2%80%94)
35. Akshat Gupta. 2023. Are chatgpt and gpt-4 good poker players? - a pre-flop analysis. ArXiv, abs/2308.12466. — [arXiv:2308.12466](https://arxiv.org/abs/2308.12466) [find](https://scholar.google.com/scholar?q=Akshat+Gupta%2E+2023%2E+Are+chatgpt+and+gpt%2D4+good+poker+players%3F+%2D+a+pre%2Dflop+analysis%2E+ArXiv%2C+abs%2F2308%2E12466%2E+%E2%80%94+%5BarXiv%3A2308%2E12466%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2308%2E12466%29)
36. Chenghao Huang, Yanbo Cao, Yinlong Wen, Tao Zhou, and Yanru Zhang. 2024. Pokergpt: An end-to-end lightweight solver for multi-player texas hold’em via large language model. ArXiv, abs/2401.06781. — [arXiv:2401.06781](https://arxiv.org/abs/2401.06781) [find](https://scholar.google.com/scholar?q=Chenghao+Huang%2C+Yanbo+Cao%2C+Yinlong+Wen%2C+Tao+Zhou%2C+and+Yanru+Zhang%2E+2024%2E+Pokergpt%3A+An+end%2Dto%2Dend+lightweight+solver+for+multi%2Dplayer+texas+hold%E2%80%99em+via+large+language+model%2E+ArXiv%2C+abs%2F2401%2E06781%2E+%E2%80%94+%5Ba)
37. Jie Huang and Kevin Chen-Chuan Chang. 2022. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/). ArXiv, abs/2212.10403.
38. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023a. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). ArXiv, abs/2310.01798.
39. Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. 2023. Boardgameqa: A dataset for natural language reasoning with contradictory information. ArXiv, abs/2306.07934. — [arXiv:2306.07934](https://arxiv.org/abs/2306.07934) [find](https://scholar.google.com/scholar?q=Mehran+Kazemi%2C+Quan+Yuan%2C+Deepti+Bhatia%2C+Najoung+Kim%2C+Xin+Xu%2C+Vaiva+Imbrasaite%2C+and+Deepak+Ramachandran%2E+2023%2E+Boardgameqa%3A+A+dataset+for+natural+language+reasoning+with+contradictory+information%2E+ArX)
40. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. [Unifiedqa: Crossing format boundaries with a single qa system](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/). In Findings.
41. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). ArXiv, abs/2205.11916.
42. Richard E. Korf. 1997. Finding optimal solutions to rubik’s cube using pattern databases. In AAAI/IAAI. [find](https://scholar.google.com/scholar?q=Richard+E%2E+Korf%2E+1997%2E+Finding+optimal+solutions+to+rubik%E2%80%99s+cube+using+pattern+databases%2E+In+AAAI%2FIAAI%2E)
43. Saurabh Kulshreshtha, Olga Kovaleva, Namrata Shivagunde, and Anna Rumshisky. 2022. Down and across: Introducing crossword-solving as a new nlp benchmark. ArXiv, abs/2205.10442. [find](https://scholar.google.com/scholar?q=Saurabh+Kulshreshtha%2C+Olga+Kovaleva%2C+Namrata+Shivagunde%2C+and+Anna+Rumshisky%2E+2022%2E+Down+and+across%3A+Introducing+crossword%2Dsolving+as+a+new+nlp+benchmark%2E+ArXiv%2C+abs%2F2205%2E10442%2E)
44. Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, De-Yong Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. 2023. Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay. ArXiv, abs/2310.14985. [find](https://scholar.google.com/scholar?q=Yihuai+Lan%2C+Zhiqiang+Hu%2C+Lei+Wang%2C+Yang+Wang%2C+De%2DYong+Ye%2C+Peilin+Zhao%2C+Ee%2DPeng+Lim%2C+Hui+Xiong%2C+and+Hao+Wang%2E+2023%2E+Llm%2Dbased+agent+society+investigation%3A+Collaboration+and+confrontation+in+avalon+game)
45. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. [Albert: A lite bert for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). ArXiv, abs/1909.11942.
46. Bin Lei, Pei-Hung Lin, Chunhua Liao, and Caiwen Ding. 2023. Boosting logical reasoning in large language models through a new framework: The graph of thought. ArXiv, abs/2308.08614. — [arXiv:2308.08614](https://arxiv.org/abs/2308.08614) [find](https://scholar.google.com/scholar?q=Bin+Lei%2C+Pei%2DHung+Lin%2C+Chunhua+Liao%2C+and+Caiwen+Ding%2E+2023%2E+Boosting+logical+reasoning+in+large+language+models+through+a+new+framework%3A+The+graph+of+thought%2E+ArXiv%2C+abs%2F2308%2E08614%2E+%E2%80%94+%5BarXiv%3A2308%2E0861)
47. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Annual Meeting of the Association for Comput…
48. Yinghao Li, Haorui Wang, and Chao Zhang. 2023. Assessing logical puzzle solving in large language models: Insights from a minesweeper case study. ArXiv, abs/2311.07387. [find](https://scholar.google.com/scholar?q=Yinghao+Li%2C+Haorui+Wang%2C+and+Chao+Zhang%2E+2023%2E+Assessing+logical+puzzle+solving+in+large+language+models%3A+Insights+from+a+minesweeper+case+study%2E+ArXiv%2C+abs%2F2311%2E07387%2E)
49. Shulin Huang, Shirong Ma, Yinghui Li, Mengzuo Huang, Wuhe Zou, Weidong Zhang, and Haitao Zheng. 2023b. Lateval: An interactive llms evaluation benchmark with incomplete information from lateral thinking puzzles. ArXiv, abs/2308.10855. [find](https://scholar.google.com/scholar?q=Shulin+Huang%2C+Shirong+Ma%2C+Yinghui+Li%2C+Mengzuo+Huang%2C+Wuhe+Zou%2C+Weidong+Zhang%2C+and+Haitao+Zheng%2E+2023b%2E+Lateval%3A+An+interactive+llms+evaluation+benchmark+with+incomplete+information+from+lateral+thinki)
50. Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. 2021. Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. In Findings. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Ziyi+Wu%2C+Yichi+Yang%2C+Dong%2DHo+Lee%2C+and+Xiang+Ren%2E+2021%2E+Riddlesense%3A+Reasoning+about+riddle+questions+featuring+linguistic+creativity+and+commonsense+knowledge%2E+In+Findings%2E)
51. Adam Ishay, Zhun Yang, and Joohyung Lee. 2023. Leveraging large language models to generate answer set programs. ArXiv, abs/2307.07699. — [arXiv:2307.07699](https://arxiv.org/abs/2307.07699) [find](https://scholar.google.com/scholar?q=Adam+Ishay%2C+Zhun+Yang%2C+and+Joohyung+Lee%2E+2023%2E+Leveraging+large+language+models+to+generate+answer+set+programs%2E+ArXiv%2C+abs%2F2307%2E07699%2E+%E2%80%94+%5BarXiv%3A2307%2E07699%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E07699%29)
52. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yuexin Zhang. 2023a. [Evaluating the logical reasoning ability of chatgpt and gpt-4](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/). ArXiv, abs/2304.03439.
53. Yifan Jiang, Filip Ilievski, and Kaixin Ma. 2023. Brainteaser: Lateral thinking puzzles for large language models. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Yifan+Jiang%2C+Filip+Ilievski%2C+and+Kaixin+Ma%2E+2023%2E+Brainteaser%3A+Lateral+thinking+puzzles+for+large+language+models%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E)
54. Hanmeng Liu, Zhiyang Teng, Ruoxi Ning, Jian Liu, Qiji Zhou, and Yuexin Zhang. 2023b. Glore: Evaluating logical reasoning of large language models. ArXiv, abs/2310.09107. — [arXiv:2310.09107](https://arxiv.org/abs/2310.09107) [find](https://scholar.google.com/scholar?q=Hanmeng+Liu%2C+Zhiyang+Teng%2C+Ruoxi+Ning%2C+Jian+Liu%2C+Qiji+Zhou%2C+and+Yuexin+Zhang%2E+2023b%2E+Glore%3A+Evaluating+logical+reasoning+of+large+language+models%2E+ArXiv%2C+abs%2F2310%2E09107%2E+%E2%80%94+%5BarXiv%3A2310%2E09107%5D%28https%3A%2F%2Fa)
55. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM Computing Surveys, 55:1 – 35.
56. Wentao Liu, Hanglei Hu, Jie Zhou, Yuyang Ding, Junsong Li, Jiayi Zeng, Mengliang He, Qin Chen, Bo Jiang, Aimin Zhou, and Liang He. 2023c. Mathematical language models: A survey. [find](https://scholar.google.com/scholar?q=Wentao+Liu%2C+Hanglei+Hu%2C+Jie+Zhou%2C+Yuyang+Ding%2C+Junsong+Li%2C+Jiayi+Zeng%2C+Mengliang+He%2C+Qin+Chen%2C+Bo+Jiang%2C+Aimin+Zhou%2C+and+Liang+He%2E+2023c%2E+Mathematical+language+models%3A+A+survey%2E)
57. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). ArXiv, abs/1907.11692.
58. Jieyi Long. 2023. Large language model guided tree-of-thought. ArXiv, abs/2305.08291. [find](https://scholar.google.com/scholar?q=Jieyi+Long%2E+2023%2E+Large+language+model+guided+tree%2Dof%2Dthought%2E+ArXiv%2C+abs%2F2305%2E08291%2E)
59. Man Luo, Shrinidhi Kumbhar, Ming shen, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, and Chitta Baral. 2023. [Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models](https://md.research.elamur.ai/free/en/papers/luo-2023-towards-logiglue-brief-survey-benchmark/). ArXiv, abs/2310.00836.
60. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. ArX… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Sean+Welleck%2C+Bodhisattwa+Prasad+Majumder%2C+Shashank+Gupta%2C)
61. Smaragda Markaki and Costas Panagiotakis. 2022. Jigsaw puzzle solving techniques and applications: a survey. The Visual Computer, 39:4405 – 4421. [find](https://scholar.google.com/scholar?q=Smaragda+Markaki+and+Costas+Panagiotakis%2E+2022%2E+Jigsaw+puzzle+solving+techniques+and+applications%3A+a+survey%2E+The+Visual+Computer%2C+39%3A4405+%E2%80%93+4421%2E)
62. Stephen McAleer, Forest Agostinelli, Alexander Shmakov, and Pierre Baldi. 2018. Solving the rubik’s cube without human knowledge. [find](https://scholar.google.com/scholar?q=Stephen+McAleer%2C+Forest+Agostinelli%2C+Alexander+Shmakov%2C+and+Pierre+Baldi%2E+2018%2E+Solving+the+rubik%E2%80%99s+cube+without+human+knowledge%2E)
63. Arindam Mitra and Chitta Baral. 2015. Learning to automatically solve logic grid puzzles. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Arindam+Mitra+and+Chitta+Baral%2E+2015%2E+Learning+to+automatically+solve+logic+grid+puzzles%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E)
64. Shentong Mo and Miao Xin. 2023. Tree of uncertain thoughts reasoning for large language models. ArXiv, abs/2309.07694. — [arXiv:2309.07694](https://arxiv.org/abs/2309.07694) [find](https://scholar.google.com/scholar?q=Shentong+Mo+and+Miao+Xin%2E+2023%2E+Tree+of+uncertain+thoughts+reasoning+for+large+language+models%2E+ArXiv%2C+abs%2F2309%2E07694%2E+%E2%80%94+%5BarXiv%3A2309%2E07694%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E07694%29)
65. David A. Noever and Ryerson Burdick. 2021. Puzzle solving without search or human knowledge: An unnatural language approach. ArXiv, abs/2109.02797. — [arXiv:2109.02797](https://arxiv.org/abs/2109.02797) [find](https://scholar.google.com/scholar?q=David+A%2E+Noever+and+Ryerson+Burdick%2E+2021%2E+Puzzle+solving+without+search+or+human+knowledge%3A+An+unnatural+language+approach%2E+ArXiv%2C+abs%2F2109%2E02797%2E+%E2%80%94+%5BarXiv%3A2109%2E02797%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2109%2E0279)
66. Theo X. Olausson, Alex Gu, Benjamin Lipkin, Cedegao Zhang, Armando Solar-Lezama, Josh Tenenbaum, and Roger Levy. 2023. [Linc: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers](https://md.research.elamur.ai/free/en/papers/olausson-2023-linc-neurosymbolic-approach-logical-reasoning/). In Conference on Empirical Methods in Natural Language Processing.
67. OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
68. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023a. [Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning](https://md.research.elamur.ai/free/en/papers/pan-2023-logic-lm-empowering-large-language/). ArXiv, abs/2305.12295.
69. Jaromir Savelka, Arav Agarwal, Christopher Bogart, and Majd Sakr. 2023. Large language models (gpt) struggle to answer multiple-choice questions about code. [find](https://scholar.google.com/scholar?q=Jaromir+Savelka%2C+Arav+Agarwal%2C+Christopher+Bogart%2C+and+Majd+Sakr%2E+2023%2E+Large+language+models+%28gpt%29+struggle+to+answer+multiple%2Dchoice+questions+about+code%2E)
70. Tal Schuster, A. Kalyan, Oleksandr Polozov, and Adam Tauman Kalai. 2021. Programming puzzles. ArXiv, abs/2106.05784. — [arXiv:2106.05784](https://arxiv.org/abs/2106.05784) [find](https://scholar.google.com/scholar?q=Tal+Schuster%2C+A%2E+Kalyan%2C+Oleksandr+Polozov%2C+and+Adam+Tauman+Kalai%2E+2021%2E+Programming+puzzles%2E+ArXiv%2C+abs%2F2106%2E05784%2E+%E2%80%94+%5BarXiv%3A2106%2E05784%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2106%2E05784%29)
71. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2017. [Mastering chess and shogi by self-play with a general reinforcement learning alg…](https://md.research.elamur.ai/free/en/papers/al-2017-mastering-chess-shogi-self-play/)
72. Helmut Simonis. 2005. Sudoku as a constraint problem. [find](https://scholar.google.com/scholar?q=Helmut+Simonis%2E+2005%2E+Sudoku+as+a+constraint+problem%2E)
73. Chris Studholme. 2001. Minesweeper as a constraint satisfaction problem. [find](https://scholar.google.com/scholar?q=Chris+Studholme%2E+2001%2E+Minesweeper+as+a+constraint+satisfaction+problem%2E)
74. Roxana Szomiu and Adrian Groza. 2021. A puzzle-based dataset for natural language inference. ArXiv, abs/2112.05742. — [arXiv:2112.05742](https://arxiv.org/abs/2112.05742) [find](https://scholar.google.com/scholar?q=Roxana+Szomiu+and+Adrian+Groza%2E+2021%2E+A+puzzle%2Dbased+dataset+for+natural+language+inference%2E+ArXiv%2C+abs%2F2112%2E05742%2E+%E2%80%94+%5BarXiv%3A2112%2E05742%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2112%2E05742%29)
75. Kyo Takano. 2023. Self-supervision is all you need for solving rubik’s cube. [find](https://scholar.google.com/scholar?q=Kyo+Takano%2E+2023%2E+Self%2Dsupervision+is+all+you+need+for+solving+rubik%E2%80%99s+cube%2E)
76. Liangming Pan, Michael Stephen Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023b. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. ArXiv, abs/2308.03188. [find](https://scholar.google.com/scholar?q=Liangming+Pan%2C+Michael+Stephen+Saxon%2C+Wenda+Xu%2C+Deepak+Nathani%2C+Xinyi+Wang%2C+and+William+Yang+Wang%2E+2023b%2E+Automatically+correcting+large+language+models%3A+Surveying+the+landscape+of+diverse+self%2Dcorrec)
77. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). ArXiv, abs/1811.00937.
78. Julien Pourcel, Cédric Colas, Pierre-Yves Oudeyer, and Laetitia Teodorescu. 2023. Aces: Generating diverse programming puzzles with autotelic language models and semantic descriptors. ArXiv, abs/2310.10692. — [arXiv:2310.10692](https://arxiv.org/abs/2310.10692) [find](https://scholar.google.com/scholar?q=Julien+Pourcel%2C+C%C3%A9dric+Colas%2C+Pierre%2DYves+Oudeyer%2C+and+Laetitia+Teodorescu%2E+2023%2E+Aces%3A+Generating+diverse+programming+puzzles+with+autotelic+language+models+and+semantic+descriptors%2E+ArXiv%2C+abs%2F2310%2E)
79. Yongqi Tong, Yifan Wang, Dawei Li, Sizhe Wang, Zi Lin, Simeng Han, and Jingbo Shang. 2023. Eliminating reasoning via inferring with planning: A new framework to guide llms’ non-linear thinking. ArXiv, abs/2310.12342. — [arXiv:2310.12342](https://arxiv.org/abs/2310.12342) [find](https://scholar.google.com/scholar?q=Yongqi+Tong%2C+Yifan+Wang%2C+Dawei+Li%2C+Sizhe+Wang%2C+Zi+Lin%2C+Simeng+Han%2C+and+Jingbo+Shang%2E+2023%2E+Eliminating+reasoning+via+inferring+with+planning%3A+A+new+framework+to+guide+llms%E2%80%99+non%2Dlinear+thinking%2E+ArXiv%2C)
80. Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022. [Reasoning with language model prompting: A survey](https://md.research.elamur.ai/free/en/papers/qiao-2022-reasoning-language-model-prompting-survey/). ArXiv, abs/2212.09597.
81. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
82. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67.
83. Josh Rozner, Christopher Potts, and Kyle Mahowald. 2021. Decrypting cryptic crosswords: Semantically complex wordplay puzzles as a target for nlp. ArXiv, abs/2104.08620. [find](https://scholar.google.com/scholar?q=Josh+Rozner%2C+Christopher+Potts%2C+and+Kyle+Mahowald%2E+2021%2E+Decrypting+cryptic+crosswords%3A+Semantically+complex+wordplay+puzzles+as+a+target+for+nlp%2E+ArXiv%2C+abs%2F2104%2E08620%2E)
84. Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim, and He He. 2023. [Testing the general deductive reasoning capacity of large language models using ood examples](https://md.research.elamur.ai/free/en/papers/saparov-2023-testing-general-deductive-reasoning-capacity/). ArXiv, abs/2305.15269.
85. Gladys Tyen, Hassan Mansoor, Peter Chen, Tony Mak, and Victor Carbune. 2023. Llms cannot find reasoning errors, but can correct them! ArXiv, abs/2311.08516. — [arXiv:2311.08516](https://arxiv.org/abs/2311.08516) [find](https://scholar.google.com/scholar?q=Gladys+Tyen%2C+Hassan+Mansoor%2C+Peter+Chen%2C+Tony+Mak%2C+and+Victor+Carbune%2E+2023%2E+Llms+cannot+find+reasoning+errors%2C+but+can+correct+them%21+ArXiv%2C+abs%2F2311%2E08516%2E+%E2%80%94+%5BarXiv%3A2311%2E08516%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
86. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023a. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lei+Wang%2C+Wanyu+Xu%2C+Yihuai+Lan%2C+Zhiqiang+Hu%2C+Yunshi+Lan%2C+Roy+Ka%2DWei+Lee%2C+and+Ee%2DPeng+Lim%2E+2023a%2E+Plan%2Dand%2Dsolve+prompting%3A+Improving+zero%2Dshot+chain%2Dof%2Dthought+reasoning+by+large+language+models%2E+In+A)
87. Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023b. Avalon’s game of thoughts: Battle against deception through recursive contemplation. ArXiv, abs/2310.01320. — [arXiv:2310.01320](https://arxiv.org/abs/2310.01320) [find](https://scholar.google.com/scholar?q=Shenzhi+Wang%2C+Chang+Liu%2C+Zilong+Zheng%2C+Siyuan+Qi%2C+Shuo+Chen%2C+Qisen+Yang%2C+Andrew+Zhao%2C+Chaofei+Wang%2C+Shiji+Song%2C+and+Gao+Huang%2E+2023b%2E+Avalon%E2%80%99s+game+of+thoughts%3A+Battle+against+deception+through+recurs)
88. Weiqi Wang, Tianqing Fang, Wenxuan Ding, Baixuan Xu, Xin Liu, Yangqiu Song, and Antoine Bosselut. 2023c. Car: Conceptualization-augmented reasoner for zero-shot commonsense question answering. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Weiqi+Wang%2C+Tianqing+Fang%2C+Wenxuan+Ding%2C+Baixuan+Xu%2C+Xin+Liu%2C+Yangqiu+Song%2C+and+Antoine+Bosselut%2E+2023c%2E+Car%3A+Conceptualization%2Daugmented+reasoner+for+zero%2Dshot+commonsense+question+answering%2E+In+Conf)
89. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). ArXiv, abs/2203.11171.
90. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). ArXiv, abs/2201.11903.
91. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. 2022. [Large language models are better reasoners with self-verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/). In Conference on Empirical Methods in Natural Language Processing.
92. Fan Xu, Yunxiang Zhang, and Xiao-Yi Wan. 2022. Cc-riddle: A question answering dataset of chinese character riddles. ArXiv, abs/2206.13778. — [arXiv:2206.13778](https://arxiv.org/abs/2206.13778) [find](https://scholar.google.com/scholar?q=Fan+Xu%2C+Yunxiang+Zhang%2C+and+Xiao%2DYi+Wan%2E+2022%2E+Cc%2Driddle%3A+A+question+answering+dataset+of+chinese+character+riddles%2E+ArXiv%2C+abs%2F2206%2E13778%2E+%E2%80%94+%5BarXiv%3A2206%2E13778%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E13778%29)
93. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. 2023a. [Are large language models really good logical reasoners? a comprehensive evaluation and beyond](https://md.research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/).
94. Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023b. [Exploring large language models for communication games: An empirical study on werewolf](https://md.research.elamur.ai/free/en/papers/xu-2023-exploring-large-language-models-communication/). ArXiv, abs/2309.04658.
95. Sen Yang, Xin Li, Leyang Cui, Li Bing, and Wai Lam. 2023a. Neuro-symbolic integration brings causal and reliable reasoning proofs. ArXiv, abs/2311.09802. [find](https://scholar.google.com/scholar?q=Sen+Yang%2C+Xin+Li%2C+Leyang+Cui%2C+Li+Bing%2C+and+Wai+Lam%2E+2023a%2E+Neuro%2Dsymbolic+integration+brings+causal+and+reliable+reasoning+proofs%2E+ArXiv%2C+abs%2F2311%2E09802%2E)
96. Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and E. Cambria. 2023b. [Logical reasoning over natural language as knowledge representation: A survey](https://md.research.elamur.ai/free/en/papers/yang-2023-logical-reasoning-over-natural-language/). ArXiv, abs/2303.12023.
97. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+ArXiv%2C+abs%2F2305%2E)
98. Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. 2023a. Natural language reasoning, a survey. [find](https://scholar.google.com/scholar?q=Fei+Yu%2C+Hongbo+Zhang%2C+Prayag+Tiwari%2C+and+Benyou+Wang%2E+2023a%2E+Natural+language+reasoning%2C+a+survey%2E)
99. Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. 2023b. Towards better chain-of-thought prompting strategies: A survey. ArXiv, abs/2310.04959. — [arXiv:2310.04959](https://arxiv.org/abs/2310.04959) [find](https://scholar.google.com/scholar?q=Zihan+Yu%2C+Liang+He%2C+Zhen+Wu%2C+Xinyu+Dai%2C+and+Jiajun+Chen%2E+2023b%2E+Towards+better+chain%2Dof%2Dthought+prompting+strategies%3A+A+survey%2E+ArXiv%2C+abs%2F2310%2E04959%2E+%E2%80%94+%5BarXiv%3A2310%2E04959%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E0)
100. Kamyar Zeinalipour, Tommaso laquinta, Asya Zanollo, Giovanni Angelini, Leonardo Rigutini, Marco Maggini, and Marco Gori. 2023a. Italian crossword generator: Enhancing education through interactive word puzzles. [find](https://scholar.google.com/scholar?q=Kamyar+Zeinalipour%2C+Tommaso+laquinta%2C+Asya+Zanollo%2C+Giovanni+Angelini%2C+Leonardo+Rigutini%2C+Marco+Maggini%2C+and+Marco+Gori%2E+2023a%2E+Italian+crossword+generator%3A+Enhancing+education+through+interactive+wor)
101. Kamyar Zeinalipour, Mohamed Saad, Marco Maggini, and Marco Gori. 2023b. [Arabicros: Ai-powered arabic crossword puzzle generation for educational applications](https://md.research.elamur.ai/free/en/papers/zeinalipour-2023-arabicros-ai-powered-arabic-crossword/). In Proceedings of ArabicNLP 2023. Association for Computational Linguistics.
102. Yunxiang Zhang and Xiaojun Wan. 2021. Birdqa: A bilingual dataset for question answering on tricky riddles. ArXiv, abs/2109.11087. — [arXiv:2109.11087](https://arxiv.org/abs/2109.11087) [find](https://scholar.google.com/scholar?q=Yunxiang+Zhang+and+Xiaojun+Wan%2E+2021%2E+Birdqa%3A+A+bilingual+dataset+for+question+answering+on+tricky+riddles%2E+ArXiv%2C+abs%2F2109%2E11087%2E+%E2%80%94+%5BarXiv%3A2109%2E11087%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2109%2E11087%29)
103. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alexander J. Smola. 2022. [Automatic chain of thought prompting in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/). ArXiv, abs/2210.03493.
104. Jingmiao Zhao and Carolyn Jane Anderson. 2023. Solving and generating npr sunday puzzles with large language models. [find](https://scholar.google.com/scholar?q=Jingmiao+Zhao+and+Carolyn+Jane+Anderson%2E+2023%2E+Solving+and+generating+npr+sunday+puzzles+with+large+language+models%2E)
105. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. [Large language models are human-level prompt engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/). ArXiv, abs/2211.01910.
106. Andrea Zugarini, Kamyar Zeinalipour, Surya Sai Kadali, Marco Maggini, Marco Gori, and Leonardo Rigutini. 2024. Clue-instruct: Text-based clue generation for educational crossword puzzles. [find](https://scholar.google.com/scholar?q=Andrea+Zugarini%2C+Kamyar+Zeinalipour%2C+Surya+Sai+Kadali%2C+Marco+Maggini%2C+Marco+Gori%2C+and+Leonardo+Rigutini%2E+2024%2E+Clue%2Dinstruct%3A+Text%2Dbased+clue+generation+for+educational+crossword+puzzles%2E)

*Updated 30 October 2026*