---
title: "Rethinking the Bounds of LLM Reasoning: Are Multi-Agent"
description: "The authors define a general pipeline for designing discussion frameworks with four parts: kick-start… 55 references, cited by 63 papers here."
authors: ["Qineng Wang", "Zihao Wang", "Ying Su", "Hanghang Tong", "Yangqiu Song"]
year: 2024
venue: "arXiv:2402.18272"
source: "https://arxiv.org/abs/2402.18272"
doi: "10.48550/arxiv.2402.18272"
arxiv: "2402.18272"
licence: "arXiv non-exclusive distribution license"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-rethinking-bounds-llm-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-rethinking-bounds-llm-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?

# Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?

**Authors:** [Qineng Wang](https://md.research.elamur.ai/free/en/authors/qineng-wang/), [Zihao Wang](https://md.research.elamur.ai/free/en/authors/zihao-wang/), [Ying Su](https://md.research.elamur.ai/free/en/authors/ying-su/), [Hanghang Tong](https://md.research.elamur.ai/free/en/authors/hanghang-tong/) and [Yangqiu Song](https://md.research.elamur.ai/free/en/authors/yangqiu-song/) · **Year:** 2024 · **Venue:** [arXiv:2402.18272](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Licence:** [arXiv non-exclusive distribution license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.18272) · [PDF](https://arxiv.org/pdf/2402.18272) · [DOI 10.48550/arxiv.2402.18272](https://doi.org/10.48550/arxiv.2402.18272) · [arXiv 2402.18272](https://arxiv.org/abs/2402.18272) · [OpenAlex](https://openalex.org/W4402672029)

**Citations:** 44 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/wang-2024-rethinking-bounds-llm-reasoning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Earlier work suggests that LLM agents discussing a problem together reason better than a single model. This paper tests that claim more rigorously. The authors define a general pipeline for designing discussion frameworks with four parts: kick-start prompt, agents, algorithm and result. They also introduce CMD (Conquer-and-Merge Discussion), a new framework modeled on human group discussion. In CMD, agents debate within groups, see only the answers of other groups, and then vote. A secretary agent breaks ties. CMD is compared with Debate, MAD, ReConcile and single-agent baselines on ECQA, GSM8K and FOLIO-wiki, using ChatGPT-3.5, Gemini Pro and Bard as backbones. Prompts are tested both with and without demonstrations. The main finding is that a strong single LLM with a good, demonstration-based prompt performs about as well as the best discussion setups. Discussion clearly helps mainly when no demonstrations are given. This tells practitioners when the extra cost of multi-agent setups is likely to pay off.

## Key points
- **Setup:** Bard runs via PaLM2 text-bison-001. Temperature is fixed at 0.25, discussions last at most 3 rounds, and accuracy is the metric. The multi-LLM CMD uses six agents split into two groups of three, with each group mixing ChatGPT-3.5, Gemini Pro and Bard.
- **With demonstrations:** average accuracies are close across setups:
  - single Gemini Pro: 78.59
  - ReConcile: 78.36
  - CMD: 78.66
- **Without demonstrations:** discussion shows a clear gain:
  - CMD: 76.93
  - ReConcile: 76.11
  - best single agent (Gemini Pro): 74.38
- **Typical discussion failures:** the authors identify two recurring error types, "judge mistake" and "wrong answer propagation."
- **Mixed-LLM discussions:** agents built on weaker models, such as Bard, improve over the rounds by interacting with stronger ones, such as Gemini Pro.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 209 works · 71,394 citations
- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 170 works · 26,341 citations
- [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) — University, Hangzhou, China · 224 works · 13,587 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong and Yangqiu Song. 2024. Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?. arXiv:2402.18272.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024rethinking,
  title = {{Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?}},
  author = {Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong and Yangqiu Song},
  year = {2024},
  doi = {10.48550/arxiv.2402.18272},
  eprint = {2402.18272},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.18272}
}
```

- [← previous Pixtral 12B](https://md.research.elamur.ai/free/en/papers/agrawal-2024-pixtral-12b/)
- [next → ChatGPT Is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://md.research.elamur.ai/free/en/papers/bian-2024-chatgpt-is-knowledgeable-but-inexperienced/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

55 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021. [Explanations for commonsenseqa: New dataset and models](https://md.research.elamur.ai/free/en/papers/aggarwal-2021-explanations-commonsenseqa-new-dataset-models/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confe…
2. Philip W Anderson. 1972. More is different: Broken symmetry and the nature of the hierarchical structure of science. Science, 177(4047):393–396. [find](https://scholar.google.com/scholar?q=Philip+W+Anderson%2E+1972%2E+More+is+different%3A+Broken+symmetry+and+the+nature+of+the+hierarchical+structure+of+science%2E+Science%2C+177%284047%29%3A393%E2%80%93396%2E)
3. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403.
4. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. 2023. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687. [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Nils+Blach%2C+Ales+Kubicek%2C+Robert+Gerstenberger%2C+Lukas+Gianinazzi%2C+Joanna+Gajda%2C+Tomasz+Lehmann%2C+Michal+Podstawski%2C+Hubert+Niewiadomski%2C+Piotr+Nyczyk%2C+et+al%2E+2023%2E+Graph+of+thoughts%3A+Solv)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
6. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. [Chateval: Towards better llm-based evaluators through multi-agent debate](https://md.research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/). arXiv preprint arXiv:2308.07201.
7. Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. 2023a. [Reconcile: Round-table conference improves reasoning via consensus among diverse llms](https://md.research.elamur.ai/free/en/papers/chen-2024-reconcile-round-table-conference-improves/). arXiv preprint arXiv:2309.13007.
8. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023b. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023b%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2E)
9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
11. Constantinos Daskalakis and Seth Matthew Weinberg. 2012. Symmetries and optimal multi-dimensional mechanism design. In Proceedings of the 13th ACM conference on Electronic commerce, pages 370–387. [find](https://scholar.google.com/scholar?q=Constantinos+Daskalakis+and+Seth+Matthew+Weinberg%2E+2012%2E+Symmetries+and+optimal+multi%2Ddimensional+mechanism+design%2E+In+Proceedings+of+the+13th+ACM+conference+on+Electronic+commerce%2C+pages+370%E2%80%93387%2E)
12. Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246. [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Pengcheng+Wang%2C+Yong+Lin%2C+and+Tong+Zhang%2E+2023%2E+Active+prompting+with+chain%2Dof%2Dthought+for+large+language+models%2E+arXiv+preprint+arXiv%3A2302%2E12246%2E)
13. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. [Improving factuality and reasoning in language models through multiagent debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). arXiv preprint arXiv:2305.14325.
14. Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Successive prompting for decomposing complex questions. arXiv preprint arXiv:2212.04092. [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Shivanshu+Gupta%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2022%2E+Successive+prompting+for+decomposing+complex+questions%2E+arXiv+preprint+arXiv%3A2212%2E04092%2E)
15. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). arXiv preprint arXiv:2210.00720.
16. Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, et al. 2022. [Folio: Natural language reasoning with first-order logic](https://md.research.elamur.ai/free/en/papers/han-2022-folio-natural-language-reasoning-first/). arXiv preprint arXiv:2209.00840.
17. Shima Imani, Liang Du, and Harsh Shrivastava. 2023. [Mathprompter: Mathematical reasoning using large language models](https://md.research.elamur.ai/free/en/papers/imani-2023-mathprompter-mathematical-reasoning-large-language/). arXiv preprint arXiv:2303.05398.
18. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. [Decomposed prompting: A modular approach for solving complex tasks](https://md.research.elamur.ai/free/en/papers/khot-2022-decomposed-prompting-modular-approach/). arXiv preprint arXiv:2210.02406.
19. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199– 22213.
20. Jean-Jacques Laffont and David Martimort. 2000. Mechanism design with collusion and correlation. Econometrica, 68(2):309–342. [find](https://scholar.google.com/scholar?q=Jean%2DJacques+Laffont+and+David+Martimort%2E+2000%2E+Mechanism+design+with+collusion+and+correlation%2E+Econometrica%2C+68%282%29%3A309%E2%80%93342%2E)
21. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336. — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Zeqi+Lin%2C+Shizhuo+Zhang%2C+Qiang+Fu%2C+Bei+Chen%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2022%2E+On+the+advance+of+making+language+models+better+reasoners%2E+arXiv+preprint+arXiv%3A2206%2E02336%2E+%E2%80%94+%5BarXiv%3A2206%2E)
22. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. [Encouraging divergent thinking in large language models through multi-agent debate](https://md.research.elamur.ai/free/en/papers/liang-2023-encouraging-divergent-thinking-large-language/). arXiv preprint arXiv:2305.19118.
23. Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. 2023. [Deductive verification of chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/ling-2023-deductive-verification-chain-thought-reasoning/). arXiv preprint arXiv:2306.03872.
24. Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te-Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M Dai. 2022. Mind’s eye: Grounded language model reasoning through simulation. arXiv preprint arXiv:2210.05359. [find](https://scholar.google.com/scholar?q=Ruibo+Liu%2C+Jason+Wei%2C+Shixiang+Shane+Gu%2C+Te%2DYen+Wu%2C+Soroush+Vosoughi%2C+Claire+Cui%2C+Denny+Zhou%2C+and+Andrew+M+Dai%2E+2022%2E+Mind%E2%80%99s+eye%3A+Grounded+language+model+reasoning+through+simulation%2E+arXiv+preprint+a)
25. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. [Chameleon: Plug-and-play compositional reasoning with large language models](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/). arXiv preprint arXiv:2304.09842.
26. Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.06009. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Peter+Clark%2C+and+Yiming+Yang%2E+2022%2E+Memory%2Dassisted+prompt+editing+to+improve+gpt%2D3+after+deployment%2E+arXiv+preprint+arXiv%3A2201%2E06009%2E)
27. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+2023%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Df)
28. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) arXiv preprint arXiv:2202.12837.
29. Marvin Minsky. 1988. Society of mind. Simon and Schuster. [find](https://scholar.google.com/scholar?q=Marvin+Minsky%2E+1988%2E+Society+of+mind%2E+Simon+and+Schuster%2E)
30. OpenAI. 2022. Chatgpt. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
31. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
32. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Muru+Zhang%2C+Sewon+Min%2C+Ludwig+Schmidt%2C+Noah+A+Smith%2C+and+Mike+Lewis%2E+2022%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2E+arXiv+preprint+arXiv%3A2210%2E03350%2E)
33. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
34. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). In Thirty-seventh Conference on Neural Information Processing Systems.
35. Kristopher Tapp. 2021. Symmetry. Springer. [find](https://scholar.google.com/scholar?q=Kristopher+Tapp%2E+2021%2E+Symmetry%2E+Springer%2E)
36. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
37. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
39. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091. [find](https://scholar.google.com/scholar?q=Lei+Wang%2C+Wanyu+Xu%2C+Yihuai+Lan%2C+Zhiqiang+Hu%2C+Yunshi+Lan%2C+Roy+Ka%2DWei+Lee%2C+and+Ee%2DPeng+Lim%2E+2023%2E+Plan%2Dand%2Dsolve+prompting%3A+Improving+zero%2Dshot+chain%2Dof%2Dthought+reasoning+by+large+language+models%2E+arXiv)
40. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
41. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
42. Lilian Weng. 2023. Llm-powered autonomous agents. lilianweng.github.io. [find](https://scholar.google.com/scholar?q=Lilian+Weng%2E+2023%2E+Llm%2Dpowered+autonomous+agents%2E+lilianweng%2Egithub%2Eio%2E)
43. Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. 2022. [Large language models are reasoners with self-verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/). arXiv preprint arXiv:2212.09561.
44. Zhiheng Xi, Senjie Jin, Yuhao Zhou, Rui Zheng, Songyang Gao, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Self-polish: Enhance reasoning in large language models via problem refinement. arXiv preprint arXiv:2305.14497. [find](https://scholar.google.com/scholar?q=Zhiheng+Xi%2C+Senjie+Jin%2C+Yuhao+Zhou%2C+Rui+Zheng%2C+Songyang+Gao%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+2023%2E+Self%2Dpolish%3A+Enhance+reasoning+in+large+language+models+via+problem+refinement%2E+arXiv+preprint)
45. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. 2023a. [Are large language models really good logical reasoners?](https://md.research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/) a comprehensive evaluation from deductive, inductive and abductive views. arXiv preprint arXiv:2306.09841.
46. Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, and Jian-guang Lou. 2023b. Re-reading improves reasoning in language models. arXiv preprint arXiv:2309.06275. — [arXiv:2309.06275](https://arxiv.org/abs/2309.06275) [find](https://scholar.google.com/scholar?q=Xiaohan+Xu%2C+Chongyang+Tao%2C+Tao+Shen%2C+Can+Xu%2C+Hongbo+Xu%2C+Guodong+Long%2C+and+Jian%2Dguang+Lou%2E+2023b%2E+Re%2Dreading+improves+reasoning+in+language+models%2E+arXiv+preprint+arXiv%3A2309%2E06275%2E+%E2%80%94+%5BarXiv%3A2309%2E06275%5D)
47. Tianci Xue, Ziqi Wang, Zhenhailong Wang, Chi Han, Pengfei Yu, and Heng Ji. 2023. Rcot: Detecting and rectifying factual inconsistency in reasoning by reversing chain-of-thought. arXiv preprint arXiv:2305.11499. — [arXiv:2305.11499](https://arxiv.org/abs/2305.11499) [find](https://scholar.google.com/scholar?q=Tianci+Xue%2C+Ziqi+Wang%2C+Zhenhailong+Wang%2C+Chi+Han%2C+Pengfei+Yu%2C+and+Heng+Ji%2E+2023%2E+Rcot%3A+Detecting+and+rectifying+factual+inconsistency+in+reasoning+by+reversing+chain%2Dof%2Dthought%2E+arXiv+preprint+arXiv%3A2)
48. Zhicheng Yang, Jinghui Qin, Jiaqi Chen, Liang Lin, and Xiaodan Liang. 2022. Logicsolver: Towards interpretable math word problem solving with logical prompt-enhanced learning. arXiv preprint arXiv:2205.08232. [find](https://scholar.google.com/scholar?q=Zhicheng+Yang%2C+Jinghui+Qin%2C+Jiaqi+Chen%2C+Liang+Lin%2C+and+Xiaodan+Liang%2E+2022%2E+Logicsolver%3A+Towards+interpretable+math+word+problem+solving+with+logical+prompt%2Denhanced+learning%2E+arXiv+preprint+arXiv%3A220)
49. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023a%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+arXiv+preprint+a)
50. Yao Yao, Zuchao Li, and Hai Zhao. 2023b. [Beyond chain-of-thought, effective graph-of-thought reasoning in large language models](https://md.research.elamur.ai/free/en/papers/yao-2023-chain-thought-effective-graph-thought/). arXiv preprint arXiv:2305.16582.
51. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488.
52. Jintian Zhang, Xin Xu, and Shumin Deng. 2023a. [Exploring collaboration mechanisms for llm agents: A social psychology view](https://md.research.elamur.ai/free/en/papers/zhang-2024-exploring-collaboration-mechanisms-llm-agents/). arXiv preprint arXiv:2310.02124.
53. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022a. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv e-prints, pages arXiv–2205.
54. Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. 2023b. [Cumulative reasoning with large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-cumulative-reasoning-large-language-models/). arXiv preprint arXiv:2308.04371.
55. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. [Automatic chain of thought prompting in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/). arXiv preprint arXiv:2210.03493.

*Updated 2 November 2026*