---
title: "More Agents Is All You Need — paper by Li et al. (2024)"
description: "The authors propose Agent Forest, a training-free method with two steps. 40 references, cited by 64 papers here."
authors: ["Junyou Li", "Qin Zhang", "Yangbin Yu", "Qiang Fu", "Deheng Ye"]
year: 2024
venue: "arXiv:2402.05120"
source: "https://arxiv.org/abs/2402.05120"
doi: "10.48550/arxiv.2402.05120"
arxiv: "2402.05120"
licence: "arXiv non-exclusive distribution license"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2024-more-agents-all-need/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2024-more-agents-all-need/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / More Agents Is All You Need

# More Agents Is All You Need

**Authors:** Junyou Li, [Qin Zhang](https://md.research.elamur.ai/free/en/authors/qin-zhang/), Yangbin Yu, [Qiang Fu](https://md.research.elamur.ai/free/en/authors/qiang-fu/) and [Deheng Ye](https://md.research.elamur.ai/free/en/authors/deheng-ye/) · **Year:** 2024 · **Venue:** arXiv:2402.05120 · **Licence:** [arXiv non-exclusive distribution license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.05120) · [PDF](https://arxiv.org/pdf/2402.05120) · [DOI 10.48550/arxiv.2402.05120](https://doi.org/10.48550/arxiv.2402.05120) · [arXiv 2402.05120](https://arxiv.org/abs/2402.05120) · [OpenAlex](https://openalex.org/W4391709247)

**Citations:** 21 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/li-2024-more-agents-all-need/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether LLM accuracy reliably improves when you run more copies of the same model and combine their outputs. Earlier work hinted at this but did not study it systematically. The authors propose Agent Forest, a training-free method with two steps. First, the same query is sent N times to one LLM, or to an LLM wrapped in another technique such as CoT or a multi-agent framework. Second, the final output is the sample most similar to the others: answer frequency is used for closed-form tasks and pairwise BLEU for open-ended ones. Tests cover arithmetic reasoning, general reasoning, and code generation with Llama2 and GPT models. Accuracy generally grows with ensemble size, and large ensembles of smaller models can match bigger models. The method also adds gains on top of more elaborate techniques. Gains are largest for harder tasks and weaker models. A controlled analysis of task difficulty leads to two refined variants, step-wise and hierarchical. The work suggests that simply adding samples can be a cheap alternative to complex prompting or collaboration designs, though it increases inference cost.

## Key points
- Benchmarks: GSM8K, MATH, MMLU, Chess (state tracking) and HumanEval. Backbones include Llama2-13B, Llama2-70B, GPT-3.5-Turbo and GPT-4. Code is published in the AgentForest GitHub repository.
- On GSM8K, Llama2-13B reaches Llama2-70B's single-run accuracy with about 15 samples. Llama2-70B (15 samples) and GPT-3.5-Turbo (20 samples) similarly approach their stronger counterparts.
- The authors split difficulty into three dimensions and report how gains behave along each:
  - **Inherent difficulty:** gains first rise, then fall.
  - **Number of reasoning steps:** gains grow as steps increase.
  - **Prior probability of the correct answer:** accuracy rises as this probability increases.
- Step-wise Agent Forest applies voting at each reasoning step. It yields 15%–42% gains, and the gains grow with difficulty.
- Hierarchical Agent Forest first solves a coarser subproblem and then the full one:
  - With GPT-3.5-Turbo alone, accuracy rises from 21% to 31%.
  - Using GPT-3.5-Turbo for the coarse step and GPT-4 for the final step raises accuracy from 35% to 47%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu and Deheng Ye. 2024. More Agents Is All You Need. arXiv:2402.05120.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2024more,
  title = {{More Agents Is All You Need}},
  author = {Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu and Deheng Ye},
  year = {2024},
  doi = {10.48550/arxiv.2402.05120},
  eprint = {2402.05120},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.05120}
}
```

- [← previous LLMs cannot find reasoning errors, but can correct them given the error location](https://md.research.elamur.ai/free/en/papers/tyen-2024-llms-cannot-find-reasoning-errors/)
- [next → On Prompt-Driven Safeguarding for Large Language Models](https://md.research.elamur.ai/free/en/papers/zheng-2024-prompt-driven-safeguarding-large-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

40 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Leo Breiman. Random forests. Machine learning, 45:5–32, 2001. [find](https://scholar.google.com/scholar?q=Leo+Breiman%2E+Random+forests%2E+Machine+learning%2C+45%3A5%E2%80%9332%2C+2001%2E)
2. Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F. Karlsson, Jie Fu, and Yemin Shi. [Autoagents: A framework for automatic agent generation](https://md.research.elamur.ai/free/en/papers/chen-2023-autoagents-framework-automatic-agent-generation/). CoRR, abs/2309.17288, 2023a.
3. Lingjiao Chen, Matei Zaharia, and James Zou. [Frugalgpt: How to use large language models while reducing cost and improving performance](https://md.research.elamur.ai/free/en/papers/chen-2023-frugalgpt-use-large-language-models/). CoRR, abs/2305.05176, 2023b.
4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mik… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
5. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. [Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors](https://md.research.elamur.ai/free/en/papers/chen-2023-agentverse-facilitating-multi-agent/) in agents. CoRR, abs/2308.10848, 2023c.
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168, 2021a.
7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168, 2021b.
8. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. [Improving factuality and reasoning in language models through multiagent debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). CoRR, abs/2305.14325, 2023.
9. Jessica Ficler and Yoav Goldberg. [Controlling linguistic style aspects in neural language generation](https://md.research.elamur.ai/free/en/papers/ficler-2017-controlling-linguistic-style-aspects-neural/). CoRR, abs/1707.02633, 2017.
10. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Dataset…
13. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework. CoRR, abs/2308.[00352, 2023](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/).
14. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. [Llm-blender: Ensembling large language models with pairwise ranking and generative fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/). In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1…
15. Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. [Ai agents that matter](https://md.research.elamur.ai/free/en/papers/kapoor-2024-ai-agents-that-matter/). arXiv preprint arXiv:2407.01502, 2024.
16. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In NeurIPS, 2022.
17. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: communicative agents for "mind" exploration of large scale language model society. CoRR, abs/2303.[17760, 2023a](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/).
18. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. [Making language models better reasoners with step-aware verifier](https://md.research.elamur.ai/free/en/papers/li-2023-making-language-models-better-reasoners/). In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguis…
19. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. [Encouraging divergent thinking in large language models through multi-agent debate](https://md.research.elamur.ai/free/en/papers/liang-2023-encouraging-divergent-thinking-large-language/). CoRR, abs/2305.19118, 2023.
20. Lei Lin, Jia-Yi Fu, Pengli Liu, Junchen Wan, Fuzheng Zhang, Zhongyuan Wang, Di Zhang, and Kun Gai. Ask one more time: Self-agreement improves reasoning of language models in (almost) all scenarios. CoRR, abs/2311.08154, 2023. [find](https://scholar.google.com/scholar?q=Lei+Lin%2C+Jia%2DYi+Fu%2C+Pengli+Liu%2C+Junchen+Wan%2C+Fuzheng+Zhang%2C+Zhongyuan+Wang%2C+Di+Zhang%2C+and+Kun+Gai%2E+Ask+one+more+time%3A+Self%2Dagreement+improves+reasoning+of+language+models+in+%28almost%29+all+scenarios%2E+Co)
21. Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. CoRR, abs/2310.[02170, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-dynamic-llm-powered-agent-network/).
22. Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. [Routing to the expert: Efficient reward-guided ensemble of large language models](https://md.research.elamur.ai/free/en/papers/lu-2023-routing-expert-efficient-reward-guided/). CoRR, abs/2311.08692, 2023.
23. Xiaoding Lu, Adian Liusie, Vyas Raina, Yuwen Zhang, and William Beauchamp. Blending is all you need: Cheaper, better alternative to trillion-parameters llm. arXiv preprint arXiv:2401.02994, 2024. — [arXiv:2401.02994](https://arxiv.org/abs/2401.02994) [find](https://scholar.google.com/scholar?q=Xiaoding+Lu%2C+Adian+Liusie%2C+Vyas+Raina%2C+Yuwen+Zhang%2C+and+William+Beauchamp%2E+Blending+is+all+you+need%3A+Cheaper%2C+better+alternative+to+trillion%2Dparameters+llm%2E+arXiv+preprint+arXiv%3A2401%2E02994%2C+2024%2E+%E2%80%94+%5Ba)
24. OpenAI. Chatgpt: Optimizing language models for dialogue, 2022. URL [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40st Annual Meeting of the Association for Computational Linguistics ACL 2002, Philadelphia, USA, July, 2002, pp. 311–318. Association for Computational…
26. Matt Post. [A call for clarity in reporting bleu scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). arXiv preprint arXiv:1804.08771, 2018.
27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
28. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
29. Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. [Large language model routing with benchmark datasets](https://md.research.elamur.ai/free/en/papers/shnitzer-2023-large-language-model-routing-benchmark/). CoRR, abs/2309.15789, 2023.
30. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzho… — [LaMDA: Language Models for Dialog Applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
32. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. [Knowledge fusion of large language models](https://md.research.elamur.ai/free/en/papers/wan-2024-knowledge-fusion-large-language-models/). arXiv preprint arXiv:2401.10491, 2024.
33. Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric P. Xing, and Mikhail Yurochkin. Fusing models with complementary expertise. CoRR, abs/2310.01542, 2023a. — [arXiv:2310.01542](https://arxiv.org/abs/2310.01542) [find](https://scholar.google.com/scholar?q=Hongyi+Wang%2C+Felipe+Maia+Polo%2C+Yuekai+Sun%2C+Souvik+Kundu%2C+Eric+P%2E+Xing%2C+and+Mikhail+Yurochkin%2E+Fusing+models+with+complementary+expertise%2E+CoRR%2C+abs%2F2310%2E01542%2C+2023a%2E+%E2%80%94+%5BarXiv%3A2310%2E01542%5D%28https%3A%2F%2Farxi)
34. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
35. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. CoRR, abs/2307.[05300, 2023c](https://md.research.elamur.ai/free/en/papers/wang-2023-unleashing-emergent-cognitive-synergy-large/).
36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In NeurIPS, 2022.
37. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. [Autogen: Enabling next-gen LLM applications via multi-agent conversation](https://md.research.elamur.ai/free/en/papers/wu-2023-autogen-enabling-next-gen-llm/) framework. CoRR, abs/2308.08155, 2023.
38. Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining the inter-consistency of large language models: An in-depth analysis via debate. CoRR, abs/2305.11595, 2023. — [arXiv:2305.11595](https://arxiv.org/abs/2305.11595) [find](https://scholar.google.com/scholar?q=Kai+Xiong%2C+Xiao+Ding%2C+Yixin+Cao%2C+Ting+Liu%2C+and+Bing+Qin%2E+Examining+the+inter%2Dconsistency+of+large+language+models%3A+An+in%2Ddepth+analysis+via+debate%2E+CoRR%2C+abs%2F2305%2E11595%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E11595%5D%28htt)
39. Jintian Zhang, Xin Xu, and Shumin Deng. [Exploring collaboration mechanisms for llm agents: A social psychology view](https://md.research.elamur.ai/free/en/papers/zhang-2024-exploring-collaboration-mechanisms-llm-agents/). arXiv preprint arXiv:2310.02124, 2023.
40. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large l… — [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

*Updated 2 November 2026*