---
title: "AFlow: Automating Agentic Workflow Generation (2024)"
description: "AFlow treats workflow design as a search over programs. 59 references, cited by 88 papers here."
authors: ["Jiayi Zhang", "Jinyu Xiang", "Zhaoyang Yu", "Fengwei Teng", "Xionghui Chen", "Jiaqi Chen", "Mingchen Zhuge", "Xin Cheng", "Sirui Hong", "Jinlin Wang", "Bingnan Zheng", "Bang Liu", "Yuyu Luo", "Chenglin Wu"]
year: 2024
venue: "arXiv:2410.10762"
source: "https://arxiv.org/abs/2410.10762"
doi: "10.48550/arxiv.2410.10762"
arxiv: "2410.10762"
licence: "arXiv non-exclusive distribution license"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-aflow-automating-agentic-workflow/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-aflow-automating-agentic-workflow/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / AFlow: Automating Agentic Workflow Generation

# AFlow: Automating Agentic Workflow Generation

**Authors:** [Jiayi Zhang](https://md.research.elamur.ai/free/en/authors/jiayi-zhang/), Jinyu Xiang, [Zhaoyang Yu](https://md.research.elamur.ai/free/en/authors/zhaoyang-yu/), Fengwei Teng, [Xionghui Chen](https://md.research.elamur.ai/free/en/authors/xionghui-chen/), [Jiaqi Chen](https://md.research.elamur.ai/free/en/authors/jiaqi-chen/), [Mingchen Zhuge](https://md.research.elamur.ai/free/en/authors/mingchen-zhuge/), [Xin Cheng](https://md.research.elamur.ai/free/en/authors/xin-cheng/), [Sirui Hong](https://md.research.elamur.ai/free/en/authors/sirui-hong/), [Jinlin Wang](https://md.research.elamur.ai/free/en/authors/jinlin-wang/), Bingnan Zheng, [Bang Liu](https://md.research.elamur.ai/free/en/authors/bang-liu/), [Yuyu Luo](https://md.research.elamur.ai/free/en/authors/yuyu-luo/) and Chenglin Wu · **Year:** 2024 · **Venue:** arXiv:2410.10762 · **Licence:** [arXiv non-exclusive distribution license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.10762) · [PDF](https://arxiv.org/pdf/2410.10762) · [DOI 10.48550/arxiv.2410.10762](https://doi.org/10.48550/arxiv.2410.10762) · [arXiv 2410.10762](https://arxiv.org/abs/2410.10762) · [OpenAlex](https://openalex.org/W4403571374)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [88](https://md.research.elamur.ai/free/en/papers/zhang-2024-aflow-automating-agentic-workflow/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Agentic workflows for LLMs are structured chains of model calls with instructions, and building them usually takes a lot of manual engineering. Earlier automation attempts have gaps. Some still need a hand-built starting workflow, some cannot express varied structures, and some search inefficiently, such as ADAS with its linear heuristic search. AFlow treats workflow design as a search over programs. Each workflow is code in which LLM-invoking nodes are connected by code-defined edges. A node is set by its model, prompt, temperature and output format, and edges can express ordering, conditions and dependencies. Reusable operators such as Ensemble or Review & Revise package common node patterns, which makes the search easier. A customized Monte Carlo Tree Search drives the optimization. It selects candidates with soft mixed-probability selection, has an LLM edit the workflow code, runs and scores the result, and feeds that experience back up the tree. On six benchmarks covering QA, coding and math, the discovered workflows beat both hand-designed and automated baselines. They also let cheaper models outperform GPT-4o at a fraction of its cost, reducing the human effort needed to adapt LLMs to new tasks.

## Key points
- Evaluation uses HumanEval and MBPP (pass@1), MATH and GSM8K (solve rate), and HotpotQA and DROP (F1).
- AFlow averages 5.7% better than manually designed state-of-the-art methods and 19.5% better than existing automated approaches such as ADAS.
- With AFlow workflows, smaller models beat GPT-4o on specific tasks at about 4.55% of its dollar inference cost.
- The search loop combines MCTS with LLM-driven code expansion, execution feedback and tree-structured experience, and an appendix discusses search-space completeness and convergence.
- Code is released at github.com/FoundationAgents/AFlow, and an appendix case shows a research-idea-generation workflow whose best version was reached after six iterations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations
- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 87 works · 91,597 citations
- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 170 works · 26,341 citations
- [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 139 works · 17,752 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 99 works · 8,510 citations
- [The Hong Kong University of Science and Technology (Guangzhou)](https://md.research.elamur.ai/free/en/universities/the-hong-kong-university-of-science-and-technology-guangzhou/) — University, Guangzhou, China · 67 works · 1,765 citations
- [King Abdullah University of Science and Technology](https://md.research.elamur.ai/free/en/universities/king-abdullah-university-of-science-and-technology/) — University, Jeddah, Saudi Arabia · 30 works · 1,270 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 52 works

## Cite

Cite as: Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo and Chenglin Wu. 2024. AFlow: Automating Agentic Workflow Generation. arXiv:2410.10762.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2024aflow,
  title = {{AFlow: Automating Agentic Workflow Generation}},
  author = {Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo and Chenglin Wu},
  year = {2024},
  doi = {10.48550/arxiv.2410.10762},
  eprint = {2410.10762},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.10762}
}
```

- [← previous A Survey on Data Selection for Language Models](https://md.research.elamur.ai/free/en/papers/albalak-2024-survey-data-selection-language-models/)
- [next → AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://md.research.elamur.ai/free/en/papers/zhan-2024-anygpt-unified-multimodal-llm-discrete/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

59 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. Introducing claude 3.5 sonnet. claude-3-5-sonnet, 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Introducing+claude+3%2E5+sonnet%2E+claude%2D3%2D5%2Dsonnet%2C+2024%2E)
2. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). CoRR, abs/2108.07732, 2021.
3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikha… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
5. Yanqi Dai, Huanran Hu, Lei Wang, Shengjie Jin, Xu Chen, and Zhiwu Lu. Mmrole: A comprehensive framework for developing and evaluating multimodal role-playing agents. arXiv preprint arXiv:2408.04203, 2024. — [arXiv:2408.04203](https://arxiv.org/abs/2408.04203) [find](https://scholar.google.com/scholar?q=Yanqi+Dai%2C+Huanran+Hu%2C+Lei+Wang%2C+Shengjie+Jin%2C+Xu+Chen%2C+and+Zhiwu+Lu%2E+Mmrole%3A+A+comprehensive+framework+for+developing+and+evaluating+multimodal+role%2Dplaying+agents%2E+arXiv+preprint+arXiv%3A2408%2E04203%2C+2)
6. Deepseek. DeepSeek-V2.5. [https://huggingface.co/deepseek-ai/DeepSeek-V2.5](https://huggingface.co/deepseek-ai/DeepSeek-V2.5), 2024. [↗](https://huggingface.co/deepseek-ai/DeepSeek-V2.5)
7. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In NAACL-HLT (1), pp. 2368–2378. Association for Computational Linguistics, 2019.
8. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. [Promptbreeder: Self-referential self-improvement via prompt evolution](https://md.research.elamur.ai/free/en/papers/fernando-2023-promptbreeder-self-referential-self-improvement/). In ICML. OpenReview.net, 2024.
9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
10. Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Wenyi Wang, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zongze Xu, an… — [Data Interpreter: An LLM Agent for Data Science](https://md.research.elamur.ai/free/en/papers/hong-2025-data-interpreter-llm-agent-data/)
11. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. [Metagpt: Meta programming for A multi-agent collaborative framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/). In ICLR. Open…
12. Shengran Hu, Cong Lu, and Jeff Clune. [Automated design of agentic systems](https://md.research.elamur.ai/free/en/papers/hu-2024-automated-design-agentic-systems/). arXiv preprint arXiv:2408.08435, 2024.
13. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into state-ofthe-art pipelines. In The… [find](https://scholar.google.com/scholar?q=Omar+Khattab%2C+Arnav+Singhvi%2C+Paridhi+Maheshwari%2C+Zhiyuan+Zhang%2C+Keshav+Santhanam%2C+Sri+Vardhamanan%2C+Saiful+Haq%2C+Ashutosh+Sharma%2C+Thomas+T%2E+Joshi%2C+Hanna+Moazam%2C+Heather+Miller%2C+Matei+Zaharia%2C+and+Christ)
14. Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. The dawn of natural language to SQL: are we fully ready? Proc. VLDB Endow., 17(11):3318–3331, 2024a. [find](https://scholar.google.com/scholar?q=Boyan+Li%2C+Yuyu+Luo%2C+Chengliang+Chai%2C+Guoliang+Li%2C+and+Nan+Tang%2E+The+dawn+of+natural+language+to+SQL%3A+are+we+fully+ready%3F+Proc%2E+VLDB+Endow%2E%2C+17%2811%29%3A3318%E2%80%933331%2C+2024a%2E)
15. Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. Autoflow: Automated workflow generation for large language model agents. CoRR, abs/2407.12821, 2024b. — [arXiv:2407.12821](https://arxiv.org/abs/2407.12821) [find](https://scholar.google.com/scholar?q=Zelong+Li%2C+Shuyuan+Xu%2C+Kai+Mei%2C+Wenyue+Hua%2C+Balaji+Rama%2C+Om+Raheja%2C+Hao+Wang%2C+He+Zhu%2C+and+Yongfeng+Zhang%2E+Autoflow%3A+Automated+workflow+generation+for+large+language+model+agents%2E+CoRR%2C+abs%2F2407%2E12821%2C)
16. Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuyu Luo, Yuxin Zhang, Ju Fan, Guoliang Li, and Nan Tang. A survey of NL2SQL with large language models: Where are we, and where are we going? CoRR, abs/2408.05109, 2024. — [arXiv:2408.05109](https://arxiv.org/abs/2408.05109) [find](https://scholar.google.com/scholar?q=Xinyu+Liu%2C+Shuyu+Shen%2C+Boyan+Li%2C+Peixian+Ma%2C+Runzhi+Jiang%2C+Yuyu+Luo%2C+Yuxin+Zhang%2C+Ju+Fan%2C+Guoliang+Li%2C+and+Nan+Tang%2E+A+survey+of+NL2SQL+with+large+language+models%3A+Where+are+we%2C+and+where+are+we+going)
17. Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.[02170, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-dynamic-llm-powered-agent-network/).
18. Yuyu Luo, Xuedi Qin, Nan Tang, and Guoliang Li. Deepeye: Towards automatic data visualization. In ICDE, pp. 101–112. IEEE Computer Society, 2018. [find](https://scholar.google.com/scholar?q=Yuyu+Luo%2C+Xuedi+Qin%2C+Nan+Tang%2C+and+Guoliang+Li%2E+Deepeye%3A+Towards+automatic+data+visualization%2E+In+ICDE%2C+pp%2E+101%E2%80%93112%2E+IEEE+Computer+Society%2C+2018%2E)
19. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedbac)
20. Boye Niu, Yiliao Song, Kai Lian, Yifan Shen, Yu Yao, Kun Zhang, and Tongliang Liu. Flow: Modularized agentic workflow automation, 2025. URL https://arxiv.org/abs/2501.07834. — [arXiv:2501.07834](https://arxiv.org/abs/2501.07834) [↗](https://arxiv.org/abs/2501.07834)
21. Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolò Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, and Eric Horvitz. Can generalist foundation models o… [find](https://scholar.google.com/scholar?q=Harsha+Nori%2C+Yin+Tat+Lee%2C+Sheng+Zhang%2C+Dean+Carignan%2C+Richard+Edgar%2C+Nicolo%CC%80+Fusi%2C+Nicholas+King%2C+Jonathan+Larson%2C+Yuanzhi+Li%2C+Weishung+Liu%2C+Renqian+Luo%2C+Scott+Mayer+McKinney%2C+Robert+Osazuwa+Ness%2C+Hoi)
22. OpenAI. Hello gpt-4o. [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/), 2024a. [↗](https://openai.com/index/hello-gpt-4o/)
23. OpenAI. GPT-4o mini: Advancing cost-efficient intelligence. [https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/), 2024b. [↗](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)
24. Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Benchmarking agentic workflow generation, 2025. URL https://arxiv.org/abs/2410.07869. — [arXiv:2410.07869](https://arxiv.org/abs/2410.07869) [↗](https://arxiv.org/abs/2410.07869)
25. Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. CoRR, abs/2401.08500, 2024. [find](https://scholar.google.com/scholar?q=Tal+Ridnik%2C+Dedy+Kredo%2C+and+Itamar+Friedman%2E+Code+generation+with+alphacodium%3A+From+prompt+engineering+to+flow+engineering%2E+CoRR%2C+abs%2F2401%2E08500%2C+2024%2E)
26. Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E. Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, and Azalia Mirhoseini. Archon: An architecture search framework for inference-time techniques. arXiv preprint arXiv:2409.15254, 2024. — [arXiv:2409.15254](https://arxiv.org/abs/2409.15254) [find](https://scholar.google.com/scholar?q=Jon+Saad%2DFalcon%2C+Adrian+Gamarra+Lafuente%2C+Shlok+Natarajan%2C+Nahum+Maru%2C+Hristo+Todorov%2C+Etash+Guha%2C+E%2E+Kelly+Buchanan%2C+Mayee+Chen%2C+Neel+Guha%2C+Christopher+Re%CC%81%2C+and+Azalia+Mirhoseini%2E+Archon%3A+An+architec)
27. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. [Reflexion: language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). In NeurIPS, 2023.
28. Chan Hee Song, Brian M Sadler, Jiaman Wu, Wei-Lun Chao, Clayton Washington, and Yu Su. [Llm-planner: Few-shot grounded planning for embodied agents with large language models](https://md.research.elamur.ai/free/en/papers/song-2022-llm-planner-few-shot-grounded/). In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2986–2997.
29. IEEE Computer Society, 2023. [find](https://scholar.google.com/scholar?q=IEEE+Computer+Society%2C+2023%2E)
30. Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang, Ji-Rong Wen, and Rui Yan. From indeterminacy to determinacy: Augmenting logical reasoning capabilities with large language models. arXiv preprint arXiv:2310.18659, 2023. — [arXiv:2310.18659](https://arxiv.org/abs/2310.18659) [find](https://scholar.google.com/scholar?q=Hongda+Sun%2C+Weikai+Xu%2C+Wei+Liu%2C+Jian+Luan%2C+Bin+Wang%2C+Shuo+Shang%2C+Ji%2DRong+Wen%2C+and+Rui+Yan%2E+From+indeterminacy+to+determinacy%3A+Augmenting+logical+reasoning+capabilities+with+large+language+models%2E+arXi)
31. Yiyou Sun, Junjie Hu, Wei Cheng, and Haifeng Chen. Chatbot meets pipeline: Augment large language model with definite finite automaton. arXiv preprint arXiv:2402.04411, 2024. — [arXiv:2402.04411](https://arxiv.org/abs/2402.04411) [find](https://scholar.google.com/scholar?q=Yiyou+Sun%2C+Junjie+Hu%2C+Wei+Cheng%2C+and+Haifeng+Chen%2E+Chatbot+meets+pipeline%3A+Augment+large+language+model+with+definite+finite+automaton%2E+arXiv+preprint+arXiv%3A2402%2E04411%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E04411%5D%28http)
32. Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. Let me speak freely? A study on the impact of format restrictions on performance of large language models. CoRR, abs/2408.02442, 2024. — [arXiv:2408.02442](https://arxiv.org/abs/2408.02442) [find](https://scholar.google.com/scholar?q=Zhi+Rui+Tam%2C+Cheng%2DKuang+Wu%2C+Yi%2DLin+Tsai%2C+Chieh%2DYen+Lin%2C+Hung%2Dyi+Lee%2C+and+Yun%2DNung+Chen%2E+Let+me+speak+freely%3F+A+study+on+the+impact+of+format+restrictions+on+performance+of+large+language+models%2E+CoRR)
33. Nan Tang, Chenyu Yang, Ju Fan, Lei Cao, Yuyu Luo, and Alon Y. Halevy. Verifai: Verified generative AI. In CIDR. www.cidrdb.org, 2024. [find](https://scholar.google.com/scholar?q=Nan+Tang%2C+Chenyu+Yang%2C+Ju+Fan%2C+Lei+Cao%2C+Yuyu+Luo%2C+and+Alon+Y%2E+Halevy%2E+Verifai%3A+Verified+generative+AI%2E+In+CIDR%2E+www%2Ecidrdb%2Eorg%2C+2024%2E)
34. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. [find](https://scholar.google.com/scholar?q=Guanzhi+Wang%2C+Yuqi+Xie%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Chaowei+Xiao%2C+Yuke+Zhu%2C+Linxi+Fan%2C+and+Anima+Anandkumar%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+arXiv+preprint+arXiv%3A23)
35. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, 2022.
36. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. [Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration](https://md.research.elamur.ai/free/en/papers/wang-2023-unleashing-emergent-cognitive-synergy-large/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Co…
37. Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, et al. [Chain-of-table: Evolving tables in the reasoning chain for table understanding](https://md.research.elamur.ai/free/en/papers/wang-2024-chain-table-evolving-tables-reasoning/). In The Twelfth International Conference on Learning Repres…
38. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
39. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. [Travelplanner: A benchmark for real-world planning with language agents](https://md.research.elamur.ai/free/en/papers/xie-2024-travelplanner-benchmark-real-world-planning/). In Forty-first International Conference on Machine Learning, 2024a.
40. Yupeng Xie, Yuyu Luo, Guoliang Li, and Nan Tang. Haichart: Human and AI paired visualization system. Proc. VLDB Endow., 17(11):3178–3191, 2024b. [find](https://scholar.google.com/scholar?q=Yupeng+Xie%2C+Yuyu+Luo%2C+Guoliang+Li%2C+and+Nan+Tang%2E+Haichart%3A+Human+and+AI+paired+visualization+system%2E+Proc%2E+VLDB+Endow%2E%2C+17%2811%29%3A3178%E2%80%933191%2C+2024b%2E)
41. Yiheng Xu, SU Hongjin, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, et al. Lemur: Harmonizing natural language and code for language agents. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Yiheng+Xu%2C+SU+Hongjin%2C+Chen+Xing%2C+Boyu+Mi%2C+Qian+Liu%2C+Weijia+Shi%2C+Binyuan+Hui%2C+Fan+Zhou%2C+Yitao+Liu%2C+Tianbao+Xie%2C+et+al%2E+Lemur%3A+Harmonizing+natural+language+and+code+for+language+agents%2E+In+The+Twelfth+)
42. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. [Large language models as optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/). In ICLR. OpenReview.net, 2024a.
43. Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonzalez, and Bin Cui. [Buffer of thoughts: Thought-augmented reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yang-2024-buffer-thoughts-thought-augmented-reasoning/). arXiv preprint arXiv:2406.04271, 2024b.
44. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In EMNLP, pp. 2369–2380. Association for Computational Linguistics, 2018.
45. Yilin Ye, Jianing Hao, Yihan Hou, Zhan Wang, Shishi Xiao, Yuyu Luo, and Wei Zeng. Generative AI for visualization: State of the art and future directions. Vis. Informatics, 8(1):43–66, 2024. [find](https://scholar.google.com/scholar?q=Yilin+Ye%2C+Jianing+Hao%2C+Yihan+Hou%2C+Zhan+Wang%2C+Shishi+Xiao%2C+Yuyu+Luo%2C+and+Wei+Zeng%2E+Generative+AI+for+visualization%3A+State+of+the+art+and+future+directions%2E+Vis%2E+Informatics%2C+8%281%29%3A43%E2%80%9366%2C+2024%2E)
46. Junchi Yu, Ran He, and Zhitao Ying. Thought propagation: An analogical approach to complex reasoning with large language models. In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Junchi+Yu%2C+Ran+He%2C+and+Zhitao+Ying%2E+Thought+propagation%3A+An+analogical+approach+to+complex+reasoning+with+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+20)
47. Mert Yüksekgönül, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic ”differentiation” via text. CoRR, abs/2406.[07496, 2024](https://md.research.elamur.ai/free/en/papers/yuksekgonul-2024-textgrad-automatic-differentiation-text/).
48. Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for llm-based multi-agent systems. arXiv preprint arXiv:2410.02506, 2024a. — [arXiv:2410.02506](https://arxiv.org/abs/2410.02506) [find](https://scholar.google.com/scholar?q=Guibin+Zhang%2C+Yanwei+Yue%2C+Zhixun+Li%2C+Sukwon+Yun%2C+Guancheng+Wan%2C+Kun+Wang%2C+Dawei+Cheng%2C+Jeffrey+Xu+Yu%2C+and+Tianlong+Chen%2E+Cut+the+crap%3A+An+economical+communication+pipeline+for+llm%2Dbased+multi%2Dagent+sy)
49. Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, and Dawei Cheng. G-designer: Architecting multi-agent communication topologies via graph neural networks. arXiv preprint arXiv:2410.11782, 2024b. — [arXiv:2410.11782](https://arxiv.org/abs/2410.11782) [find](https://scholar.google.com/scholar?q=Guibin+Zhang%2C+Yanwei+Yue%2C+Xiangguo+Sun%2C+Guancheng+Wan%2C+Miao+Yu%2C+Junfeng+Fang%2C+Kun+Wang%2C+and+Dawei+Cheng%2E+G%2Ddesigner%3A+Architecting+multi%2Dagent+communication+topologies+via+graph+neural+networks%2E+arXiv+)
50. Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, and Jianping Fan. Mobileexperts: A dynamic tool-enabled agent team in mobile devices. CoRR, abs/2407.03913, 2024c. — [arXiv:2407.03913](https://arxiv.org/abs/2407.03913) [find](https://scholar.google.com/scholar?q=Jiayi+Zhang%2C+Chuang+Zhao%2C+Yihan+Zhao%2C+Zhaoyang+Yu%2C+Ming+He%2C+and+Jianping+Fan%2E+Mobileexperts%3A+A+dynamic+tool%2Denabled+agent+team+in+mobile+devices%2E+CoRR%2C+abs%2F2407%2E03913%2C+2024c%2E+%E2%80%94+%5BarXiv%3A2407%2E03913%5D%28http)
51. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
52. Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step by step. In ACL (Findings), pp. 851–870. Association for Computational Linguistics, 2024a. — [Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step](https://md.research.elamur.ai/free/en/papers/zhong-2024-debug-like-human-large-language/)
53. Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, Bo Du, and Dacheng Tao. Achieving¿ 97% on gsm8k: Deeply understanding the problems makes llms perfect reasoners. arXiv preprint arXiv:2404.14963, 2024b. — [arXiv:2404.14963](https://arxiv.org/abs/2404.14963) [find](https://scholar.google.com/scholar?q=Qihuang+Zhong%2C+Kang+Wang%2C+Ziyang+Xu%2C+Juhua+Liu%2C+Liang+Ding%2C+Bo+Du%2C+and+Dacheng+Tao%2E+Achieving%C2%BF+97%25+on+gsm8k%3A+Deeply+understanding+the+problems+makes+llms+perfect+reasoners%2E+arXiv+preprint+arXiv%3A2404%2E1)
54. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Forty-first International Conference on Machine Learning, 2024a. [find](https://scholar.google.com/scholar?q=Andy+Zhou%2C+Kai+Yan%2C+Michal+Shlapentokh%2DRothman%2C+Haohan+Wang%2C+and+Yu%2DXiong+Wang%2E+Language+agent+tree+search+unifies+reasoning%2C+acting%2C+and+planning+in+language+models%2E+In+Forty%2Dfirst+International+Conf)
55. Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, Huajun Chen, and Yuchen Eleanor Jiang. [Symbolic learning enables self-evolving agents](https://md.research.elamur.ai/free/en/papers/zhou-2024-symbolic-learning-enables-self-evolving/). CoRR, abs/2406.18532, 2024b.
56. Xuanhe Zhou, Guoliang Li, and Zhiyuan Liu. Llm as dba. arXiv preprint arXiv:2308.05481, 2023. — [arXiv:2308.05481](https://arxiv.org/abs/2308.05481) [find](https://scholar.google.com/scholar?q=Xuanhe+Zhou%2C+Guoliang+Li%2C+and+Zhiyuan+Liu%2E+Llm+as+dba%2E+arXiv+preprint+arXiv%3A2308%2E05481%2C+2023%2E+%E2%80%94+%5BarXiv%3A2308%2E05481%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2308%2E05481%29)
57. Yizhang Zhu, Shiyin Du, Boyan Li, Yuyu Luo, and Nan Tang. Are large language models good statisticians? In NeurIPS, 2024. [find](https://scholar.google.com/scholar?q=Yizhang+Zhu%2C+Shiyin+Du%2C+Boyan+Li%2C+Yuyu+Luo%2C+and+Nan+Tang%2E+Are+large+language+models+good+statisticians%3F+In+NeurIPS%2C+2024%2E)
58. Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. Mindstorms in natural language-based societies of mind. arXiv preprint arXiv:2305.17066, 2023. — [arXiv:2305.17066](https://arxiv.org/abs/2305.17066) [find](https://scholar.google.com/scholar?q=Mingchen+Zhuge%2C+Haozhe+Liu%2C+Francesco+Faccio%2C+Dylan+R+Ashley%2C+Ro%CC%81bert+Csorda%CC%81s%2C+Anand+Gopalakrishnan%2C+Abdullah+Hamdi%2C+Hasan+Abed+Al+Kader+Hammoud%2C+Vincent+Herrmann%2C+Kazuki+Irie%2C+et+al%2E+Mindstorms+in+n)
59. Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: [Language agents as optimizable graphs](https://md.research.elamur.ai/free/en/papers/zhuge-2024-language-agents-optimizable-graphs/). In Forty-first International Conference on Machine Learning, 2024.

*Updated 2 November 2026*