---
title: "Meta-Prompting — paper by Suzgun and Kalai (2024)"
description: "The authors propose meta-prompting, a zero-shot method that reuses one fixed, high-level prompt for… 75 references, cited by 44 papers here."
authors: ["Mirac Suzgun", "Adam Tauman Kalai"]
year: 2024
venue: "arXiv:2401.12954"
source: "https://arxiv.org/abs/2401.12954"
doi: "10.48550/arxiv.2401.12954"
arxiv: "2401.12954"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/suzgun-2024-meta-prompting-enhancing-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/suzgun-2024-meta-prompting-enhancing-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding

# Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding

**Authors:** [Mirac Suzgun](https://md.research.elamur.ai/free/en/authors/mirac-suzgun/) and [Adam Tauman Kalai](https://md.research.elamur.ai/free/en/authors/adam-tauman-kalai/) · **Year:** 2024 · **Venue:** arXiv:2401.12954 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.12954) · [PDF](https://arxiv.org/pdf/2401.12954) · [DOI 10.48550/arxiv.2401.12954](https://doi.org/10.48550/arxiv.2401.12954) · [arXiv 2401.12954](https://arxiv.org/abs/2401.12954) · [OpenAlex](https://openalex.org/W4391212860)

**Citations:** 18 citations (OpenAlex, 2026-10-14) · [44](https://md.research.elamur.ai/free/en/papers/suzgun-2024-meta-prompting-enhancing-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models like GPT-4 still make mistakes. Most scaffolding methods that try to fix this depend on hand-written instructions or examples for each task. The authors propose meta-prompting, a zero-shot method that reuses one fixed, high-level prompt for every task. Under this prompt, a single LM acts as a coordinating "Meta Model." It splits a problem into subtasks, writes targeted instructions for "expert" calls, collects their answers, and checks the result before giving a final response. The experts are new calls to the same LM. Each sees only the instructions written for it, not the whole conversation, so it judges the subtask with fresh eyes. The framework can also send work to a Python interpreter. The authors tested it with GPT-4 against several zero-shot baselines on puzzles, chess, programming and creative writing. With code execution, meta-prompting gave double-digit average gains. This shows that a general, task-independent wrapper can substantially improve a black-box model without users designing a prompt for each task.

## Key points
- The base model is GPT-4. Baselines are standard zero-shot (Std), zero-shot chain-of-thought (0-CoT), static and dynamic expert prompting (Ex-St, Ex-Dy), and multipersona prompting (MP).
- Benchmarks include:
  - Game of 24
  - Checkmate-in-One from BIG-Bench
  - Python Programming Puzzles
  - a new "Shakespearean Sonnet Writing" task created by the authors
- When paired with a Python interpreter, meta-prompting's average lead across tasks is 17.1% over standard prompting, 17.3% over dynamic expert prompting, and 15.2% over multipersona prompting.
- A typical session alternates three steps: injected guidance for the Meta Model, the Meta Model's output based on the full history, and expert replies based only on their own instructions. Verification by a second expert is encouraged before the final answer.
- Data, prompts and model outputs are released at github.com/suzgunmirac/meta-prompting.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Mirac Suzgun and Adam Tauman Kalai. 2024. Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding. arXiv:2401.12954.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{suzgun2024meta,
  title = {{Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding}},
  author = {Mirac Suzgun and Adam Tauman Kalai},
  year = {2024},
  doi = {10.48550/arxiv.2401.12954},
  eprint = {2401.12954},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.12954}
}
```

- [← previous Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens](https://md.research.elamur.ai/free/en/papers/liu-2024-infini-gram-scaling-unbounded-n/)
- [next → Multimodal Table Understanding](https://md.research.elamur.ai/free/en/papers/zheng-2024-multimodal-table-understanding/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

75 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gati V Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/). In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma B…
2. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403 (2023). https://arxiv.org/abs/2305.10403 [↗](https://arxiv.org/abs/2305.10403)
3. Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. 2023. Ask Me Anything: A simple strategy for prompting language models. In The Eleventh International Conference on Learning Representations. [https://openreview.net/forum?id=bhUPJnS2g0X](https://openreview.net/forum?id=bhUPJnS2g0X) [↗](https://openreview.net/forum?id=bhUPJnS2g0X)
4. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. 2023. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687 (2023). [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Nils+Blach%2C+Ales+Kubicek%2C+Robert+Gerstenberger%2C+Lukas+Gianinazzi%2C+Joanna+Gajda%2C+Tomasz+Lehmann%2C+Michal+Podstawski%2C+Hubert+Niewiadomski%2C+Piotr+Nyczyk%2C+et+al%2E+2023%2E+Graph+of+thoughts%3A+Solv)
5. BIG-Bench authors. 2023. [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Transactions on Machine Learning Research (2023). [https://openreview.net/forum?](https://openreview.net/forum?) id=uyTL5Bvosj Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2023. Large language m… [↗](https://openreview.net/forum?)
6. Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. 2023a. AutoAgents: A Framework for Automatic Agent Generation. arXiv preprint arXiv:2309.17288 (2023). [find](https://scholar.google.com/scholar?q=Guangyao+Chen%2C+Siwei+Dong%2C+Yu+Shu%2C+Ge+Zhang%2C+Jaward+Sesay%2C+B%C3%B6rje+F+Karlsson%2C+Jie+Fu%2C+and+Yemin+Shi%2E+2023a%2E+AutoAgents%3A+A+Framework+for+Automatic+Agent+Generation%2E+arXiv+preprint+arXiv%3A2309%2E17288+%282023)
7. Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2023b. [Iterative Translation Refinement with Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-iterative-translation-refinement-large-language/). arXiv preprint arXiv:2306.03856 (2023).
8. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023d. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. Transactions on Machine Learning Research (2023). [https://openreview.net/forum?id=YfZ4ZPt8zd](https://openreview.net/forum?id=YfZ4ZPt8zd) [↗](https://openreview.net/forum?id=YfZ4ZPt8zd)
9. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. 2023e. [Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors](https://md.research.elamur.ai/free/en/papers/chen-2023-agentverse-facilitating-multi-agent/) in agents. arXiv preprint arXiv:2308.10848 (2023).
10. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023c. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 (2023). [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023c%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128+%282023%29%2E)
11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168 (2021).
12. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. [Toxicity in chatgpt: Analyzing persona-assigned language models](https://md.research.elamur.ai/free/en/papers/deshpande-2023-toxicity-chatgpt-analyzing-persona-assigned/). arXiv preprint arXiv:2304.05335 (2023).
13. Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2023. Compositional Semantic Parsing with Large Language Models. In The Eleventh International Conference on Learning Representations. [https://openreview.net/forum?id=](https://openreview.net/forum?id=) gJW8hSG… [↗](https://openreview.net/forum?id=)
14. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). arXiv preprint arXiv:2305.14325 (2023).
15. Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. 2023a. [Improving language model negotiation with self-play and in-context learning from ai feedback](https://md.research.elamur.ai/free/en/papers/fu-2023-improving-language-model-negotiation-self/). arXiv preprint arXiv:2305.10142 (2023).
16. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023b. [Complexity-Based Prompting for Multi-step Reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). In The Eleventh International Conference on Learning Representations. [https://openreview.net/forum?id=yf1icZHC-l9](https://openreview.net/forum?id=yf1icZHC-l9) [↗](https://openreview.net/forum?id=yf1icZHC-l9)
17. Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. 2023. AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn. arXiv preprint arXiv:2306.08640 (2023). — [arXiv:2306.08640](https://arxiv.org/abs/2306.08640) [find](https://scholar.google.com/scholar?q=Difei+Gao%2C+Lei+Ji%2C+Luowei+Zhou%2C+Kevin+Qinghong+Lin%2C+Joya+Chen%2C+Zihan+Fan%2C+and+Mike+Zheng+Shou%2E+2023%2E+AssistGPT%3A+A+General+Multi%2Dmodal+Assistant+that+can+Plan%2C+Execute%2C+Inspect%2C+and+Learn%2E+arXiv+prepri)
18. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. [CRITIC: Large language models can self-correct with tool-interactive critiquing](https://md.research.elamur.ai/free/en/papers/gou-2023-critic-large-language-models-self/). arXiv preprint arXiv:2305.11738 (2023).
19. Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. 2022. Language Models Can Teach Themselves to Program Better. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Patrick+Haluptzok%2C+Matthew+Bowers%2C+and+Adam+Tauman+Kalai%2E+2022%2E+Language+Models+Can+Teach+Themselves+to+Program+Better%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2E)
20. Rui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yirui Zhang, and Liqiang Nie. 2023b. ChatLLM Network: More brains, More intelligence. arXiv preprint arXiv:2304.12998 (2023). — [arXiv:2304.12998](https://arxiv.org/abs/2304.12998) [find](https://scholar.google.com/scholar?q=Rui+Hao%2C+Linmei+Hu%2C+Weijian+Qi%2C+Qingliu+Wu%2C+Yirui+Zhang%2C+and+Liqiang+Nie%2E+2023b%2E+ChatLLM+Network%3A+More+brains%2C+More+intelligence%2E+arXiv+preprint+arXiv%3A2304%2E12998+%282023%29%2E+%E2%80%94+%5BarXiv%3A2304%2E12998%5D%28https%3A%2F%2Fa)
21. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023a. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992 (2023). [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haodi+Ma%2C+Joshua+Jiahua+Hong%2C+Zhen+Wang%2C+Daisy+Zhe+Wang%2C+and+Zhiting+Hu%2E+2023a%2E+Reasoning+with+language+model+is+planning+with+world+model%2E+arXiv+preprint+arXiv%3A2305%2E14992+%282023%29%2E)
22. Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023c. [ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://md.research.elamur.ai/free/en/papers/hao-2023-toolkengpt-augmenting-frozen-language-models/). arXiv preprint arXiv:2305.11554 (2023).
23. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2023. Inner Monologue: Embodied Reasoning through Planning with Language Models. In Conference on Robot Learning. PMLR, 1769–1782. [find](https://scholar.google.com/scholar?q=Wenlong+Huang%2C+Fei+Xia%2C+Ted+Xiao%2C+Harris+Chan%2C+Jacky+Liang%2C+Pete+Florence%2C+Andy+Zeng%2C+Jonathan+Tompson%2C+Igor+Mordatch%2C+Yevgen+Chebotar%2C+et+al%2E+2023%2E+Inner+Monologue%3A+Embodied+Reasoning+through+Plannin)
24. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023a. [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/). arXiv preprint arXiv:2306.02561 (2023).
25. Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023b. SelfEvolve: A Code Evolution Framework via Large Language Models. arXiv preprint arXiv:2306.02907 (2023). — [arXiv:2306.02907](https://arxiv.org/abs/2306.02907) [find](https://scholar.google.com/scholar?q=Shuyang+Jiang%2C+Yuhao+Wang%2C+and+Yu+Wang%2E+2023b%2E+SelfEvolve%3A+A+Code+Evolution+Framework+via+Large+Language+Models%2E+arXiv+preprint+arXiv%3A2306%2E02907+%282023%29%2E+%E2%80%94+%5BarXiv%3A2306%2E02907%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306)
26. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. 2023. [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://md.research.elamur.ai/free/en/papers/khattab-2023-dspy-compiling-declarative-language-model/). arXiv preprint arXiv:2310.03714 (2023).
27. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed Prompting: A Modular Approach for Solving Complex Tasks. In The Eleventh International Conference on Learning Representations. [https://openreview.net/forum?id=_nGgzQjzaRy](https://openreview.net/forum?id=_nGgzQjzaRy) [↗](https://openreview.net/forum?id=_nGgzQjzaRy)
28. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. [Language models can solve computer tasks](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/). arXiv preprint arXiv:2303.17491 (2023).
29. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large Language Models are Zero-Shot Reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In Advances in Neural Information Processing Systems, Vol. 35. 22199–22213.
30. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. [Camel: Communicative agents for" mind" exploration of large scale language model society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/). arXiv preprint arXiv:2303.17760 (2023).
31. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://md.research.elamur.ai/free/en/papers/liang-2023-encouraging-divergent-thinking-large-language/). arXiv preprint arXiv:2305.19118 (2023).
32. Zeyi Liu, Arpit Bahety, and Shuran Song. 2023a. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306.15724 (2023). — [arXiv:2306.15724](https://arxiv.org/abs/2306.15724) [find](https://scholar.google.com/scholar?q=Zeyi+Liu%2C+Arpit+Bahety%2C+and+Shuran+Song%2E+2023a%2E+Reflect%3A+Summarizing+robot+experiences+for+failure+explanation+and+correction%2E+arXiv+preprint+arXiv%3A2306%2E15724+%282023%29%2E+%E2%80%94+%5BarXiv%3A2306%2E15724%5D%28https%3A%2F%2Farxi)
33. Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2023b. Dynamic LLM-Agent Network: An LLM-agent Collaboration Framework with Agent Team Optimization. arXiv preprint arXiv:2310.02170 (2023). [find](https://scholar.google.com/scholar?q=Zijun+Liu%2C+Yanzhe+Zhang%2C+Peng+Li%2C+Yang+Liu%2C+and+Diyi+Yang%2E+2023b%2E+Dynamic+LLM%2DAgent+Network%3A+An+LLM%2Dagent+Collaboration+Framework+with+Agent+Team+Optimization%2E+arXiv+preprint+arXiv%3A2310%2E02170+%282023%29%2E)
34. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/). arXiv preprint arXiv:2304.09842 (2023).
35. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 (2023). [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+2023%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Df)
36. Aman Madaan and Amir Yazdanbakhsh. 2022. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.07686 (2022). [find](https://scholar.google.com/scholar?q=Aman+Madaan+and+Amir+Yazdanbakhsh%2E+2022%2E+Text+and+patterns%3A+For+effective+chain+of+thought%2C+it+takes+two+to+tango%2E+arXiv+preprint+arXiv%3A2209%2E07686+%282022%29%2E)
37. Masa. 2023. 7 Challenges and Potential Risks of AutoGPT Technology. [find](https://scholar.google.com/scholar?q=Masa%2E+2023%2E+7+Challenges+and+Potential+Risks+of+AutoGPT+Technology%2E)
38. AutoGPT Official (Apr 2023). [https://autogpt.net/challenges-and-potential-risks-of-autogpt-technology/](https://autogpt.net/challenges-and-potential-risks-of-autogpt-technology/) Xuefei Ning, Zinan Lin, Zixuan Zhou, Huazhong Yang, and Yu Wang. 2023. Skeleton-of-thought: Large language models can do parallel decoding. arXiv preprint arXiv:2307.15337 (2023). [↗](https://autogpt.net/challenges-and-potential-risks-of-autogpt-technology/)
39. OpenAI. 2023. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774 (2023). https://arxiv.org/abs/ 2303.08774 [↗](https://arxiv.org/abs/)
40. Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. ART: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014 (2023). [find](https://scholar.google.com/scholar?q=Bhargavi+Paranjape%2C+Scott+Lundberg%2C+Sameer+Singh%2C+Hannaneh+Hajishirzi%2C+Luke+Zettlemoyer%2C+and+Marco+Tulio+Ribeiro%2E+2023%2E+ART%3A+Automatic+multi%2Dstep+reasoning+and+tool%2Duse+for+large+language+models%2E+arXi)
41. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23) (San Francisco, CA, USA) (UIST ’… [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+C%2E+O%E2%80%99Brien%2C+Carrie+J%2E+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S%2E+Bernstein%2E+2023%2E+Generative+Agents%3A+Interactive+Simulacra+of+Human+Behavior%2E+In+In+the+36th+Annual)
42. Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2022. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–18. [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Lindsay+Popowski%2C+Carrie+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S+Bernstein%2E+2022%2E+Social+simulacra%3A+Creating+populated+prototypes+for+social+computing+systems%2E+In+Proce)
43. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. [Gorilla: Large language model connected with massive apis](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/). arXiv preprint arXiv:2305.15334 (2023).
44. Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. [Refiner: Reasoning feedback on intermediate representations](https://md.research.elamur.ai/free/en/papers/paul-2023-refiner-reasoning-feedback-intermediate-representations/). arXiv preprint arXiv:2304.01904 (2023).
45. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350 (2022). [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Muru+Zhang%2C+Sewon+Min%2C+Ludwig+Schmidt%2C+Noah+A+Smith%2C+and+Mike+Lewis%2E+2022%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2E+arXiv+preprint+arXiv%3A2210%2E03350+%282022%29%2E)
46. Shuofei Qiao, Honghao Gui, Huajun Chen, and Ningyu Zhang. 2023. Making Language Models Better Tool Learners with Execution Feedback. arXiv preprint arXiv:2305.13068 (2023). [find](https://scholar.google.com/scholar?q=Shuofei+Qiao%2C+Honghao+Gui%2C+Huajun+Chen%2C+and+Ningyu+Zhang%2E+2023%2E+Making+Language+Models+Better+Tool+Learners+with+Execution+Feedback%2E+arXiv+preprint+arXiv%3A2305%2E13068+%282023%29%2E)
47. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. [Self-critiquing models for assisting human evaluators](https://md.research.elamur.ai/free/en/papers/saunders-2022-self-critiquing-models-assisting-human/). arXiv preprint arXiv:2206.05802 (2022).
48. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023a. [Toolformer: Language Models Can Teach Themselves to Use Tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/).
49. Timo Schick, Jane A. Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2023b. PEER: A Collaborative Language Model. In The Eleventh International Conference on Learning Representations. [https://openreview.net…](https://openreview.net…) [↗](https://openreview.net…)
50. KbYevcLjnc Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. 2021. Programming Puzzles. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://arxiv.org/abs/2106.05784 — [arXiv:2106.05784](https://arxiv.org/abs/2106.05784) [↗](https://arxiv.org/abs/2106.05784)
51. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). In The Eleventh International Conference on Learning Representa…
52. Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. [Reflexion: an autonomous agent with dynamic memory and self-reflection](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). arXiv preprint arXiv:2303.11366 (2023).
53. Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2023. AdaPlanner: Adaptive Planning from Feedback with Language Models. arXiv preprint arXiv:2305.16653 (2023). [find](https://scholar.google.com/scholar?q=Haotian+Sun%2C+Yuchen+Zhuang%2C+Lingkai+Kong%2C+Bo+Dai%2C+and+Chao+Zhang%2E+2023%2E+AdaPlanner%3A+Adaptive+Planning+from+Feedback+with+Language+Models%2E+arXiv+preprint+arXiv%3A2305%2E16653+%282023%29%2E)
54. Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. 2023a. Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding. In Findings of the Association for Computational Linguistics: ACL 2023, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Co… [find](https://scholar.google.com/scholar?q=Mirac+Suzgun%2C+Luke+Melas%2DKyriazi%2C+and+Dan+Jurafsky%2E+2023a%2E+Follow+the+Wisdom+of+the+Crowd%3A+Effective+Text+Generation+via+Minimum+Bayes+Risk+Decoding%2E+In+Findings+of+the+Association+for+Computational+L)
55. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023b. [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). In Findings of the Association for Computational Linguistics:…
56. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288 (2023).
57. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023. [Unleashing Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration](https://md.research.elamur.ai/free/en/papers/wang-2023-unleashing-emergent-cognitive-synergy-large/). arXiv preprint arXiv:2307.05300 (2023).
58. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. [Emergent Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Transactions on M…
59. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022b. [Chain of Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and…
60. Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023. [Generating Sequences by Learning to Self-Correct](https://md.research.elamur.ai/free/en/papers/welleck-2022-generating-sequences-learning-self-correct/). In The Eleventh International Conference on Learning Representations. [https://openreview.net/forum?id=hH36JeQZDaO](https://openreview.net/forum?id=hH36JeQZDaO) [↗](https://openreview.net/forum?id=hH36JeQZDaO)
61. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://md.research.elamur.ai/free/en/papers/wu-2023-autogen-enabling-next-gen-llm/) Framework. arXiv:2308.08155 [cs.AI]
62. Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. 2023. Diving into the Inter-Consistency of Large Language Models: An Insightful Analysis through Debate. arXiv preprint arXiv:2305.11595 (2023). — [arXiv:2305.11595](https://arxiv.org/abs/2305.11595) [find](https://scholar.google.com/scholar?q=Kai+Xiong%2C+Xiao+Ding%2C+Yixin+Cao%2C+Ting+Liu%2C+and+Bing+Qin%2E+2023%2E+Diving+into+the+Inter%2DConsistency+of+Large+Language+Models%3A+An+Insightful+Analysis+through+Debate%2E+arXiv+preprint+arXiv%3A2305%2E11595+%282023%29)
63. Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2023. ExpertPrompting: Instructing Large Language Models to be Distinguished Experts. arXiv preprint arXiv:2305.14688 (2023). [find](https://scholar.google.com/scholar?q=Benfeng+Xu%2C+An+Yang%2C+Junyang+Lin%2C+Quan+Wang%2C+Chang+Zhou%2C+Yongdong+Zhang%2C+and+Zhendong+Mao%2E+2023%2E+ExpertPrompting%3A+Instructing+Large+Language+Models+to+be+Distinguished+Experts%2E+arXiv+preprint+arXiv%3A23)
64. Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating Longer Stories With Recursive Reprompting and Revision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Comp… [find](https://scholar.google.com/scholar?q=Kevin+Yang%2C+Yuandong+Tian%2C+Nanyun+Peng%2C+and+Dan+Klein%2E+2022%2E+Re3%3A+Generating+Longer+Stories+With+Recursive+Reprompting+and+Revision%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natur)
65. Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. Gpt4tools: Teaching large language model to use tools via self-instruction. arXiv preprint arXiv:2305.18752 (2023). [find](https://scholar.google.com/scholar?q=Rui+Yang%2C+Lin+Song%2C+Yanwei+Li%2C+Sijie+Zhao%2C+Yixiao+Ge%2C+Xiu+Li%2C+and+Ying+Shan%2E+2023%2E+Gpt4tools%3A+Teaching+large+language+model+to+use+tools+via+self%2Dinstruction%2E+arXiv+preprint+arXiv%3A2305%2E18752+%282023%29%2E)
66. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601 (2023). [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023a%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+arXiv+preprint+a)
67. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023b. [ReAct: Synergizing Reasoning and Acting in Language Models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In International Conference on Learning Representations (ICLR).
68. Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R Fung, Hao Peng, and Heng Ji. 2023. Craft: Customizing llms by creating and retrieving from specialized toolsets. arXiv preprint arXiv:2309.17428 (2023). — [arXiv:2309.17428](https://arxiv.org/abs/2309.17428) [find](https://scholar.google.com/scholar?q=Lifan+Yuan%2C+Yangyi+Chen%2C+Xingyao+Wang%2C+Yi+R+Fung%2C+Hao+Peng%2C+and+Heng+Ji%2E+2023%2E+Craft%3A+Customizing+llms+by+creating+and+retrieving+from+specialized+toolsets%2E+arXiv+preprint+arXiv%3A2309%2E17428+%282023%29%2E+%E2%80%94+%5B)
69. Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. 2023. Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation. arXiv preprint arXiv:2310.02304 (2023). — [arXiv:2310.02304](https://arxiv.org/abs/2310.02304) [find](https://scholar.google.com/scholar?q=Eric+Zelikman%2C+Eliana+Lorch%2C+Lester+Mackey%2C+and+Adam+Tauman+Kalai%2E+2023%2E+Self%2DTaught+Optimizer+%28STOP%29%3A+Recursively+Self%2DImproving+Code+Generation%2E+arXiv+preprint+arXiv%3A2310%2E02304+%282023%29%2E+%E2%80%94+%5BarXiv%3A2310)
70. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. [STaR: Bootstrapping Reasoning With Reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). In Advances in Neural Information Processing Systems, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (Eds.). [https://openreview.net/forum?id=_3ELRdg2sgI](https://openreview.net/forum?id=_3ELRdg2sgI) [↗](https://openreview.net/forum?id=_3ELRdg2sgI)
71. Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023a. Self-Edit: Fault-Aware Code Editor for Code Generation. arXiv preprint arXiv:2305.04087 (2023). [find](https://scholar.google.com/scholar?q=Kechi+Zhang%2C+Zhuo+Li%2C+Jia+Li%2C+Ge+Li%2C+and+Zhi+Jin%2E+2023a%2E+Self%2DEdit%3A+Fault%2DAware+Code+Editor+for+Code+Generation%2E+arXiv+preprint+arXiv%3A2305%2E04087+%282023%29%2E)
72. Muru Zhang, Ofir Press, Will Merrill, Alisa Liu, and Noah A. Smith. 2023b. [How Language Model Hallucinations Can Snowball](https://md.research.elamur.ai/free/en/papers/zhang-2023-language-model-hallucinations-snowball/). ArXiv abs/2305.13534 (2023). [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 258841857 [↗](https://api.semanticscholar.org/CorpusID:)
73. Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. 2023c. Cumulative reasoning with large language models. arXiv preprint arXiv:2308.04371 (2023). [find](https://scholar.google.com/scholar?q=Yifan+Zhang%2C+Jingqin+Yang%2C+Yang+Yuan%2C+and+Andrew+Chi%2DChih+Yao%2E+2023c%2E+Cumulative+reasoning+with+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E04371+%282023%29%2E)
74. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023d. [Automatic Chain of Thought Prompting in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/). In The Eleventh International Conference on Learning Representations. [https://openreview.net/forum?id=5NTt8GFjUHkr](https://openreview.net/forum?id=5NTt8GFjUHkr) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scal… [↗](https://openreview.net/forum?id=5NTt8GFjUHkr)
75. Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. 2023. Mindstorms in Natural Language-Based Societies of Mind. arXiv preprint arXiv:2305.17066 (2023). — [arXiv:2305.17066](https://arxiv.org/abs/2305.17066) [find](https://scholar.google.com/scholar?q=Mingchen+Zhuge%2C+Haozhe+Liu%2C+Francesco+Faccio%2C+Dylan+R+Ashley%2C+R%C3%B3bert+Csord%C3%A1s%2C+Anand+Gopalakrishnan%2C+Abdullah+Hamdi%2C+Hasan+Abed+Al+Kader+Hammoud%2C+Vincent+Herrmann%2C+Kazuki+Irie%2C+et+al%2E+2023%2E+Mindstorms+)

*Updated 29 October 2026*