---
title: "Promptbreeder — Fernando 2023"
description: "Hand-written prompting strategies like Chain-of-Thought help LLMs reason, but they are generic and seldom the best choice for a particular task."
authors: ["Chrisantha Fernando", "Dylan Banarse", "Henryk Michalewski", "Simon Osindero", "Tim Rocktaschel"]
year: 2023
venue: "arXiv:2309.16797"
source: "https://arxiv.org/abs/2309.16797"
doi: "10.48550/arxiv.2309.16797"
arxiv: "2309.16797"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fernando-2023-promptbreeder-self-referential-self-improvement/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fernando-2023-promptbreeder-self-referential-self-improvement/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution

# Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution

**Authors:** Chrisantha Fernando, Dylan Banarse, [Henryk Michalewski](https://md.research.elamur.ai/free/en/authors/henryk-michalewski/), Simon Osindero, [Tim Rocktaschel](https://md.research.elamur.ai/free/en/authors/tim-rocktaschel/) · **Year:** 2023 · **Venue:** arXiv:2309.16797 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.16797) · [PDF](https://arxiv.org/pdf/2309.16797) · [DOI 10.48550/arxiv.2309.16797](https://doi.org/10.48550/arxiv.2309.16797) · [arXiv 2309.16797](https://arxiv.org/abs/2309.16797) · [OpenAlex](https://openalex.org/W4387293261)

**Citations:** 26 citations (openalex-snapshot · 2026-10-14) · [137](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Hand-written prompting strategies like Chain-of-Thought help LLMs reason, but they are generic and seldom the best choice for a particular task. Earlier automatic methods such as APE also stopped improving after a few rounds. Researchers at Google DeepMind propose Promptbreeder (PB), an evolutionary system in which the LLM itself acts as the mutation operator. Each unit in the population holds task-prompts and a mutation-prompt. The starting population is built by combining a problem description with generic "thinking styles". Candidates are scored on random batches of training examples and selected with a binary tournament genetic algorithm. The key feature is self-reference: the instructions that rewrite task-prompts are also mutated and selected, so the system improves how it improves. Because no model weights change, the method also works with models available only through an API. With PaLM 2-L, PB beats Chain-of-Thought, Plan-and-Solve, APE and OPRO on most arithmetic and commonsense benchmarks. It also evolves complex prompts for hate speech classification on ETHOS. This points to prompt optimisation that gets better as LLMs get stronger.

## Key points
- Zero-shot results with PaLM 2-L: GSM8K 83.9% (OPRO 80.2%, PS+ 60.5%), AQuA-RAT 62.2% (best baseline 45.7%), SVAMP 90.2% and StrategyQA 71.8%. In the few-shot setting PB reaches 100% on MultiArith but falls behind the text-davinci-003 baselines on AddSub.
- There are five classes of mutation operators: direct mutation, estimation-of-distribution mutation, hypermutation (rewriting mutation-prompts), Lamarckian mutation (building a prompt from a correct worked solution), and crossover with context shuffling.
- For MultiArith, SingleEq, AddSub and SVAMP, half of the examples were randomly chosen for training and the other half were used for testing.
- Ablations used a population of 10 and 200 evaluations (about 20 generations). In most cases, removing a self-referential component lowered fitness. The thinking-style-guided initialisation of task-prompts had the largest positive effect.
- On ETHOS with the vague problem description "Solve the Problem", PB scored 81.6%. Without the Lamarckian operator it fell to 64.6%, and keeping only context mutation gave 68.7%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

137 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 132 not yet.

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://md.research.elamur.ai/free/en/papers/novikov-2025-alphaevolve-coding-agent-scientific-algorithmic/)
- [Large Language Models as Optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{fernando2023promptbreeder,
  title = {{Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution}},
  author = {Fernando, Chrisantha and Banarse, Dylan and Michalewski, Henryk and Osindero, Simon and Rocktaschel, Tim},
  year = {2023},
  doi = {10.48550/arxiv.2309.16797},
  eprint = {2309.16797},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.16797}
}
```

- [← previous Prompt-based Logical Semantics Enhancement for Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/en/papers/wang-2023-prompt-based-logical-semantics-enhancement/)
- [next → Prompting Large Language Model for Machine Translation: A Case Study](https://md.research.elamur.ai/free/en/papers/zhang-2023-prompting-large-language-model-machine/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

84 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… [find](https://scholar.google.com/scholar?q=Rohan+Anil%2C+Andrew+M%2E+Dai%2C+Orhan+Firat%2C+Melvin+Johnson%2C+Dmitry+Lepikhin%2C+Alexandre+Passos%2C+Siamak+Shakeri%2C+Emanuel+Taropa%2C+Paige+Bailey%2C+Zhifeng+Chen%2C+Eric+Chu%2C+Jonathan+H%2E+Clark%2C+Laurent+El+Shafey%2C+Y) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
2. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. CoRR, abs/2308.09687, 2023. doi: 10.48550… [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Nils+Blach%2C+Ales+Kubicek%2C+Robert+Gerstenberger%2C+Lukas+Gianinazzi%2C+Joanna+Gajda%2C+Tomasz+Lehmann%2C+Michal+Podstawski%2C+Hubert+Niewiadomski%2C+Piotr+Nyczyk%2C+and+Torsten+Hoefler%2E+Graph+of+though)
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
4. Angelica Chen, David M. Dohan, and David R. So. Evoprompting: Language models for code-level neural architecture search. CoRR, abs/2302.14838, 2023. doi: 10.48550/arXiv.2302.14838. URL https://doi.org/10.48550/arXiv.2302.14838. — [doi:10.48550/arxiv.2302.14838](https://doi.org/10.48550/arxiv.2302.14838) [↗](https://doi.org/10.48550/arXiv.2302.14838)
5. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks, November 2022. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W%2E+Cohen%2E+Program+of+Thoughts+Prompting%3A+Disentangling+Computation+from+Reasoning+for+Numerical+Reasoning+Tasks%2C+November+2022%2E)
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.1416… [↗](https://arxiv.org/abs/2110.1416…) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
7. Richard Dawkins. 13 - The evolution of evolvability. In Sanjeev Kumar and Peter J. Bentley (eds.), On Growth, Form and Computers, pp. 239–255. Academic Press, London, January 2003. ISBN 978-0-12-428765-5. doi: 10.1016/B978-012428765-5/50046-3. — [doi:10.1016/b978-012428765-5/50046-3](https://doi.org/10.1016/b978-012428765-5/50046-3) [find](https://scholar.google.com/scholar?q=Richard+Dawkins%2E+13+%2D+The+evolution+of+evolvability%2E+In+Sanjeev+Kumar+and+Peter+J%2E+Bentley+%28eds%2E%29%2C+On+Growth%2C+Form+and+Computers%2C+pp%2E+239%E2%80%93255%2E+Academic+Press%2C+London%2C+January+2003%2E+ISBN+978%2D0%2D12%2D42876)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Comp… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+BERT%3A+pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Jill+Burstein%2C+Christy+Doran%2C+and+Thamar+Solorio+)
9. Alexander Gajewski, Jeff Clune, Kenneth O. Stanley, and Joel Lehman. Evolvability ES: scalable and direct optimization of evolvability. In Anne Auger and Thomas Stützle (eds.), Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2019, Prague, Czech Republic, July 13-17, 2019,… [find](https://scholar.google.com/scholar?q=Alexander+Gajewski%2C+Jeff+Clune%2C+Kenneth+O%2E+Stanley%2C+and+Joel+Lehman%2E+Evolvability+ES%3A+scalable+and+direct+optimization+of+evolvability%2E+In+Anne+Auger+and+Thomas+Stu%CC%88tzle+%28eds%2E%29%2C+Proceedings+of+the+Gen)
10. ACM, 2019. doi: 10.1145/3321707.3321876. URL https://doi.org/10.1145/3321707.3321876. — [doi:10.1145/3321707.3321876](https://doi.org/10.1145/3321707.3321876) [↗](https://doi.org/10.1145/3321707.3321876)
11. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Trans. Assoc. Comput. Linguistics, 9:346–361, 2021. doi: 10.1162/tacl\\ a\\ 00370. URL https://doi. org/10.1162/tacl_a_00370… [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+Did+aristotle+use+a+laptop%3F+A+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Trans%2E+Assoc%2E+Comput%2E+)
12. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers, September 2023. [find](https://scholar.google.com/scholar?q=Qingyan+Guo%2C+Rui+Wang%2C+Junliang+Guo%2C+Bei+Li%2C+Kaitao+Song%2C+Xu+Tan%2C+Guoqing+Liu%2C+Jiang+Bian%2C+and+Yujiu+Yang%2E+Connecting+Large+Language+Models+with+Evolutionary+Algorithms+Yields+Powerful+Prompt+Optimize)
13. Inman Harvey. The microbial genetic algorithm. In Advances in Artificial Life. Darwin Meets von Neumann: 10th European Conference, ECAL 2009, Budapest, Hungary, September 13-16, 2009, Revised Selected Papers, Part II 10, pp. 126–133. [find](https://scholar.google.com/scholar?q=Inman+Harvey%2E+The+microbial+genetic+algorithm%2E+In+Advances+in+Artificial+Life%2E+Darwin+Meets+von+Neumann%3A+10th+European+Conference%2C+ECAL+2009%2C+Budapest%2C+Hungary%2C+September+13%2D16%2C+2009%2C+Revised+Selected)
14. Mark Hauschild and Martin Pelikan. An introduction and survey of estimation of distribution algorithms. Swarm and evolutionary computation, 1(3):111–128, 2011. [find](https://scholar.google.com/scholar?q=Mark+Hauschild+and+Martin+Pelikan%2E+An+introduction+and+survey+of+estimation+of+distribution+algorithms%2E+Swarm+and+evolutionary+computation%2C+1%283%29%3A111%E2%80%93128%2C+2011%2E)
15. Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume… [find](https://scholar.google.com/scholar?q=Or+Honovich%2C+Uri+Shaham%2C+Samuel+R%2E+Bowman%2C+and+Omer+Levy%2E+Instruction+induction%3A+From+few+examples+to+natural+language+task+descriptions%2E+In+Anna+Rogers%2C+Jordan+L%2E+Boyd%2DGraber%2C+and+Naoaki+Okazaki+%28eds)
16. Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 523–533, Doha, Qatar, October 2014. Association fo… [find](https://scholar.google.com/scholar?q=Mohammad+Javad+Hosseini%2C+Hannaneh+Hajishirzi%2C+Oren+Etzioni%2C+and+Nate+Kushman%2E+Learning+to+solve+arithmetic+word+problems+with+verb+categorization%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Me) [our card](https://md.research.elamur.ai/free/en/papers/hosseini-2014-learning-solve-arithmetic-word-problems/)
17. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki… [find](https://scholar.google.com/scholar?q=Cheng%2DYu+Hsieh%2C+Chun%2DLiang+Li%2C+Chih%2DKuan+Yeh%2C+Hootan+Nakhost%2C+Yasuhisa+Fujii%2C+Alex+Ratner%2C+Ranjay+Krishna%2C+Chen%2DYu+Lee%2C+and+Tomas+Pfister%2E+Distilling+step%2Dby%2Dstep%21+outperforming+larger+language+models) [our card](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/)
18. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. CoRR, abs/2210.11610, 2022. doi: 10.48550/ arXiv.2210.11610. URL https://doi.org/10.48550/arXiv.2210.11610. [↗](https://doi.org/10.48550/arXiv.2210.11610) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/)
19. Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. A modern self-referential weight matrix that learns to modify itself. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022… [find](https://scholar.google.com/scholar?q=Kazuki+Irie%2C+Imanol+Schlag%2C+Ro%CC%81bert+Csorda%CC%81s%2C+and+Ju%CC%88rgen+Schmidhuber%2E+A+modern+self%2Dreferential+weight+matrix+that+learns+to+modify+itself%2E+In+Kamalika+Chaudhuri%2C+Stefanie+Jegelka%2C+Le+Song%2C+Csaba+Sze)
20. PMLR, 2022. URL [https://proceedings.mlr.press/v162/irie22b.html](https://proceedings.mlr.press/v162/irie22b.html). [↗](https://proceedings.mlr.press/v162/irie22b.html)
21. Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. CoRR, abs/1711.09846, 2017a. URL http://arxiv.org/abs/1… — [arXiv:1711.09846](https://arxiv.org/abs/1711.09846) [↗](http://arxiv.org/abs/1…)
22. Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Tr… [find](https://scholar.google.com/scholar?q=Max+Jaderberg%2C+Volodymyr+Mnih%2C+Wojciech+Marian+Czarnecki%2C+Tom+Schaul%2C+Joel+Z%2E+Leibo%2C+David+Silver%2C+and+Koray+Kavukcuoglu%2E+Reinforcement+learning+with+unsupervised+auxiliary+tasks%2E+In+5th+International)
23. OpenReview.net, 2017b. URL [https://openreview.net/forum?id=SJ6yPD5xg](https://openreview.net/forum?id=SJ6yPD5xg). [↗](https://openreview.net/forum?id=SJ6yPD5xg)
24. Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob N. Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information P… [find](https://scholar.google.com/scholar?q=Minqi+Jiang%2C+Michael+Dennis%2C+Jack+Parker%2DHolder%2C+Jakob+N%2E+Foerster%2C+Edward+Grefenstette%2C+and+Tim+Rockta%CC%88schel%2E+Replay%2Dguided+adversarial+environment+design%2E+In+Marc%E2%80%99Aurelio+Ranzato%2C+Alina+Beygelzimer%2C)
25. Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 4940–4… [find](https://scholar.google.com/scholar?q=Minqi+Jiang%2C+Edward+Grefenstette%2C+and+Tim+Rockta%CC%88schel%2E+Prioritized+level+replay%2E+In+Marina+Meila+and+Tong+Zhang+%28eds%2E%29%2C+Proceedings+of+the+38th+International+Conference+on+Machine+Learning%2C+ICML+2021)
26. PMLR, 2021b. URL [http://proceedings.mlr.press/v139/](http://proceedings.mlr.press/v139/) jiang21b.html. [↗](http://proceedings.mlr.press/v139/)
27. Minqi Jiang, Tim Rocktäschel, and Edward Grefenstette. General intelligence requires rethinking exploration. CoRR, abs/2211.07819, 2022. doi: 10.48550/arXiv.2211.07819. URL https://doi.org/10.48550/arXiv.2211.07819. — [doi:10.48550/arxiv.2211.07819](https://doi.org/10.48550/arxiv.2211.07819) [↗](https://doi.org/10.48550/arXiv.2211.07819)
28. Louis Kirsch and Jürgen Schmidhuber. Eliminating meta optimization through self-referential meta learning. CoRR, abs/2212.14392, 2022. doi: 10.48550/arXiv.2212.14392. URL https://doi.org/10.48550/arXiv.2212.14392. — [doi:10.48550/arxiv.2212.14392](https://doi.org/10.48550/arxiv.2212.14392) [↗](https://doi.org/10.48550/arXiv.2212.14392)
29. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022. URL [http://papers.nips.cc/paper_files/paper/2022/hash/](http://papers.nips.cc/paper_files/paper/2022/hash/) 8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html. [↗](http://papers.nips.cc/paper_files/paper/2022/hash/) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
30. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597, 2015. doi: 10.1162/tacl a 00160. URL https://aclanthology.org/Q15-1042. [↗](https://aclanthology.org/Q15-1042)
31. Joel Lehman and Kenneth O. Stanley. Evolving a diversity of virtual creatures through novelty search and local competition. In Natalio Krasnogor and Pier Luca Lanzi (eds.), 13th Annual Genetic and Evolutionary Computation Conference, GECCO 2011, Proceedings, Dublin, Ireland, July 12-16, 2011, pp. 21… [find](https://scholar.google.com/scholar?q=Joel+Lehman+and+Kenneth+O%2E+Stanley%2E+Evolving+a+diversity+of+virtual+creatures+through+novelty+search+and+local+competition%2E+In+Natalio+Krasnogor+and+Pier+Luca+Lanzi+%28eds%2E%29%2C+13th+Annual+Genetic+and+Evo)
32. ACM, 2011a. doi: 10.1145/2001576.2001606. URL https://doi.org/10.1145/2001576.2001606. — [doi:10.1145/2001576.2001606](https://doi.org/10.1145/2001576.2001606) [↗](https://doi.org/10.1145/2001576.2001606)
33. Joel Lehman and Kenneth O. Stanley. Abandoning Objectives: Evolution Through the Search for Novelty Alone. Evolutionary Computation, 19(2):189–223, June 2011b. ISSN 1063-6560. doi: 10.1162/EVCO a 00025. — [doi:10.1162/evco](https://doi.org/10.1162/evco) [find](https://scholar.google.com/scholar?q=Joel+Lehman+and+Kenneth+O%2E+Stanley%2E+Abandoning+Objectives%3A+Evolution+Through+the+Search+for+Novelty+Alone%2E+Evolutionary+Computation%2C+19%282%29%3A189%E2%80%93223%2C+June+2011b%2E+ISSN+1063%2D6560%2E+doi%3A+10%2E1162%2FEVCO+a+0002)
34. Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley. Evolution through large models. CoRR, abs/2206.08896, 2022. doi: 10.48550/arXiv.2206.08896. URL https://doi.org/10.48550/arXiv.2206.08896. — [doi:10.48550/arxiv.2206.08896](https://doi.org/10.48550/arxiv.2206.08896) [↗](https://doi.org/10.48550/arXiv.2206.08896)
35. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Ev… [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+In+Marie%2DFrancine+Moens%2C+Xuanjing+Huang%2C+Lucia+Specia%2C+and+Scott+Wen%2Dtau+Yih+%28eds%2E%29%2C+Proceeding)
36. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158–167, Vancouver, Canada… [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+t)
37. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. CoRR, abs/2307.03172, 2023. doi: 10.48550/arXiv.2307.03172. URL https://doi.org/10.48550/arXiv. 2307.03172. [↗](https://doi.org/10.48550/arXiv) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/)
38. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. CoRR, abs/2103.10385, 2021. URL https://arxiv.org/abs/2103. 10385. [↗](https://arxiv.org/abs/2103) [our card](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/)
39. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Associati… [find](https://scholar.google.com/scholar?q=Yao+Lu%2C+Max+Bartolo%2C+Alastair+Moore%2C+Sebastian+Riedel%2C+and+Pontus+Stenetorp%2E+Fantastically+ordered+prompts+and+where+to+find+them%3A+Overcoming+few%2Dshot+prompt+order+sensitivity%2E+In+Smaranda+Muresan%2C+Pr) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/)
40. Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two to tango. CoRR, abs/2209.07686, 2022. doi: 10.48550/arXiv.2209.07686. URL https://doi.org/10.48550/arXiv.2209.07686. [↗](https://doi.org/10.48550/arXiv.2209.07686)
41. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. CoRR, abs… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Sean+Welleck%2C+Bodhisattwa+Prasad+Majumder%2C+Shashank+Gupta%2C)
42. Elliot Meyerson, Mark J. Nelson, Herbie Bradley, Arash Moradi, Amy K. Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting. CoRR, abs/2302.12170, 2023. doi: 10.48550/arXiv.2302.12170. URL https://doi.org/10.48550/arXiv.2302. 12170. — [doi:10.48550/arxiv.2302.12170](https://doi.org/10.48550/arxiv.2302.12170) [↗](https://doi.org/10.48550/arXiv.2302)
43. Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. CoRR, abs/2307.04721, 2023. doi: 10.48550/arXiv.2307.04721. URL https://doi.org/10.48550/arXiv.2307.04721. [↗](https://doi.org/10.48550/arXiv.2307.04721)
44. Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. ETHOS: a multi-label hate speech detection dataset. Complex and Intelligent Systems, 8(6):4663–4678, jan 2022. doi: 10.1007/s40747-021-00608-2. URL https://doi.org/10.1007% 2Fs40747-021-00608-2. — [doi:10.1007/s40747-021-00608-2](https://doi.org/10.1007/s40747-021-00608-2) [find](https://scholar.google.com/scholar?q=Ioannis+Mollas%2C+Zoe+Chrysopoulou%2C+Stamatis+Karlos%2C+and+Grigorios+Tsoumakas%2E+ETHOS%3A+a+multi%2Dlabel+hate+speech+detection+dataset%2E+Complex+and+Intelligent+Systems%2C+8%286%29%3A4663%E2%80%934678%2C+jan+2022%2E+doi%3A+10%2E1007%2F)
45. Milad Moradi and Matthias Samwald. Evaluating the robustness of neural language models to input perturbations. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virt… [find](https://scholar.google.com/scholar?q=Milad+Moradi+and+Matthias+Samwald%2E+Evaluating+the+robustness+of+neural+language+models+to+input+perturbations%2E+In+Marie%2DFrancine+Moens%2C+Xuanjing+Huang%2C+Lucia+Specia%2C+and+Scott+Wen%2Dtau+Yih+%28eds%2E%29%2C+Proc)
46. Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. abs/1504.04909, 2015. URL http://arxiv.org/abs/1504.04909. — [arXiv:1504.04909](https://arxiv.org/abs/1504.04909) [↗](http://arxiv.org/abs/1504.04909)
47. CoRR, Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. CoRR, abs/2112.00114… [find](https://scholar.google.com/scholar?q=CoRR%2C+Maxwell+I%2E+Nye%2C+Anders+Johan+Andreassen%2C+Guy+Gur%2DAri%2C+Henryk+Michalewski%2C+Jacob+Austin%2C+David+Bieber%2C+David+Dohan%2C+Aitor+Lewkowycz%2C+Maarten+Bosma%2C+David+Luan%2C+Charles+Sutton%2C+and+Augustus+Odena%2E) [our card](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/)
48. Michael Öllinger and Günther Knoblich. Psychological research on insight problem solving. In Recasting reality: Wolfgang Pauli’s philosophical ideas and contemporary science, pp. 275–300. Springer, 2009. [find](https://scholar.google.com/scholar?q=Michael+O%CC%88llinger+and+Gu%CC%88nther+Knoblich%2E+Psychological+research+on+insight+problem+solving%2E+In+Recasting+reality%3A+Wolfgang+Pauli%E2%80%99s+philosophical+ideas+and+contemporary+science%2C+pp%2E+275%E2%80%93300%2E+Springer%2C+)
49. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. CoRR, abs/2304.03442, 2023. doi: 10.48550/arXiv.2304.03442. URL https://doi.org/10. 48550/arXiv.2304.03442. [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+C%2E+O%E2%80%99Brien%2C+Carrie+J%2E+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S%2E+Bernstein%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+CoRR%2C+abs%2F2304%2E03442%2C+2023%2E)
50. Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob N. Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), Internati… [find](https://scholar.google.com/scholar?q=Jack+Parker%2DHolder%2C+Minqi+Jiang%2C+Michael+Dennis%2C+Mikayel+Samvelyan%2C+Jakob+N%2E+Foerster%2C+Edward+Grefenstette%2C+and+Tim+Rockta%CC%88schel%2E+Evolving+curricula+with+regret%2Dbased+environment+design%2E+In+Kamalika+C)
51. PMLR, 2022. URL [https://proceedings.mlr.press/](https://proceedings.mlr.press/) v162/parker-holder22a.html. [↗](https://proceedings.mlr.press/)
52. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 20… [find](https://scholar.google.com/scholar?q=Arkil+Patel%2C+Satwik+Bhattamishra%2C+and+Navin+Goyal%2E+Are+NLP+models+really+able+to+solve+simple+math+word+problems%3F+In+Kristina+Toutanova%2C+Anna+Rumshisky%2C+Luke+Zettlemoyer%2C+Dilek+Hakkani%2DTu%CC%88r%2C+Iz+Beltag) [our card](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/)
53. Joshua L. Payne and Andreas Wagner. The causes of evolvability and their evolution. Nature Reviews Genetics, 20(1):24–38, January 2019. ISSN 1471-0064. doi: 10.1038/s41576-018-0069-z. — [doi:10.1038/s41576-018-0069-z](https://doi.org/10.1038/s41576-018-0069-z) [find](https://scholar.google.com/scholar?q=Joshua+L%2E+Payne+and+Andreas+Wagner%2E+The+causes+of+evolvability+and+their+evolution%2E+Nature+Reviews+Genetics%2C+20%281%29%3A24%E2%80%9338%2C+January+2019%2E+ISSN+1471%2D0064%2E+doi%3A+10%2E1038%2Fs41576%2D018%2D0069%2Dz%2E+%E2%80%94+%5Bdoi%3A10%2E1038%2Fs)
54. Massimo Pigliucci. Is evolvability evolvable? Nature Reviews Genetics, 9(1):75–82, January 2008. ISSN 1471-0064. doi: 10.1038/nrg2278. — [doi:10.1038/nrg2278](https://doi.org/10.1038/nrg2278) [find](https://scholar.google.com/scholar?q=Massimo+Pigliucci%2E+Is+evolvability+evolvable%3F+Nature+Reviews+Genetics%2C+9%281%29%3A75%E2%80%9382%2C+January+2008%2E+ISSN+1471%2D0064%2E+doi%3A+10%2E1038%2Fnrg2278%2E+%E2%80%94+%5Bdoi%3A10%2E1038%2Fnrg2278%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1038%2Fnrg2278%29)
55. Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with” gradient descent” and beam search. arXiv preprint arXiv:2305.03495, 2023. [find](https://scholar.google.com/scholar?q=Reid+Pryzant%2C+Dan+Iter%2C+Jerry+Li%2C+Yin+Tat+Lee%2C+Chenguang+Zhu%2C+and+Michael+Zeng%2E+Automatic+prompt+optimization+with%E2%80%9D+gradient+descent%E2%80%9D+and+beam+search%2E+arXiv+preprint+arXiv%3A2305%2E03495%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/)
56. Guanghui Qin and Jason Eisner. Learning How to Ask: Querying LMs with Mixtures of Soft Prompts, April 2021. [find](https://scholar.google.com/scholar?q=Guanghui+Qin+and+Jason+Eisner%2E+Learning+How+to+Ask%3A+Querying+LMs+with+Mixtures+of+Soft+Prompts%2C+April+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/)
57. Subhro Roy and Dan Roth. arXiv:1608.01413, 2016. [find](https://scholar.google.com/scholar?q=Subhro+Roy+and+Dan+Roth%2E+arXiv%3A1608%2E01413%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/)
58. Solving general arithmetic word problems. arXiv preprint Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools, February 2023. [find](https://scholar.google.com/scholar?q=Solving+general+arithmetic+word+problems%2E+arXiv+preprint+Timo+Schick%2C+Jane+Dwivedi%2DYu%2C+Roberto+Dess%C4%B1%CC%80%2C+Roberta+Raileanu%2C+Maria+Lomeli%2C+Luke+Zettlemoyer%2C+Nicola+Cancedda%2C+and+Thomas+Scialom%2E+Toolformer) [our card](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/)
59. J. Schmidhuber. A ‘Self-Referential’ Weight Matrix. In Stan Gielen and Bert Kappen (eds.), ICANN ’93, pp. 446–450, London, 1993. Springer. ISBN 978-1-4471-2063-6. doi: 10.1007/ 978-1-4471-2063-6 107. [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber%2E+A+%E2%80%98Self%2DReferential%E2%80%99+Weight+Matrix%2E+In+Stan+Gielen+and+Bert+Kappen+%28eds%2E%29%2C+ICANN+%E2%80%9993%2C+pp%2E+446%E2%80%93450%2C+London%2C+1993%2E+Springer%2E+ISBN+978%2D1%2D4471%2D2063%2D6%2E+doi%3A+10%2E1007%2F+978%2D1%2D4471%2D2063%2D6+107%2E)
60. Jürgen Schmidhuber. Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments. 1990. [find](https://scholar.google.com/scholar?q=Ju%CC%88rgen+Schmidhuber%2E+Making+the+world+differentiable%3A+On+using+fully+recurrent+self%2Dsupervised+neural+networks+for+dynamic+reinforcement+learning+and+planning+in+non%2Dstationary+environments%2E+1990%2E)
61. Jürgen Schmidhuber. Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks. Neural Computation, 4(1):131–139, January 1992. ISSN 0899-7667. doi: 10.1162/neco.1992.4.1.131. — [doi:10.1162/neco.1992.4.1.131](https://doi.org/10.1162/neco.1992.4.1.131) [find](https://scholar.google.com/scholar?q=Ju%CC%88rgen+Schmidhuber%2E+Learning+to+Control+Fast%2DWeight+Memories%3A+An+Alternative+to+Dynamic+Recurrent+Networks%2E+Neural+Computation%2C+4%281%29%3A131%E2%80%93139%2C+January+1992%2E+ISSN+0899%2D7667%2E+doi%3A+10%2E1162%2Fneco%2E1992%2E4%2E1%2E)
62. Jürgen Schmidhuber. Gödel machines: self-referential universal problem solvers making provably optimal self-improvements. arXiv preprint cs/0309048, 2003. [find](https://scholar.google.com/scholar?q=Ju%CC%88rgen+Schmidhuber%2E+Go%CC%88del+machines%3A+self%2Dreferential+universal+problem+solvers+making+provably+optimal+self%2Dimprovements%2E+arXiv+preprint+cs%2F0309048%2C+2003%2E)
63. Jimmy Secretan, Nicholas Beato, David B. D Ambrosio, Adelein Rodriguez, Adam Campbell, and Kenneth O. Stanley. Picbreeder: Evolving pictures collaboratively online. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’08, pp. 1759–1768, New York, NY, USA, April 2008. A… — [doi:10.1145/1357054.1357328](https://doi.org/10.1145/1357054.1357328) [find](https://scholar.google.com/scholar?q=Jimmy+Secretan%2C+Nicholas+Beato%2C+David+B%2E+D+Ambrosio%2C+Adelein+Rodriguez%2C+Adam+Campbell%2C+and+Kenneth+O%2E+Stanley%2E+Picbreeder%3A+Evolving+pictures+collaboratively+online%2E+In+Proceedings+of+the+SIGCHI+Confer)
64. Ofer M Shir and Thomas Bäck. Niching in evolution strategies. In Proceedings of the 7th annual conference on Genetic and evolutionary computation, pp. 915–916, 2005. [find](https://scholar.google.com/scholar?q=Ofer+M+Shir+and+Thomas+Ba%CC%88ck%2E+Niching+in+evolution+strategies%2E+In+Proceedings+of+the+7th+annual+conference+on+Genetic+and+evolutionary+computation%2C+pp%2E+915%E2%80%93916%2C+2005%2E)
65. Kashun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with chain-of-thought from labeled data. CoRR, abs/2302.12822, 2023. doi: 10.48550/arXiv.2302. 12822. URL https://doi.org/10.48550/arXiv.2302.12822. [↗](https://doi.org/10.48550/arXiv.2302.12822)
66. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… [find](https://scholar.google.com/scholar?q=Alon+Talmor%2C+Jonathan+Herzig%2C+Nicholas+Lourie%2C+and+Jonathan+Berant%2E+CommonsenseQA%3A+A+question+answering+challenge+targeting+commonsense+knowledge%2E+In+Proceedings+of+the+2019+Conference+of+the+North+Am)
67. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. CoRR, abs/2305.16291, 2023a. doi: 10.48550/arXiv.2305.16291. URL https://doi.org/ 10.48550/arXiv.2305.16291. [find](https://scholar.google.com/scholar?q=Guanzhi+Wang%2C+Yuqi+Xie%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Chaowei+Xiao%2C+Yuke+Zhu%2C+Linxi+Fan%2C+and+Anima+Anandkumar%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+CoRR%2C+abs%2F2305%2E16291%2C+2)
68. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the… [find](https://scholar.google.com/scholar?q=Lei+Wang%2C+Wanyu+Xu%2C+Yihuai+Lan%2C+Zhiqiang+Hu%2C+Yunshi+Lan%2C+Roy+Ka%2DWei+Lee%2C+and+Ee%2DPeng+Lim%2E+Plan%2Dand%2Dsolve+prompting%3A+Improving+zero%2Dshot+chain%2Dof%2Dthought+reasoning+by+large+language+models%2E+In+Anna+Rog)
69. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. [find](https://scholar.google.com/scholar?q=Xuezhi+Wang%2C+Jason+Wei%2C+Dale+Schuurmans%2C+Quoc+Le%2C+Ed+Chi%2C+Sharan+Narang%2C+Aakanksha+Chowdhery%2C+and+Denny+Zhou%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
70. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the A… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Anna+Roger)
71. Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. CoRR, abs/2302.01560, 2023d. doi: 10.48550/arXiv.2302.01560. URL https://doi.org/ 10.48550/arXiv.2302.01560. [find](https://scholar.google.com/scholar?q=Zihao+Wang%2C+Shaofei+Cai%2C+Anji+Liu%2C+Xiaojian+Ma%2C+and+Yitao+Liang%2E+Describe%2C+explain%2C+plan+and+select%3A+Interactive+planning+with+large+language+models+enables+open%2Dworld+multi%2Dtask+agents%2E+CoRR%2C+abs%2F230) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-describe-explain-plan-select-interactive/)
72. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. URL [http://papers.nips.cc/paper_files/paper/](http://papers.nips.cc/paper_files/paper/) 2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstr… [↗](http://papers.nips.cc/paper_files/paper/) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
73. Yue Wu, Shrimai Prabhumoye, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom M. Mitchell, and Yuanzhi Li. SPRING: GPT-4 out-performs RL algorithms by studying papers and reasoning. CoRR, abs/2305.15486, 2023. doi: 10.48550/arXiv.2305.15486. URL https://doi.org/10.48550/arXiv.2305.15… — [doi:10.48550/arxiv.2305.15486](https://doi.org/10.48550/arxiv.2305.15486) [↗](https://doi.org/10.48550/arXiv.2305.15…)
74. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. CoRR, abs/2309.03409, 2023a. doi: 10.48550/ arXiv.2309.03409. URL https://doi.org/10.48550/arXiv.2309.03409. [↗](https://doi.org/10.48550/arXiv.2309.03409) [our card](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/)
75. Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023b. [find](https://scholar.google.com/scholar?q=Zhengyuan+Yang%2C+Linjie+Li%2C+Jianfeng+Wang%2C+Kevin+Lin%2C+Ehsan+Azarnasab%2C+Faisal+Ahmed%2C+Zicheng+Liu%2C+Ce+Liu%2C+Michael+Zeng%2C+and+Lijuan+Wang%2E+Mm%2Dreact%3A+Prompting+chatgpt+for+multimodal+reasoning+and+action%2E)
76. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Jeffrey+Zhao%2C+Dian+Yu%2C+Nan+Du%2C+Izhak+Shafran%2C+Karthik+Narasimhan%2C+and+Yuan+Cao%2E+React%3A+Synergizing+reasoning+and+acting+in+language+models%2E+arXiv+preprint+arXiv%3A2210%2E03629%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/)
77. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models, May 2023. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+Thoughts%3A+Deliberate+Problem+Solving+with+Large+Language+Models%2C+May+2023%2E)
78. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. In NeurIPS, 2022. URL [http://papers.nips.cc/paper_files/paper/](http://papers.nips.cc/paper_files/paper/) 2022/hash/639a9a172c044fbb64175b5fad42e9a5-Abstract-Conference. html. [↗](http://papers.nips.cc/paper_files/paper/) [our card](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/)
79. Jenny Zhang, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. OMNI: open-endedness via models of human notions of interestingness. CoRR, abs/2306.01711, 2023a. doi: 10.48550/arXiv.2306. 01711. URL https://doi.org/10.48550/arXiv.2306.01711. [↗](https://doi.org/10.48550/arXiv.2306.01711)
80. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Aston+Zhang%2C+Mu+Li%2C+and+Alex+Smola%2E+Automatic+chain+of+thought+prompting+in+large+language+models%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+ICLR+2023%2C+Kig) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/)
81. OpenReview.net, 2023b. URL [https://openreview.net/pdf?id=5NTt8GFjUHkr](https://openreview.net/pdf?id=5NTt8GFjUHkr). [↗](https://openreview.net/pdf?id=5NTt8GFjUHkr)
82. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. [find](https://scholar.google.com/scholar?q=Denny+Zhou%2C+Nathanael+Scha%CC%88rli%2C+Le+Hou%2C+Jason+Wei%2C+Nathan+Scales%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Claire+Cui%2C+Olivier+Bousquet%2C+Quoc+Le%2C+et+al%2E+Least%2Dto%2Dmost+prompting+enables+complex+reasoning+in+large) [our card](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/)
83. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. [find](https://scholar.google.com/scholar?q=Yongchao+Zhou%2C+Andrei+Ioan+Muresanu%2C+Ziwen+Han%2C+Keiran+Paster%2C+Silviu+Pitis%2C+Harris+Chan%2C+and+Jimmy+Ba%2E+Large+language+models+are+human%2Dlevel+prompt+engineers%2E+In+The+Eleventh+International+Conference) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/)
84. OpenReview.net, 2023. URL [https://openreview.net/pdf?id=92gvk82DE-](https://openreview.net/pdf?id=92gvk82DE-). [↗](https://openreview.net/pdf?id=92gvk82DE-)