---
title: "Automatic Prompt Optimization with “Gradient — Pryzant 2023"
description: "Writing prompts for LLMs is still mostly manual trial and error."
authors: ["Reid Pryzant", "Dan Iter", "Jerry Li", "Yin Lee", "Chenguang Zhu", "Michael Zeng"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.494/"
doi: "10.18653/v1/2023.emnlp-main.494"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / Automatic Prompt Optimization with “Gradient Descent” and Beam Search

# Automatic Prompt Optimization with “Gradient Descent” and Beam Search

**Authors:** [Reid Pryzant](https://md.research.elamur.ai/free/en/authors/reid-pryzant/), [Dan Iter](https://md.research.elamur.ai/free/en/authors/dan-iter/), [Jerry Li](https://md.research.elamur.ai/free/en/authors/jerry-li/), Yin Lee, [Chenguang Zhu](https://md.research.elamur.ai/free/en/authors/chenguang-zhu/), [Michael Zeng](https://md.research.elamur.ai/free/en/authors/michael-zeng/) · **Year:** 2023 · **Venue:** Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.494/) · [PDF](https://aclanthology.org/2023.emnlp-main.494.pdf) · [DOI 10.18653/v1/2023.emnlp-main.494](https://doi.org/10.18653/v1/2023.emnlp-main.494) · [OpenAlex](https://openalex.org/W4389520756)

**Citations:** 187 citations (openalex-snapshot · 2026-10-14) · [210](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Writing prompts for LLMs is still mostly manual trial and error. Many existing automatic methods need access to model internals, or they search the prompt space without a clear direction. The authors introduce ProTeGi (Prompt Optimization with Textual Gradients), a nonparametric method that needs only labelled training examples and a black-box LLM API. It imitates gradient descent in text. The LLM examines a prompt's errors on a minibatch and writes a critique, which serves as the textual "gradient". A second prompt then rewrites the original to fix the problems the critique describes, and paraphrasing adds more candidates. These candidates feed a beam search. Deciding which candidates survive is treated as a best arm identification problem and solved with bandit algorithms to limit evaluation cost. Across three benchmark NLP classification tasks and a new LLM jailbreak detection task, ProTeGi improved starting prompts by up to 31%. It also beat earlier prompt-editing baselines by 4–8% on average while using fewer API calls. This matters because it turns vague task descriptions into precise, readable instructions without model training or hyperparameter tuning.

## Key points
- **Core loop:** a feedback prompt (∇) summarizes the current prompt's flaws from its minibatch errors. An editing prompt (δ) produces improved versions, and a paraphrasing step (LLM_mc) explores nearby wordings.
- **Candidate selection:** the paper considers UCB, UCB-E and Successive Rejects bandits. Successive Rejects is provably optimal for best arm identification and needs no hyperparameters.
- **Evaluation:** four classification tasks, including Ethos (hate speech) and the newly proposed jailbreak detection problem. Baselines include Monte Carlo (MC) paraphrase search and an RL-based prompt editor.
- **Results:** up to 31% gain over the initial prompt and an average 4–8% gain over state-of-the-art prompt learning baselines, with fewer LLM API calls.
- **Limitations and resources:**
  - API rate limits and the many calls required can push runtime past an hour.
  - Testing was limited to classification tasks.
  - Code and data are available at github.com/microsoft/LMOps (prompt_optimization).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

210 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 200 not yet.

- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://md.research.elamur.ai/free/en/papers/fernando-2023-promptbreeder-self-referential-self-improvement/)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://md.research.elamur.ai/free/en/papers/kamoi-2024-when-can-llms-actually-correct/)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://md.research.elamur.ai/free/en/papers/khattab-2023-dspy-compiling-declarative-language-model/)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/)
- [To Write or to Automate Linguistic Prompts, That Is the Question](https://md.research.elamur.ai/free/en/papers/torron-2026-write-or-automate-linguistic-prompts/)
- [Large Language Models as Optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/)
- [GBC: Gradient-Based Connections for Optimizing Multi-Agent Systems](https://md.research.elamur.ai/free/en/papers/yang-2026-gbc-gradient-based-connections-optimizing/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pryzant2023automatic,
  title = {{Automatic Prompt Optimization with “Gradient Descent” and Beam Search}},
  author = {Pryzant, Reid and Iter, Dan and Li, Jerry and Lee, Yin and Zhu, Chenguang and Zeng, Michael},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.494},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.494}
}
```

- [← previous Automatic Evaluation and Analysis of Idioms in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/baziotis-2023-automatic-evaluation-analysis-idioms-neural/)
- [next → Automatic Standardization of Arabic Dialects for Machine Translation](https://md.research.elamur.ai/free/en/papers/alnassan-2023-automatic-standardization-arabic-dialects-machine/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)

## References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jean-Yves Audibert, Sébastien Bubeck, and Rémi Munos. 2010. Best arm identification in multi-armed bandits. In COLT, pages 41–53. [find](https://scholar.google.com/scholar?q=Jean%2DYves+Audibert%2C+S%C3%A9bastien+Bubeck%2C+and+R%C3%A9mi+Munos%2E+2010%2E+Best+arm+identification+in+multi%2Darmed+bandits%2E+In+COLT%2C+pages+41%E2%80%9353%2E)
2. Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72. [find](https://scholar.google.com/scholar?q=Steven+Bird%2E+2006%2E+Nltk%3A+the+natural+language+toolkit%2E+In+Proceedings+of+the+COLING%2FACL+2006+Interactive+Presentation+Sessions%2C+pages+69%E2%80%9372%2E) [our card](https://md.research.elamur.ai/free/en/papers/bird-2006-nltk-natural-language-toolkit/)
3. Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. 2012. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning, 5(1):1–122. [find](https://scholar.google.com/scholar?q=S%C3%A9bastien+Bubeck%2C+Nicolo+Cesa%2DBianchi%2C+et+al%2E+2012%2E+Regret+analysis+of+stochastic+and+nonstochastic+multi%2Darmed+bandit+problems%2E+Foundations+and+Trends%C2%AE+in+Machine+Learning%2C+5%281%29%3A1%E2%80%93122%2E)
4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. [find](https://scholar.google.com/scholar?q=S%C3%A9bastien+Bubeck%2C+Varun+Chandrasekaran%2C+Ronen+Eldan%2C+Johannes+Gehrke%2C+Eric+Horvitz%2C+Ece+Kamar%2C+Peter+Lee%2C+Yin+Tat+Lee%2C+Yuanzhi+Li%2C+Scott+Lundberg%2C+et+al%2E+2023%2E+Sparks+of+artificial+general+intelligenc) [our card](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)
5. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2E)
6. Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. 2022. Rlprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205.12548. [find](https://scholar.google.com/scholar?q=Mingkai+Deng%2C+Jianyu+Wang%2C+Cheng%2DPing+Hsieh%2C+Yihan+Wang%2C+Han+Guo%2C+Tianmin+Shu%2C+Meng+Song%2C+Eric+P+Xing%2C+and+Zhiting+Hu%2E+2022%2E+Rlprompt%3A+Optimizing+discrete+text+prompts+with+reinforcement+learning%2E+arX) [our card](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/)
7. Ibrahim Abu Farha and Walid Magdy. 2020. From arabic sentiment analysis to sarcasm detection: The arsarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39. [find](https://scholar.google.com/scholar?q=Ibrahim+Abu+Farha+and+Walid+Magdy%2E+2020%2E+From+arabic+sentiment+analysis+to+sarcasm+detection%3A+The+arsarcasm+dataset%2E+In+Proceedings+of+the+4th+Workshop+on+Open%2DSource+Arabic+Corpora+and+Processing+Too) [our card](https://md.research.elamur.ai/free/en/papers/farha-2020-arabic-sentiment-analysis-sarcasm-detection/)
8. Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Adam+Fisch%2C+and+Danqi+Chen%2E+2020%2E+Making+pre%2Dtrained+language+models+better+few%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2012%2E15723%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/)
9. Yiduo Guo, Yaobo Liang, Chenfei Wu, Wenshan Wu, Dongyan Zhao, and Nan Duan. 2023. Learning to program with natural language. arXiv preprint arXiv:2304.10464. — [arXiv:2304.10464](https://arxiv.org/abs/2304.10464) [find](https://scholar.google.com/scholar?q=Yiduo+Guo%2C+Yaobo+Liang%2C+Chenfei+Wu%2C+Wenshan+Wu%2C+Dongyan+Zhao%2C+and+Nan+Duan%2E+2023%2E+Learning+to+program+with+natural+language%2E+arXiv+preprint+arXiv%3A2304%2E10464%2E+%E2%80%94+%5BarXiv%3A2304%2E10464%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
10. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. Warp: Word-level adversarial reprogramming. arXiv preprint arXiv:2101.00121. [find](https://scholar.google.com/scholar?q=Karen+Hambardzumyan%2C+Hrant+Khachatrian%2C+and+Jonathan+May%2E+2021%2E+Warp%3A+Word%2Dlevel+adversarial+reprogramming%2E+arXiv+preprint+arXiv%3A2101%2E00121%2E)
11. Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. 2022. Optimizing prompts for text-to-image generation. arXiv preprint arXiv:2212.09611. — [arXiv:2212.09611](https://arxiv.org/abs/2212.09611) [find](https://scholar.google.com/scholar?q=Yaru+Hao%2C+Zewen+Chi%2C+Li+Dong%2C+and+Furu+Wei%2E+2022%2E+Optimizing+prompts+for+text%2Dto%2Dimage+generation%2E+arXiv+preprint+arXiv%3A2212%2E09611%2E+%E2%80%94+%5BarXiv%3A2212%2E09611%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2212%2E09611%29)
12. Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. 2022. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.10782. [find](https://scholar.google.com/scholar?q=Or+Honovich%2C+Uri+Shaham%2C+Samuel+R+Bowman%2C+and+Omer+Levy%2E+2022%2E+Instruction+induction%3A+From+few+examples+to+natural+language+task+descriptions%2E+arXiv+preprint+arXiv%3A2205%2E10782%2E)
13. Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022. Promptmaker: Prompt-based prototyping with large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pages 1–8. [find](https://scholar.google.com/scholar?q=Ellen+Jiang%2C+Kristen+Olson%2C+Edwin+Toh%2C+Alejandra+Molina%2C+Aaron+Donsbach%2C+Michael+Terry%2C+and+Carrie+J+Cai%2E+2022%2E+Promptmaker%3A+Prompt%2Dbased+prototyping+with+large+language+models%2E+In+CHI+Conference+on+H)
14. Zohar Karnin, Tomer Koren, and Oren Somekh. 2013. Almost optimal exploration in multi-armed bandits. In International Conference on Machine Learning, pages 1238–1246. PMLR. [find](https://scholar.google.com/scholar?q=Zohar+Karnin%2C+Tomer+Koren%2C+and+Oren+Somekh%2E+2013%2E+Almost+optimal+exploration+in+multi%2Darmed+bandits%2E+In+International+Conference+on+Machine+Learning%2C+pages+1238%E2%80%931246%2E+PMLR%2E)
15. Volodymyr Kuleshov and Doina Precup. 2014. Algorithms for multi-armed bandit problems. arXiv preprint arXiv:1402.6028. — [arXiv:1402.6028](https://arxiv.org/abs/1402.6028) [find](https://scholar.google.com/scholar?q=Volodymyr+Kuleshov+and+Doina+Precup%2E+2014%2E+Algorithms+for+multi%2Darmed+bandit+problems%2E+arXiv+preprint+arXiv%3A1402%2E6028%2E+%E2%80%94+%5BarXiv%3A1402%2E6028%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1402%2E6028%29)
16. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691. [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+2021%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+arXiv+preprint+arXiv%3A2104%2E08691%2E)
17. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092–1097. [find](https://scholar.google.com/scholar?q=Yujia+Li%2C+David+Choi%2C+Junyoung+Chung%2C+Nate+Kushman%2C+Julian+Schrittwieser%2C+R%C3%A9mi+Leblond%2C+Tom+Eccles%2C+James+Keeling%2C+Felix+Gimeno%2C+Agustin+Dal+Lago%2C+et+al%2E+2022%2E+Competition%2Dlevel+code+generation+with+a) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
18. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786. [find](https://scholar.google.com/scholar?q=Yao+Lu%2C+Max+Bartolo%2C+Alastair+Moore%2C+Sebastian+Riedel%2C+and+Pontus+Stenetorp%2E+2021%2E+Fantastically+ordered+prompts+and+where+to+find+them%3A+Overcoming+few%2Dshot+prompt+order+sensitivity%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/)
19. Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2020. Ethos: an online hate speech detection dataset. arXiv preprint arXiv:2006.08328. — [arXiv:2006.08328](https://arxiv.org/abs/2006.08328) [find](https://scholar.google.com/scholar?q=Ioannis+Mollas%2C+Zoe+Chrysopoulou%2C+Stamatis+Karlos%2C+and+Grigorios+Tsoumakas%2E+2020%2E+Ethos%3A+an+online+hate+speech+detection+dataset%2E+arXiv+preprint+arXiv%3A2006%2E08328%2E+%E2%80%94+%5BarXiv%3A2006%2E08328%5D%28https%3A%2F%2Farxiv%2Eor)
20. OpenAI. 2023. Gpt-4 technical report. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+Gpt%2D4+technical+report%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
21. Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2022. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.07281. [find](https://scholar.google.com/scholar?q=Archiki+Prasad%2C+Peter+Hase%2C+Xiang+Zhou%2C+and+Mohit+Bansal%2E+2022%2E+Grips%3A+Gradient%2Dfree%2C+edit%2Dbased+instruction+search+for+prompting+large+language+models%2E+arXiv+preprint+arXiv%3A2203%2E07281%2E) [our card](https://md.research.elamur.ai/free/en/papers/prasad-2023-grips-gradient-free-edit-based/)
22. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. arXiv preprint arXiv:2104.06599. [find](https://scholar.google.com/scholar?q=Guanghui+Qin+and+Jason+Eisner%2E+2021%2E+Learning+how+to+ask%3A+Querying+lms+with+mixtures+of+soft+prompts%2E+arXiv+preprint+arXiv%3A2104%2E06599%2E) [our card](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/)
23. Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7. [find](https://scholar.google.com/scholar?q=Laria+Reynolds+and+Kyle+McDonell%2E+2021%2E+Prompt+programming+for+large+language+models%3A+Beyond+the+few%2Dshot+paradigm%2E+In+Extended+Abstracts+of+the+2021+CHI+Conference+on+Human+Factors+in+Computing+Syste) [our card](https://md.research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/)
24. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980. [find](https://scholar.google.com/scholar?q=Taylor+Shin%2C+Yasaman+Razeghi%2C+Robert+L+Logan+IV%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+2020%2E+Autoprompt%3A+Eliciting+knowledge+from+language+models+with+automatically+generated+prompts%2E+arXiv+preprint+arXiv%3A2) [our card](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/)
25. William Yang Wang. 2017. " liar, liar pants on fire": A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648. [find](https://scholar.google.com/scholar?q=William+Yang+Wang%2E+2017%2E+%22+liar%2C+liar+pants+on+fire%22%3A+A+new+benchmark+dataset+for+fake+news+detection%2E+arXiv+preprint+arXiv%3A1705%2E00648%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2017-liar-liar-pants-fire-new/)
26. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+pre)
27. Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. 2022. Gps: Genetic prompt search for efficient few-shot learning. arXiv preprint arXiv:2210.17041. [find](https://scholar.google.com/scholar?q=Hanwei+Xu%2C+Yujun+Chen%2C+Yulun+Du%2C+Nan+Shao%2C+Yanggang+Wang%2C+Haiyu+Li%2C+and+Zhilin+Yang%2E+2022%2E+Gps%3A+Genetic+prompt+search+for+efficient+few%2Dshot+learning%2E+arXiv+preprint+arXiv%3A2210%2E17041%2E)
28. J Zamfirescu-Pereira, Richmond Wong, Bjoern Hartmann, and Qian Yang. 2023. Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI conference on human factors in computing systems (CHI’23). [find](https://scholar.google.com/scholar?q=J+Zamfirescu%2DPereira%2C+Richmond+Wong%2C+Bjoern+Hartmann%2C+and+Qian+Yang%2E+2023%2E+Why+johnny+can%E2%80%99t+prompt%3A+how+non%2Dai+experts+try+%28and+fail%29+to+design+llm+prompts%2E+In+Proceedings+of+the+2023+CHI+conference+o)
29. Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. 2022. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598. [find](https://scholar.google.com/scholar?q=Andy+Zeng%2C+Adrian+Wong%2C+Stefan+Welker%2C+Krzysztof+Choromanski%2C+Federico+Tombari%2C+Aveek+Purohit%2C+Michael+Ryoo%2C+Vikas+Sindhwani%2C+Johnny+Lee%2C+Vincent+Vanhoucke%2C+et+al%2E+2022%2E+Socratic+models%3A+Composing+zer) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2022-socratic-models-composing-zero-shot/)
30. Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. 2023. Tempera: Test-time prompt editing via reinforcement learning. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianjun+Zhang%2C+Xuezhi+Wang%2C+Denny+Zhou%2C+Dale+Schuurmans%2C+and+Joseph+E+Gonzalez%2E+2023%2E+Tempera%3A+Test%2Dtime+prompt+editing+via+reinforcement+learning%2E+In+The+Eleventh+International+Conference+on+Learning)
31. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910. [find](https://scholar.google.com/scholar?q=Yongchao+Zhou%2C+Andrei+Ioan+Muresanu%2C+Ziwen+Han%2C+Keiran+Paster%2C+Silviu+Pitis%2C+Harris+Chan%2C+and+Jimmy+Ba%2E+2022%2E+Large+language+models+are+human%2Dlevel+prompt+engineers%2E+arXiv+preprint+arXiv%3A2211%2E01910%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/)