--- title: "Automatic Chain of Thought Prompting in Large — Zhang 2022" description: "Chain-of-thought (CoT) prompting helps large language models solve multi-step problems." authors: ["Zhuosheng Zhang", "Aston Zhang", "Mu Li", "Alex Smola"] year: 2022 venue: "arXiv:2210.03493" source: "https://arxiv.org/abs/2210.03493" doi: "10.48550/arxiv.2210.03493" arxiv: "2210.03493" licence: "CC BY-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / Automatic Chain of Thought Prompting in Large Language Models # Automatic Chain of Thought Prompting in Large Language Models **Authors:** [Zhuosheng Zhang](https://md.research.elamur.ai/free/en/authors/zhuosheng-zhang/), [Aston Zhang](https://md.research.elamur.ai/free/en/authors/aston-zhang/), [Mu Li](https://md.research.elamur.ai/free/en/authors/mu-li/), [Alex Smola](https://md.research.elamur.ai/free/en/authors/alex-smola/) · **Year:** 2022 · **Venue:** arXiv:2210.03493 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2210.03493) · [PDF](https://arxiv.org/pdf/2210.03493) · [DOI 10.48550/arxiv.2210.03493](https://doi.org/10.48550/arxiv.2210.03493) · [arXiv 2210.03493](https://arxiv.org/abs/2210.03493) · [OpenAlex](https://openalex.org/W4304194220) **Citations:** 229 citations (openalex-snapshot · 2026-10-14) · [459](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Chain-of-thought (CoT) prompting helps large language models solve multi-step problems. The best-performing variant, Manual-CoT, needs people to write example questions with worked reasoning, and each task needs its own set. This paper replaces that manual work with Auto-CoT. The authors first show that simple automation does not work well. If you retrieve questions similar to the test question and let Zero-Shot-CoT ("Let's think step by step") write their rationales, the model's reasoning errors tend to repeat across the demonstrations. Their fix is to make the demonstration questions diverse. Auto-CoT encodes the questions of a dataset with Sentence-BERT and groups them into clusters. It then takes one representative question from each cluster, has Zero-Shot-CoT generate its reasoning chain, and filters these chains with simple heuristics. Experiments used GPT-3 (text-davinci-002, 175B) on ten arithmetic, commonsense and symbolic reasoning benchmarks. Auto-CoT matched or beat Manual-CoT on every one of them. This matters because good CoT prompting becomes practical for any task without hand-written demonstrations, and each dataset gets its own prompts. ## Key points - Benchmarks: MultiArith, GSM8K, AddSub, AQuA, SingleEq, SVAMP (arithmetic); CSQA, StrategyQA (commonsense); Last Letter Concatenation, Coin Flip (symbolic). Only unlabeled test questions are used, with no training set. - With GPT-3, Auto-CoT vs Manual-CoT: MultiArith 92.0 vs 91.7, GSM8K 47.9 vs 46.9, CSQA 74.4 vs 73.5, Coin Flip 99.9 vs 97.2. On StrategyQA both scored 65.4. - With Codex, Auto-CoT stays competitive. It is ahead on GSM8K (62.8 vs 59.4) and AddSub (91.9 vs 84.6) but behind on MultiArith (93.2 vs 96.8). - Diverse sampling makes the method robust to bad demonstrations: on MultiArith, accuracy barely drops even when half the demonstrations have wrong answers. Sampling from the test question's own cluster is much less robust. - A bootstrapping variant, Auto-CoT*, handles questions that arrive in batches of 30. From the second batch on, it performs about as well as Manual-CoT. Code is at github.com/amazon-research/auto-cot. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 459 papers of our library cite it (of the 140673 whose bibliographies we read); 28 of them have a page here, 431 not yet. - [Many-Shot In-Context Learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/) - [Quiet-STaR: Language models can teach themselves to think before speaking](https://md.research.elamur.ai/free/en/papers/al-2024-quiet-star-language-models-teach/) - [A Survey on In-context Learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/) - [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://md.research.elamur.ai/free/en/papers/fan-2024-survey-rag-meeting-llms-towards/) - [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://md.research.elamur.ai/free/en/papers/fernando-2023-promptbreeder-self-referential-self-improvement/) - [Exploring Human-Like Translation Strategy with Large Language Models](https://md.research.elamur.ai/free/en/papers/he-2024-exploring-human-like-translation-strategy/) - [Large Language Models Can Self-Improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/) - [Towards Reasoning in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/) - [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/) - [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/) - [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://md.research.elamur.ai/free/en/papers/khattab-2022-demonstrate-search-predict-composing-retrieval/) - [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://md.research.elamur.ai/free/en/papers/khattab-2023-dspy-compiling-declarative-language-model/) - [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/) - [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://md.research.elamur.ai/free/en/papers/liang-2023-encouraging-divergent-thinking-large-language/) - [Deductive Verification of Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/ling-2023-deductive-verification-chain-thought-reasoning/) - [Query Rewriting in Retrieval-Augmented Large Language Models](https://md.research.elamur.ai/free/en/papers/ma-2023-query-rewriting-retrieval-augmented-large/) - [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://md.research.elamur.ai/free/en/papers/pourreza-2023-din-sql-decomposed-context-learning/) - [Reasoning with Language Model Prompting: A Survey](https://md.research.elamur.ai/free/en/papers/qiao-2022-reasoning-language-model-prompting-survey/) - [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) - [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://md.research.elamur.ai/free/en/papers/sahoo-2024-systematic-survey-prompt-engineering-large/) - [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://md.research.elamur.ai/free/en/papers/wang-2023-understanding-chain-thought-prompting-empirical/) - [Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration](https://md.research.elamur.ai/free/en/papers/wang-2023-unleashing-emergent-cognitive-synergy-large/) - [Large Language Models are Better Reasoners with Self-Verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/) - [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://md.research.elamur.ai/free/en/papers/wu-2023-visual-chatgpt-talking-drawing-editing/) - [Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?](https://md.research.elamur.ai/free/en/papers/yao-2024-sarcasm-detection-step-step-reasoning/) - [Multimodal Chain-of-Thought Reasoning in Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-multimodal-chain-thought-reasoning-language/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) - [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{zhang2022automatic, title = {{Automatic Chain of Thought Prompting in Large Language Models}}, author = {Zhang, Zhuosheng and Zhang, Aston and Li, Mu and Smola, Alex}, year = {2022}, doi = {10.48550/arxiv.2210.03493}, eprint = {2210.03493}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2210.03493} } ``` - [← previous Autoformalization with Large Language Models](https://md.research.elamur.ai/free/en/papers/wu-2022-autoformalization-large-language-models/) - [next → Automatic Dialect Density Estimation for African American English](https://md.research.elamur.ai/free/en/papers/johnson-2022-automatic-dialect-density-estimation-african/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/) ## References 9 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche) 2. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzho… [find](https://scholar.google.com/scholar?q=Romal+Thoppilan%2C+Daniel+De+Freitas%2C+Jamie+Hall%2C+Noam+Shazeer%2C+Apoorv+Kulshreshtha%2C+Heng%2DTze+Cheng%2C+Alicia+Jin%2C+Taylor+Bos%2C+Leslie+Baker%2C+Yu+Du%2C+YaGuang+Li%2C+Hongrae+Lee%2C+Huaixiu+Steven+Zheng%2C+Amin+Ghaf) [our card](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/) 3. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick%2C+A) 4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+Parker+Schuh%2C+Kensen+Shi%2C+Sasha+Tsvyashc) 5. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022), 2022a. URL https://arxiv.org/abs/2201.11903. [↗](https://arxiv.org/abs/2201.11903) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) 6. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022), 2022. URL https://arxiv.org/abs/2205.11916. [↗](https://arxiv.org/abs/2205.11916) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) 7. Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752, Lisbon, Portugal, 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1202. URL https://aclanthology.org/D1… [↗](https://aclanthology.org/D1…) [our card](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/) 8. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… [find](https://scholar.google.com/scholar?q=Alon+Talmor%2C+Jonathan+Herzig%2C+Nicholas+Lourie%2C+and+Jonathan+Berant%2E+CommonsenseQA%3A+A+question+answering+challenge+targeting+commonsense+knowledge%2E+In+Proceedings+of+the+2019+Conference+of+the+North+Am) 9. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, 2021. doi: 10.18653/v1/2021.emnlp-ma… [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Peter+West%2C+Vered+Shwartz%2C+Yejin+Choi%2C+and+Luke+Zettlemoyer%2E+Surface+form+competition%3A+Why+the+highest+probability+answer+isn%E2%80%99t+always+right%2E+In+Proceedings+of+the+2021+Conference+on+Emp) [our card](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/)