---
title: "LAMBADA — paper by Kazemi et al. (2023) — summary and ideas"
description: "The authors borrow the classical AI preference for goal-directed search and propose… 0.94 accuracy. 45 references, cited by 57 papers here."
authors: ["Mehran Kazemi", "Najoung Kim", "Deepti Bhatia", "Xin Xu", "Deepak Ramachandran"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.361/"
doi: "10.18653/v1/2023.acl-long.361"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kazemi-2023-lambada-backward-chaining-automated-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kazemi-2023-lambada-backward-chaining-automated-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / LAMBADA: Backward Chaining for Automated Reasoning in Natural Language

# LAMBADA: Backward Chaining for Automated Reasoning in Natural Language

**Authors:** [Mehran Kazemi](https://md.research.elamur.ai/free/en/authors/mehran-kazemi/), [Najoung Kim](https://md.research.elamur.ai/free/en/authors/najoung-kim/), Deepti Bhatia, [Xin Xu](https://md.research.elamur.ai/free/en/authors/xin-xu/) and [Deepak Ramachandran](https://md.research.elamur.ai/free/en/authors/deepak-ramachandran/) · **Year:** 2023 · **Venue:** [Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2023](https://md.research.elamur.ai/free/en/venues/acl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.361/) · [PDF](https://aclanthology.org/2023.acl-long.361.pdf) · [DOI 10.18653/v1/2023.acl-long.361](https://doi.org/10.18653/v1/2023.acl-long.361) · [OpenAlex](https://openalex.org/W4385570291)

**Citations:** 32 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/kazemi-2023-lambada-backward-chaining-automated-reasoning/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Language models still struggle with multi-step deductive reasoning over text. Existing prompting methods such as Chain-of-Thought and Selection-Inference (SI) build proofs forward from facts toward the goal. This forces a combinatorial choice of which facts and rules to combine, and it gives no clear signal for when to stop. The authors borrow the classical AI preference for goal-directed search and propose LAMBADA, a backward chaining algorithm. It starts from the statement to be proved and recursively splits it into sub-goals until they can be checked against the given facts. The symbolic control loop handles proof planning. A few-shot prompted LM (PaLM) executes each step using four modules: one checks facts, one selects rules, one decomposes goals and one verifies sign agreement (Fact Check, Rule Selection, Goal Decomposition, Sign Agreement). Every goal is labelled PROVED, DISPROVED or UNKNOWN. On two logical reasoning benchmarks, including ProofWriter (with the paraphrased ParaRules subset), LAMBADA outperforms forward-reasoning baselines, especially on deep proofs. It also produces more valid proof traces and needs fewer LM calls than SI. The results suggest that LM reasoning systems benefit from goal-directed planning.

## Key points
- The benchmarks include proofs up to 5 hops deep, as well as goals that the theory can neither prove nor disprove. LAMBADA's advantage over CoT and SI grows with depth.
- Compared with SI on ProofWriter-PUD, LAMBADA makes 3.8x fewer LM inference calls at Depth-1 and 11.8x fewer at Depth-5. It still needs more calls than CoT.
- Module-level analysis:
  - Fact Check reaches 0.94 accuracy when it may select one fact, and near-perfect accuracy with two.
  - Sign Agreement is near perfect.
  - With PaLM 62B, Fact Check and Rule Selection degrade noticeably. With PaLM 8B, all modules fall sharply, some to near chance.
- Accuracy stays almost unchanged on modified ProofWriter-PUD test sets that use novel tokens and novel rule templates, which indicates lexical robustness.
- Stated limitations:
  - The method only covers classification-style entailment with modus ponens.
  - All rules must fit in the prompt.
  - LM calls are sequential and cannot be batched.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu and Deepak Ramachandran. 2023. LAMBADA: Backward Chaining for Automated Reasoning in Natural Language. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kazemi2023lambada,
  title = {{LAMBADA: Backward Chaining for Automated Reasoning in Natural Language}},
  author = {Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu and Deepak Ramachandran},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.361},
  url = {https://doi.org/10.18653/v1/2023.acl-long.361}
}
```

- [← previous Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks](https://md.research.elamur.ai/free/en/papers/sottana-2023-evaluation-metrics-era-gpt-4/)
- [next → Language Modeling Is Compression](https://md.research.elamur.ai/free/en/papers/deletang-2023-language-modeling-compression/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

45 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022. [Exploring length generalization in large language models](https://md.research.elamur.ai/free/en/papers/anil-2022-exploring-length-generalization-large-language/). In Advances in Neural Information Processing Systems, volume 35, pages 38546–38556. Cur…
2. Gregor Betz, Christian Voigt, and Kyle Richardson. 2021. Critical thinking for language models. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 63–75, Groningen, The Netherlands (online). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gregor+Betz%2C+Christian+Voigt%2C+and+Kyle+Richardson%2E+2021%2E+Critical+thinking+for+language+models%2E+In+Proceedings+of+the+14th+International+Conference+on+Computational+Semantics+%28IWCS%29%2C+pages+63%E2%80%9375%2C+Gron)
3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
4. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. [Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi…](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
6. Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2021. [Transformers as soft reasoners over language](https://md.research.elamur.ai/free/en/papers/clark-2020-transformers-soft-reasoners-over-language/). In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI’20.
7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv:2110.14168.
8. Antonia Creswell and Murray Shanahan. 2022. Faithful reasoning using large language models. arXiv:2208.14271. [find](https://scholar.google.com/scholar?q=Antonia+Creswell+and+Murray+Shanahan%2E+2022%2E+Faithful+reasoning+using+large+language+models%2E+arXiv%3A2208%2E14271%2E)
9. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2023. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Antonia+Creswell%2C+Murray+Shanahan%2C+and+Irina+Higgins%2E+2023%2E+Selection%2Dinference%3A+Exploiting+large+language+models+for+interpretable+logical+reasoning%2E+In+The+Eleventh+International+Conference+on+Learn)
10. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1–220. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Dan+Roth%2C+Mark+Sammons%2C+and+Fabio+Massimo+Zanzotto%2E+2013%2E+Recognizing+textual+entailment%3A+Models+and+applications%2E+Synthesis+Lectures+on+Human+Language+Technologies%2C+6%284%29%3A1%E2%80%93220%2E)
11. Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021. [Explaining answers with entailment trees](https://md.research.elamur.ai/free/en/papers/dalvi-2021-explaining-answers-entailment-trees/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7358–7370, Online and Punta Cana, Domin…
12. Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Successive prompting for decomposing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1251–1265, Abu Dhabi, United Arab Emirates. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Shivanshu+Gupta%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2022%2E+Successive+prompting+for+decomposing+complex+questions%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Languag)
13. Artur d’Avila Garcez and Luis C Lamb. 2020. Neurosymbolic ai: the 3rd wave. arXiv:2012.05876. [find](https://scholar.google.com/scholar?q=Artur+d%E2%80%99Avila+Garcez+and+Luis+C+Lamb%2E+2020%2E+Neurosymbolic+ai%3A+the+3rd+wave%2E+arXiv%3A2012%2E05876%2E)
14. Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Chris Pal. 2020. Measuring systematic generalization in neural proof generation with transformers. In Advances in Neural Information Processing Systems, volume 33, pages 22231–22242. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Nicolas+Gontier%2C+Koustuv+Sinha%2C+Siva+Reddy%2C+and+Chris+Pal%2E+2020%2E+Measuring+systematic+generalization+in+neural+proof+generation+with+transformers%2E+In+Advances+in+Neural+Information+Processing+Systems%2C)
15. Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, et al. 2022. [FOLIO: Natural language reasoning with first-order logic](https://md.research.elamur.ai/free/en/papers/han-2022-folio-natural-language-reasoning-first/). arXiv:2209.00840.
16. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran.
17. Carl Hewitt. 1969. Planner: A language for proving theorems in robots. In Proceedings of the 1st International Joint Conference on Artificial Intelligence, IJCAI’69, page 295–301, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=Carl+Hewitt%2E+1969%2E+Planner%3A+A+language+for+proving+theorems+in+robots%2E+In+Proceedings+of+the+1st+International+Joint+Conference+on+Artificial+Intelligence%2C+IJCAI%E2%80%9969%2C+page+295%E2%80%93301%2C+San+Francisco%2C+CA%2C+U)
18. Jie Huang and Kevin Chen-Chuan Chang. 2022. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/). arXiv:2212.10403.
19. Harsh Jhamtani and Peter Clark. 2020. Learning to explain: Datasets and models for identifying valid reasoning chains in multihop question-answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 137–150, Online. Association for Computationa… [find](https://scholar.google.com/scholar?q=Harsh+Jhamtani+and+Peter+Clark%2E+2020%2E+Learning+to+explain%3A+Datasets+and+models+for+identifying+valid+reasoning+chains+in+multihop+question%2Danswering%2E+In+Proceedings+of+the+2020+Conference+on+Empirical)
20. Nora Kassner, Benno Krojer, and Hinrich Schütze. 2020. [Are pretrained language models symbolic reasoners over knowledge?](https://md.research.elamur.ai/free/en/papers/kassner-2020-are-pretrained-language-models-symbolic/) In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 552–564, Online. Association for Computational Linguistics.
21. Mehran Kazemi, Sid Mittal, and Deepak Ramachandran. 2023. Understanding finetuning for factual knowledge extraction from language models. arXiv:2301.11293. [find](https://scholar.google.com/scholar?q=Mehran+Kazemi%2C+Sid+Mittal%2C+and+Deepak+Ramachandran%2E+2023%2E+Understanding+finetuning+for+factual+knowledge+extraction+from+language+models%2E+arXiv%3A2301%2E11293%2E)
22. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Harsh+Trivedi%2C+Matthew+Finlayson%2C+Yao+Fu%2C+Kyle+Richardson%2C+Peter+Clark%2C+and+Ashish+Sabharwal%2E+2023%2E+Decomposed+prompting%3A+A+modular+approach+for+solving+complex+tasks%2E+In+The+Eleventh+Int)
23. Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. 2023. Transformers learn shortcuts to automata. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bingbin+Liu%2C+Jordan+T%2E+Ash%2C+Surbhi+Goel%2C+Akshay+Krishnamurthy%2C+and+Cyril+Zhang%2E+2023%2E+Transformers+learn+shortcuts+to+automata%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2E)
24. Gary Marcus. 2020. The next decade in AI: four steps towards robust artificial intelligence. arXiv:2002.06177. [find](https://scholar.google.com/scholar?q=Gary+Marcus%2E+2020%2E+The+next+decade+in+AI%3A+four+steps+towards+robust+artificial+intelligence%2E+arXiv%3A2002%2E06177%2E)
25. John McCarthy. 1959. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes, pages 75–91, London. Her Majesty’s Stationary Office. [find](https://scholar.google.com/scholar?q=John+McCarthy%2E+1959%2E+Programs+with+common+sense%2E+In+Proceedings+of+the+Teddington+Conference+on+the+Mechanization+of+Thought+Processes%2C+pages+75%E2%80%9391%2C+London%2E+Her+Majesty%E2%80%99s+Stationary+Office%2E)
26. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2022. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). In Deep Learning for Co…
27. David L Poole and Alan K Mackworth. 2010. Artificial Intelligence: foundations of computational agents. Cambridge University Press. [find](https://scholar.google.com/scholar?q=David+L+Poole+and+Alan+K+Mackworth%2E+2010%2E+Artificial+Intelligence%3A+foundations+of+computational+agents%2E+Cambridge+University+Press%2E)
28. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick%2C+A)
29. Mohammed Saeed, Naser Ahmadi, Preslav Nakov, and Paolo Papotti. 2021. RuleBERT: Teaching soft rules to pre-trained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1460–1476, Online and Punta Cana, Dominican Republic. Association for C… [find](https://scholar.google.com/scholar?q=Mohammed+Saeed%2C+Naser+Ahmadi%2C+Preslav+Nakov%2C+and+Paolo+Papotti%2E+2021%2E+RuleBERT%3A+Teaching+soft+rules+to+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natura)
30. Abulhair Saparov and He He. 2023. [Language models are greedy reasoners: A systematic formal analysis of chain-of-thought](https://md.research.elamur.ai/free/en/papers/saparov-2022-language-models-greedy-reasoners-systematic/). In The Eleventh International Conference on Learning Representations.
31. Imanol Schlag, Sainbayar Sukhbaatar, Asli Celikyilmaz, Wen-tau Yih, Jason Weston, Jürgen Schmidhuber, and Xian Li. 2023. Large language model programs. arXiv preprint arXiv:2305.05364. [find](https://scholar.google.com/scholar?q=Imanol+Schlag%2C+Sainbayar+Sukhbaatar%2C+Asli+Celikyilmaz%2C+Wen%2Dtau+Yih%2C+Jason+Weston%2C+J%C3%BCrgen+Schmidhuber%2C+and+Xian+Li%2E+2023%2E+Large+language+model+programs%2E+arXiv+preprint+arXiv%3A2305%2E05364%2E)
32. Viktor Schlegel, Kamen Pavlov, and Ian Pratt-Hartmann. 2022. Can transformers reason in fragments of natural language? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11184–11199, Abu Dhabi, United Arab Emirates. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Viktor+Schlegel%2C+Kamen+Pavlov%2C+and+Ian+Pratt%2DHartmann%2E+2022%2E+Can+transformers+reason+in+fragments+of+natural+language%3F+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Pr)
33. Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A multi-task framework for math word problems. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2269–2279, Punta Cana, Dominican Republic. Association for Computa… [find](https://scholar.google.com/scholar?q=Jianhao+Shen%2C+Yichun+Yin%2C+Lin+Li%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Ming+Zhang%2C+and+Qun+Liu%2E+2021%2E+Generate+%26+rank%3A+A+multi%2Dtask+framework+for+math+word+problems%2E+In+Findings+of+the+Association+for+Computation)
34. Zayne Sprague, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2022. Natural language deduction with incomplete information. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8230–8258, Abu Dhabi, United Arab Emirates. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Zayne+Sprague%2C+Kaj+Bostrom%2C+Swarat+Chaudhuri%2C+and+Greg+Durrett%2E+2022%2E+Natural+language+deduction+with+incomplete+information%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Lang)
35. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv:2210.09261.
36. Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. [ProofWriter: Generating implications, proofs, and abductive statements over natural language](https://md.research.elamur.ai/free/en/papers/tafjord-2021-proofwriter-generating-implications-proofs-abductive/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3621–3634, Online. Association for Computational Linguistics.
37. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop. [find](https://scholar.google.com/scholar?q=Karthik+Valmeekam%2C+Alberto+Olmo%2C+Sarath+Sreedharan%2C+and+Subbarao+Kambhampati%2E+2022%2E+Large+language+models+still+can%E2%80%99t+plan+%28a+benchmark+for+LLMs+on+planning+and+reasoning+about+change%29%2E+In+NeurIPS+202)
38. Boshi Wang, Xiang Deng, and Huan Sun. 2022. Iteratively prompt pre-trained language models for chain of thought. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2714–2730, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Boshi+Wang%2C+Xiang+Deng%2C+and+Huan+Sun%2E+2022%2E+Iteratively+prompt+pre%2Dtrained+language+models+for+chain+of+thought%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
40. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association fo… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+the+North+American+Chapter+of+the+Association+for+Computation)
41. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. [STaR: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). In Advances in Neural Information Processing Systems, volume 35, pages 15476–15488. Curran Associates, Inc.
42. Hanlin Zhang, Ziyang Li, Jiani Huang, Mayur Naik, and Eric Xing. 2022a. Improved logical reasoning of language models via differentiable symbolic programming. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022. [find](https://scholar.google.com/scholar?q=Hanlin+Zhang%2C+Ziyang+Li%2C+Jiani+Huang%2C+Mayur+Naik%2C+and+Eric+Xing%2E+2022a%2E+Improved+logical+reasoning+of+language+models+via+differentiable+symbolic+programming%2E+In+First+Workshop+on+Pre%2Dtraining%3A+Perspe)
43. Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. 2022b. On the paradox of learning to reason from data. arXiv:2205.11502. — [arXiv:2205.11502](https://arxiv.org/abs/2205.11502) [find](https://scholar.google.com/scholar?q=Honghua+Zhang%2C+Liunian+Harold+Li%2C+Tao+Meng%2C+Kai%2DWei+Chang%2C+and+Guy+Van+den+Broeck%2E+2022b%2E+On+the+paradox+of+learning+to+reason+from+data%2E+arXiv%3A2205%2E11502%2E+%E2%80%94+%5BarXiv%3A2205%2E11502%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
44. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). In The Eleventh International Conference on Learning Representation…
45. Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. 2022. Teaching algorithmic reasoning via in-context learning. arXiv:2211.09066. [find](https://scholar.google.com/scholar?q=Hattie+Zhou%2C+Azade+Nova%2C+Hugo+Larochelle%2C+Aaron+Courville%2C+Behnam+Neyshabur%2C+and+Hanie+Sedghi%2E+2022%2E+Teaching+algorithmic+reasoning+via+in%2Dcontext+learning%2E+arXiv%3A2211%2E09066%2E)

*Updated 24 October 2026*