---
title: "Answering Questions by Meta-Reasoning over Multiple Chains of"
description: "The authors propose Multi-Chain Reasoning (MCR) to address this. 52 references, cited by 61 papers here."
authors: ["Ori Yoran", "Tomer Wolfson", "Ben Bogin", "Uri Katz", "Daniel Deutch", "Jonathan Berant"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.364/"
doi: "10.18653/v1/2023.emnlp-main.364"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yoran-2023-answering-questions-meta-reasoning-over/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yoran-2023-answering-questions-meta-reasoning-over/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Answering Questions by Meta-Reasoning over Multiple Chains of Thought

# Answering Questions by Meta-Reasoning over Multiple Chains of Thought

**Authors:** [Ori Yoran](https://md.research.elamur.ai/free/en/authors/ori-yoran/), [Tomer Wolfson](https://md.research.elamur.ai/free/en/authors/tomer-wolfson/), [Ben Bogin](https://md.research.elamur.ai/free/en/authors/ben-bogin/), Uri Katz, [Daniel Deutch](https://md.research.elamur.ai/free/en/authors/daniel-deutch/) and [Jonathan Berant](https://md.research.elamur.ai/free/en/authors/jonathan-berant/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.364/) · [PDF](https://aclanthology.org/2023.emnlp-main.364.pdf) · [DOI 10.18653/v1/2023.emnlp-main.364](https://doi.org/10.18653/v1/2023.emnlp-main.364) · [OpenAlex](https://openalex.org/W4389523807)

**Citations:** 36 citations (OpenAlex, 2026-10-14) · [61](https://md.research.elamur.ai/free/en/papers/yoran-2023-answering-questions-meta-reasoning-over/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Multi-hop question answering with large language models often relies on self-consistency (SC). SC samples several chains of thought and takes a majority vote over their final answers. This throws away useful facts from the intermediate steps, fails when answers vary widely, and yields no single explanation. The authors propose Multi-Chain Reasoning (MCR) to address this. A prompted LLM breaks the question into intermediate questions and interleaves them with retrieved evidence, producing several reasoning chains. A separate prompted "meta-reasoner" LLM then reads all chains together, combines the relevant facts, and writes an explanation along with the final answer. MCR was tested in an open-domain setting on seven benchmarks covering implicit-reasoning and explicit-reasoning questions. It beat every baseline on all of them, including SC and retrieval-augmented Self-Ask and CoT, while using the same sampled chains. Manual analysis showed that MCR's explanations are mostly of high quality and let humans check its answers. This makes multi-chain reasoning both more accurate and easier to interpret.

## Key points
- The seven datasets are StrategyQA, Fermi, QuaRTz, HotpotQA, 2WikiMQA, Bamboogle and FEVEROUS. MCR outperformed SC by as much as 5.7%.
- Over 82% of MCR explanations were judged high quality, and under 3% were unhelpful. Human answers based on the explanations agreed with MCR's predictions in 90% of cases, or 95% if Fermi is excluded.
- The authors manually examined 700 errors (100 per dataset):
  - In four datasets, more than 20% of the "errors" were actually acceptable answers.
  - Decomposition problems dominated in the implicit datasets.
  - Retrieval problems and contradicting facts were more common in the explicit datasets.
- The main experiments used code-davinci-002. For reproducibility, the authors also report results with the open-source Vicuna model and an open-source retriever.
- Code and data are released at github.com/oriyor/reasoning-on-cots. A fine-tuned meta-reasoner was not tested and is left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations
- [Bar-Ilan University](https://md.research.elamur.ai/free/en/universities/bar-ilan-university/) — University, Ramat Gan, Israel · 118 works · 21,077 citations
- [Tel Aviv University](https://md.research.elamur.ai/free/en/universities/tel-aviv-university/) — University, Tel Aviv, Israel · 67 works · 10,006 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 242 works

## Cite

Cite as: Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch and Jonathan Berant. 2023. Answering Questions by Meta-Reasoning over Multiple Chains of Thought. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yoran2023answering,
  title = {{Answering Questions by Meta-Reasoning over Multiple Chains of Thought}},
  author = {Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch and Jonathan Berant},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.364},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.364}
}
```

- [← previous MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/)
- [next → VeRA: Vector-based Random Matrix Adaptation](https://md.research.elamur.ai/free/en/papers/kopiczko-2023-vera-vector-based-random-matrix/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

52 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran,…](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
2. Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. [Towards a human-like open-domain chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/).
3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). ArXiv preprint, abs/2110.14168.
4. Rami Aly, Zhijiang Guo, M. Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. Feverous: Fact extraction and verification over unstructured and structured information. ArXiv, abs/2106.05707. — [arXiv:2106.05707](https://arxiv.org/abs/2106.05707) [find](https://scholar.google.com/scholar?q=Rami+Aly%2C+Zhijiang+Guo%2C+M%2E+Schlichtkrull%2C+James+Thorne%2C+Andreas+Vlachos%2C+Christos+Christodoulopoulos%2C+Oana+Cocarascu%2C+and+Arpit+Mittal%2E+2021%2E+Feverous%3A+Fact+extraction+and+verification+over+unstructur)
5. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. ArXiv, abs/2205.09712. \#\# Acknowledgements We would like to thank Harsh Trivedi, Ofir Press, Mor Geva, Peter Clark and Ashish Sabharwal for their fee… [find](https://scholar.google.com/scholar?q=Antonia+Creswell%2C+Murray+Shanahan%2C+and+Irina+Higgins%2E+2022%2E+Selection%2Dinference%3A+Exploiting+large+language+models+for+interpretable+logical+reasoning%2E+ArXiv%2C+abs%2F2205%2E09712%2E+%5C%23%5C%23+Acknowledgements+We+w)
6. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/)
7. Yao Fu, Hao-Chun Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). ArXiv, abs/2210.00720.
8. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2022. [Rarr: Researching and revising what language models say, using language models](https://md.research.elamur.ai/free/en/papers/gao-2023-rarr-researching-revising-what-language/).
9. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+2021%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+)
10. Hangfeng He, Hongming Zhang, and Dan Roth. 2022. Rethinking with retrieval: Faithful large language model inference. [find](https://scholar.google.com/scholar?q=Hangfeng+He%2C+Hongming+Zhang%2C+and+Dan+Roth%2E+2022%2E+Rethinking+with+retrieval%3A+Faithful+large+language+model+inference%2E)
11. Jie Huang and Kevin Chen-Chuan Chang. 2022. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/).
12. Alon Jacovi and Yoav Goldberg. 2020. [Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?](https://md.research.elamur.ai/free/en/papers/jacovi-2020-faithfully-interpretable-nlp-systems-how/) In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online. Association for Computational Linguistics.
13. Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Her… [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+T%2E+J%2E+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zachary+Dodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+Scott+Johnston%2C+Sheer+El%2DShowk%2C+Andy+Jones%2C+Nel)
14. Ashwin Kalyan, Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal, and Peter Clark. 2021. How much coffee was consumed during EMNLP 2019? fermi problems: A new reasoning challenge for AI. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7318–7328, O… [find](https://scholar.google.com/scholar?q=Ashwin+Kalyan%2C+Abhinav+Kumar%2C+Arjun+Chandrasekaran%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2021%2E+How+much+coffee+was+consumed+during+EMNLP+2019%3F+fermi+problems%3A+A+new+reasoning+challenge+for+AI%2E+In+Procee)
15. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
16. O. Khattab, Keshav Santhanam, Xiang Lisa Li, David Leo Wright Hall, Percy Liang, Christopher Potts, and Matei A. Zaharia. 2022. [Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp](https://md.research.elamur.ai/free/en/papers/khattab-2022-demonstrate-search-predict-composing-retrieval/). ArXiv, abs/2212.14024.
17. Omar Khattab and Matei Zaharia. 2020. [Colbert: Efficient and effective passage search via contextualized late interaction over BERT](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/). In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020,…
18. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Harsh+Trivedi%2C+Matthew+Finlayson%2C+Yao+Fu%2C+Kyle+Richardson%2C+Peter+Clark%2C+and+Ashish+Sabharwal%2E+2023%2E+Decomposed+prompting%3A+A+modular+approach+for+solving+complex+tasks%2E)
19. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
20. Angeliki Lazaridou, Elena Gribovskaya, Wojciech Jan Stokowiec, and Nikolai Grigorev. 2023. [Internet-augmented language models through few-shot prompting for open-domain question answering](https://md.research.elamur.ai/free/en/papers/lazaridou-2022-internet-augmented-language-models-through/).
21. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022. On the advance of making language models better reasoners. ArXiv, abs/2206.02336. — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Zeqi+Lin%2C+Shizhuo+Zhang%2C+Qiang+Fu%2C+Bei+Chen%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2022%2E+On+the+advance+of+making+language+models+better+reasoners%2E+ArXiv%2C+abs%2F2206%2E02336%2E+%E2%80%94+%5BarXiv%3A2206%2E02336%5D%28htt)
22. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
23. Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. 2022. A survey of deep learning for mathematical reasoning. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Liang+Qiu%2C+Wenhao+Yu%2C+Sean+Welleck%2C+and+Kai%2DWei+Chang%2E+2022%2E+A+survey+of+deep+learning+for+mathematical+reasoning%2E)
24. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
25. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2022. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/).
26. Bhargavi Paranjape, Scott M. Lundberg, Sameer Singh, Hanna Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. ArXiv, abs/2303.09014. [find](https://scholar.google.com/scholar?q=Bhargavi+Paranjape%2C+Scott+M%2E+Lundberg%2C+Sameer+Singh%2C+Hanna+Hajishirzi%2C+Luke+Zettlemoyer%2C+and+Marco+Tulio+Ribeiro%2E+2023%2E+Art%3A+Automatic+multi%2Dstep+reasoning+and+tool%2Duse+for+large+language+models%2E+ArXi)
27. Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. [Refiner: Reasoning feedback on intermediate representations](https://md.research.elamur.ai/free/en/papers/paul-2023-refiner-reasoning-feedback-intermediate-representations/).
28. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. ArXiv, abs/2210.03350. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Muru+Zhang%2C+Sewon+Min%2C+Ludwig+Schmidt%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+2022%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2E+ArXiv%2C+abs%2F2210%2E03350%2E)
29. Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen. 2022. [Evaluating explanations: How much do explanations from the teacher aid students?](https://md.research.elamur.ai/free/en/papers/pruthi-2022-evaluating-explanations-how-much-do/) Transactions of the Association for Computational Linguistics, 10:359–37…
30. Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022. [Reasoning with language model prompting: A survey](https://md.research.elamur.ai/free/en/papers/qiao-2022-reasoning-language-model-prompting-survey/).
31. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ori+Ram%2C+Yoav+Levine%2C+Itay+Dalmedigos%2C+Dor+Muhlgay%2C+Amnon+Shashua%2C+Kevin+Leyton%2DBrown%2C+and+Yoav+Shoham%2E+2023%2E+In%2Dcontext+retrieval%2Daugmented+language+models%2E+Transactions+of+the+Association+for+Comput)
32. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Col-BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan… [find](https://scholar.google.com/scholar?q=Keshav+Santhanam%2C+Omar+Khattab%2C+Jon+Saad%2DFalcon%2C+Christopher+Potts%2C+and+Matei+Zaharia%2E+2022%2E+Col%2DBERTv2%3A+Effective+and+efficient+retrieval+via+lightweight+late+interaction%2E+In+Proceedings+of+the+2022+)
33. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/).
34. Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023. Recitation-augmented language models. ICLR. [find](https://scholar.google.com/scholar?q=Zhiqing+Sun%2C+Xuezhi+Wang%2C+Yi+Tay%2C+Yiming+Yang%2C+and+Denny+Zhou%2E+2023%2E+Recitation%2Daugmented+language+models%2E+ICLR%2E)
35. Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. QuaRTz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (E… [find](https://scholar.google.com/scholar?q=Oyvind+Tafjord%2C+Matt+Gardner%2C+Kevin+Lin%2C+and+Peter+Clark%2E+2019%2E+QuaRTz%3A+An+open%2Ddomain+dataset+of+qualitative+relationship+questions%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natu)
36. Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. Multimodalqa: complex question answering over text, tables and images. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Aus… [find](https://scholar.google.com/scholar?q=Alon+Talmor%2C+Ori+Yoran%2C+Amnon+Catav%2C+Dan+Lahav%2C+Yizhong+Wang%2C+Akari+Asai%2C+Gabriel+Ilharco%2C+Hannaneh+Hajishirzi%2C+and+Jonathan+Berant%2E+2021%2E+Multimodalqa%3A+complex+question+answering+over+text%2C+tables+an)
37. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
38. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). ArXiv,…
39. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022a. [Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions](https://md.research.elamur.ai/free/en/papers/trivedi-2022-interleaving-retrieval-chain-thought-reasoning/).
40. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022b. MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. [find](https://scholar.google.com/scholar?q=Harsh+Trivedi%2C+Niranjan+Balasubramanian%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2022b%2E+MuSiQue%3A+Multi%2Dhop+questions+via+single%2Dhop+question+composition%2E+Transactions+of+the+Association+for+Computational+L)
41. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations.
42. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). NeurIPS.
43. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. [Constructing datasets for multi-hop reading comprehension across documents](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/). Transactions of the Association for Computational Linguistics, 6:287– 302.
44. Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2022. Entailer: Answering questions with faithful and truthful chains of reasoning. [find](https://scholar.google.com/scholar?q=Oyvind+Tafjord%2C+Bhavana+Dalvi+Mishra%2C+and+Peter+Clark%2E+2022%2E+Entailer%3A+Answering+questions+with+faithful+and+truthful+chains+of+reasoning%2E)
45. Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. [Break it down: A question understanding benchmark](https://md.research.elamur.ai/free/en/papers/wolfson-2020-break-it-down-question-understanding/). Transactions of the Association for Computational Linguistics, 8:183–198.
46. Alon Talmor and Jonathan Berant. 2018. [The web as a knowledge-base for answering complex questions](https://md.research.elamur.ai/free/en/papers/talmor-2018-web-as-knowledge-base-answering/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651, New Orleans, L…
47. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
48. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E)
49. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). ArXiv preprint, abs/2210.03629.
50. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2023. [Making retrieval-augmented language models robust to irrelevant context](https://md.research.elamur.ai/free/en/papers/yoran-2023-making-retrieval-augmented-language-models/).
51. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. [STar: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). In Advances in Neural Information Processing Systems.
52. Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Huai hsin Chi. 2022. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). ArXiv, abs/2205.10625.

*Updated 24 October 2026*