---
title: "Improving Factuality and Reasoning in Language — Du 2023"
description: "Large language models often state invented facts with confidence or make unjustified leaps in their reasoning."
authors: ["Yilun Du", "Shuang Li", "Antonio Torralba", "Joshua B. Tenenbaum", "Igor Mordatch"]
year: 2023
venue: "arXiv:2305.14325"
source: "https://arxiv.org/abs/2305.14325"
doi: "10.48550/arxiv.2305.14325"
arxiv: "2305.14325"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / Improving Factuality and Reasoning in Language Models through Multiagent Debate

# Improving Factuality and Reasoning in Language Models through Multiagent Debate

**Authors:** Yilun Du, [Shuang Li](https://md.research.elamur.ai/free/en/authors/shuang-li/), [Antonio Torralba](https://md.research.elamur.ai/free/en/authors/antonio-torralba/), [Joshua B. Tenenbaum](https://md.research.elamur.ai/free/en/authors/joshua-b-tenenbaum/), [Igor Mordatch](https://md.research.elamur.ai/free/en/authors/igor-mordatch/) · **Institutions:** [Google](https://md.research.elamur.ai/free/en/companies/google/) · [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) · [MIT Computer Science and Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/mit-computer-science-and-artificial-intelligence-laboratory/) · **Year:** 2023 · **Venue:** arXiv:2305.14325 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.14325) · [PDF](https://arxiv.org/pdf/2305.14325) · [DOI 10.48550/arxiv.2305.14325](https://doi.org/10.48550/arxiv.2305.14325) · [arXiv 2305.14325](https://arxiv.org/abs/2305.14325) · [OpenAlex](https://openalex.org/W4378509427)

**Citations:** 95 citations (OpenAlex, 2026-10-14) · [469](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Large language models often state invented facts with confidence or make unjustified leaps in their reasoning. Earlier remedies, such as chain-of-thought prompting, self-consistency and reflection, all work with a single model instance. Du et al. instead treat several copies of an LLM as agents. Each agent first answers a query independently. It then reads the other agents' answers, critiques them and revises its own, and this exchange repeats over several rounds until the group reaches a common answer.

The method needs only black-box text generation and uses identical prompts for every task. It can also be stacked with zero-shot chain of thought. On six benchmarks (arithmetic, grade-school math, MMLU, chess move validity and optimality, and factual biographies), debate outperformed single-model baselines, including zero-shot chain of thought and reflection. Increasing both the number of agents and the number of rounds helped. The authors also introduce a new benchmark of computer scientist biographies. The approach matters because it improves accuracy without access to model internals and could produce better data for model self-improvement.

## Key points
- Six evaluation tasks: Arithmetic, Grade School Math (GSM8K), MMLU, Chess Move Validity, Chess Move Optimality and Biographies. The Biographies benchmark is a new dataset on famous computer scientists, where models frequently get institutions and dates wrong.
- Agents often begin with different answers but nearly always converge on one shared answer. In some cases every agent starts out wrong and the group still reaches the correct result. Facts the agents are unsure about tend to be disputed and then dropped.
- In a mixed-model experiment, chatGPT and Bard debated 20 GSM8K problems. Bard alone solved 11, chatGPT alone solved 14, and their joint debate solved 17.
- The method needs no likelihoods or gradients, so it works with public model APIs. It is also complementary to retrieval and prompt engineering.
- Limitations:
  - Debate requires many more model calls, which makes it expensive.
  - In long debates, models tend to attend mainly to the most recent messages.
  - Converged answers can still be wrong, and models then wrongly insist they are correct.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{du2023improving,
  title = {{Improving Factuality and Reasoning in Language Models through Multiagent Debate}},
  author = {Du, Yilun and Li, Shuang and Torralba, Antonio and Tenenbaum, Joshua B. and Mordatch, Igor},
  year = {2023},
  doi = {10.48550/arxiv.2305.14325},
  eprint = {2305.14325},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.14325}
}
```

- [← previous Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://md.research.elamur.ai/free/en/papers/turpin-2023-language-models-don-t-always/)
- [next → ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)

## References

33 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: A visual language model for few-shot learning. NeurIPS, 2022. URL https://arxiv.org/abs/2204.14198. 9 [↗](https://arxiv.org/abs/2204.14198) [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
2. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Neural Information Processing Systems, 2017. 9 [find](https://scholar.google.com/scholar?q=P%2E+F%2E+Christiano%2C+J%2E+Leike%2C+T%2E+Brown%2C+M%2E+Martic%2C+S%2E+Legg%2C+and+D%2E+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+Neural+Information+Processing+Systems%2C+2017%2E+9)
3. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 5, 9 [find](https://scholar.google.com/scholar?q=K%2E+Cobbe%2C+V%2E+Kosaraju%2C+M%2E+Bavarian%2C+M%2E+Chen%2C+H%2E+Jun%2C+L%2E+Kaiser%2C+M%2E+Plappert%2C+J%2E+Tworek%2C+J%2E+Hilton%2C+R%2E+Nakano%2C+et+al%2E+Training+verifiers+to+solve+math+word+problems%2E+arXiv+preprint+arXiv%3A2110%2E14168%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
4. Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, 2020. 9 [find](https://scholar.google.com/scholar?q=Y%2E+Du%2C+S%2E+Li%2C+and+I%2E+Mordatch%2E+Compositional+visual+generation+with+energy+based+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2020%2E+9)
5. Y. Du, C. Durkan, R. Strudel, J. B. Tenenbaum, S. Dieleman, R. Fergus, J. Sohl-Dickstein, A. Doucet, and W. Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. arXiv preprint arXiv:2302.11552, 2023. 9 — [arXiv:2302.11552](https://arxiv.org/abs/2302.11552) [find](https://scholar.google.com/scholar?q=Y%2E+Du%2C+C%2E+Durkan%2C+R%2E+Strudel%2C+J%2E+B%2E+Tenenbaum%2C+S%2E+Dieleman%2C+R%2E+Fergus%2C+J%2E+Sohl%2DDickstein%2C+A%2E+Doucet%2C+and+W%2E+Grathwohl%2E+Reduce%2C+reuse%2C+recycle%3A+Compositional+generation+with+energy%2Dbased+diffusion+mode)
6. Fsmosca. Fsmosca/pgn-standard: Portable game notation specification and implementation guide. URL [https://github.com/fsmosca/PGN-Standard](https://github.com/fsmosca/PGN-Standard). 5 [↗](https://github.com/fsmosca/PGN-Standard)
7. K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang. REALM: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. 9 [find](https://scholar.google.com/scholar?q=K%2E+Guu%2C+K%2E+Lee%2C+Z%2E+Tung%2C+P%2E+Pasupat%2C+and+M%2E%2DW%2E+Chang%2E+REALM%3A+Retrieval%2Daugmented+language+model+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2002%2E08909%2C+2020%2E+9)
8. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 7 [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+C%2E+Burns%2C+S%2E+Basart%2C+A%2E+Zou%2C+M%2E+Mazeika%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2E+Measuring+massive+multitask+language+understanding%2E+arXiv+preprint+arXiv%3A2009%2E03300%2C+2020%2E+7) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
9. G. Irving, P. Christiano, and D. Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899, 2018. 9 [find](https://scholar.google.com/scholar?q=G%2E+Irving%2C+P%2E+Christiano%2C+and+D%2E+Amodei%2E+Ai+safety+via+debate%2E+arXiv+preprint+arXiv%3A1805%2E00899%2C+2018%2E+9)
10. S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. H. Dodds, N. DasSarma, E. Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. 7, 9 [find](https://scholar.google.com/scholar?q=S%2E+Kadavath%2C+T%2E+Conerly%2C+A%2E+Askell%2C+T%2E+Henighan%2C+D%2E+Drain%2C+E%2E+Perez%2C+N%2E+Schiefer%2C+Z%2E+H%2E+Dodds%2C+N%2E+DasSarma%2C+E%2E+Tran%2DJohnson%2C+et+al%2E+Language+models+%28mostly%29+know+what+they+know%2E+arXiv+preprint+arXiv%3A2)
11. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. 2, 6, 9 [find](https://scholar.google.com/scholar?q=T%2E+Kojima%2C+S%2E+S%2E+Gu%2C+M%2E+Reid%2C+Y%2E+Matsuo%2C+and+Y%2E+Iwasawa%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+arXiv+preprint+arXiv%3A2205%2E11916%2C+2022%2E+2%2C+6%2C+9) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
12. N. Lee, W. Ping, P. Xu, M. Patwary, P. N. Fung, M. Shoeybi, and B. Catanzaro. Factuality enhanced language models for open-ended text generation. Advances in Neural Information Processing Systems, 35:34586–34599, 2022. 9 [find](https://scholar.google.com/scholar?q=N%2E+Lee%2C+W%2E+Ping%2C+P%2E+Xu%2C+M%2E+Patwary%2C+P%2E+N%2E+Fung%2C+M%2E+Shoeybi%2C+and+B%2E+Catanzaro%2E+Factuality+enhanced+language+models+for+open%2Dended+text+generation%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A)
13. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. 9 [find](https://scholar.google.com/scholar?q=A%2E+Lewkowycz%2C+A%2E+Andreassen%2C+D%2E+Dohan%2C+E%2E+Dyer%2C+H%2E+Michalewski%2C+V%2E+Ramasesh%2C+A%2E+Slone%2C+C%2E+Anil%2C+I%2E+Schlag%2C+T%2E+Gutman%2DSolo%2C+et+al%2E+Solving+quantitative+reasoning+problems+with+language+models%2E+arXiv+pr) [our card](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/)
14. S. Li, Y. Du, J. B. Tenenbaum, A. Torralba, and I. Mordatch. Composing ensembles of pre-trained models via iterative consensus. arXiv preprint arXiv:2210.11522, 2022. 9 — [arXiv:2210.11522](https://arxiv.org/abs/2210.11522) [find](https://scholar.google.com/scholar?q=S%2E+Li%2C+Y%2E+Du%2C+J%2E+B%2E+Tenenbaum%2C+A%2E+Torralba%2C+and+I%2E+Mordatch%2E+Composing+ensembles+of+pre%2Dtrained+models+via+iterative+consensus%2E+arXiv+preprint+arXiv%3A2210%2E11522%2C+2022%2E+9+%E2%80%94+%5BarXiv%3A2210%2E11522%5D%28https%3A%2F%2Far)
15. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science, 378 (6624):1092–1097, 2022. 5, 9 [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+D%2E+Choi%2C+J%2E+Chung%2C+N%2E+Kushman%2C+J%2E+Schrittwieser%2C+R%2E+Leblond%2C+T%2E+Eccles%2C+J%2E+Keeling%2C+F%2E+Gimeno%2C+A%2E+Dal+Lago%2C+et+al%2E+Competition%2Dlevel+code+generation+with+alphacode%2E+Science%2C+378+%286624%29%3A1092%E2%80%93109) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
16. H. Liu, L. Lee, K. Lee, and P. Abbeel. Instruction-following agents with jointly pre-trained vision-language models. arXiv preprint arXiv:2210.13431, 2022. 9 — [arXiv:2210.13431](https://arxiv.org/abs/2210.13431) [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+L%2E+Lee%2C+K%2E+Lee%2C+and+P%2E+Abbeel%2E+Instruction%2Dfollowing+agents+with+jointly+pre%2Dtrained+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2210%2E13431%2C+2022%2E+9+%E2%80%94+%5BarXiv%3A2210%2E13431%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
17. N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum. Compositional visual generation with composable diffusion models. arXiv preprint arXiv:2206.01714, 2022. 9 — [arXiv:2206.01714](https://arxiv.org/abs/2206.01714) [find](https://scholar.google.com/scholar?q=N%2E+Liu%2C+S%2E+Li%2C+Y%2E+Du%2C+A%2E+Torralba%2C+and+J%2E+B%2E+Tenenbaum%2E+Compositional+visual+generation+with+composable+diffusion+models%2E+arXiv+preprint+arXiv%3A2206%2E01714%2C+2022%2E+9+%E2%80%94+%5BarXiv%3A2206%2E01714%5D%28https%3A%2F%2Farxiv%2Eor)
18. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. 2, 5, 9 [find](https://scholar.google.com/scholar?q=A%2E+Madaan%2C+N%2E+Tandon%2C+P%2E+Gupta%2C+S%2E+Hallinan%2C+L%2E+Gao%2C+S%2E+Wiegreffe%2C+U%2E+Alon%2C+N%2E+Dziri%2C+S%2E+Prabhumoye%2C+Y%2E+Yang%2C+et+al%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedback%2E+arXiv+preprint+arXiv%3A2303%2E176)
19. M. Minsky. Society of mind. Simon and Schuster, 1988. 1 [find](https://scholar.google.com/scholar?q=M%2E+Minsky%2E+Society+of+mind%2E+Simon+and+Schuster%2C+1988%2E+1)
20. M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. 9 [find](https://scholar.google.com/scholar?q=M%2E+Nye%2C+A%2E+J%2E+Andreassen%2C+G%2E+Gur%2DAri%2C+H%2E+Michalewski%2C+J%2E+Austin%2C+D%2E+Bieber%2C+D%2E+Dohan%2C+A%2E+Lewkowycz%2C+M%2E+Bosma%2C+D%2E+Luan%2C+et+al%2E+Show+your+work%3A+Scratchpads+for+intermediate+computation+with+language+mod) [our card](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/)
21. OpenAI. Chatgpt: Optimizing language models for dialogue, Dec 2022. URL [https://openai.com/blog/chatgpt/](https://openai.com/blog/chatgpt/). 2, 5 [↗](https://openai.com/blog/chatgpt/)
22. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. 4 [find](https://scholar.google.com/scholar?q=L%2E+Ouyang%2C+J%2E+Wu%2C+X%2E+Jiang%2C+D%2E+Almeida%2C+C%2E+L%2E+Wainwright%2C+P%2E+Mishkin%2C+C%2E+Zhang%2C+S%2E+Agarwal%2C+K%2E+Slama%2C+A%2E+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+arXiv+preprint) [our card](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
23. S. Pichai. An important next step on our ai journey, Feb 2023. URL [https://blog.google/](https://blog.google/) technology/ai/bard-google-ai-search-updates/. 2, 9 [↗](https://blog.google/)
24. N. F. Rajani, B. McCann, C. Xiong, and R. Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361, 2019. 9 [find](https://scholar.google.com/scholar?q=N%2E+F%2E+Rajani%2C+B%2E+McCann%2C+C%2E+Xiong%2C+and+R%2E+Socher%2E+Explain+yourself%21+leveraging+language+models+for+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1906%2E02361%2C+2019%2E+9) [our card](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/)
25. L. Reynolds and K. McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7, 2021. 9 [find](https://scholar.google.com/scholar?q=L%2E+Reynolds+and+K%2E+McDonell%2E+Prompt+programming+for+large+language+models%3A+Beyond+the+few%2Dshot+paradigm%2E+In+Extended+Abstracts+of+the+2021+CHI+Conference+on+Human+Factors+in+Computing+Systems%2C+pages+1) [our card](https://md.research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/)
26. N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023. 2, 5, 9 [find](https://scholar.google.com/scholar?q=N%2E+Shinn%2C+B%2E+Labash%2C+and+A%2E+Gopinath%2E+Reflexion%3A+an+autonomous+agent+with+dynamic+memory+and+self%2Dreflection%2E+arXiv+preprint+arXiv%3A2303%2E11366%2C+2023%2E+2%2C+5%2C+9) [our card](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/)
27. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. 7, 14 [find](https://scholar.google.com/scholar?q=A%2E+Srivastava%2C+A%2E+Rastogi%2C+A%2E+Rao%2C+A%2E+A%2E+M%2E+Shoeb%2C+A%2E+Abid%2C+A%2E+Fisch%2C+A%2E+R%2E+Brown%2C+A%2E+Santoro%2C+A%2E+Gupta%2C+A%2E+Garriga%2DAlonso%2C+et+al%2E+Beyond+the+imitation+game%3A+Quantifying+and+extrapolating+the+capabili) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
28. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. 9 [find](https://scholar.google.com/scholar?q=R%2E+Thoppilan%2C+D%2E+De+Freitas%2C+J%2E+Hall%2C+N%2E+Shazeer%2C+A%2E+Kulshreshtha%2C+H%2E%2DT%2E+Cheng%2C+A%2E+Jin%2C+T%2E+Bos%2C+L%2E+Baker%2C+Y%2E+Du%2C+et+al%2E+Lamda%3A+Language+models+for+dialog+applications%2E+arXiv+preprint+arXiv%3A2201%2E08239%2C) [our card](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
29. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. 9 [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+J%2E+Wei%2C+D%2E+Schuurmans%2C+Q%2E+Le%2C+E%2E+Chi%2C+and+D%2E+Zhou%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+arXiv+preprint+arXiv%3A2203%2E11171%2C+2022%2E+9) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
30. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. 9 [find](https://scholar.google.com/scholar?q=J%2E+Wei%2C+X%2E+Wang%2C+D%2E+Schuurmans%2C+M%2E+Bosma%2C+E%2E+Chi%2C+Q%2E+Le%2C+and+D%2E+Zhou%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2201%2E11903%2C+2022%2E+9) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
31. E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022. 9 [find](https://scholar.google.com/scholar?q=E%2E+Zelikman%2C+Y%2E+Wu%2C+J%2E+Mu%2C+and+N%2E+Goodman%2E+Star%3A+Bootstrapping+reasoning+with+reasoning%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A15476%E2%80%9315488%2C+2022%2E+9) [our card](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/)
32. A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022. URL https://arxiv.org/abs/2204. 00598. 9 [↗](https://arxiv.org/abs/2204) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2022-socratic-models-composing-zero-shot/)
33. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 9 [find](https://scholar.google.com/scholar?q=D%2E+M%2E+Ziegler%2C+N%2E+Stiennon%2C+J%2E+Wu%2C+T%2E+B%2E+Brown%2C+A%2E+Radford%2C+D%2E+Amodei%2C+P%2E+Christiano%2C+and+G%2E+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A1909%2E08593%2C+2019%2E+9)