---
title: "Debating with More Persuasive LLMs Leads to More Truthful"
description: "The paper asks whether weaker overseers can reliably check stronger AI systems, a problem that… 43 references, cited by 90 papers here."
authors: ["Akbir Khan", "John Hughes", "Dan Valentine", "Laura Ruis", "Kshitij Sachan", "Ansh Radhakrishnan", "Edward Grefenstette", "Samuel R. Bowman", "Tim Rocktaschel", "Ethan Perez"]
year: 2024
venue: "arXiv:2402.06782"
source: "https://arxiv.org/abs/2402.06782"
doi: "10.48550/arxiv.2402.06782"
arxiv: "2402.06782"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/khan-2024-debating-more-persuasive-llms-leads/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/khan-2024-debating-more-persuasive-llms-leads/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Debating with More Persuasive LLMs Leads to More Truthful Answers

# Debating with More Persuasive LLMs Leads to More Truthful Answers

**Authors:** [Akbir Khan](https://md.research.elamur.ai/free/en/authors/akbir-khan/), [John Hughes](https://md.research.elamur.ai/free/en/authors/john-hughes/), Dan Valentine, [Laura Ruis](https://md.research.elamur.ai/free/en/authors/laura-ruis/), Kshitij Sachan, [Ansh Radhakrishnan](https://md.research.elamur.ai/free/en/authors/ansh-radhakrishnan/), [Edward Grefenstette](https://md.research.elamur.ai/free/en/authors/edward-grefenstette/), [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/), [Tim Rocktaschel](https://md.research.elamur.ai/free/en/authors/tim-rocktaschel/) and [Ethan Perez](https://md.research.elamur.ai/free/en/authors/ethan-perez/) · **Year:** 2024 · **Venue:** arXiv:2402.06782 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.06782) · [PDF](https://arxiv.org/pdf/2402.06782) · [DOI 10.48550/arxiv.2402.06782](https://doi.org/10.48550/arxiv.2402.06782) · [arXiv 2402.06782](https://arxiv.org/abs/2402.06782) · [OpenAlex](https://openalex.org/W4391800466)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [90](https://md.research.elamur.ai/free/en/papers/khan-2024-debating-more-persuasive-llms-leads/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether weaker overseers can reliably check stronger AI systems, a problem that grows as models outpace human expertise and ground-truth labels become unavailable. The authors simulate this gap with information asymmetry on the QuALITY reading-comprehension benchmark. "Expert" LLMs can read a long science-fiction story, while "non-expert" judges, either LLMs or humans, see only the question and two candidate answers. Three protocols are compared. In debate, two experts defend opposite answers. Interactive debate is the same, but the judge may also pose questions. In consultancy, a single expert advocates one answer. Experts can cite passages, and an automatic tool marks each citation as verified or not. The authors also make debaters more persuasive without labels by using inference-time selection guided by judge approval. Debate produced clearly more accurate judgements than both consultancy and no help. Stronger debaters further increased judge accuracy, while stronger consultants reduced it. This is early empirical evidence that adversarial debate could support scalable oversight.

## Key points
- **Data:** HARD questions from the Project Gutenberg sci-fi portion of QuALITY, with stories around 7,000 tokens at the median. Each question is reduced to the correct answer plus the strongest distractor.
- **Accuracy with the most persuasive debaters:**
  - Human judges: 88% with debate, 78% with consultancy, 60% naive. Annotators who read the text score 89%.
  - LLM judges: 76% with debate, 54% with consultancy, 48% naive.
- **Models and setup:**
  - Debaters and consultants: GPT-4-Turbo, GPT-3.5-Turbo, Claude 2.1, Claude 1.3 and Mixtral 8x7B.
  - Persuasiveness is boosted with best-of-N (up to best-of-16) and critique-and-refinement.
  - Each run has three rounds and a 900-word transcript cap.
- **Persuasiveness and truth:** Optimising for judge approval helps debaters more when they argue the true answer than when they argue the false one. This benefit carries over to judges that were not used during optimisation.
- **Human study and code:** The human study used over 1,000 annotator-hours, and judges were better calibrated under debate than under consultancy. The code is available at github.com/ucl-dark/llm_debate (ICML 2024).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktaschel and Ethan Perez. 2024. Debating with More Persuasive LLMs Leads to More Truthful Answers. arXiv:2402.06782.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{khan2024debating,
  title = {{Debating with More Persuasive LLMs Leads to More Truthful Answers}},
  author = {Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktaschel and Ethan Perez},
  year = {2024},
  doi = {10.48550/arxiv.2402.06782},
  eprint = {2402.06782},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.06782}
}
```

- [← previous A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://md.research.elamur.ai/free/en/papers/zhang-2024-careful-examination-large-language-model/)
- [next → Do Multilingual Language Models Think Better in English?](https://md.research.elamur.ai/free/en/papers/etxaniz-2024-do-multilingual-language-models-think/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

43 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. [Concrete problems in AI safety](https://md.research.elamur.ai/free/en/papers/amodei-2016-concrete-problems-ai-safety/). arXiv preprint arXiv:1606.06565, 2016.
2. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. [Constitutional AI: Harmlessness from AI feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022.
3. Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., and Weld, D. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–16, 2021. [find](https://scholar.google.com/scholar?q=Bansal%2C+G%2E%2C+Wu%2C+T%2E%2C+Zhou%2C+J%2E%2C+Fok%2C+R%2E%2C+Nushi%2C+B%2E%2C+Kamar%2C+E%2E%2C+Ribeiro%2C+M%2E+T%2E%2C+and+Weld%2C+D%2E+Does+the+whole+exceed+its+parts%3F+the+effect+of+ai+explanations+on+complementary+team+performance%2E+In+Proceedin)
4. Barnes, B. Debate update: Obfuscated arguments problem. AI Alignment Forum, 2020. [find](https://scholar.google.com/scholar?q=Barnes%2C+B%2E+Debate+update%3A+Obfuscated+arguments+problem%2E+AI+Alignment+Forum%2C+2020%2E)
5. Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukosuite, K., Askell, A., Jones, A., Chen, A., et al. [Measuring progress on scalable oversight for large language models](https://md.research.elamur.ai/free/en/papers/bowman-2022-measuring-progress-scalable-oversight-large/). arXiv preprint arXiv:2211.03540, 2022.
6. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/). arXiv preprint arXiv:2312.09390, 2023.
7. Christiano, P., Shlegeris, B., and Amodei, D. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. [find](https://scholar.google.com/scholar?q=Christiano%2C+P%2E%2C+Shlegeris%2C+B%2E%2C+and+Amodei%2C+D%2E+Supervising+strong+learners+by+amplifying+weak+experts%2E+arXiv+preprint+arXiv%3A1810%2E08575%2C+2018%2E)
8. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Christiano%2C+P%2E+F%2E%2C+Leike%2C+J%2E%2C+Brown%2C+T%2E%2C+Martic%2C+M%2E%2C+Legg%2C+S%2E%2C+and+Amodei%2C+D%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
9. Chromik, M., Eiband, M., Buchner, F., Krüger, A., and Butz, A. I think i get your point, ai! the illusion of explanatory depth in explainable ai. In 26th International Conference on Intelligent User Interfaces, pp. 307–317, 2021. [find](https://scholar.google.com/scholar?q=Chromik%2C+M%2E%2C+Eiband%2C+M%2E%2C+Buchner%2C+F%2E%2C+Kru%CC%88ger%2C+A%2E%2C+and+Butz%2C+A%2E+I+think+i+get+your+point%2C+ai%21+the+illusion+of+explanatory+depth+in+explainable+ai%2E+In+26th+International+Conference+on+Intelligent+User+)
10. Cotra, A. The case for aligning narrowly superhuman models. AI Alignment Forum, 2021. [find](https://scholar.google.com/scholar?q=Cotra%2C+A%2E+The+case+for+aligning+narrowly+superhuman+models%2E+AI+Alignment+Forum%2C+2021%2E)
11. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. [Improving factuality and reasoning in language models through multiagent debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). arXiv preprint arXiv:2305.14325, 2023.
12. Elo, A. The Rating of Chessplayers: Past and Present. Ishi Press International, 1978. ISBN 9780923891275. URL [https://books.google.com/books?id=syjcPQAACAAJ](https://books.google.com/books?id=syjcPQAACAAJ). [↗](https://books.google.com/books?id=syjcPQAACAAJ)
13. Gemini Team, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
14. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. [Generative adversarial nets](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-generative-adversarial-nets/). Advances in neural information processing systems, 27, 2014.
15. Goswami, M., Sanil, V., Choudhry, A., Srinivasan, A., Udompanyawit, C., and Dubrawski, A. Aqua: A benchmarking tool for label quality assessment. arXiv preprint arXiv:2306.09467, 2023. — [arXiv:2306.09467](https://arxiv.org/abs/2306.09467) [find](https://scholar.google.com/scholar?q=Goswami%2C+M%2E%2C+Sanil%2C+V%2E%2C+Choudhry%2C+A%2E%2C+Srinivasan%2C+A%2E%2C+Udompanyawit%2C+C%2E%2C+and+Dubrawski%2C+A%2E+Aqua%3A+A+benchmarking+tool+for+label+quality+assessment%2E+arXiv+preprint+arXiv%3A2306%2E09467%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E0)
16. Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. Ai control: Improving safety despite intentional subversion. arXiv preprint arXiv:2312.06942, 2023. — [arXiv:2312.06942](https://arxiv.org/abs/2312.06942) [find](https://scholar.google.com/scholar?q=Greenblatt%2C+R%2E%2C+Shlegeris%2C+B%2E%2C+Sachan%2C+K%2E%2C+and+Roger%2C+F%2E+Ai+control%3A+Improving+safety+despite+intentional+subversion%2E+arXiv+preprint+arXiv%3A2312%2E06942%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E06942%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
17. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations, 2020.
18. Herbrich, R., Minka, T., and Graepel, T. Trueskill™: a bayesian skill rating system. Advances in neural information processing systems, 19, 2006. [find](https://scholar.google.com/scholar?q=Herbrich%2C+R%2E%2C+Minka%2C+T%2E%2C+and+Graepel%2C+T%2E+Trueskill%E2%84%A2%3A+a+bayesian+skill+rating+system%2E+Advances+in+neural+information+processing+systems%2C+19%2C+2006%2E)
19. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. [Sleeper agents: Training deceptive llms that persist through safety training](https://md.research.elamur.ai/free/en/papers/hubinger-2024-sleeper-agents-training-deceptive-llms/). arXiv preprint arXiv:2401.05566, 2024.
20. Irving, G., Christiano, P., and Amodei, D. [AI safety via debate](https://md.research.elamur.ai/free/en/papers/irving-2018-ai-safety-debate/). arXiv preprint arXiv:1805.00899, 2018.
21. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Roux%2C+A%2E%2C+Mensch%2C+A%2E%2C+Savary%2C+B%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Hanna%2C+E%2E+B%2E%2C+Bressand%2C+F%2E%2C+et+al%2E+Mixtral+of+experts%2E+arXiv+preprint+arXiv%3A2401%2E04088%2C+)
22. Lin, S., Hilton, J., and Evans, O. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214– 3252, 2022.
23. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. [find](https://scholar.google.com/scholar?q=Madaan%2C+A%2E%2C+Tandon%2C+N%2E%2C+Gupta%2C+P%2E%2C+Hallinan%2C+S%2E%2C+Gao%2C+L%2E%2C+Wiegreffe%2C+S%2E%2C+Alon%2C+U%2E%2C+Dziri%2C+N%2E%2C+Prabhumoye%2C+S%2E%2C+Yang%2C+Y%2E%2C+Gupta%2C+S%2E%2C+Majumder%2C+B%2E+P%2E%2C+Hermann%2C+K%2E%2C+Welleck%2C+S%2E%2C+Yazdanbakhsh%2C+A%2E%2C+and+Clar)
24. Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. [Teaching language models to support answers with verified quotes](https://md.research.elamur.ai/free/en/papers/menick-2022-teaching-language-models-support-answers/). arXiv preprint arXiv:2203.11147, 2022.
25. Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Padmakumar, V., and Bowman, S. R. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702, 2023. [find](https://scholar.google.com/scholar?q=Michael%2C+J%2E%2C+Mahdi%2C+S%2E%2C+Rein%2C+D%2E%2C+Petty%2C+J%2E%2C+Dirani%2C+J%2E%2C+Padmakumar%2C+V%2E%2C+and+Bowman%2C+S%2E+R%2E+Debate+helps+supervise+unreliable+experts%2E+arXiv+preprint+arXiv%3A2311%2E08702%2C+2023%2E)
26. Nikola. Jailbreaking gpt-4’s code interpreter. LessWrong, July 2023. URL [https://www.lesswrong.com/posts/KSroBnxCHodGmPPJ8/](https://www.lesswrong.com/posts/KSroBnxCHodGmPPJ8/) jailbreaking-gpt-4-s-code-interpreter. [↗](https://www.lesswrong.com/posts/KSroBnxCHodGmPPJ8/)
27. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). In Deep Learning for Code Workshop, 2022.
28. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774, 2023. arXiv preprint Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Proce…
29. Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. [QuALITY: Question answering with long input texts, yes!](https://md.research.elamur.ai/free/en/papers/pang-2022-quality-question-answering-long-input/) In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
30. Parrish, A., Trivedi, H., Nangia, N., Phang, J., Padmakumar, V., Saimbhi, A. S., and Bowman, S. R. Two-turn debate does not help humans answer hard reading comprehension questions. In NeurIPS ML Safety Workshop, 2022a. [find](https://scholar.google.com/scholar?q=Parrish%2C+A%2E%2C+Trivedi%2C+H%2E%2C+Nangia%2C+N%2E%2C+Phang%2C+J%2E%2C+Padmakumar%2C+V%2E%2C+Saimbhi%2C+A%2E+S%2E%2C+and+Bowman%2C+S%2E+R%2E+Two%2Dturn+debate+does+not+help+humans+answer+hard+reading+comprehension+questions%2E+In+NeurIPS+ML+Safet)
31. Parrish, A., Trivedi, H., Perez, E., Chen, A., Nangia, N., Phang, J., and Bowman, S. Single-turn debate does not help humans answer hard reading-comprehension questions. In Proceedings of the First Workshop on Learning with Natural Language Supervision, pp. 17–28, 2022b. [find](https://scholar.google.com/scholar?q=Parrish%2C+A%2E%2C+Trivedi%2C+H%2E%2C+Perez%2C+E%2E%2C+Chen%2C+A%2E%2C+Nangia%2C+N%2E%2C+Phang%2C+J%2E%2C+and+Bowman%2C+S%2E+Single%2Dturn+debate+does+not+help+humans+answer+hard+reading%2Dcomprehension+questions%2E+In+Proceedings+of+the+First+Wo)
32. Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., and Cho, K. [Finding generalizable evidence by learning to convince Q&A models](https://md.research.elamur.ai/free/en/papers/perez-2019-finding-generalizable-evidence-learning-convince/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proces…
33. Perez, E., Kiela, D., and Cho, K. [True few-shot learning with language models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/). Advances in neural information processing systems, 34:11054–11070, 2021.
34. Radhakrishnan, A. Anthropic fall 2023 debate progress update. [find](https://scholar.google.com/scholar?q=Radhakrishnan%2C+A%2E+Anthropic+fall+2023+debate+progress+update%2E)
35. AI Alignment Forum, 2023. [find](https://scholar.google.com/scholar?q=AI+Alignment+Forum%2C+2023%2E)
36. Radhakrishnan, A., Nguyen, K., Chen, A., Chen, C., Denison, C., Hernandez, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiūtė, K., et al. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307.11768, 2023. [find](https://scholar.google.com/scholar?q=Radhakrishnan%2C+A%2E%2C+Nguyen%2C+K%2E%2C+Chen%2C+A%2E%2C+Chen%2C+C%2E%2C+Denison%2C+C%2E%2C+Hernandez%2C+D%2E%2C+Durmus%2C+E%2E%2C+Hubinger%2C+E%2E%2C+Kernion%2C+J%2E%2C+Lukos%CC%8Ciu%CC%84te%CC%87%2C+K%2E%2C+et+al%2E+Question+decomposition+improves+the+faithfulness+of+model)
37. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). arXiv preprint arXiv:2311.12022, 2023.
38. Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. [Self-critiquing models for assisting human evaluators](https://md.research.elamur.ai/free/en/papers/saunders-2022-self-critiquing-models-assisting-human/). arXiv preprint arXiv:2206.05802, 2022.
39. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 3008–3021, 2020. [find](https://scholar.google.com/scholar?q=Stiennon%2C+N%2E%2C+Ouyang%2C+L%2E%2C+Wu%2C+J%2E%2C+Ziegler%2C+D%2E%2C+Lowe%2C+R%2E%2C+Voss%2C+C%2E%2C+Radford%2C+A%2E%2C+Amodei%2C+D%2E%2C+and+Christiano%2C+P%2E+F%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Information+Processing+S)
40. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
41. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. [find](https://scholar.google.com/scholar?q=Yao%2C+S%2E%2C+Yu%2C+D%2E%2C+Zhao%2C+J%2E%2C+Shafran%2C+I%2E%2C+Griffiths%2C+T%2E+L%2E%2C+Cao%2C+Y%2E%2C+and+Narasimhan%2C+K%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E10601%2C+2023%2E)
42. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023.
43. Debaters: Consultancy Judge Accuracy (%) Aggregate Win Rate (%) [find](https://scholar.google.com/scholar?q=Debaters%3A+Consultancy+Judge+Accuracy+%28%25%29+Aggregate+Win+Rate+%28%25%29)

*Updated 2 November 2026*