---
title: "Demystifying Multi-Agent Debate — paper by Zhu et al. (2026)"
description: "They propose two complementary fixes: Diversity-aware initialisation: a training-free step that picks a… 57 references, cited by 4 papers here."
authors: ["Xiaochen Zhu", "Caiqi Zhang", "Yizhou Chi", "Tom Stafford", "Nigel Collier", "Andreas Vlachos"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1694/"
doi: "10.18653/v1/2026.findings-acl.1694"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2026-demystifying-multi-agent-debate-role/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2026-demystifying-multi-agent-debate-role/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / Demystifying Multi-Agent Debate: The Role of Confidence and Diversity

# Demystifying Multi-Agent Debate: The Role of Confidence and Diversity

**Authors:** [Xiaochen Zhu](https://md.research.elamur.ai/free/en/authors/xiaochen-zhu/), [Caiqi Zhang](https://md.research.elamur.ai/free/en/authors/caiqi-zhang/), Yizhou Chi, [Tom Stafford](https://md.research.elamur.ai/free/en/authors/tom-stafford/), [Nigel Collier](https://md.research.elamur.ai/free/en/authors/nigel-collier/) and [Andreas Vlachos](https://md.research.elamur.ai/free/en/authors/andreas-vlachos/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1694/) · [PDF](https://aclanthology.org/2026.findings-acl.1694.pdf) · [DOI 10.18653/v1/2026.findings-acl.1694](https://doi.org/10.18653/v1/2026.findings-acl.1694) · [OpenAlex](https://openalex.org/W7166837933)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/zhu-2026-demystifying-multi-agent-debate-role/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Multi-agent debate (MAD) is a popular test-time scaling technique for LLMs, but its standard form often does no better than a plain majority vote while costing more. Earlier theory explains this: with identical agents and unweighted updates, expected correctness stays constant over rounds (a martingale). Drawing on research about human group deliberation, the authors argue that vanilla MAD lacks two things: varied starting opinions and calibrated confidence signals. They propose two complementary fixes:

- **Diversity-aware initialisation:** a training-free step that picks a more varied set of candidate answers to start the debate.
- **Confidence-modulated debate:** a trained protocol in which agents state calibrated confidence and weigh peers' answers by it.

Their theory shows that diversity raises the prior chance that a correct answer is present without changing the update dynamics. Confidence weighting turns those dynamics into a submartingale that drifts toward correctness. On six reasoning-oriented QA benchmarks, both methods beat vanilla MAD and majority vote. The work offers practical, principled guidance for building multi-agent LLM systems.

## Key points

- The theoretical framework extends Choi et al. (2025), adopting their Dirichlet–categorical model of debate along with its assumptions of agent homogeneity.
- Diversity of initial answers correlates significantly with debate gains. MAD helps most on harder datasets, where answers naturally vary more.
- Diversity-aware initialisation increases both the number of unique initial answers and initial-round Pass@5. Table 3 lists values that appear to rise from 1.45 to 1.92 and from 1.61 to 2.32 under random versus diverse initialisation for the two models, though the extracted table is partly garbled.
- Confidence-modulated debate outperforms a Learn2Agg ablation (aggregation training without the confidence reward) on every dataset, including two OOD benchmarks. It corrects more initially wrong answers and keeps correct ones stable.
- Limitations include homogeneous agents, fully connected debate graphs, English-only QA evaluation and dependence on calibration. Code is released as DMAD on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 243 works · 121,396 citations
- [University of Sheffield](https://md.research.elamur.ai/free/en/universities/university-of-sheffield/) — University, Sheffield, United Kingdom · 113 works · 21,542 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Xiaochen Zhu, Caiqi Zhang, Yizhou Chi, Tom Stafford, Nigel Collier and Andreas Vlachos. 2026. Demystifying Multi-Agent Debate: The Role of Confidence and Diversity. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhu2026demystifying,
  title = {{Demystifying Multi-Agent Debate: The Role of Confidence and Diversity}},
  author = {Xiaochen Zhu, Caiqi Zhang, Yizhou Chi, Tom Stafford, Nigel Collier and Andreas Vlachos},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1694},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1694}
}
```

- [← previous CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic](https://md.research.elamur.ai/free/en/papers/zhang-2026-criticsearch-fine-grained-credit-assignment/)
- [next → Dissecting Failure Dynamics in Large Language Model Reasoning](https://md.research.elamur.ai/free/en/papers/zhu-2026-dissecting-failure-dynamics-large-language/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

57 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Oron Anschel, Alon Shoshan, Adam Botach, Shunit Haviv Hakimi, Asaf Gendler, Emanuel Ben Baruch, Nadav Bhonker, Igor Kviatkovsky, Manoj Aggarwal, and Gerard Medioni. 2025. Group-aware reinforcement learning for output diversity in large language models. In Proceedings of the 2025 Conference on Empiri… [find](https://scholar.google.com/scholar?q=Oron+Anschel%2C+Alon+Shoshan%2C+Adam+Botach%2C+Shunit+Haviv+Hakimi%2C+Asaf+Gendler%2C+Emanuel+Ben+Baruch%2C+Nadav+Bhonker%2C+Igor+Kviatkovsky%2C+Manoj+Aggarwal%2C+and+Gerard+Medioni%2E+2025%2E+Group%2Daware+reinforcement+lea)
2. Solomon E Asch. 1955. Opinions and social pressure. Scientific american, 193(5):31–35. [find](https://scholar.google.com/scholar?q=Solomon+E+Asch%2E+1955%2E+Opinions+and+social+pressure%2E+Scientific+american%2C+193%285%29%3A31%E2%80%9335%2E)
3. Bahador Bahrami, Karsten Olsen, Peter E Latham, Andreas Roepstorff, Geraint Rees, and Chris D Frith. 2010. Optimally interacting minds. Science, 329(5995):1081–1085. [find](https://scholar.google.com/scholar?q=Bahador+Bahrami%2C+Karsten+Olsen%2C+Peter+E+Latham%2C+Andreas+Roepstorff%2C+Geraint+Rees%2C+and+Chris+D+Frith%2E+2010%2E+Optimally+interacting+minds%2E+Science%2C+329%285995%29%3A1081%E2%80%931085%2E)
4. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. [Chateval: Towards better llm-based evaluators through multi-agent debate](https://md.research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenR…
5. Hyeong Kyu Choi, Xiaojin Zhu, and Yixuan Li. 2025. Debate or vote: Which yields better decisions in multi-agent large language models? ArXiv preprint, abs/2508.17536. — [arXiv:2508.17536](https://arxiv.org/abs/2508.17536) [find](https://scholar.google.com/scholar?q=Hyeong+Kyu+Choi%2C+Xiaojin+Zhu%2C+and+Yixuan+Li%2E+2025%2E+Debate+or+vote%3A+Which+yields+better+decisions+in+multi%2Dagent+large+language+models%3F+ArXiv+preprint%2C+abs%2F2508%2E17536%2E+%E2%80%94+%5BarXiv%3A2508%2E17536%5D%28https%3A%2F%2Farxi)
6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). ArXiv preprint, abs/1803.05457.
7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). ArXiv preprint, abs/2110.14168.
8. David Crecraft and Stephen Gergely. 2002. Analog Electronics: circuits, systems and signal processing. Elsevier. [find](https://scholar.google.com/scholar?q=David+Crecraft+and+Stephen+Gergely%2E+2002%2E+Analog+Electronics%3A+circuits%2C+systems+and+signal+processing%2E+Elsevier%2E)
9. Guillaume Dezecache, Martin Dockendorff, Dardo N Ferreiro, Ophelia Deroy, and Bahador Bahrami. 2022. Democratic forecast: Small groups predict the future better than individuals and crowds. Journal of Experimental Psychology: Applied, 28(3):525. [find](https://scholar.google.com/scholar?q=Guillaume+Dezecache%2C+Martin+Dockendorff%2C+Dardo+N+Ferreiro%2C+Ophelia+Deroy%2C+and+Bahador+Bahrami%2E+2022%2E+Democratic+forecast%3A+Small+groups+predict+the+future+better+than+individuals+and+crowds%2E+Journal+of)
10. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. [Improving factuality and reasoning in language models through multiagent debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv e-prints, pages arXiv–2407.
12. Andrew Estornell and Yang Liu. 2024. Multi-llm debate: Framework, principals, and interventions. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. [find](https://scholar.google.com/scholar?q=Andrew+Estornell+and+Yang+Liu%2E+2024%2E+Multi%2Dllm+debate%3A+Framework%2C+principals%2C+and+interventions%2E+In+Advances+in+Neural+Information+Processing+Systems+38%3A+Annual+Conference+on+Neural+Information+Proces)
13. Riccardo Fusaroli, Bahador Bahrami, Karsten Olsen, Andreas Roepstorff, Geraint Rees, Chris Frith, and Kristian Tylén. 2012. Coming to terms: Quantifying the benefits of linguistic coordination. Psychological science, 23(8):931–939. [find](https://scholar.google.com/scholar?q=Riccardo+Fusaroli%2C+Bahador+Bahrami%2C+Karsten+Olsen%2C+Andreas+Roepstorff%2C+Geraint+Rees%2C+Chris+Frith%2C+and+Kristian+Tyl%C3%A9n%2E+2012%2E+Coming+to+terms%3A+Quantifying+the+benefits+of+linguistic+coordination%2E+Psycho)
14. Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. [Large language model based multi-agents: A survey of progress and challenges](https://md.research.elamur.ai/free/en/papers/guo-2024-large-language-model-based/). In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI…
15. Joshua J Guyer, Pablo Briñol, Thomas I Vaughan-Johnston, Leandre R Fabrigar, Lorena Moreno, and Richard E Petty. 2021. Paralinguistic features communicated through voice can affect appraisals of confidence and evaluative judgments. Journal of nonverbal behavior, 45(4):479–504. [find](https://scholar.google.com/scholar?q=Joshua+J+Guyer%2C+Pablo+Bri%C3%B1ol%2C+Thomas+I+Vaughan%2DJohnston%2C+Leandre+R+Fabrigar%2C+Lorena+Moreno%2C+and+Richard+E+Petty%2E+2021%2E+Paralinguistic+features+communicated+through+voice+can+affect+appraisals+of+confi)
16. Andre Wang He, Daniel Fried, and Sean Welleck. 2025. [Rewarding the unlikely: Lifting grpo beyond distribution sharpening](https://md.research.elamur.ai/free/en/papers/he-2025-rewarding-unlikely-lifting-grpo-beyond/). In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25559–25571.
17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
18. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
19. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
20. Tiancheng Hu, Benjamin Minixhofer, and Nigel Collier. 2025. Navigating the alignment-calibration trade-off: A pareto-superior frontier via model merging. arXiv preprint arXiv:2510.17426. [find](https://scholar.google.com/scholar?q=Tiancheng+Hu%2C+Benjamin+Minixhofer%2C+and+Nigel+Collier%2E+2025%2E+Navigating+the+alignment%2Dcalibration+trade%2Doff%3A+A+pareto%2Dsuperior+frontier+via+model+merging%2E+arXiv+preprint+arXiv%3A2510%2E17426%2E)
21. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 1051–1068.
22. Irving L Janis. 1972. Victims of groupthink: A psychological study of foreign-policy decisions and fiascoes. [find](https://scholar.google.com/scholar?q=Irving+L+Janis%2E+1972%2E+Victims+of+groupthink%3A+A+psychological+study+of+foreign%2Dpolicy+decisions+and+fiascoes%2E)
23. Georgi Karadzhov, Andreas Vlachos, and Tom Stafford. 2024. The effect of diversity on group decision-making. ArXiv preprint, abs/2402.01427. — [arXiv:2402.01427](https://arxiv.org/abs/2402.01427) [find](https://scholar.google.com/scholar?q=Georgi+Karadzhov%2C+Andreas+Vlachos%2C+and+Tom+Stafford%2E+2024%2E+The+effect+of+diversity+on+group+decision%2Dmaking%2E+ArXiv+preprint%2C+abs%2F2402%2E01427%2E+%E2%80%94+%5BarXiv%3A2402%2E01427%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E01427%29)
24. Norbert L Kerr and R Scott Tindale. 2004. Group performance and decision making. Annu. Rev. Psychol., 55(1):623–655. [find](https://scholar.google.com/scholar?q=Norbert+L+Kerr+and+R+Scott+Tindale%2E+2004%2E+Group+performance+and+decision+making%2E+Annu%2E+Rev%2E+Psychol%2E%2C+55%281%29%3A623%E2%80%93655%2E)
25. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. [Understanding the effects of RLHF on LLM generalisation and diversity](https://md.research.elamur.ai/free/en/papers/kirk-2023-understanding-effects-rlhf-llm-generalisation/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austri…
26. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. [Encouraging divergent thinking in large language models through multi-agent debate](https://md.research.elamur.ai/free/en/papers/liang-2023-encouraging-divergent-thinking-large-language/). In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 17…
27. Zijie Lin and Bryan Hooi. 2025. Enhancing multi-agent debate system performance via confidence expression. ArXiv preprint, abs/2509.14034. [find](https://scholar.google.com/scholar?q=Zijie+Lin+and+Bryan+Hooi%2E+2025%2E+Enhancing+multi%2Dagent+debate+system+performance+via+confidence+expression%2E+ArXiv+preprint%2C+abs%2F2509%2E14034%2E)
28. Hugo Mercier. 2020. Not born yesterday: The science of who we trust and what we believe. In Not born yesterday. Princeton University Press. [find](https://scholar.google.com/scholar?q=Hugo+Mercier%2E+2020%2E+Not+born+yesterday%3A+The+science+of+who+we+trust+and+what+we+believe%2E+In+Not+born+yesterday%2E+Princeton+University+Press%2E)
29. Hugo Mercier and Dan Sperber. 2011. Why do humans reason? arguments for an argumentative theory. Behavioral and brain sciences, 34(2):57–74. [find](https://scholar.google.com/scholar?q=Hugo+Mercier+and+Dan+Sperber%2E+2011%2E+Why+do+humans+reason%3F+arguments+for+an+argumentative+theory%2E+Behavioral+and+brain+sciences%2C+34%282%29%3A57%E2%80%9374%2E)
30. Hugo Mercier and Dan Sperber. 2017. The enigma of reason. Harvard University Press. [find](https://scholar.google.com/scholar?q=Hugo+Mercier+and+Dan+Sperber%2E+2017%2E+The+enigma+of+reason%2E+Harvard+University+Press%2E)
31. David Moshman and Molly Geil. 1998. Collaborative reasoning: Evidence for collective rationality. Thinking & Reasoning, 4(3):231–248. [find](https://scholar.google.com/scholar?q=David+Moshman+and+Molly+Geil%2E+1998%2E+Collaborative+reasoning%3A+Evidence+for+collective+rationality%2E+Thinking+%26+Reasoning%2C+4%283%29%3A231%E2%80%93248%2E)
32. Vishakh Padmakumar and He He. 2024. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. [find](https://scholar.google.com/scholar?q=Vishakh+Padmakumar+and+He+He%2E+2024%2E+Does+writing+with+language+models+reduce+content+diversity%3F+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+ICLR+2024%2C+Vienna%2C+Austria%2C+May+7%2D1)
33. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling.
34. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). ArXiv preprint, abs/2402.03300.
35. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2024. [Towards underst…](https://md.research.elamur.ai/free/en/papers/sharma-2023-towards-understanding-sycophancy-language-models/)
36. Paul E Smaldino, Cody Moser, Alejandro Perez Velilla, and Mikkel Werling. 2024. Maintaining transient diversity is a general principle for improving collective problem solving. Perspectives on Psychological Science, 19(2):454–464. [find](https://scholar.google.com/scholar?q=Paul+E+Smaldino%2C+Cody+Moser%2C+Alejandro+Perez+Velilla%2C+and+Mikkel+Werling%2E+2024%2E+Maintaining+transient+diversity+is+a+general+principle+for+improving+collective+problem+solving%2E+Perspectives+on+Psychol)
37. Paul Stangel, David Bani-Harouni, Chantal Pellegrini, Ege Özsoy, Kamilia Zaripova, Matthias Keicher, and Nassir Navab. 2025. Rewarding doubt: A reinforcement learning approach to calibrated confidence expression of large language models. ArXiv preprint, abs/2503.02623. — [arXiv:2503.02623](https://arxiv.org/abs/2503.02623) [find](https://scholar.google.com/scholar?q=Paul+Stangel%2C+David+Bani%2DHarouni%2C+Chantal+Pellegrini%2C+Ege+%C3%96zsoy%2C+Kamilia+Zaripova%2C+Matthias+Keicher%2C+and+Nassir+Navab%2E+2025%2E+Rewarding+doubt%3A+A+reinforcement+learning+approach+to+calibrated+confidence)
38. Fengfei Sun, Ningke Li, Kailong Wang, and Lorenz Goette. 2025. Large language models are overconfident and amplify human bias. ArXiv preprint, abs/2505.02151. — [arXiv:2505.02151](https://arxiv.org/abs/2505.02151) [find](https://scholar.google.com/scholar?q=Fengfei+Sun%2C+Ningke+Li%2C+Kailong+Wang%2C+and+Lorenz+Goette%2E+2025%2E+Large+language+models+are+overconfident+and+amplify+human+bias%2E+ArXiv+preprint%2C+abs%2F2505%2E02151%2E+%E2%80%94+%5BarXiv%3A2505%2E02151%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
39. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
40. Qwen Team. 2024. Qwen2.5: A party of foundation models. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E)
41. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and 1 others. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag… [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Zefan+Cai%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Lingpeng+Kong%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+1+others%2E+2024%2E+Large+language+models+are+not+fair+evaluators%2E+In+Proceedings+of+t)
42. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,…
43. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processi…
44. Haolun Wu, Zhenkun Li, and Lingyao Li. 2025. Can llm agents really debate? a controlled study of multi-agent debate in logical reasoning. ArXiv preprint, abs/2511.07784. — [arXiv:2511.07784](https://arxiv.org/abs/2511.07784) [find](https://scholar.google.com/scholar?q=Haolun+Wu%2C+Zhenkun+Li%2C+and+Lingyao+Li%2E+2025%2E+Can+llm+agents+really+debate%3F+a+controlled+study+of+multi%2Dagent+debate+in+logical+reasoning%2E+ArXiv+preprint%2C+abs%2F2511%2E07784%2E+%E2%80%94+%5BarXiv%3A2511%2E07784%5D%28https%3A%2F%2Fa)
45. Andrea Wynn, Harsh Satija, and Gillian Hadfield. 2025. Talk isn’t always cheap: Understanding failure modes in multi-agent debate. ArXiv preprint, abs/2509.05396. — [arXiv:2509.05396](https://arxiv.org/abs/2509.05396) [find](https://scholar.google.com/scholar?q=Andrea+Wynn%2C+Harsh+Satija%2C+and+Gillian+Hadfield%2E+2025%2E+Talk+isn%E2%80%99t+always+cheap%3A+Understanding+failure+modes+in+multi%2Dagent+debate%2E+ArXiv+preprint%2C+abs%2F2509%2E05396%2E+%E2%80%94+%5BarXiv%3A2509%2E05396%5D%28https%3A%2F%2Farxiv%2Eor)
46. Ruihan Yang, Caiqi Zhang, Zhisong Zhang, Xinting Huang, Sen Yang, Nigel Collier, Dong Yu, and Deqing Yang. 2025a. [LoGU: Long-form generation with uncertainty expressions](https://md.research.elamur.ai/free/en/papers/yang-2025-logu-long-form-generation-uncertainty/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18947–1…
47. Ruihan Yang, Caiqi Zhang, Zhisong Zhang, Xinting Huang, Dong Yu, Nigel Collier, and Deqing Yang. 2025b. UNCLE: Benchmarking uncertainty expressions in long-form generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 30328–30344, Suzhou, China. As… [find](https://scholar.google.com/scholar?q=Ruihan+Yang%2C+Caiqi+Zhang%2C+Zhisong+Zhang%2C+Xinting+Huang%2C+Dong+Yu%2C+Nigel+Collier%2C+and+Deqing+Yang%2E+2025b%2E+UNCLE%3A+Benchmarking+uncertainty+expressions+in+long%2Dform+generation%2E+In+Proceedings+of+the+2025+)
48. Yongjin Yang, Euiin Yi, Jongwoo Ko, Kimin Lee, Zhijing Jin, and Se-Young Yun. 2025c. Revisiting multi-agent debate as test-time scaling: A systematic study of conditional effectiveness. ArXiv preprint, abs/2505.22960. — [arXiv:2505.22960](https://arxiv.org/abs/2505.22960) [find](https://scholar.google.com/scholar?q=Yongjin+Yang%2C+Euiin+Yi%2C+Jongwoo+Ko%2C+Kimin+Lee%2C+Zhijing+Jin%2C+and+Se%2DYoung+Yun%2E+2025c%2E+Revisiting+multi%2Dagent+debate+as+test%2Dtime+scaling%3A+A+systematic+study+of+conditional+effectiveness%2E+ArXiv+preprint)
49. Luke Yoffe, Alfonso Amayuelas, and William Yang Wang. 2024. Debunc: Improving large language model agent communication with uncertainty metrics. ArXiv preprint, abs/2407.06426. [find](https://scholar.google.com/scholar?q=Luke+Yoffe%2C+Alfonso+Amayuelas%2C+and+William+Yang+Wang%2E+2024%2E+Debunc%3A+Improving+large+language+model+agent+communication+with+uncertainty+metrics%2E+ArXiv+preprint%2C+abs%2F2407%2E06426%2E)
50. Paul Zarnoth and Janet A Sniezek. 1997. The social influence of confidence in group decision making. Journal of Experimental Social Psychology, 33(4):345–366. [find](https://scholar.google.com/scholar?q=Paul+Zarnoth+and+Janet+A+Sniezek%2E+1997%2E+The+social+influence+of+confidence+in+group+decision+making%2E+Journal+of+Experimental+Social+Psychology%2C+33%284%29%3A345%E2%80%93366%2E)
51. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
52. Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. 2024. [LUQ: Long-text uncertainty quantification for LLMs](https://md.research.elamur.ai/free/en/papers/zhang-2024-luq-long-text-uncertainty-quantification/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5244–5262, Miami, Florida, USA. Association for Computational Linguistics.
53. Caiqi Zhang, Ruihan Yang, Zhisong Zhang, Xinting Huang, Sen Yang, Dong Yu, and Nigel Collier. 2025a. Atomic calibration of LLMs in long-form generations. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the… [find](https://scholar.google.com/scholar?q=Caiqi+Zhang%2C+Ruihan+Yang%2C+Zhisong+Zhang%2C+Xinting+Huang%2C+Sen+Yang%2C+Dong+Yu%2C+and+Nigel+Collier%2E+2025a%2E+Atomic+calibration+of+LLMs+in+long%2Dform+generations%2E+In+Proceedings+of+the+14th+International+Joint)
54. Caiqi Zhang, Ruihan Yang, Xiaochen Zhu, Chengzu Li, Tiancheng Hu, Yijiang River Dong, Deqing Yang, and Nigel Collier. 2026. [Confidence estimation for llms in multi-turn interactions](https://md.research.elamur.ai/free/en/papers/zhang-2026-confidence-estimation-llms-multi-turn/). Preprint, arXiv:2601.02179.
55. Caiqi Zhang, Xiaochen Zhu, Chengzu Li, Nigel Collier, and Andreas Vlachos. 2025b. Reinforcement learning for better verbalized confidence in long-form generation. ArXiv preprint, abs/2505.23912. — [arXiv:2505.23912](https://arxiv.org/abs/2505.23912) [find](https://scholar.google.com/scholar?q=Caiqi+Zhang%2C+Xiaochen+Zhu%2C+Chengzu+Li%2C+Nigel+Collier%2C+and+Andreas+Vlachos%2E+2025b%2E+Reinforcement+learning+for+better+verbalized+confidence+in+long%2Dform+generation%2E+ArXiv+preprint%2C+abs%2F2505%2E23912%2E+%E2%80%94+%5Bar)
56. Wenting Zhao, Pranjal Aggarwal, Swarnadeep Saha, Asli Celikyilmaz, Jason Weston, and Ilia Kulikov. 2025. The majority is not always right: Rl training for solution aggregation. ArXiv preprint, abs/2509.06870. — [arXiv:2509.06870](https://arxiv.org/abs/2509.06870) [find](https://scholar.google.com/scholar?q=Wenting+Zhao%2C+Pranjal+Aggarwal%2C+Swarnadeep+Saha%2C+Asli+Celikyilmaz%2C+Jason+Weston%2C+and+Ilia+Kulikov%2E+2025%2E+The+majority+is+not+always+right%3A+Rl+training+for+solution+aggregation%2E+ArXiv+preprint%2C+abs%2F250)
57. Xiaochen Zhu, Caiqi Zhang, Tom Stafford, Nigel Collier, and Andreas Vlachos. 2025. [Conformity in large language models](https://md.research.elamur.ai/free/en/papers/zhu-2025-conformity-large-language-models/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3854–3872.

*Updated 1 November 2026*