--- title: "Crowd Comparative Reasoning — paper by Zhang et al. (2025)" description: "The authors propose Crowd-based Comparative Evaluation (CCE). 80.3 accuracy. 2 references, cited by 10 papers here." authors: ["Qiyuan Zhang", "Yufei Wang", "Yuxin Jiang", "Liangyou Li", "Chuhan Wu", "Yasheng Wang", "Xin Jiang", "Lifeng Shang", "Ruiming Tang", "Fuyuan Lyu", "Chen Ma"] year: 2025 venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2025.acl-long.252/" doi: "10.18653/v1/2025.acl-long.252" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zhang-2025-crowd-comparative-reasoning-unlocking-comprehensive/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2025-crowd-comparative-reasoning-unlocking-comprehensive/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) / Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge # Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge **Authors:** [Qiyuan Zhang](https://md.research.elamur.ai/free/en/authors/qiyuan-zhang/), [Yufei Wang](https://md.research.elamur.ai/free/en/authors/yufei-wang/), [Yuxin Jiang](https://md.research.elamur.ai/free/en/authors/yuxin-jiang/), [Liangyou Li](https://md.research.elamur.ai/free/en/authors/liangyou-li/), [Chuhan Wu](https://md.research.elamur.ai/free/en/authors/chuhan-wu/), [Yasheng Wang](https://md.research.elamur.ai/free/en/authors/yasheng-wang/), [Xin Jiang](https://md.research.elamur.ai/free/en/authors/xin-jiang/), [Lifeng Shang](https://md.research.elamur.ai/free/en/authors/lifeng-shang/), [Ruiming Tang](https://md.research.elamur.ai/free/en/authors/ruiming-tang/), [Fuyuan Lyu](https://md.research.elamur.ai/free/en/authors/fuyuan-lyu/) and [Chen Ma](https://md.research.elamur.ai/free/en/authors/chen-ma/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.252/) · [PDF](https://aclanthology.org/2025.acl-long.252.pdf) · [DOI 10.18653/v1/2025.acl-long.252](https://doi.org/10.18653/v1/2025.acl-long.252) · [OpenAlex](https://openalex.org/W4412886639) **Citations:** 3 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/zhang-2025-crowd-comparative-reasoning-unlocking-comprehensive/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary LLM-as-a-Judge systems often produce chain-of-thought (CoT) judgments that focus on only a few aspects of the responses, so their verdicts can be shallow or wrong. Majority voting and criteria expansion only partly fix this. The former is costly and passive, and the latter ignores what is actually in each response. The authors propose Crowd-based Comparative Evaluation (CCE). Several extra "crowd" responses are generated for the same instruction, and each candidate is judged against each crowd response. The resulting crowd judgments are then filtered and fed back as context for the final pairwise CoT judgment. The filter keeps judgments where the candidate loses and strips their final verdicts. Across five judge benchmarks, CCE raises accuracy by 6.7% on average. Its richer CoTs also help in two other settings: distilling smaller judge models and selecting SFT data through "crowd rejection sampling". This matters because CCE offers a response-aware, test-time-scalable way to make automatic evaluation more thorough and less biased. ## Key points - **Benchmarks:** RewardBench, HelpSteer2, MTBench Human, JudgeBench and EvalBias, with an average accuracy gain of 6.7% over the vanilla judge. - **Crowd responses:** These come from models of different sizes, such as Qwen2.5-0.5B-Instruct and Mistral-Nemo-Instruct-2407, sampled at varied temperatures. - **Selection strategy:** "Criticizing Selection" with outcome removal performs best. With 16 crowd judgments it averages 80.3 accuracy versus 78.6 for random selection. Accuracy and CoT length grow as crowd judgments increase from 0 to 16. - **Distillation and data selection:** - Distilling CCE judgments into Qwen 2.5-7B yields gains of 4.5–5.6%, with notably better bias robustness. - Whether 1K or 10K samples are drawn, crowd rejection sampling beats both random selection and the plain rejection-sampling baseline when evaluated on MTBench and AlpacaEval-v2. - **CoT analysis and code:** GPT-4o key-point counting and Bart-base cross-attention coverage both show that CCE's CoTs are more comprehensive. Code is available at github.com/Don-Joey/CCE. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 75 works · 12,975 citations - [McGill University](https://md.research.elamur.ai/free/en/universities/mcgill-university/) — University, Montreal, Canada · 87 works · 11,027 citations - [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 54 works · 13,565 citations - [City University of Hong Kong](https://md.research.elamur.ai/free/en/universities/city-university-of-hong-kong/) — University, Hong Kong, China · 45 works · 3,014 citations - [The Hong Kong University of Science and Technology (Guangzhou)](https://md.research.elamur.ai/free/en/universities/the-hong-kong-university-of-science-and-technology-guangzhou/) — University, Guangzhou, China · 53 works · 1,367 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 182 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works ## Cite Cite as: Qiyuan Zhang, Yufei Wang, Yuxin Jiang, Liangyou Li, Chuhan Wu, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Fuyuan Lyu and Chen Ma. 2025. Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{zhang2025crowd, title = {{Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge}}, author = {Qiyuan Zhang, Yufei Wang, Yuxin Jiang, Liangyou Li, Chuhan Wu, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Fuyuan Lyu and Chen Ma}, booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2025}, doi = {10.18653/v1/2025.acl-long.252}, url = {https://doi.org/10.18653/v1/2025.acl-long.252} } ``` - [← previous ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/zhu-2025-convsearch-r1-enhancing-query-reformulation/) - [next → Decoding Echo Chambers: LLM-Powered Simulations Revealing Polarization in Social Networks](https://md.research.elamur.ai/free/en/papers/wang-2025-decoding-echo-chambers-llm-powered/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Instruction: Classify this tweet from the categories of positive, negative, or neutral. Tweet: I’m so thankful for all the wonderful people in my life Sher Badshah and Hassan Sajjad. 2024. Reference-guided verdict: Llms-as-judges in automatic evaluation of free-form text. In arXiv. [find](https://scholar.google.com/scholar?q=Instruction%3A+Classify+this+tweet+from+the+categories+of+positive%2C+negative%2C+or+neutral%2E+Tweet%3A+I%E2%80%99m+so+thankful+for+all+the+wonderful+people+in+my+life+Sher+Badshah+and+Hassan+Sajjad%2E+2024%2E+Reference%2Dg) 2. Response A The sentiment of this social media comment is positive. The use of the word "grateful" implies appreciation and thankfulness, while "amazing experiences" suggest that these experiences have been very fulfilling or enjoyable. Response B Positive Adrien Bibal, Rémi Cardon, David Alfter, Rod… [find](https://scholar.google.com/scholar?q=Response+A+The+sentiment+of+this+social+media+comment+is+positive%2E+The+use+of+the+word+%22grateful%22+implies+appreciation+and+thankfulness%2C+while+%22amazing+experiences%22+suggest+that+these+experiences+have) *Updated 31 October 2026*