---
title: "Mitigating Judgment Preference Bias in Large Language Models"
description: "The authors propose Genii (Group-Based Polling Optimization), an unsupervised training framework that places several… 58 references."
authors: ["Shuliang Liu", "Zhipeng Xu", "Zhenghao Liu", "Yukun Yan", "Minghe Yu", "Yu Gu", "Chong Chen", "Huiyuan Xie", "Ge Yu"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.71/"
doi: "10.18653/v1/2026.findings-acl.71"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2026-mitigating-judgment-preference-bias-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2026-mitigating-judgment-preference-bias-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/) / Mitigating Judgment Preference Bias in Large Language Models through Group-Based Polling

# Mitigating Judgment Preference Bias in Large Language Models through Group-Based Polling

**Authors:** [Shuliang Liu](https://md.research.elamur.ai/free/en/authors/shuliang-liu/), [Zhipeng Xu](https://md.research.elamur.ai/free/en/authors/zhipeng-xu/), [Zhenghao Liu](https://md.research.elamur.ai/free/en/authors/zhenghao-liu/), [Yukun Yan](https://md.research.elamur.ai/free/en/authors/yukun-yan/), [Minghe Yu](https://md.research.elamur.ai/free/en/authors/minghe-yu/), [Yu Gu](https://md.research.elamur.ai/free/en/authors/yu-gu/), [Chong Chen](https://md.research.elamur.ai/free/en/authors/chong-chen/), [Huiyuan Xie](https://md.research.elamur.ai/free/en/authors/huiyuan-xie/) and [Ge Yu](https://md.research.elamur.ai/free/en/authors/ge-yu/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.71/) · [PDF](https://aclanthology.org/2026.findings-acl.71.pdf) · [DOI 10.18653/v1/2026.findings-acl.71](https://doi.org/10.18653/v1/2026.findings-acl.71) · [OpenAlex](https://openalex.org/W4416383312)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LLM-based evaluators ("LLM-as-a-Judge") tend to rate their own outputs, or outputs resembling their style, more favorably. This judgment preference bias makes automatic assessment less trustworthy. Earlier remedies either prompt a single model to reason more carefully or self-correct, or they combine several judges only at inference time, typically with a strong model steering weaker ones.

The authors propose Genii (Group-Based Polling Optimization), an unsupervised training framework that places several heterogeneous judge models in one multi-agent system. One model acts as the client and samples K candidate answers to a query. The remaining models act as servers and provide feedback, from which a group consistency score is derived for each answer. The client role rotates through every agent, and the resulting scores are turned into preference pairs. Each model is then tuned with DPO toward the collective view of the group.

Without any human labels, Genii outperforms judges trained on annotated judgment data and improves every agent in the group. Strong models benefit even when weaker models serve as servers. The work therefore offers a cheap route to fairer automatic evaluation.

## Key points
- Experiments use small open models such as Qwen2.5-7B-Instruct, Gemma2-9B-Instruct and Llama3.1-8B-Instruct. Similarity between responses is computed with MiniCPM-Embedding. Code is released at github.com/NEUIR/Genii.
- The bias analysis covers NQ, HotpotQA and TriviaQA, where the vanilla models' incorrect answers receive clearly higher perplexity after Genii training. This indicates weaker attachment to their own errors, with a larger gain for Gemma2-9B-Instruct than for the already strong Qwen2.5-7B-Instruct.
- Judgment accuracy correlates positively with generation quality measured by ROUGE-L. The link is strong for Llama3.1-8B-Instruct, weak for Gemma2-9B-Instruct and unstable for Qwen2.5-7B-Instruct.
- All extra cost falls in offline preference-data construction and scales with K and the number of agents, while inference remains a single forward pass. Raising K improves both judgment accuracy and the HSPP fairness metric.
- The stated limitations are dependence on the quality of embedding-based similarity and the absence of tests on larger-scale LLMs because of resource constraints.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 75 works · 12,975 citations
- [Northeastern University](https://md.research.elamur.ai/free/en/universities/northeastern-university-cn/) — University, Shenyang, China · 42 works · 1,940 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Shuliang Liu, Zhipeng Xu, Zhenghao Liu, Yukun Yan, Minghe Yu, Yu Gu, Chong Chen, Huiyuan Xie and Ge Yu. 2026. Mitigating Judgment Preference Bias in Large Language Models through Group-Based Polling. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2026mitigating,
  title = {{Mitigating Judgment Preference Bias in Large Language Models through Group-Based Polling}},
  author = {Shuliang Liu, Zhipeng Xu, Zhenghao Liu, Yukun Yan, Minghe Yu, Yu Gu, Chong Chen, Huiyuan Xie and Ge Yu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.71},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.71}
}
```

- [← previous Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise Perturbation](https://md.research.elamur.ai/free/en/papers/wang-2026-mitigating-hallucinations-vlms-enhancing-visual/)
- [next → Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling](https://md.research.elamur.ai/free/en/papers/wang-2026-mitigating-position-bias-transformers-layer/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/)

## References

58 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, and Xiaodong Deng. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). ArXiv preprint, abs/2309.16609.
2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. [Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). ArXiv preprint, abs/1611.09268.
3. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. [Semantic parsing on Freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of EMNLP, pages 1533–1544.
4. Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. 2023. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. ArXiv preprint, abs/2308.06595. — [arXiv:2308.06595](https://arxiv.org/abs/2308.06595) [find](https://scholar.google.com/scholar?q=Yonatan+Bitton%2C+Hritik+Bansal%2C+Jack+Hessel%2C+Rulin+Shao%2C+Wanrong+Zhu%2C+Anas+Awadalla%2C+Josh+Gardner%2C+Rohan+Taori%2C+and+Ludwig+Schmidt%2E+2023%2E+Visit%2Dbench%3A+A+benchmark+for+vision%2Dlanguage+instruction+follow)
5. Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Shi Feng, and Yu Meng. 2025a. Do llm evaluators prefer themselves for a reason? ArXiv preprint, abs/2504.03846. — [arXiv:2504.03846](https://arxiv.org/abs/2504.03846) [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chen%2C+Zhepei+Wei%2C+Xinyu+Zhu%2C+Shi+Feng%2C+and+Yu+Meng%2E+2025a%2E+Do+llm+evaluators+prefer+themselves+for+a+reason%3F+ArXiv+preprint%2C+abs%2F2504%2E03846%2E+%E2%80%94+%5BarXiv%3A2504%2E03846%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2504%2E038)
6. Zhi-Yuan Chen, Hao Wang, Xinyu Zhang, Enrui Hu, and Yankai Lin. 2025b. [Beyond the surface: Measuring self-preference in llm judgments](https://md.research.elamur.ai/free/en/papers/chen-2025-beyond-surface-measuring-self-preference/). ArXiv preprint, abs/2506.02592.
7. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2024. ULTRAFEEDBACK: boosting language models with scaled AI feedback. In Proceedings of ICML. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024%2E+ULTRAFEEDBACK%3A+boosting+language+models+with+scaled+AI+feedback%2E+In+Proceedings+of+IC)
8. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Tong Yu, Hanieh Deilamsalehy, Ruiyi Zhang, Sungchul Kim, and Franck Dernoncourt. 2024. Self-debiasing large language models: Zero-shot recognition and reduction of stereotypes. ArXiv preprint, abs/2402.01981. [find](https://scholar.google.com/scholar?q=Isabel+O+Gallegos%2C+Ryan+A+Rossi%2C+Joe+Barrow%2C+Md+Mehrab+Tanjim%2C+Tong+Yu%2C+Hanieh+Deilamsalehy%2C+Ruiyi+Zhang%2C+Sungchul+Kim%2C+and+Franck+Dernoncourt%2E+2024%2E+Self%2Ddebiasing+large+language+models%3A+Zero%2Dshot+re)
9. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. 2023. [The capacity for moral self-correction in large language models](https://md.research.elamur.ai/free/en/papers/ganguli-2023-capacity-moral-self-correction-large/). ArXiv preprint, abs/2302.07459.
10. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. ArXiv preprint, abs/2411.15594. [find](https://scholar.google.com/scholar?q=Jiawei+Gu%2C+Xuhui+Jiang%2C+Zhichao+Shi%2C+Hexiang+Tan%2C+Xuehao+Zhai%2C+Chengjin+Xu%2C+Wei+Li%2C+Yinghan+Shen%2C+Shengjie+Ma%2C+Honghao+Liu%2C+et+al%2E+2024%2E+A+survey+on+llm%2Das%2Da%2Djudge%2E+ArXiv+preprint%2C+abs%2F2411%2E15594%2E)
11. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). ArXiv preprint, abs/2501.12948.
12. Michael Hassid, Gabriel Synnaeve, Yossi Adi, and Roy Schwartz. 2025. Don’t overthink it. preferring shorter thinking chains for improved llm reasoning. ArXiv preprint, abs/2505.17813. — [arXiv:2505.17813](https://arxiv.org/abs/2505.17813) [find](https://scholar.google.com/scholar?q=Michael+Hassid%2C+Gabriel+Synnaeve%2C+Yossi+Adi%2C+and+Roy+Schwartz%2E+2025%2E+Don%E2%80%99t+overthink+it%2E+preferring+shorter+thinking+chains+for+improved+llm+reasoning%2E+ArXiv+preprint%2C+abs%2F2505%2E17813%2E+%E2%80%94+%5BarXiv%3A2505%2E17)
13. Hangfeng He, Hongming Zhang, and Dan Roth. 2024. SocREval: Large language models with the socratic method for reference-free reasoning evaluation. In Proceedings of NAACL, pages 2736–2764. [find](https://scholar.google.com/scholar?q=Hangfeng+He%2C+Hongming+Zhang%2C+and+Dan+Roth%2E+2024%2E+SocREval%3A+Large+language+models+with+the+socratic+method+for+reference%2Dfree+reasoning+evaluation%2E+In+Proceedings+of+NAACL%2C+pages+2736%E2%80%932764%2E)
14. Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8):1–79. [find](https://scholar.google.com/scholar?q=Xinyi+Hou%2C+Yanjie+Zhao%2C+Yue+Liu%2C+Zhou+Yang%2C+Kailong+Wang%2C+Li+Li%2C+Xiapu+Luo%2C+David+Lo%2C+John+Grundy%2C+and+Haoyu+Wang%2E+2024%2E+Large+language+models+for+software+engineering%3A+A+systematic+literature+review%2E)
15. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In Proceedings of ICLR.
16. Pengcheng Huang, Zhenghao Liu, Yukun Yan, Xiaoyuan Yi, Hao Chen, Zhiyuan Liu, Maosong Sun, Tong Xiao, Ge Yu, and Chenyan Xiong. 2025. Pip-kag: Mitigating knowledge conflicts in knowledge-augmented generation via parametric pruning. ArXiv preprint, abs/2502.15543. — [arXiv:2502.15543](https://arxiv.org/abs/2502.15543) [find](https://scholar.google.com/scholar?q=Pengcheng+Huang%2C+Zhenghao+Liu%2C+Yukun+Yan%2C+Xiaoyuan+Yi%2C+Hao+Chen%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+Tong+Xiao%2C+Ge+Yu%2C+and+Chenyan+Xiong%2E+2025%2E+Pip%2Dkag%3A+Mitigating+knowledge+conflicts+in+knowledge%2Daugmented+gen)
17. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of ACL, pages 1601–1611.
18. Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Akbir+Khan%2C+John+Hughes%2C+Dan+Valentine%2C+Laura+Ruis%2C+Kshitij+Sachan%2C+Ansh+Radhakrishnan%2C+Edward+Grefenstette%2C+Samuel+R%2E+Bowman%2C+Tim+Rockt%C3%A4schel%2C+and+Ethan+Perez%2E+2024%2E+Debating+with+more+persuasive+llm)
19. Seungone Kim, Jamin Shin, Yejin Choi, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024a. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). In Proceedings of ICLR.
20. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024b. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/). ArXiv preprint, abs/2405.01535.
21. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. ArXiv preprint, abs/2309.17012. [find](https://scholar.google.com/scholar?q=Ryan+Koo%2C+Minhwa+Lee%2C+Vipul+Raheja%2C+Jong+Inn+Park%2C+Zae+Myung+Kim%2C+and+Dongyeop+Kang%2E+2023%2E+Benchmarking+cognitive+biases+in+large+language+models+as+evaluators%2E+ArXiv+preprint%2C+abs%2F2309%2E17012%2E)
22. Marius Köppel, Alexander Segner, Martin Wagener, Lukas Pensel, Andreas Karwath, and Stefan Kramer. 2019. Pairwise learning to rank by neural networks revisited: Reconstruction, theoretical analysis and practical performance. ArXiv preprint, abs/1909.02768. — [arXiv:1909.02768](https://arxiv.org/abs/1909.02768) [find](https://scholar.google.com/scholar?q=Marius+K%C3%B6ppel%2C+Alexander+Segner%2C+Martin+Wagener%2C+Lukas+Pensel%2C+Andreas+Karwath%2C+and+Stefan+Kramer%2E+2019%2E+Pairwise+learning+to+rank+by+neural+networks+revisited%3A+Reconstruction%2C+theoretical+analysis+an)
23. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
24. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. ArXiv preprint, abs/2403.13787. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+et+al%2E+2024%2E+Rewardbench%3A+Evaluating+reward+models+for+)
25. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2025a. From generation to judgment: Opportunities and challenges of llm-as-a-judge, 2025. ArXiv preprint, abs/2411.16594. [find](https://scholar.google.com/scholar?q=Dawei+Li%2C+Bohan+Jiang%2C+Liangjie+Huang%2C+Alimohammad+Beigi%2C+Chengshuai+Zhao%2C+Zhen+Tan%2C+Amrita+Bhattacharjee%2C+Yuxuan+Jiang%2C+Canyu+Chen%2C+Tianhao+Wu%2C+et+al%2E+2025a%2E+From+generation+to+judgment%3A+Opportunitie)
26. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2024. [Generative judge for evaluating alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/). In Proceedings of ICLR.
27. Xinze Li, Sen Mei, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Hao Chen, Ge Yu, Zhiyuan Liu, Maosong Sun, and Chenyan Xiong. 2025b. RAG-DDR: optimizing retrieval-augmented generation using differentiable data rewards. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Xinze+Li%2C+Sen+Mei%2C+Zhenghao+Liu%2C+Yukun+Yan%2C+Shuo+Wang%2C+Shi+Yu%2C+Zheni+Zeng%2C+Hao+Chen%2C+Ge+Yu%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+and+Chenyan+Xiong%2E+2025b%2E+RAG%2DDDR%3A+optimizing+retrieval%2Daugmented+generation+using)
28. Yen-Ting Lin and Yun-Nung Chen. 2023. [LLM-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models](https://md.research.elamur.ai/free/en/papers/lin-2023-llm-eval-unified-multi-dimensional/). In Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023), pages 47–58.
29. Shuliang Liu, Xinze Li, Zhenghao Liu, Yukun Yan, Cheng Yang, Zheni Zeng, Zhiyuan Liu, Maosong Sun, and Ge Yu. 2025a. Judge as a judge: Improving the evaluation of retrieval-augmented generation through the judge-consistency of large language models. ArXiv preprint, abs/2502.18817. [find](https://scholar.google.com/scholar?q=Shuliang+Liu%2C+Xinze+Li%2C+Zhenghao+Liu%2C+Yukun+Yan%2C+Cheng+Yang%2C+Zheni+Zeng%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+and+Ge+Yu%2E+2025a%2E+Judge+as+a+judge%3A+Improving+the+evaluation+of+retrieval%2Daugmented+generation+throug)
30. Zhuo Liu, Moxin Li, Xun Deng, Qifan Wang, and Fuli Feng. 2025b. Assistant-guided mitigation of teacher preference bias in llm-as-a-judge. ArXiv preprint, abs/2505.19176. [find](https://scholar.google.com/scholar?q=Zhuo+Liu%2C+Moxin+Li%2C+Xun+Deng%2C+Qifan+Wang%2C+and+Fuli+Feng%2E+2025b%2E+Assistant%2Dguided+mitigation+of+teacher+preference+bias+in+llm%2Das%2Da%2Djudge%2E+ArXiv+preprint%2C+abs%2F2505%2E19176%2E)
31. R OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). preprint, abs/2303.08774.
32. ArXiv Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. 2024. [Offsetbias: Leveraging debiased data for tuning evaluators](https://md.research.elamur.ai/free/en/papers/park-2024-offsetbias-leveraging-debiased-data-tuning/). ArXiv preprint, abs/2407.06551.
33. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Proceedings of NeurIPS.
34. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. [Asqa: Factoid questions meet long-form answers](https://md.research.elamur.ai/free/en/papers/stelmakh-2022-asqa-factoid-questions-meet-long/). In Proceedings of EMNLP, pages 8273–8288.
35. Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. ArXiv preprint, abs/2410.12784. [find](https://scholar.google.com/scholar?q=Sijun+Tan%2C+Siyuan+Zhuang%2C+Kyle+Montgomery%2C+William+Y+Tang%2C+Alejandro+Cuadron%2C+Chenguang+Wang%2C+Raluca+Ada+Popa%2C+and+Ion+Stoica%2E+2024%2E+Judgebench%3A+A+benchmark+for+evaluating+llm%2Dbased+judges%2E+ArXiv+prep)
36. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). ArXiv preprint, abs/2312.11805.
37. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. ArXiv preprint, abs/2408.00118. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+et+al%2E+2024%2E+Gemma+2%3A+Improving+o)
38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). ArXiv preprint, abs/2307.09288.
39. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In Proceedings of ICLR.
40. Yicheng Wang, Jiayi Yuan, Yu-Neng Chuang, Zhuoer Wang, Yingchi Liu, Mark Cusick, Param Kulkarni, Zhengping Ji, Yasser Ibrahim, and Xia Hu. 2024. [find](https://scholar.google.com/scholar?q=Yicheng+Wang%2C+Jiayi+Yuan%2C+Yu%2DNeng+Chuang%2C+Zhuoer+Wang%2C+Yingchi+Liu%2C+Mark+Cusick%2C+Param+Kulkarni%2C+Zhengping+Ji%2C+Yasser+Ibrahim%2C+and+Xia+Hu%2E+2024%2E)
41. Dhp benchmark: Are llms good nlg evaluators? ArXiv preprint, abs/2408.13704. [find](https://scholar.google.com/scholar?q=Dhp+benchmark%3A+Are+llms+good+nlg+evaluators%3F+ArXiv+preprint%2C+abs%2F2408%2E13704%2E)
42. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2024. [Self-preference bias in llm-as-a-judge](https://md.research.elamur.ai/free/en/papers/wataoka-2024-self-preference-bias-llm-judge/). ArXiv preprint, abs/2410.21819.
43. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). ArXiv preprint, abs/2206.07682.
44. Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-ameta-judge. ArXiv preprint, abs/2407.19594. — [arXiv:2407.19594](https://arxiv.org/abs/2407.19594) [find](https://scholar.google.com/scholar?q=Tianhao+Wu%2C+Weizhe+Yuan%2C+Olga+Golovneva%2C+Jing+Xu%2C+Yuandong+Tian%2C+Jiantao+Jiao%2C+Jason+Weston%2C+and+Sainbayar+Sukhbaatar%2E+2024%2E+Meta%2Drewarding+language+models%3A+Self%2Dimproving+alignment+with+llm%2Das%2Dameta%2D)
45. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. [Wizardlm: Empowering large pre-trained language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). In Proceedings of ICLR.
46. Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue Liu, Nirav Shah, Rakesh Wang… [find](https://scholar.google.com/scholar?q=Xiao+Yang%2C+Kai+Sun%2C+Hao+Xin%2C+Yushi+Sun%2C+Nikita+Bhalla%2C+Xiangsen+Chen%2C+Sajal+Choudhary%2C+Rongze+Daniel+Gui%2C+Ziran+Will+Jiang%2C+Ziyu+Jiang%2C+Lingkun+Kong%2C+Brian+Moran%2C+Jiaqi+Wang%2C+Yifan+Xu%2C+An+Yan%2C+Chenyu+)
47. Xinyi Yang, Runzhe Zhan, Derek F Wong, Shu Yang, Junchao Wu, and Lidia S Chao. 2025. Rethinking prompt-based debiasing in large language models. ArXiv preprint, abs/2503.09219. — [arXiv:2503.09219](https://arxiv.org/abs/2503.09219) [find](https://scholar.google.com/scholar?q=Xinyi+Yang%2C+Runzhe+Zhan%2C+Derek+F+Wong%2C+Shu+Yang%2C+Junchao+Wu%2C+and+Lidia+S+Chao%2E+2025%2E+Rethinking+prompt%2Dbased+debiasing+in+large+language+models%2E+ArXiv+preprint%2C+abs%2F2503%2E09219%2E+%E2%80%94+%5BarXiv%3A2503%2E09219%5D%28ht)
48. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. [WikiQA: A challenge dataset for open-domain question answering](https://md.research.elamur.ai/free/en/papers/yang-2015-wikiqa-challenge-dataset-open-domain/). In Proceedings of EMNLP, pages 2013–2018.
49. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of EMNLP, pages 2369–2380.
50. Sijia Yao, Pengcheng Huang, Zhenghao Liu, Yu Gu, Yukun Yan, Shi Yu, and Ge Yu. 2025. Llm-qe: Improving query expansion by aligning large language models with ranking preferences. ArXiv preprint, abs/2502.17057. — [arXiv:2502.17057](https://arxiv.org/abs/2502.17057) [find](https://scholar.google.com/scholar?q=Sijia+Yao%2C+Pengcheng+Huang%2C+Zhenghao+Liu%2C+Yu+Gu%2C+Yukun+Yan%2C+Shi+Yu%2C+and+Ge+Yu%2E+2025%2E+Llm%2Dqe%3A+Improving+query+expansion+by+aligning+large+language+models+with+ranking+preferences%2E+ArXiv+preprint%2C+abs%2F2)
51. Hengyuan Zhang, Yanru Wu, Dawei Li, Sak Yang, Rui Zhao, Yong Jiang, and Fei Tan. 2024a. Balancing speciality and versatility: a coarse to fine framework for supervised fine-tuning large language model. In Proceedings of ACL, pages 7467–7509. [find](https://scholar.google.com/scholar?q=Hengyuan+Zhang%2C+Yanru+Wu%2C+Dawei+Li%2C+Sak+Yang%2C+Rui+Zhao%2C+Yong+Jiang%2C+and+Fei+Tan%2E+2024a%2E+Balancing+speciality+and+versatility%3A+a+coarse+to+fine+framework+for+supervised+fine%2Dtuning+large+language+model)
52. Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. 2024b. [Self-alignment for factuality: Mitigating hallucinations in llms via self-evaluation](https://md.research.elamur.ai/free/en/papers/zhang-2024-self-alignment-factuality-mitigating-hallucinations/). ArXiv preprint, abs/2402.09267.
53. Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In Proceedings of IEEE, pages 4320–4328. [find](https://scholar.google.com/scholar?q=Ying+Zhang%2C+Tao+Xiang%2C+Timothy+M%2E+Hospedales%2C+and+Huchuan+Lu%2E+2018%2E+Deep+mutual+learning%2E+In+Proceedings+of+IEEE%2C+pages+4320%E2%80%934328%2E)
54. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). ArXiv preprint, abs/2303.18223.
55. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Proceedings of NeurIPS.
56. Lianghui Zhu et al. 2023. [JudgeLM: Fine-tuned large language models are scalable judges](https://md.research.elamur.ai/free/en/papers/zhu-2023-judgelm-fine-tuned-large-language/). ArXiv preprint, abs/2310.17631.
57. Split Dataset Total Training Evol-Instruct (2024) ASQA (2022) MARCO QA (2016) Web Questions (2013) WikiQA (2015) Yahoo!QA 5,328 190 675 754 208 837 [find](https://scholar.google.com/scholar?q=Split+Dataset+Total+Training+Evol%2DInstruct+%282024%29+ASQA+%282022%29+MARCO+QA+%282016%29+Web+Questions+%282013%29+WikiQA+%282015%29+Yahoo%21QA+5%2C328+190+675+754+208+837)
58. Development UltraFeedback (2024) *Table 5: Data Statistics of the Training and Development Datasets.* [find](https://scholar.google.com/scholar?q=Development+UltraFeedback+%282024%29+%2ATable+5%3A+Data+Statistics+of+the+Training+and+Development+Datasets%2E%2A)

*Updated 1 November 2026*