---
title: "An Empirical Study of LLM-as-a-Judge: How Design Choices"
description: "They vary three design choices: what the judge prompt contains (reference answers, score rubrics)… 44 references, cited by 7 papers here."
authors: ["Yusuke Yamauchi", "Taro Yano", "Masafumi Oyamada"]
year: 2026
venue: "Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)"
source: "https://aclanthology.org/2026.gem-main.19/"
doi: "10.18653/v1/2026.gem-main.19"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yamauchi-2026-empirical-study-llm-as-judge/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yamauchi-2026-empirical-study-llm-as-judge/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/) / An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability

# An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability

**Authors:** Yusuke Yamauchi, Taro Yano and [Masafumi Oyamada](https://md.research.elamur.ai/free/en/authors/masafumi-oyamada/) · **Year:** 2026 · **Venue:** [Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)](https://md.research.elamur.ai/free/en/venues/gem/) ([GEM 2026](https://md.research.elamur.ai/free/en/venues/gem/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.gem-main.19/) · [PDF](https://aclanthology.org/2026.gem-main.19.pdf) · [DOI 10.18653/v1/2026.gem-main.19](https://doi.org/10.18653/v1/2026.gem-main.19) · [OpenAlex](https://openalex.org/W4415109192)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/yamauchi-2026-empirical-study-llm-as-judge/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM-as-a-Judge is widely used to score open-ended model outputs, but it is unclear how far its verdicts can be trusted. This GEM 2026 paper examines two properties of judge reliability: how well judge scores match human ratings, measured by correlation, and how stable they are across repeated runs, measured by Krippendorff's alpha. The authors use GPT-4o-2024-08-06 and LLaMA-3.1-70B-Instruct as evaluators on BIGGEN-Bench and EvalBiasBench, which contains correct and deliberately biased answers. They vary three design choices: what the judge prompt contains (reference answers, score rubrics), how the judge decodes (greedy versus sampling), and whether the judge reasons with Chain-of-Thought before scoring. The clearest finding is that explicit scoring criteria matter most. Averaging several sampled scores tracks humans better than a single greedy score. CoT adds little once good rubrics are in place. These results point to a cheap and effective setup: direct scoring with rubrics and averaged samples. All experiments are English-only, so the conclusions still need checking for other languages, including Ukrainian.

## Key points
- **Data and setup:**
  - 765 BIGGEN-Bench instances with hand-written criteria and 80 EvalBiasBench instances.
  - For EvalBiasBench, criteria and reference answers were generated with GPT-4o, and the rubrics were designed to penalize biased answers.
  - The judge assigns 1–5 scores, and consistency is computed over five sampled scores.
- **Prompt content:**
  - With the full prompt, GPT-4o correlates with humans at 0.666. This falls to 0.591 without criteria, 0.638 without the reference, and 0.487 without both.
  - LLaMA3.1 is more sensitive, falling from 0.641 to 0.555, 0.581 and 0.346 respectively.
- **Rubric granularity:** Describing only the extreme scores (1 and 5) performed as well as or slightly better than describing every level. This questions whether descriptions for intermediate scores are needed.
- **Decoding:**
  - Sampling five scores and aggregating them consistently beat greedy decoding.
  - Taking the mean worked better than the median or majority vote, because it preserves in-between judgments such as 4.5. For example, LLaMA3.1's correlation rose from 0.593 with greedy decoding to 0.641 with the mean.
- **CoT:** Under the full prompt, direct scoring and scoring with CoT give almost identical results with GPT-4o: correlation of 0.664 versus 0.666 and alpha of 0.912 versus 0.908.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Tokyo](https://md.research.elamur.ai/free/en/universities/the-university-of-tokyo/) — University, Tokyo, Japan · 90 works · 9,156 citations
- [NEC](https://md.research.elamur.ai/free/en/companies/nec-jp/) — Company, Tokyo, Japan · 6 works · 546 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Yusuke Yamauchi, Taro Yano and Masafumi Oyamada. 2026. An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yamauchi2026empirical,
  title = {{An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability}},
  author = {Yusuke Yamauchi, Taro Yano and Masafumi Oyamada},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.19},
  url = {https://doi.org/10.18653/v1/2026.gem-main.19}
}
```

- [← previous AbjadGenEval: Abjad AI Generated Text Detection Shared Task for Languages Using Arabic Script at AbjadNLP 2026](https://md.research.elamur.ai/free/en/papers/ezzini-2026-abjadgeneval-abjad-ai-generated-text/)
- [next → Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward](https://md.research.elamur.ai/free/en/papers/guo-2026-backdoors-rlvr-jailbreak-backdoors-llms/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/)

## References

44 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra,… — [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
2. Anthropic. 2024. Claude 3.5 sonnet. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Claude+3%2E5+sonnet%2E)
3. Ron Artstein and Massimo Poesio. 2008. [Survey article: Inter-coder agreement for computational linguistics](https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/). Computational Linguistics, 34:555–596.
4. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. 2024. Benchmarking foundation models with language-model-as-an-examiner. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Jiahao+Ying%2C+Yixin+Cao%2C+Xin+Lv%2C+Yuze+He%2C+Xiaozhi+Wang%2C+Jifan+Yu%2C+Kaisheng+Zeng%2C+Yijia+Xiao%2C+Haozhe+Lyu%2C+et+al%2E+2024%2E+Benchmarking+foundation+models+with+language%2Dmodel%2Das%2Dan%2Dexaminer%2E+Advan)
5. Guiming Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. [Humans or llms as the judge? A study on judgement bias](https://md.research.elamur.ai/free/en/papers/chen-2024-humans-or-llms-as-judge/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 8301–8327. Association…
6. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, H… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
7. Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Sshubam Verma, and Mitesh M. Khapra. 2024. Finding blind spots in evaluator llms with interpretable checklists. Preprint, arXiv:2406.13439. [find](https://scholar.google.com/scholar?q=Sumanth+Doddapaneni%2C+Mohammed+Safi+Ur+Rahman+Khan%2C+Sshubam+Verma%2C+and+Mitesh+M%2E+Khapra%2E+2024%2E+Finding+blind+spots+in+evaluator+llms+with+interpretable+checklists%2E+Preprint%2C+arXiv%3A2406%2E13439%2E)
8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
9. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. CoRR, abs/2404.04475. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B%2E+Hashimoto%2E+2024%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+CoRR%2C+abs%2F2404%2E04475%2E)
10. Google. 2024. Our next-generation model: Gemini 1.5. [find](https://scholar.google.com/scholar?q=Google%2E+2024%2E+Our+next%2Dgeneration+model%3A+Gemini+1%2E5%2E)
11. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024a. A survey on llm-as-a-judge. CoRR, abs/2411.15594. [find](https://scholar.google.com/scholar?q=Jiawei+Gu%2C+Xuhui+Jiang%2C+Zhichao+Shi%2C+Hexiang+Tan%2C+Xuehao+Zhai%2C+Chengjin+Xu%2C+Wei+Li%2C+Yinghan+Shen%2C+Shengjie+Ma%2C+Honghao+Liu%2C+Yuanzhuo+Wang%2C+and+Jian+Guo%2E+2024a%2E+A+survey+on+llm%2Das%2Da%2Djudge%2E+CoRR%2C+abs%2F24)
12. [Aligning with human judgement: The role of pairwise preference in large language model evaluators](https://md.research.elamur.ai/free/en/papers/liu-2024-aligning-human-judgement-role-pairwise/). arXiv preprint arXiv:2403.16950.
13. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024b. A survey on llm-as-a-judge. ArXiv, abs/2411.15594. [find](https://scholar.google.com/scholar?q=Jiawei+Gu%2C+Xuhui+Jiang%2C+Zhichao+Shi%2C+Hexiang+Tan%2C+Xuehao+Zhai%2C+Chengjin+Xu%2C+Wei+Li%2C+Yinghan+Shen%2C+Shengjie+Ma%2C+Honghao+Liu%2C+Yuanzhuo+Wang%2C+and+Jian+Guo%2E+2024b%2E+A+survey+on+llm%2Das%2Da%2Djudge%2E+ArXiv%2C+abs%2F2)
14. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2023b. [Calibrating llm-based evaluator](https://md.research.elamur.ai/free/en/papers/liu-2024-calibrating-llm-based-evaluator/). Preprint, arXiv:2309.13308.
15. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
16. Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. [When can llms actually correct their own mistakes? A critical survey of self-correction of llms](https://md.research.elamur.ai/free/en/papers/kamoi-2024-when-can-llms-actually-correct/). CoRR, abs/2406.01297.
17. Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo… [find](https://scholar.google.com/scholar?q=Seungone+Kim%2C+Juyoung+Suk%2C+Ji+Yong+Cho%2C+Shayne+Longpre%2C+Chaeeun+Kim%2C+Dongkeun+Yoon%2C+Guijin+Son%2C+Yejin+Cho%2C+Sheikh+Shafayat%2C+Jinheon+Baek%2C+Sue+Hyun+Park%2C+Hyeonbin+Hwang%2C+Jinkyung+Jo%2C+Hyowon+Cho%2C+Haebin)
18. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2011%2E+Computing+krippendorff%E2%80%99s+alpha%2Dreliability%2E)
19. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024a. [Llms-as-judges: A comprehensive survey on llm-based evaluation methods](https://md.research.elamur.ai/free/en/papers/li-2024-llms-judges-comprehensive-survey-llm/). CoRR, abs/2412.05579.
20. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024b. [From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/). CoRR, abs/2406.11939.
21. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
22. Chengyuan Liu, Fubang Zhao, Lizhi Qing, Yangyang Kang, Changlong Sun, Kun Kuang, and Fei Wu. 2023a. Goal-oriented prompt attack and safety evaluation for llms. Preprint, arXiv:2309.11830. — [arXiv:2309.11830](https://arxiv.org/abs/2309.11830) [find](https://scholar.google.com/scholar?q=Chengyuan+Liu%2C+Fubang+Zhao%2C+Lizhi+Qing%2C+Yangyang+Kang%2C+Changlong+Sun%2C+Kun+Kuang%2C+and+Fei+Wu%2E+2023a%2E+Goal%2Doriented+prompt+attack+and+safety+evaluation+for+llms%2E+Preprint%2C+arXiv%3A2309%2E11830%2E+%E2%80%94+%5BarXiv%3A230)
23. Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2024a. [find](https://scholar.google.com/scholar?q=Yinhong+Liu%2C+Han+Zhou%2C+Zhijiang+Guo%2C+Ehsan+Shareghi%2C+Ivan+Vulic%CC%81%2C+Anna+Korhonen%2C+and+Nigel+Collier%2E+2024a%2E)
24. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024b. HD-eval: Aligning large language model evaluators through hierarchical criteria decomposition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Yuxuan+Liu%2C+Tianchi+Yang%2C+Shaohan+Huang%2C+Zihan+Zhang%2C+Haizhen+Huang%2C+Furu+Wei%2C+Weiwei+Deng%2C+Feng+Sun%2C+and+Qi+Zhang%2E+2024b%2E+HD%2Deval%3A+Aligning+large+language+model+evaluators+through+hierarchical+criter)
25. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
26. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
27. CoRR, Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computa…
28. Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. 2024a. [Offsetbias: Leveraging debiased data for tuning evaluators](https://md.research.elamur.ai/free/en/papers/park-2024-offsetbias-leveraging-debiased-data-tuning/). Preprint, arXiv:2407.06551.
29. Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. 2024b. [Offsetbias: Leveraging debiased data for tuning evaluators](https://md.research.elamur.ai/free/en/papers/park-2024-offsetbias-leveraging-debiased-data-tuning/). In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 1043–1067. Association for Computati…
30. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [Comet: A neural framework for mt evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). Preprint, arXiv:2009.09025.
31. Kayla Schroeder and Zach Wood-Doughty. 2024. Can you trust LLM judgments? reliability of llm-as-ajudge. CoRR, abs/2412.12509. — [arXiv:2412.12509](https://arxiv.org/abs/2412.12509) [find](https://scholar.google.com/scholar?q=Kayla+Schroeder+and+Zach+Wood%2DDoughty%2E+2024%2E+Can+you+trust+LLM+judgments%3F+reliability+of+llm%2Das%2Dajudge%2E+CoRR%2C+abs%2F2412%2E12509%2E+%E2%80%94+%5BarXiv%3A2412%2E12509%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2412%2E12509%29)
32. Yuda Song, Hanlin Zhang, Carson Eisenach, Sham M. Kakade, Dean P. Foster, and Udaya Ghai. 2024. Mind the gap: Examining the self-improvement capabilities of large language models. abs/2412.02674. — [arXiv:2412.02674](https://arxiv.org/abs/2412.02674) [find](https://scholar.google.com/scholar?q=Yuda+Song%2C+Hanlin+Zhang%2C+Carson+Eisenach%2C+Sham+M%2E+Kakade%2C+Dean+P%2E+Foster%2C+and+Udaya+Ghai%2E+2024%2E+Mind+the+gap%3A+Examining+the+self%2Dimprovement+capabilities+of+large+language+models%2E+abs%2F2412%2E02674%2E+%E2%80%94+%5Ba)
33. CoRR, Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Preprint, arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=CoRR%2C+Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+Preprint%2C+arXiv%3A1904%2E09675%2E)
34. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2025. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. Preprint, arXiv:2406.12624. [find](https://scholar.google.com/scholar?q=Aman+Singh+Thakur%2C+Kartik+Choudhary%2C+Venkat+Srinik+Ramayapally%2C+Sankaran+Vaidyanathan%2C+and+Dieuwke+Hupkes%2E+2025%2E+Judging+the+judges%3A+Evaluating+alignment+and+vulnerabilities+in+llms%2Das%2Djudges%2E+Preprin)
35. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685.
36. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). Preprint, arXiv:1411.5726.
37. A Qwen Team. 2024. Introducing qwen1.5. [find](https://scholar.google.com/scholar?q=A+Qwen+Team%2E+2024%2E+Introducing+qwen1%2E5%2E)
38. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,…
39. Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. 2024. Systematic evaluation of llm-as-a-judge in LLM alignment tasks: Explainable metrics and diverse prompt templates. CoRR, abs/2408.13006. — [arXiv:2408.13006](https://arxiv.org/abs/2408.13006) [find](https://scholar.google.com/scholar?q=Hui+Wei%2C+Shenghua+He%2C+Tian+Xia%2C+Andy+Wong%2C+Jingyang+Lin%2C+and+Mei+Han%2E+2024%2E+Systematic+evaluation+of+llm%2Das%2Da%2Djudge+in+LLM+alignment+tasks%3A+Explainable+metrics+and+diverse+prompt+templates%2E+CoRR%2C+abs%2F)
40. Ziyou Yan. 2024. Evaluating the effectiveness of llm-evaluators (aka llm-as-judge). eugeneyan.com. [find](https://scholar.google.com/scholar?q=Ziyou+Yan%2E+2024%2E+Evaluating+the+effectiveness+of+llm%2Devaluators+%28aka+llm%2Das%2Djudge%29%2E+eugeneyan%2Ecom%2E)
41. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
42. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. 2024. [Justice or prejudice? quantifying biases in llm-as-a-judge](https://md.research.elamur.ai/free/en/papers/ye-2024-justice-prejudice-quantifying-biases-llm/). CoRR, abs/2410.02736.
43. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Xian+Li%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+Weston%2E+2024%2E+Self%2Drewarding+language+models%2E+In+Forty%2Dfirst+International+Conference+on+Machine+Lea)
44. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Jiatong+Yu%2C+Tianyu+Gao%2C+Yu+Meng%2C+Tanya+Goyal%2C+and+Danqi+Chen%2E+2024%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+In+The+Twelfth+International+Conference+on+Learn)

*Updated 1 November 2026*