---
title: "JuStRank: Benchmarking LLM Judges for System Ranking (2025)"
description: "The authors argue that instance-level accuracy does not predict ranking quality, because a judge's… 49 references, cited by 13 papers here."
authors: ["Ariel Gera", "Odellia Boni", "Yotam Perlitz", "Roy Bar-Haim", "Lilach Eden", "Asaf Yehudai"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.34/"
doi: "10.18653/v1/2025.acl-long.34"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gera-2025-justrank-benchmarking-llm-judges-system/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gera-2025-justrank-benchmarking-llm-judges-system/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) / JuStRank: Benchmarking LLM Judges for System Ranking

# JuStRank: Benchmarking LLM Judges for System Ranking

**Authors:** [Ariel Gera](https://md.research.elamur.ai/free/en/authors/ariel-gera/), Odellia Boni, [Yotam Perlitz](https://md.research.elamur.ai/free/en/authors/yotam-perlitz/), [Roy Bar-Haim](https://md.research.elamur.ai/free/en/authors/roy-bar-haim/), [Lilach Eden](https://md.research.elamur.ai/free/en/authors/lilach-eden/) and [Asaf Yehudai](https://md.research.elamur.ai/free/en/authors/asaf-yehudai/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.34/) · [PDF](https://aclanthology.org/2025.acl-long.34.pdf) · [DOI 10.18653/v1/2025.acl-long.34](https://doi.org/10.18653/v1/2025.acl-long.34) · [OpenAlex](https://openalex.org/W4412886854)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [13](https://md.research.elamur.ai/free/en/papers/gera-2025-justrank-benchmarking-llm-judges-system/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM judges are usually validated on individual responses or response pairs, without regard to which system produced them. In practice, they are mostly used to decide which model or configuration is better overall. The authors argue that instance-level accuracy does not predict ranking quality, because a judge's errors may cluster on particular systems. To measure this, they built JuStRank, a benchmark that treats judges as system rankers.

Responses from 63 systems to the 500 Arena Hard v0.1 instructions were scored by 48 judge realizations: reward models plus generative LLMs prompted in several ways. The scores were then aggregated into system rankings and compared with a human-derived Chatbot Arena ranking using Kendall's Tau.

Key findings:
- Ranking ability is not simply a function of model size or general quality.
- Some compact reward models match leading LLM judges.
- Two behaviours strongly relate to ranking accuracy: decisiveness, meaning the judge widens the gap between strong and weak systems, and system-specific bias.

The released data lets others extend the benchmark to new judges without collecting new human labels.

## Key points
- **Data and gold ranking:** The responses come from Arena Hard v0.1 (500 instructions, 63 systems, about 32K responses). Chatbot Arena's Hard Prompts category, restricted to English, serves as the gold ranking.
- **Judges:** There are 8 reward models (e.g., ArmoRM-Llama3-8B-v0.1, URM-LLaMa-3.1-8B, Skywork-Reward-Llama-3.1-8B-v0.2). There are also 10 LLMs (e.g., GPT-4o, Llama-3.1-405B-Instruct, Qwen2.5-72B-Instruct), each used in 4 realizations:
  - Numeric (a 0–100 score)
  - Likert (5 labels)
  - TokenProbs (probability of "yes" to a good/bad question)
  - Anchor (comparison against GPT-4-0314)
- **Released resource:** About 1.5M judgment scores, with scores aggregated by methods such as Mean, Win-Rate and BT.
- **Top results (Kendall's τ):**
  - Qwen2.5-72B-Instruct (Likert, Win-Rate): .83
  - URM-LLaMa-3.1-8B: .82
  - GPT-4o-2024-11-20 (Anchor): .82
- **Limitations:** The study covers only English and general-purpose instructions. Results may be sensitive to the exact prompt wording. Human preference is treated as one dimension, defined through Bradley-Terry wins.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 139 works · 70,691 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

## Cite

Cite as: Ariel Gera, Odellia Boni, Yotam Perlitz, Roy Bar-Haim, Lilach Eden and Asaf Yehudai. 2025. JuStRank: Benchmarking LLM Judges for System Ranking. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gera2025justrank,
  title = {{JuStRank: Benchmarking LLM Judges for System Ranking}},
  author = {Ariel Gera, Odellia Boni, Yotam Perlitz, Roy Bar-Haim, Lilach Eden and Asaf Yehudai},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.34},
  url = {https://doi.org/10.18653/v1/2025.acl-long.34}
}
```

- [← previous Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large Language Models](https://md.research.elamur.ai/free/en/papers/xu-2025-interactive-evolution-neural-symbolic-self/)
- [next → L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models](https://md.research.elamur.ai/free/en/papers/jeon-2025-l4q-parameter-efficient-quantization-aware/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/)

## References

49 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shir Ashury Tahan, Ariel Gera, Benjamin Sznajder, Leshem Choshen, Liat Ein-Dor, and Eyal Shnarch. 2024. Label-efficient model selection for text generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8384–8402, Bangkok, T… [find](https://scholar.google.com/scholar?q=Shir+Ashury+Tahan%2C+Ariel+Gera%2C+Benjamin+Sznajder%2C+Leshem+Choshen%2C+Liat+Ein%2DDor%2C+and+Eyal+Shnarch%2E+2024%2E+Label%2Defficient+model+selection+for+text+generation%2E+In+Proceedings+of+the+62nd+Annual+Meeting+o)
2. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. 2024. [LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks](https://md.research.elamur.ai/free/en/papers/bavaresco-2024-llms-instead-human-judges-large/). arXiv:2406.18403.
3. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E+Terry%2E+1952%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93+345%2E)
4. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. [InternLM2 technical report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/). arXiv:2403.17297.
5. Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. ODIN: Disentangled reward mitigates hacking in RLHF. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Lichang+Chen%2C+Chen+Zhu%2C+Jiuhai+Chen%2C+Davit+Soselia%2C+Tianyi+Zhou%2C+Tom+Goldstein%2C+Heng+Huang%2C+Mohammad+Shoeybi%2C+and+Bryan+Catanzaro%2E+2024%2E+ODIN%3A+Disentangled+reward+mitigates+hacking+in+RLHF%2E+In+Forty%2Df)
6. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. [Chatbot Arena: An open platform for evaluating LLMs by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/). In Forty-first International Conference on Machine Learn…
7. Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H Holliday, Bob M Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, et al. 2024. Social choice should guide ai alignment in dealing with diverse human feedback. arXiv:2404.10271. — [arXiv:2404.10271](https://arxiv.org/abs/2404.10271) [find](https://scholar.google.com/scholar?q=Vincent+Conitzer%2C+Rachel+Freedman%2C+Jobst+Heitzig%2C+Wesley+H+Holliday%2C+Bob+M+Jacobs%2C+Nathan+Lambert%2C+Milan+Moss%C3%A9%2C+Eric+Pacuit%2C+Stuart+Russell%2C+Hailey+Schoelkopf%2C+et+al%2E+2024%2E+Social+choice+should+guide+)
8. Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. Re-examining system-level correlations of automatic summarization evaluation metrics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 6038–6052, Se… [find](https://scholar.google.com/scholar?q=Daniel+Deutsch%2C+Rotem+Dror%2C+and+Dan+Roth%2E+2022%2E+Re%2Dexamining+system%2Dlevel+correlations+of+automatic+summarization+evaluation+metrics%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapte)
9. Florian E Dorner, Vivian Y Nastl, and Moritz Hardt. 2024. Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data. arXiv:2410.13341. — [arXiv:2410.13341](https://arxiv.org/abs/2410.13341) [find](https://scholar.google.com/scholar?q=Florian+E+Dorner%2C+Vivian+Y+Nastl%2C+and+Moritz+Hardt%2E+2024%2E+Limits+to+scalable+evaluation+at+the+frontier%3A+LLM+as+judge+won%E2%80%99t+beat+twice+the+data%2E+arXiv%3A2410%2E13341%2E+%E2%80%94+%5BarXiv%3A2410%2E13341%5D%28https%3A%2F%2Farxiv%2Eor)
10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv:2407.21783.
11. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv:2404.04475. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2024%2E+Length%2Dcontrolled+AlpacaEval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv%3A2404%2E04475%2E)
12. Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, and John P Dickerson. 2024. Style outweighs substance: Failure modes of LLM judges in alignment benchmarking. arXiv:2409.15268. — [arXiv:2409.15268](https://arxiv.org/abs/2409.15268) [find](https://scholar.google.com/scholar?q=Benjamin+Feuer%2C+Micah+Goldblum%2C+Teresa+Datta%2C+Sanjana+Nambiar%2C+Raz+Besaleli%2C+Samuel+Dooley%2C+Max+Cembalest%2C+and+John+P+Dickerson%2E+2024%2E+Style+outweighs+substance%3A+Failure+modes+of+LLM+judges+in+alignme)
13. Srishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzieh Fadaee. 2024. [M-RewardBench: Evaluating reward models in multilingual settings](https://md.research.elamur.ai/free/en/papers/gureja-2025-m-rewardbench-evaluating-reward-models/). arXiv:2410.15522.
14. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+2024%2E+Mixtral+o)
15. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver…
16. Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A Hale. 2024. The PRISM alignment project: What participatory, representative and individualised human feedback reveals abou… [find](https://scholar.google.com/scholar?q=Hannah+Rose+Kirk%2C+Alexander+Whitefield%2C+Paul+R%C3%B6ttger%2C+Andrew+Bean%2C+Katerina+Margatina%2C+Juan+Ciro%2C+Rafael+Mosquera%2C+Max+Bartolo%2C+Adina+Williams%2C+He+He%2C+Bertie+Vidgen%2C+and+Scott+A+Hale%2E+2024%2E+The+PRISM+)
17. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems, volume 37, pages 68772–68802. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Arjun+Panickssery%2C+Samuel+R%2E+Bowman%2C+and+Shi+Feng%2E+2024%2E+Llm+evaluators+recognize+and+favor+their+own+generations%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+37%2C+pages+68772%E2%80%9368802%2E+C)
18. Meelis Kull, Telmo Silva Filho, and Peter Flach. 2017. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machin… [find](https://scholar.google.com/scholar?q=Meelis+Kull%2C+Telmo+Silva+Filho%2C+and+Peter+Flach%2E+2017%2E+Beta+calibration%3A+a+well%2Dfounded+and+easily+implemented+improvement+on+logistic+calibration+for+binary+classifiers%2E+In+Proceedings+of+the+20th+In)
19. Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. 2024. [OffsetBias: Leveraging debiased data for tuning evaluators](https://md.research.elamur.ai/free/en/papers/park-2024-offsetbias-leveraging-debiased-data-tuning/). In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1043–1067, Miami, Florida, USA. Association for Computational Linguistics.
20. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. RewardBench: Evaluating reward models for language modeling. arXiv:2403.13787. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+et+al%2E+2024%2E+RewardBench%3A+Evaluating+reward+models+for+)
21. Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, and Kyomin Jung. 2024a. Are LLM-judges robust to expressions of uncertainty? investigating the effect of epistemic markers on LLM-based evaluation. arXiv:2410.20774. [find](https://scholar.google.com/scholar?q=Dongryeol+Lee%2C+Yerin+Hwang%2C+Yongil+Kim%2C+Joonsuk+Park%2C+and+Kyomin+Jung%2E+2024a%2E+Are+LLM%2Djudges+robust+to+expressions+of+uncertainty%3F+investigating+the+effect+of+epistemic+markers+on+LLM%2Dbased+evaluation)
22. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024b. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with ai feedback. arXiv:2309.00267. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Thomas+Mesnard%2C+Johan+Ferret%2C+Kellie+Lu%2C+Colton+Bishop%2C+Ethan+Hall%2C+Victor+Carbune%2C+Abhinav+Rastogi%2C+and+Sushant+Prakash%2E+2024b%2E+RLAIF+vs%2E+RLHF%3A+Scaling+r)
23. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. [From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/). arXiv:2406.11939.
24. Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology. [find](https://scholar.google.com/scholar?q=Rensis+Likert%2E+1932%2E+A+technique+for+the+measurement+of+attitudes%2E+Archives+of+Psychology%2E)
25. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024a. [Skywork-reward: Bag of tricks for reward modeling in LLMs](https://md.research.elamur.ai/free/en/papers/liu-2024-skywork-reward-bag-tricks/). arXiv:2410.18451.
26. Yotam Perlitz, Ariel Gera, Ofir Arviv, Asaf Yehudai, Elron Bandel, Eyal Shnarch, Michal Shmueli-Scheuer, and Leshem Choshen. 2024. Do these LLM benchmarks agree? Fixing benchmark evaluation with BenchBench. arXiv:2407.13696. — [arXiv:2407.13696](https://arxiv.org/abs/2407.13696) [find](https://scholar.google.com/scholar?q=Yotam+Perlitz%2C+Ariel+Gera%2C+Ofir+Arviv%2C+Asaf+Yehudai%2C+Elron+Bandel%2C+Eyal+Shnarch%2C+Michal+Shmueli%2DScheuer%2C+and+Leshem+Choshen%2E+2024%2E+Do+these+LLM+benchmarks+agree%3F+Fixing+benchmark+evaluation+with+Bench)
27. Maxime Peyrard, Wei Zhao, Steffen Eger, and Robert West. 2021. Better than average: Paired evaluation of NLP systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Maxime+Peyrard%2C+Wei+Zhao%2C+Steffen+Eger%2C+and+Robert+West%2E+2021%2E+Better+than+average%3A+Paired+evaluation+of+NLP+systems%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Lin)
28. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529–558. [find](https://scholar.google.com/scholar?q=Ehud+Reiter+and+Anja+Belz%2E+2009%2E+An+investigation+into+the+validity+of+some+metrics+for+automatically+evaluating+natural+language+generation+systems%2E+Computational+Linguistics%2C+35%284%29%3A529%E2%80%93558%2E)
29. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. arXiv:2310.10076. — [arXiv:2310.10076](https://arxiv.org/abs/2310.10076) [find](https://scholar.google.com/scholar?q=Keita+Saito%2C+Akifumi+Wachi%2C+Koki+Wataoka%2C+and+Youhei+Akimoto%2E+2023%2E+Verbosity+bias+in+preference+labeling+by+large+language+models%2E+arXiv%3A2310%2E10076%2E+%E2%80%94+%5BarXiv%3A2310%2E10076%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E10)
30. Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez, Meelis Kull, and Peter Flach. 2023. Classifier calibration: a survey on how to assess and improve predicted class probabilities. Machine Learning, 112(9):3211–3260. [find](https://scholar.google.com/scholar?q=Telmo+Silva+Filho%2C+Hao+Song%2C+Miquel+Perello%2DNieto%2C+Raul+Santos%2DRodriguez%2C+Meelis+Kull%2C+and+Peter+Flach%2E+2023%2E+Classifier+calibration%3A+a+survey+on+how+to+assess+and+improve+predicted+class+probabilitie)
31. Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2024. JudgeBench: A benchmark for evaluating LLM-based judges. arXiv:2410.12784. [find](https://scholar.google.com/scholar?q=Sijun+Tan%2C+Siyuan+Zhuang%2C+Kyle+Montgomery%2C+William+Y+Tang%2C+Alejandro+Cuadron%2C+Chenguang+Wang%2C+Raluca+Ada+Popa%2C+and+Ion+Stoica%2E+2024%2E+JudgeBench%3A+A+benchmark+for+evaluating+LLM%2Dbased+judges%2E+arXiv%3A2410)
32. Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2024b. [RM-bench: Benchmarking reward models of language models with subtlety and style](https://md.research.elamur.ai/free/en/papers/liu-2024-rm-bench-benchmarking-reward-models/). arXiv:2410.16184.
33. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges. arXiv:2406.12624. [find](https://scholar.google.com/scholar?q=Aman+Singh+Thakur%2C+Kartik+Choudhary%2C+Venkat+Srinik+Ramayapally%2C+Sankaran+Vaidyanathan%2C+and+Dieuwke+Hupkes%2E+2024%2E+Judging+the+judges%3A+Evaluating+alignment+and+vulnerabilities+in+LLMs%2Das%2Djudges%2E+arXiv%3A2)
34. Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. 2024. Uncertainty-aware reward model: Teaching reward models to know what is unknown. arXiv:2410.00847. — [arXiv:2410.00847](https://arxiv.org/abs/2410.00847) [find](https://scholar.google.com/scholar?q=Xingzhou+Lou%2C+Dong+Yan%2C+Wei+Shen%2C+Yuzi+Yan%2C+Jian+Xie%2C+and+Junge+Zhang%2E+2024%2E+Uncertainty%2Daware+reward+model%3A+Teaching+reward+models+to+know+what+is+unknown%2E+arXiv%3A2410%2E00847%2E+%E2%80%94+%5BarXiv%3A2410%2E00847%5D%28http)
35. Annie Louis and Ani Nenkova. 2013. [Automatically assessing machine summary content without a gold standard](https://md.research.elamur.ai/free/en/papers/louis-2013-automatically-assessing-machine-summary-content/). Computational Linguistics, 39(2):267– 300.
36. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt LLM evaluation. arXiv:2401.00595. [find](https://scholar.google.com/scholar?q=Moran+Mizrahi%2C+Guy+Kaplan%2C+Dan+Malkin%2C+Rotem+Dror%2C+Dafna+Shahaf%2C+and+Gabriel+Stanovsky%2E+2024%2E+State+of+what+art%3F+a+call+for+multi%2Dprompt+LLM+evaluation%2E+arXiv%3A2401%2E00595%2E)
37. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conferen… [find](https://scholar.google.com/scholar?q=Katherine+Tian%2C+Eric+Mitchell%2C+Allan+Zhou%2C+Archit+Sharma%2C+Rafael+Rafailov%2C+Huaxiu+Yao%2C+Chelsea+Finn%2C+and+Christopher+Manning%2E+2023%2E+Just+ask+for+calibration%3A+Strategies+for+eliciting+calibrated+confid)
38. John W Tukey. 1949. Comparing individual means in the analysis of variance. Biometrics, pages 99–114. [find](https://scholar.google.com/scholar?q=John+W+Tukey%2E+1949%2E+Comparing+individual+means+in+the+analysis+of+variance%2E+Biometrics%2C+pages+99%E2%80%93114%2E)
39. Pius von Däniken, Jan Deriu, and Mark Cieliebak. 2024. A measure of the system dependence of automated metrics. arXiv:2412.03152. [find](https://scholar.google.com/scholar?q=Pius+von+D%C3%A4niken%2C+Jan+Deriu%2C+and+Mark+Cieliebak%2E+2024%2E+A+measure+of+the+system+dependence+of+automated+metrics%2E+arXiv%3A2412%2E03152%2E)
40. Pius Von Däniken, Jan Deriu, Don Tuggener, and Mark Cieliebak. 2024. Favi-Score: A measure for favoritism in automated preference ratings for generative AI evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4437– 4454… [find](https://scholar.google.com/scholar?q=Pius+Von+D%C3%A4niken%2C+Jan+Deriu%2C+Don+Tuggener%2C+and+Mark+Cieliebak%2E+2024%2E+Favi%2DScore%3A+A+measure+for+favoritism+in+automated+preference+ratings+for+generative+AI+evaluation%2E+In+Proceedings+of+the+62nd+Annua)
41. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Advances in Neural Information Processing Systems, volume 3…
42. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024. [Interpretable preferences via multi-objective reward modeling and mixture-of-experts](https://md.research.elamur.ai/free/en/papers/wang-2024-interpretable-preferences-multi-objective-reward/). In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10582– 10592, Miami, Florida, USA. Association for Computation…
43. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Zefan+Cai%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2023%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv%3A2305%2E17926%2E)
44. Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. 2024. Systematic evaluation of LLM-as-a-judge in LLM alignment tasks: Explainable metrics and diverse prompt templates. arXiv:2408.13006. — [arXiv:2408.13006](https://arxiv.org/abs/2408.13006) [find](https://scholar.google.com/scholar?q=Hui+Wei%2C+Shenghua+He%2C+Tian+Xia%2C+Andy+Wong%2C+Jingyang+Lin%2C+and+Mei+Han%2E+2024%2E+Systematic+evaluation+of+LLM%2Das%2Da%2Djudge+in+LLM+alignment+tasks%3A+Explainable+metrics+and+diverse+prompt+templates%2E+arXiv%3A2408)
45. Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. 2024. [Pride and prejudice: LLM amplifies self-bias in self-refinement](https://md.research.elamur.ai/free/en/papers/xu-2024-pride-prejudice-llm-amplifies-self/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15474–15492, Bangkok, Thail…
46. Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024. Regularizing hidden states enables learning generalizable reward model for LLMs. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Rui+Yang%2C+Ruomeng+Ding%2C+Yong+Lin%2C+Huan+Zhang%2C+and+Tong+Zhang%2E+2024%2E+Regularizing+hidden+states+enables+learning+generalizable+reward+model+for+LLMs%2E+In+Advances+in+Neural+Information+Processing+System)
47. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2024. [Justice or prejudice? quantifying biases in LLM-as-a-judge](https://md.research.elamur.ai/free/en/papers/ye-2024-justice-prejudice-quantifying-biases-llm/). arXiv:2410.02736.
48. Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Eyal Shnarch, and Leshem Choshen. 2024. Achieving human parity in content-grounded datasets generation. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Asaf+Yehudai%2C+Boaz+Carmeli%2C+Yosi+Mass%2C+Ofir+Arviv%2C+Nathaniel+Mills%2C+Eyal+Shnarch%2C+and+Leshem+Choshen%2E+2024%2E+Achieving+human+parity+in+content%2Dgrounded+datasets+generation%2E+In+The+Twelfth+International)
49. Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. 2024. Advancing LLM reasoning generalists with preference trees. arXiv:2404.02078. [find](https://scholar.google.com/scholar?q=Lifan+Yuan%2C+Ganqu+Cui%2C+Hanbin+Wang%2C+Ning+Ding%2C+Xingyao+Wang%2C+Jia+Deng%2C+Boji+Shan%2C+Huimin+Chen%2C+Ruobing+Xie%2C+Yankai+Lin%2C+Zhenghao+Liu%2C+Bowen+Zhou%2C+Hao+Peng%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024%2E+Advancin)

*Updated 31 October 2026*