---
title: "Graders Should Cheat — paper by Zhou et al. (2025)"
description: "Using a language model to grade other models is cheap, but it breaks down on frontier tasks such as… 50 references, cited by 3 papers here."
authors: ["Jin Peng Zhou", "Séb Arnold", "Nan Ding", "Kilian Q Weinberger", "Nan Hua", "Fei Sha"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.838/"
doi: "10.18653/v1/2025.emnlp-main.838"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2025-graders-should-cheat-privileged-information/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2025-graders-should-cheat-privileged-information/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations

# Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations

**Authors:** [Jin Peng Zhou](https://md.research.elamur.ai/free/en/authors/jin-peng-zhou/), Séb Arnold, [Nan Ding](https://md.research.elamur.ai/free/en/authors/nan-ding/), [Kilian Q Weinberger](https://md.research.elamur.ai/free/en/authors/kilian-q-weinberger/), [Nan Hua](https://md.research.elamur.ai/free/en/authors/nan-hua/) and [Fei Sha](https://md.research.elamur.ai/free/en/authors/fei-sha/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.838/) · [PDF](https://aclanthology.org/2025.emnlp-main.838.pdf) · [DOI 10.18653/v1/2025.emnlp-main.838](https://doi.org/10.18653/v1/2025.emnlp-main.838) · [OpenAlex](https://openalex.org/W4416036191)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/zhou-2025-graders-should-cheat-privileged-information/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Using a language model to grade other models is cheap, but it breaks down on frontier tasks such as Olympiad math or graduate-level physics. On these tasks the grader may be no more capable than the model it judges. The authors borrow Vapnik's notion of privileged information (PI) and apply it to evaluation. PI is extra material shown only to the grader, never to the candidates, that makes judging easier. Examples include reference solutions, prompt-specific rating guidelines, earlier human ratings and detailed image captions. PI can be written by people or synthesized by an LM (for instance, guidelines distilled from pooled human votes) and is simply added to the grading prompt. Experiments mostly use pairwise comparisons on RewardBench, Vibe-Eval and MathOdyssey. With PI, general-purpose graders reach top RewardBench accuracy, beat single human raters on Vibe-Eval and get close to expert graders on Olympiad problems. PI also yields hint-based, easier versions of hard problems, which separate models that would otherwise all score near chance. Together, these results make it more credible for weaker graders to assess stronger models.

## Key points
- PI types studied include ground-truth answers, per-prompt rating guidelines, reused prior ratings and multimodal annotations such as image descriptions. These can be combined, and they come from manual, automatic or mixed pipelines.
- On RewardBench, PI-augmented graders match state-of-the-art accuracy and gain more than 6 percentage points over the same graders without PI. They outperform nearly all specially tuned reward models.
- On Vibe-Eval, Gemini 1.5 Flash and Pro with PI exceed individual human raters. Their Spearman correlation with the 5-annotator average rises by over 0.35, and the gains are largest on the Hard split.
- On MathOdyssey, graders reach more than 0.7 correlation with expert human scores. Hints synthesized from solutions improve how clearly models are separated.
- Models differ in how much hints help them. GPT-4o's win rate against Claude 3.5 Sonnet falls from 40.2% to 26.6% as hints are added, while Gemini models benefit more.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 120 works · 59,940 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Jin Peng Zhou, Séb Arnold, Nan Ding, Kilian Q Weinberger, Nan Hua and Fei Sha. 2025. Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhou2025graders,
  title = {{Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations}},
  author = {Jin Peng Zhou, Séb Arnold, Nan Ding, Kilian Q Weinberger, Nan Hua and Fei Sha},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.838},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.838}
}
```

- [← previous German Aspect-based Sentiment Analysis in the Wild: B2B Dataset Creation and Cross-Domain Evaluation](https://md.research.elamur.ai/free/en/papers/fehle-2025-german-aspect-based-sentiment-analysis/)
- [next → How to age BERT Well: Continuous Training for Historical Language Adaptation](https://md.research.elamur.ai/free/en/papers/harju-2025-how-age-bert-well-continuous/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/)

## References

50 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. 2023. TouchStone: Evaluating vision-language models by language models. arXiv [cs.CV]. [find](https://scholar.google.com/scholar?q=Shuai+Bai%2C+Shusheng+Yang%2C+Jinze+Bai%2C+Peng+Wang%2C+Xingxuan+Zhang%2C+Junyang+Lin%2C+Xinggang+Wang%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+2023%2E+TouchStone%3A+Evaluating+vision%2Dlanguage+models+by+language+models%2E+arXiv+)
2. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
3. Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. 2023. Elo uncovered: Robustness and best practices in language model evaluation. arXiv [cs.CL]. [find](https://scholar.google.com/scholar?q=Meriem+Boubdir%2C+Edward+Kim%2C+Beyza+Ermis%2C+Sara+Hooker%2C+and+Marzieh+Fadaee%2E+2023%2E+Elo+uncovered%3A+Robustness+and+best+practices+in+language+model+evaluation%2E+arXiv+%5Bcs%2ECL%5D%2E)
4. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Ch… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. 2024. Ticking all the boxes: Generated checklists improve llm evaluation and generation. arXiv preprint arXiv:2410.03608. [find](https://scholar.google.com/scholar?q=Jonathan+Cook%2C+Tim+Rockt%C3%A4schel%2C+Jakob+Foerster%2C+Dennis+Aumiller%2C+and+Alex+Wang%2E+2024%2E+Ticking+all+the+boxes%3A+Generated+checklists+improve+llm+evaluation+and+generation%2E+arXiv+preprint+arXiv%3A2410%2E03608)
6. Ehsan Doostmohammadi, Oskar Holmström, and Marco Kuhlmann. 2024. How reliable are automatic evaluation methods for instruction-tuned llms? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6321–6336. [find](https://scholar.google.com/scholar?q=Ehsan+Doostmohammadi%2C+Oskar+Holmstr%C3%B6m%2C+and+Marco+Kuhlmann%2E+2024%2E+How+reliable+are+automatic+evaluation+methods+for+instruction%2Dtuned+llms%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A)
7. Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. 2024. MathOdyssey: Benchmarking mathematical problem-solving skills in large language models using odyssey math data. arXiv [cs.CL]. [find](https://scholar.google.com/scholar?q=Meng+Fang%2C+Xiangpeng+Wan%2C+Fei+Lu%2C+Fei+Xing%2C+and+Kai+Zou%2E+2024%2E+MathOdyssey%3A+Benchmarking+mathematical+problem%2Dsolving+skills+in+large+language+models+using+odyssey+math+data%2E+arXiv+%5Bcs%2ECL%5D%2E)
8. Mara Finkelstein, Dan Deutsch, Parker Riley, Juraj Juraska, Geza Kovacs, and Markus Freitag. 2024. From jack of all trades to master of one: Specializing llm-based autoraters to a test set. arXiv preprint arXiv:2411.15387. — [arXiv:2411.15387](https://arxiv.org/abs/2411.15387) [find](https://scholar.google.com/scholar?q=Mara+Finkelstein%2C+Dan+Deutsch%2C+Parker+Riley%2C+Juraj+Juraska%2C+Geza+Kovacs%2C+and+Markus+Freitag%2E+2024%2E+From+jack+of+all+trades+to+master+of+one%3A+Specializing+llm%2Dbased+autoraters+to+a+test+set%2E+arXiv+prep)
9. Gemini Team, Google. 2024. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). Technical report, Google.
10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv [cs.LG].
11. Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. [MetricX-23: The google submission to the WMT 2023 metrics shared task](https://md.research.elamur.ai/free/en/papers/juraska-2023-metricx-23-google-submission-wmt/). In Proceedings of the Eighth Conference on Machine Translation, pages 756– 767, Stroudsburg, PA, USA. Association for C…
12. Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. In International Conference on Machine Learning, pages 23662–23733.… [find](https://scholar.google.com/scholar?q=Akbir+Khan%2C+John+Hughes%2C+Dan+Valentine%2C+Laura+Ruis%2C+Kshitij+Sachan%2C+Ansh+Radhakrishnan%2C+Edward+Grefenstette%2C+Samuel+R+Bowman%2C+Tim+Rockt%C3%A4schel%2C+and+Ethan+Perez%2E+2024%2E+Debating+with+more+persuasive+llms)
13. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). In The Twelfth International Conference on Learning Representations.
14. Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, et al. 2024a. The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models. arXiv preprint arXiv:2406.05761. [find](https://scholar.google.com/scholar?q=Seungone+Kim%2C+Juyoung+Suk%2C+Ji+Yong+Cho%2C+Shayne+Longpre%2C+Chaeeun+Kim%2C+Dongkeun+Yoon%2C+Guijin+Son%2C+Yejin+Cho%2C+Sheikh+Shafayat%2C+Jinheon+Baek%2C+et+al%2E+2024a%2E+The+biggen+bench%3A+A+principled+benchmark+for+fin)
15. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024b. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/). In EMNLP.
16. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+et+al%2E+2024%2E+Rewardbench%3A+Evaluating+reward+models+for+)
17. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Pengfei Liu, et al. [Generative judge for evaluating alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/). In The Twelfth International Conference on Learning Representations.
18. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023a. [Generative judge for evaluating alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/). arXiv preprint arXiv:2310.05470.
19. Qintong Li, Leyang Cui, Lingpeng Kong, and Wei Bi. 2023b. [Exploring the reliability of large language models as customized evaluators for diverse nlp tasks](https://md.research.elamur.ai/free/en/papers/li-2025-exploring-reliability-large-language-models/). arXiv preprint arXiv:2310.19740.
20. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
21. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634.
22. Kun-Peng Ning, Shuo Yang, Yu-Yang Liu, Jia-Yu Yao, Zhen-Hui Liu, Yu Wang, Ming Pang, and Li Yuan. 2024. Peer-review-in-LLMs: Automatic evaluation method for LLMs in open-environment. arXiv [cs.CL]. [find](https://scholar.google.com/scholar?q=Kun%2DPeng+Ning%2C+Shuo+Yang%2C+Yu%2DYang+Liu%2C+Jia%2DYu+Yao%2C+Zhen%2DHui+Liu%2C+Yu+Wang%2C+Ming+Pang%2C+and+Li+Yuan%2E+2024%2E+Peer%2Dreview%2Din%2DLLMs%3A+Automatic+evaluation+method+for+LLMs+in+open%2Denvironment%2E+arXiv+%5Bcs%2ECL%5D%2E)
23. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv [cs.CL].
24. OpenAI. 2024. Learning to reason with LLMs. [https://openai.com/index/](https://openai.com/index/) learning-to-reason-with-llms/. Accessed: 2024-9-12. [↗](https://openai.com/index/)
25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Traini…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
26. Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, Ethan Yeo, Eugenie Lamprecht, Qi Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng, Cyprien de Masson d’Autume, Dani Yogatama, Mikel Artetxe, and Yi T… — [Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models](https://md.research.elamur.ai/free/en/papers/padlewski-2024-vibe-eval-hard-evaluation-suite/)
27. Arjun Panickssery, Sam Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. [find](https://scholar.google.com/scholar?q=Arjun+Panickssery%2C+Sam+Bowman%2C+and+Shi+Feng%2E+LLM+evaluators+recognize+and+favor+their+own+generations%2E)
28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. [BLEU: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, pages 311–318, Morristown, NJ, USA. Association for Computational Lingu…
29. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2023. [GPQA: A graduate-level google-proof Q&A benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). arXiv [cs.AI].
30. Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2023. Branch-solve-merge improves large language model evaluation and generation. arXiv preprint arXiv:2310.15123. [find](https://scholar.google.com/scholar?q=Swarnadeep+Saha%2C+Omer+Levy%2C+Asli+Celikyilmaz%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Xian+Li%2E+2023%2E+Branch%2Dsolve%2Dmerge+improves+large+language+model+evaluation+and+generation%2E+arXiv+preprint+arXiv%3A2310%2E15123)
31. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. [Bleurt: Learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/). arXiv preprint arXiv:2004.04696.
32. Shreya Shankar, JD Zamfirescu-Pereira, Björn Hartmann, Aditya Parameswaran, and Ian Arawjo. 2024. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–1… [find](https://scholar.google.com/scholar?q=Shreya+Shankar%2C+JD+Zamfirescu%2DPereira%2C+Bj%C3%B6rn+Hartmann%2C+Aditya+Parameswaran%2C+and+Ian+Arawjo%2E+2024%2E+Who+validates+the+validators%3F+aligning+llm%2Dassisted+evaluation+of+llm+outputs+with+human+preferences%2E+)
33. Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2024. [Aligning large multimodal models with factually augmented rlhf](https://md.research.elamur.ai/free/en/papers/sun-2023-aligning-large-multimodal-models-factually/). In Findings of the Association for Computational Linguistics ACL 2024, pages 13088–13110.
34. Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. 2024. [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature, 625(7995):476– 482.
35. Vladimir Vapnik. 1982. Estimation of dependences based on empirical data. Springer Series in Statistics. Springer, New York, NY. [find](https://scholar.google.com/scholar?q=Vladimir+Vapnik%2E+1982%2E+Estimation+of+dependences+based+on+empirical+data%2E+Springer+Series+in+Statistics%2E+Springer%2C+New+York%2C+NY%2E)
36. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. [Replacing judges with juries: Evaluating LLM generations with a panel of diverse models](https://md.research.elamur.ai/free/en/papers/verga-2024-replacing-judges-juries-evaluating-llm/). arXiv [cs.CL].
37. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, et al. 2023. [Freshllms: Refreshing large language models with search engine augmentation](https://md.research.elamur.ai/free/en/papers/vu-2023-freshllms-refreshing-large-language-models/). arXiv preprint arXiv:2310.03214.
38. Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. 2024. Foundational autoraters: Taming large language models for better automatic evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17086–17105. [find](https://scholar.google.com/scholar?q=Tu+Vu%2C+Kalpesh+Krishna%2C+Salaheddin+Alzubi%2C+Chris+Tar%2C+Manaal+Faruqui%2C+and+Yun%2DHsuan+Sung%2E+2024%2E+Foundational+autoraters%3A+Taming+large+language+models+for+better+automatic+evaluation%2E+In+Proceedings+of)
39. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024. [Interpretable preferences via multi-objective reward modeling and mixture-of-experts](https://md.research.elamur.ai/free/en/papers/wang-2024-interpretable-preferences-multi-objective-reward/). Preprint, arXiv:2406.12845.
40. Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023. [Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization](https://md.research.elamur.ai/free/en/papers/wang-2023-pandalm-automatic-evaluation-benchmark-llm/). arXiv preprint arXiv:2306.05087.
41. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837.
42. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). In Forty-first International Conference on Machine Learning.
43. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). Advances in Neural Information Processing Systems, 34:27263–27277.
44. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In International Conference on Machine Learning, pages 57905–57923. PMLR. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Xian+Li%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+E+Weston%2E+2024%2E+Self%2Drewarding+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pag)
45. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Jiatong+Yu%2C+Tianyu+Gao%2C+Yu+Meng%2C+Tanya+Goyal%2C+and+Danqi+Chen%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+In+The+Twelfth+International+Conference+on+Learning+Re)
46. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024a. [Generative verifiers: Reward modeling as next-token prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/). arXiv [cs.LG].
47. Qiyuan Zhang, Yufei Wang, Tiezheng Yu, Yuxin Jiang, Chuhan Wu, Liangyou Li, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, et al. 2024b. Reviseval: Improving llm-as-a-judge via response-adapted references. arXiv preprint arXiv:2410.05193. — [arXiv:2410.05193](https://arxiv.org/abs/2410.05193) [find](https://scholar.google.com/scholar?q=Qiyuan+Zhang%2C+Yufei+Wang%2C+Tiezheng+Yu%2C+Yuxin+Jiang%2C+Chuhan+Wu%2C+Liangyou+Li%2C+Yasheng+Wang%2C+Xin+Jiang%2C+Lifeng+Shang%2C+Ruiming+Tang%2C+et+al%2E+2024b%2E+Reviseval%3A+Improving+llm%2Das%2Da%2Djudge+via+response%2Dadapted+)
48. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
49. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in neural information processing systems, 36:46595–46623.
50. Jin Peng Zhou, Charles Staats, Wenda Li, Christian Szegedy, Kilian Q Weinberger, and Yuhuai Wu. 2024. Don’t trust: Verify – grounding LLM quantitative reasoning with autoformalization. arXiv [cs.AI]. [find](https://scholar.google.com/scholar?q=Jin+Peng+Zhou%2C+Charles+Staats%2C+Wenda+Li%2C+Christian+Szegedy%2C+Kilian+Q+Weinberger%2C+and+Yuhuai+Wu%2E+2024%2E+Don%E2%80%99t+trust%3A+Verify+%E2%80%93+grounding+LLM+quantitative+reasoning+with+autoformalization%2E+arXiv+%5Bcs%2EAI%5D%2E)

*Updated 31 October 2026*