---
title: "Learning to Judge — paper by Siro et al. (2026)"
description: "The authors propose GER-Eval, a diagnostic framework that splits evaluation into two stages. 29 references, cited by 3 papers here."
authors: ["Clemencia Siro", "Pourya Aliannejadi", "Mohammad Aliannejadi"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.335/"
doi: "10.18653/v1/2026.findings-eacl.335"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/siro-2026-learning-judge-llms-designing-applying/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/siro-2026-learning-judge-llms-designing-applying/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / Learning to Judge: LLMs Designing and Applying Evaluation Rubrics

# Learning to Judge: LLMs Designing and Applying Evaluation Rubrics

**Authors:** [Clemencia Siro](https://md.research.elamur.ai/free/en/authors/clemencia-siro/), Pourya Aliannejadi and [Mohammad Aliannejadi](https://md.research.elamur.ai/free/en/authors/mohammad-aliannejadi/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.335/) · [PDF](https://aclanthology.org/2026.findings-eacl.335.pdf) · [DOI 10.18653/v1/2026.findings-eacl.335](https://doi.org/10.18653/v1/2026.findings-eacl.335) · [OpenAlex](https://openalex.org/W7140126704)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM judges usually score generated text using rubrics written by humans, but such fixed criteria may not match how a model itself represents quality. Newer approaches let models write their own criteria and assume these self-made rubrics are more dependable, though this assumption has rarely been tested. The authors propose GER-Eval, a diagnostic framework that splits evaluation into two stages. First, a model proposes criteria, each with a name, definition, scale and scoring instruction. Second, it uses those criteria to rate candidate outputs, either zero-shot or with demonstrations. They test closed and open-weight models on four English benchmarks covering dialogue, summarization and instruction following. They compare model rubrics with human ones and measure agreement within and across models. Models produce sensible, task-aware criteria and apply their own rubrics consistently. Agreement drops when a model uses another model's rubric, and reliability weakens on factual, domain-heavy data. GPT-4o is more consistent and closer to human ratings than Llama. The authors conclude that each model has its own "evaluation dialect". LLM judges therefore suit checks of linguistic quality better than factual verification.

## Key points
- **Datasets and tasks:** GER-Eval covers three NLG tasks on four datasets with human quality scores: USR, SummEval, SumPubMed and HelpSteer2.
- **Models:** GPT-4o, GPT-4o-mini, Mixtral, Qwen2.5 and Llama-3.3.
- **Prompting conditions:**
  - Rubric generation: Task-only, Task + Contexts, and Task + Contrastive Examples (one high- and one low-rated output, chosen by human scores).
  - Rubric application: Rubric-only (zero-shot) or Rubric + Demonstrations (few-shot).
- **Reliability metrics:** ICC, Krippendorff's α and Fleiss' κ.
- **Main findings:**
  - Consistency is highest when models use rubrics they wrote themselves, intermediate with human-written rubrics, and lowest with rubrics produced by other models.
  - Human-model alignment is good for fluency and coherence but weak for factual criteria, especially on SumPubMed.
  - The authors plan to release data and scripts. Limitations include English-only data and a restricted model set.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Amsterdam](https://md.research.elamur.ai/free/en/universities/university-of-amsterdam/) — University, Amsterdam, Netherlands · 144 works · 173,906 citations
- [Centrum Wiskunde & Informatica](https://md.research.elamur.ai/free/en/labs/centrum-wiskunde-informatica/) — Research institute, Amsterdam, Netherlands · 3 works · 5,881 citations
- [Shahid Beheshti University](https://md.research.elamur.ai/free/en/universities/shahid-beheshti-university/) — University, Tehran, Iran · 5 works · 14 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Clemencia Siro, Pourya Aliannejadi and Mohammad Aliannejadi. 2026. Learning to Judge: LLMs Designing and Applying Evaluation Rubrics. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{siro2026learning,
  title = {{Learning to Judge: LLMs Designing and Applying Evaluation Rubrics}},
  author = {Clemencia Siro, Pourya Aliannejadi and Mohammad Aliannejadi},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.335},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.335}
}
```

- [← previous Language-Coupled Reinforcement Learning for Multilingual Retrieval-Augmented Generation](https://md.research.elamur.ai/free/en/papers/qi-2026-language-coupled-reinforcement-learning-multilingual/)
- [next → LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification](https://md.research.elamur.ai/free/en/papers/yang-2026-longspec-long-context-lossless-speculative/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

29 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wei-Lin Chiang, Chengrun Yang Mao, Yuhuai Wu, Xinyun Chen, and James Zou. 2024. Can large language models evaluate themselves accurately? arXiv preprint arXiv:2401.10020. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Chengrun+Yang+Mao%2C+Yuhuai+Wu%2C+Xinyun+Chen%2C+and+James+Zou%2E+2024%2E+Can+large+language+models+evaluate+themselves+accurately%3F+arXiv+preprint+arXiv%3A2401%2E10020%2E)
2. Yvan Dubois, Adam Jacob, Camille Dubois, Yonatan Belinkov, and Aviv Efrat. 2023. Alpacaeval: A strong set of reference rankings for instruction following llms. arXiv preprint arXiv:2310.01377. [find](https://scholar.google.com/scholar?q=Yvan+Dubois%2C+Adam+Jacob%2C+Camille+Dubois%2C+Yonatan+Belinkov%2C+and+Aviv+Efrat%2E+2023%2E+Alpacaeval%3A+A+strong+set+of+reference+rankings+for+instruction+following+llms%2E+arXiv+preprint+arXiv%3A2310%2E01377%2E)
3. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R%2E+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+SummEval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+fo)
4. Xiang Fu, Zifan Zhou, Aman Madaan, Jiani He, Hao Tan, and Luke Zettlemoyer. 2023. Debiasing llms for fair evaluation. arXiv preprint arXiv:2308.07309. — [arXiv:2308.07309](https://arxiv.org/abs/2308.07309) [find](https://scholar.google.com/scholar?q=Xiang+Fu%2C+Zifan+Zhou%2C+Aman+Madaan%2C+Jiani+He%2C+Hao+Tan%2C+and+Luke+Zettlemoyer%2E+2023%2E+Debiasing+llms+for+fair+evaluation%2E+arXiv+preprint+arXiv%3A2308%2E07309%2E+%E2%80%94+%5BarXiv%3A2308%2E07309%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2308%2E0)
5. Vivek Gupta, Prerna Bharti, Pegah Nokhiz, and Harish Karnick. 2021. SumPubMed: Summarization dataset of PubMed scientific articles. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: S… [find](https://scholar.google.com/scholar?q=Vivek+Gupta%2C+Prerna+Bharti%2C+Pegah+Nokhiz%2C+and+Harish+Karnick%2E+2021%2E+SumPubMed%3A+Summarization+dataset+of+PubMed+scientific+articles%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Com)
6. Youngseok Jang, Yuxuan Ruan, Aniruddh Jain, et al. 2023. Correcting for position bias in llm evaluators. arXiv preprint arXiv:2311.08889. — [arXiv:2311.08889](https://arxiv.org/abs/2311.08889) [find](https://scholar.google.com/scholar?q=Youngseok+Jang%2C+Yuxuan+Ruan%2C+Aniruddh+Jain%2C+et+al%2E+2023%2E+Correcting+for+position+bias+in+llm+evaluators%2E+arXiv+preprint+arXiv%3A2311%2E08889%2E+%E2%80%94+%5BarXiv%3A2311%2E08889%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E08889%29)
7. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
8. Erik Jones, Arjun Patrawala, and Jacob Steinhardt. 2025. Uncovering gaps in how humans and llms interpret subjective language. ArXiv, abs/2503.04113. [find](https://scholar.google.com/scholar?q=Erik+Jones%2C+Arjun+Patrawala%2C+and+Jacob+Steinhardt%2E+2025%2E+Uncovering+gaps+in+how+humans+and+llms+interpret+subjective+language%2E+ArXiv%2C+abs%2F2503%2E04113%2E)
9. Ryan Koo, Minhwa Lee, Vipul Raheja, et al. 2023. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309.17012. [find](https://scholar.google.com/scholar?q=Ryan+Koo%2C+Minhwa+Lee%2C+Vipul+Raheja%2C+et+al%2E+2023%2E+Benchmarking+cognitive+biases+in+large+language+models+as+evaluators%2E+arXiv+preprint+arXiv%3A2309%2E17012%2E)
10. Sneha Kudugunta, Xi Victoria Lin, Balakrishnan Narayanaswamy, et al. 2023. Investigating and reducing preference bias in llms for pairwise comparison. arXiv preprint arXiv:2312.11592. — [arXiv:2312.11592](https://arxiv.org/abs/2312.11592) [find](https://scholar.google.com/scholar?q=Sneha+Kudugunta%2C+Xi+Victoria+Lin%2C+Balakrishnan+Narayanaswamy%2C+et+al%2E+2023%2E+Investigating+and+reducing+preference+bias+in+llms+for+pairwise+comparison%2E+arXiv+preprint+arXiv%3A2312%2E11592%2E+%E2%80%94+%5BarXiv%3A2312%2E11)
11. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: Nlg evaluation using gpt-4 with better prompts](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634.
12. Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2024a. [Aligning with human judgement: The role of pairwise preference in large language model evaluators](https://md.research.elamur.ai/free/en/papers/liu-2024-aligning-human-judgement-role-pairwise/).
13. In Proceedings of the Conference on Language Modeling (COLM). [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+Conference+on+Language+Modeling+%28COLM%29%2E)
14. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024b. Hd-eval: Aligning large language model evaluators through hierarchical criteria decomposition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Yuxuan+Liu%2C+Tianchi+Yang%2C+Shaohan+Huang%2C+Zihan+Zhang%2C+Haizhen+Huang%2C+Furu+Wei%2C+Weiwei+Deng%2C+Feng+Sun%2C+and+Qi+Zhang%2E+2024b%2E+Hd%2Deval%3A+Aligning+large+language+model+evaluators+through+hierarchical+criter)
15. Shikib Mehri and Maxine Eskenazi. 2020. [USR: An unsupervised and reference free evaluation metric for dialog generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707, Online. Association for Computational Linguistics.
16. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
17. CoRR, Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2024. Branch-solve-merge improves large language model evaluation and generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=CoRR%2C+Swarnadeep+Saha%2C+Omer+Levy%2C+Asli+Celikyilmaz%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Xian+Li%2E+2024%2E+Branch%2Dsolve%2Dmerge+improves+large+language+model+evaluation+and+generation%2E+In+Proceedings+of+the+202)
18. Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, and Ian Arawjo. 2024. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST). [find](https://scholar.google.com/scholar?q=Shreya+Shankar%2C+J%2ED%2E+Zamfirescu%2DPereira%2C+Bj%C3%B6rn+Hartmann%2C+Aditya+G%2E+Parameswaran%2C+and+Ian+Arawjo%2E+2024%2E+Who+validates+the+validators%3F+aligning+llm%2Dassisted+evaluation+of+llm+outputs+with+human+preferen)
19. Weijia Shi, Runzhe Zhang, Jianyu Chi, et al. 2024. [Judging the judges: A systematic study of position bias in llm-as-a-judge](https://md.research.elamur.ai/free/en/papers/shi-2025-judging-judges-systematic-study-position/). arXiv preprint arXiv:2406.07791.
20. Oscar Stureborg, Iason Papadimitriou, Ruiqi Zhang, and Byron C. Wallace. 2024. [Large language models are inconsistent and biased evaluators](https://md.research.elamur.ai/free/en/papers/stureborg-2024-large-language-models-inconsistent-biased/). arXiv preprint arXiv:2405.01724.
21. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
22. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2407.19594. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Xian+Li%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+Weston%2E+2024%2E+Self%2Drewarding+language+models%2E+arXiv+preprint+arXiv%3A2407%2E19594%2E)
23. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685.
24. Wanjun Zhong, Tao Yu, and Danqi Chen. 2020. Qags: Evaluating factual consistency with qa-based metrics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 519–529. [find](https://scholar.google.com/scholar?q=Wanjun+Zhong%2C+Tao+Yu%2C+and+Danqi+Chen%2E+2020%2E+Qags%3A+Evaluating+factual+consistency+with+qa%2Dbased+metrics%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP)
25. Weizhi Zhou, Jie Gao, Yangqiu Song, and Tong Zhang. 2023. Promptbench: Towards evaluating the robustness of large language models with respect to prompting strategies. arXiv preprint arXiv:2310.04455. — [arXiv:2310.04455](https://arxiv.org/abs/2310.04455) [find](https://scholar.google.com/scholar?q=Weizhi+Zhou%2C+Jie+Gao%2C+Yangqiu+Song%2C+and+Tong+Zhang%2E+2023%2E+Promptbench%3A+Towards+evaluating+the+robustness+of+large+language+models+with+respect+to+prompting+strategies%2E+arXiv+preprint+arXiv%3A2310%2E04455%2E)
26. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, ab…
27. Peiyi Wang, Lei Li, Liang Chen, et al. 2023. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+et+al%2E+2023%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2E)
28. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024. Helpsteer2: Open-source dataset for training top-performing reward models. Preprint, arXiv:2406.08673. [find](https://scholar.google.com/scholar?q=Zhilin+Wang%2C+Yi+Dong%2C+Olivier+Delalleau%2C+Jiaqi+Zeng%2C+Gerald+Shen%2C+Daniel+Egert%2C+Jimmy+J%2E+Zhang%2C+Makesh+Narsimhan+Sreedhar%2C+and+Oleksii+Kuchaiev%2E+2024%2E+Helpsteer2%3A+Open%2Dsource+dataset+for+training+top%2D)
29. Yizhou Xu, Fan Yang, Yuntao Bai, Xiang Gao, and Xiang Ren. 2023. [Steering llms to produce more helpful dialogues via self-alignment](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). arXiv preprint arXiv:2305.14325.

*Updated 31 October 2026*