---
title: "Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to"
description: "The Cohere authors test 11 judge models from five families (Llama3, Claude3, GPT4, Command R… 34 references, cited by 9 papers here."
authors: ["Hongyu Chen", "Seraphina Goldfarb-Tarrant"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.970/"
doi: "10.18653/v1/2025.acl-long.970"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-safer-or-luckier-llms-as/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-safer-or-luckier-llms-as/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts

# Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts

**Authors:** [Hongyu Chen](https://md.research.elamur.ai/free/en/authors/hongyu-chen/) and [Seraphina Goldfarb-Tarrant](https://md.research.elamur.ai/free/en/authors/seraphina-goldfarb-tarrant/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.970/) · [PDF](https://aclanthology.org/2025.acl-long.970.pdf) · [DOI 10.18653/v1/2025.acl-long.970](https://doi.org/10.18653/v1/2025.acl-long.970) · [OpenAlex](https://openalex.org/W4412944925)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/chen-2025-safer-or-luckier-llms-as/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Many teams let an LLM judge decide which of two models gives safer output, but this practice has rarely been stress-tested for safety. The Cohere authors test 11 judge models from five families (Llama3, Claude3, GPT4, Command R, Mistral), taking one larger and one smaller model from each family. Their data are 576 prompts written by trained safety annotators across five areas: Misinformation, CSAM, Toxicity, Sexually Explicit and Self-harm. These prompts yield 2303 completion pairs, each labelled by three humans. Each judge is checked for three things: stability across repeated runs, agreement with humans, and sensitivity to five surface artifacts. Two artifacts come from earlier work (position, verbosity), and three are new and safety-specific (apology, authority, halo). Every judge proved vulnerable. Injected artifacts shift individual verdicts and model-level win rates, sometimes enough to change which system appears safer. Bigger judges were not reliably more robust, and strong human agreement did not predict artifact resistance. Panels of jurors picked with artifact behaviour in mind performed best, but bias remained. The authors conclude that single-judge safety comparisons can produce illusory differences.

## Key points
- Judges evaluated: llama3-70B-chat, llama3-8b-chat, command-r-plus, command-r, mistral-large, mistral-8x7b, claude-3-sonnet, claude-3-haiku, gpt-4-1106-preview, gpt-4o-2024-05-13 and gpt-4o-mini-2024-07-18.
- Dataset: 576 human-written prompts and 4606 completions forming 2303 pairs, each triply annotated for safety and quality.
- Artifacts tested:
  - **Apology:** a refusal-style apology is prepended to the completion.
  - **Authority:** a fake citation is added.
  - **Halo:** a closing offer of further help is appended.
  - **Verbosity:** the completion is rewritten to be longer with the same meaning, using Command R.
  - **Position:** the order of the two completions is swapped.
- Severity of effects:
  - The abstract reports that apologetic wording alone can skew judge preferences by up to 98%.
  - The conclusion names apology and position as the most severe biases.
  - Most judges handled verbosity well.
  - Completions with authority-style citations tended to be dispreferred.
- Robustness and human agreement are separate dimensions, and judges can be inconsistent across repeated runs. Juries help but do not eliminate artifact sensitivity. The authors suggest asking narrower questions (e.g., whether a completion contains misinformation) and trying chain-of-thought prompting.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: Hongyu Chen and Seraphina Goldfarb-Tarrant. 2025. Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2025safer,
  title = {{Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts}},
  author = {Hongyu Chen and Seraphina Goldfarb-Tarrant},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.970},
  url = {https://doi.org/10.18653/v1/2025.acl-long.970}
}
```

- [← previous QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/wan-2025-qwenlong-l1-long-context/)
- [next → Safety Alignment via Constrained Knowledge Unlearning](https://md.research.elamur.ai/free/en/papers/shi-2025-safety-alignment-constrained-knowledge-unlearning/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

34 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. 2024. [The multilingual alignment prism: Aligning global and local preferences to reduce harm](https://md.research.elamur.ai/free/en/papers/aakanksha-2024-multilingual-alignment-prism-aligning-global/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing…
2. Bhashithe Abeysinghe and Ruhan Circi. 2024. The challenges of evaluating llm applications: An analysis of automated, human, and llm-based approaches. arXiv preprint arXiv:2406.03339. — [arXiv:2406.03339](https://arxiv.org/abs/2406.03339) [find](https://scholar.google.com/scholar?q=Bhashithe+Abeysinghe+and+Ruhan+Circi%2E+2024%2E+The+challenges+of+evaluating+llm+applications%3A+An+analysis+of+automated%2C+human%2C+and+llm%2Dbased+approaches%2E+arXiv+preprint+arXiv%3A2406%2E03339%2E+%E2%80%94+%5BarXiv%3A2406%2E033)
3. ActiveFence. Protecting Children from Online Grooming. AI@Meta. 2024. Llama 3 model card. [find](https://scholar.google.com/scholar?q=ActiveFence%2E+Protecting+Children+from+Online+Grooming%2E+AI%40Meta%2E+2024%2E+Llama+3+model+card%2E)
4. Anthropic. The claude 3 model family: Opus, sonnet, haiku. [find](https://scholar.google.com/scholar?q=Anthropic%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E)
5. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, Melbourne, Australia. Association for Co… [find](https://scholar.google.com/scholar?q=Arun+Chaganty%2C+Stephen+Mussmann%2C+and+Percy+Liang%2E+2018%2E+The+price+of+debiasing+automatic+metrics+in+natural+language+evalaution%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Comput)
6. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. [Humans or LLMs as the judge? a study on judgement bias](https://md.research.elamur.ai/free/en/papers/chen-2024-humans-or-llms-as-judge/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327, Miami, Florida, USA. Association for Computational Lingu…
7. Cohere. 2024a. Command-r. [https://docs.cohere.com/v2/docs/command-r](https://docs.cohere.com/v2/docs/command-r). [↗](https://docs.cohere.com/v2/docs/command-r)
8. Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, et al. 2025. Command a: An enterprise-ready large language model. arXiv preprint arXiv:2504.00698. — [arXiv:2504.00698](https://arxiv.org/abs/2504.00698) [find](https://scholar.google.com/scholar?q=Team+Cohere%2C+Arash+Ahmadian%2C+Marwan+Ahmed%2C+Jay+Alammar%2C+Milad+Alizadeh%2C+Yazeed+Alnumay%2C+Sophia+Althammer%2C+Arkady+Arkhangorodsky%2C+Viraat+Aryabumi%2C+Dennis+Aumiller%2C+et+al%2E+2025%2E+Command+a%3A+An+enterprise)
9. Aparna Elangovan, Ling Liu, Lei Xu, Sravan Babu Bodapati, and Dan Roth. 2024. ConSiDERS-the-human evaluation framework: Rethinking human evaluation for generative large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)… [find](https://scholar.google.com/scholar?q=Aparna+Elangovan%2C+Ling+Liu%2C+Lei+Xu%2C+Sravan+Babu+Bodapati%2C+and+Dan+Roth%2E+2024%2E+ConSiDERS%2Dthe%2Dhuman+evaluation+framework%3A+Rethinking+human+evaluation+for+generative+large+language+models%2E+In+Proceedings)
10. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673. [find](https://scholar.google.com/scholar?q=Robert+Geirhos%2C+J%C3%B6rn%2DHenrik+Jacobsen%2C+Claudio+Michaelis%2C+Richard+Zemel%2C+Wieland+Brendel%2C+Matthias+Bethge%2C+and+Felix+A+Wichmann%2E+2020%2E+Shortcut+learning+in+deep+neural+networks%2E+Nature+Machine+Intellig)
11. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
12. Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. 2023. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925, 10. [find](https://scholar.google.com/scholar?q=Timo+Kaufmann%2C+Paul+Weng%2C+Viktor+Bengs%2C+and+Eyke+H%C3%BCllermeier%2E+2023%2E+A+survey+of+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv%3A2312%2E14925%2C+10%2E)
13. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics: ACL 2024, pages 517–545, Bangkok, Thailand. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Ryan+Koo%2C+Minhwa+Lee%2C+Vipul+Raheja%2C+Jong+Inn+Park%2C+Zae+Myung+Kim%2C+and+Dongyeop+Kang%2E+2024%2E+Benchmarking+cognitive+biases+in+large+language+models+as+evaluators%2E+In+Findings+of+the+Association+for+Comp)
14. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2025. RewardBench: Evaluating reward models for language modeling. In Findings of the Association for Computational… [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+Noah+A%2E+Smith%2C+and+Hannaneh+Hajishirzi%2E+2025%2E+RewardBen)
15. Ruosen Li, Teerth Patel, and Xinya Du. 2024. Prd: Peer rank and discussion improve large language model based evaluations. Preprint, arXiv:2307.02762. [find](https://scholar.google.com/scholar?q=Ruosen+Li%2C+Teerth+Patel%2C+and+Xinya+Du%2E+2024%2E+Prd%3A+Peer+rank+and+discussion+improve+large+language+model+based+evaluations%2E+Preprint%2C+arXiv%3A2307%2E02762%2E)
16. Adian Liusie, Potsawee Manakul, and Mark Gales. 2024. [LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models](https://md.research.elamur.ai/free/en/papers/liusie-2024-llm-comparative-assessment-zero-shot/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers),…
17. Mistral AI. 2024. Au large. [https://mistral.ai/](https://mistral.ai/) en/news/mistral-large. [↗](https://mistral.ai/)
18. Juan Luis Nicolau, Juan Pedro Mellinas, and Eva Martín-Fuentes. 2020. The halo effect: A longitudinal approach. Annals of Tourism Research, 83:102938. [find](https://scholar.google.com/scholar?q=Juan+Luis+Nicolau%2C+Juan+Pedro+Mellinas%2C+and+Eva+Mart%C3%ADn%2DFuentes%2E+2020%2E+The+halo+effect%3A+A+longitudinal+approach%2E+Annals+of+Tourism+Research%2C+83%3A102938%2E)
19. OpenAI. 2024a. Gpt-4 turbo and gpt-4. [https://platform.openai.com/docs/models\#](https://platform.openai.com/docs/models\#) gpt-4-turbo-and-gpt-4. [↗](https://platform.openai.com/docs/models\#)
20. OpenAI. 2024b. Gpt-4o. [https://platform.openai.com/docs/models/gpt-4o](https://platform.openai.com/docs/models/gpt-4o). [↗](https://platform.openai.com/docs/models/gpt-4o)
21. Vyas Raina, Adian Liusie, and Mark Gales. 2024. [Is LLM-as-a-judge robust? investigating universal adversarial attacks on zero-shot LLM assessment](https://md.research.elamur.ai/free/en/papers/raina-2024-is-llm-as-judge-robust/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7499–7517, Miami, Florida, USA. Association for Computat…
22. Yuqi Ren and Deyi Xiong. 2023. HuaSLIM: Human attention motivated shortcut learning identification and mitigation for large language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12350–12365, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yuqi+Ren+and+Deyi+Xiong%2E+2023%2E+HuaSLIM%3A+Human+attention+motivated+shortcut+learning+identification+and+mitigation+for+large+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistic)
23. Rui Song, Yingji Li, Lida Shi, Fausto Giunchiglia, and Hao Xu. 2024. Shortcut learning in in-context learning: A survey. arXiv preprint arXiv:2411.02018. — [arXiv:2411.02018](https://arxiv.org/abs/2411.02018) [find](https://scholar.google.com/scholar?q=Rui+Song%2C+Yingji+Li%2C+Lida+Shi%2C+Fausto+Giunchiglia%2C+and+Hao+Xu%2E+2024%2E+Shortcut+learning+in+in%2Dcontext+learning%3A+A+survey%2E+arXiv+preprint+arXiv%3A2411%2E02018%2E+%E2%80%94+%5BarXiv%3A2411%2E02018%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F241)
24. Ruixiang Tang, Dehan Kong, Longtao Huang, and Hui Xue. 2023. Large language models can be lazy learners: Analyze shortcuts in in-context learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4645–4657, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ruixiang+Tang%2C+Dehan+Kong%2C+Longtao+Huang%2C+and+Hui+Xue%2E+2023%2E+Large+language+models+can+be+lazy+learners%3A+Analyze+shortcuts+in+in%2Dcontext+learning%2E+In+Findings+of+the+Association+for+Computational+Ling)
25. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. arXiv preprint arXiv:2406.12624. [find](https://scholar.google.com/scholar?q=Aman+Singh+Thakur%2C+Kartik+Choudhary%2C+Venkat+Srinik+Ramayapally%2C+Sankaran+Vaidyanathan%2C+and+Dieuwke+Hupkes%2E+2024%2E+Judging+the+judges%3A+Evaluating+alignment+and+vulnerabilities+in+llms%2Das%2Djudges%2E+arXiv+p)
26. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. [Replacing judges with juries: Evaluating llm generations with a panel of diverse models](https://md.research.elamur.ai/free/en/papers/verga-2024-replacing-judges-juries-evaluating-llm/). Preprint, arXiv:2404.18796.
27. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Zefan+Cai%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Lingpeng+Kong%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2024%2E+Large+language+models+are+not+fair+evaluators%2E+In+Proceedings+o)
28. Minghao Wu and Alham Fikri Aji. 2023. [Style over substance: Evaluation biases for large language models](https://md.research.elamur.ai/free/en/papers/wu-2025-style-over-substance-evaluation-biases/). Preprint, arXiv:2307.03025.
29. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. 2024. Sorry-bench: Systematically evaluating large language model safety refusal behaviors.… [find](https://scholar.google.com/scholar?q=Tinghao+Xie%2C+Xiangyu+Qi%2C+Yi+Zeng%2C+Yangsibo+Huang%2C+Udari+Madhushani+Sehwag%2C+Kaixuan+Huang%2C+Luxi+He%2C+Boyi+Wei%2C+Dacheng+Li%2C+Ying+Sheng%2C+Ruoxi+Jia%2C+Bo+Li%2C+Kai+Li%2C+Danqi+Chen%2C+Peter+Henderson%2C+and+Prateek+)
30. Guangxuan Xu, Kai Xu, Shivchander Sudalairaj, Hao Wang, and Akash Srivastava. 2024. Fantastic llms for preference data annotation and how to (not) find them. arXiv e-prints, pages arXiv–2411. [find](https://scholar.google.com/scholar?q=Guangxuan+Xu%2C+Kai+Xu%2C+Shivchander+Sudalairaj%2C+Hao+Wang%2C+and+Akash+Srivastava%2E+2024%2E+Fantastic+llms+for+preference+data+annotation+and+how+to+%28not%29+find+them%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932411%2E)
31. Yi Zeng, Yu Yang, Andy Zhou, Jeffrey Ziwei Tan, Yuheng Tu, Yifan Mai, Kevin Klyman, Minzhou Pan, Ruoxi Jia, Dawn Song, Percy Liang, and Bo Li. 2024a. [find](https://scholar.google.com/scholar?q=Yi+Zeng%2C+Yu+Yang%2C+Andy+Zhou%2C+Jeffrey+Ziwei+Tan%2C+Yuheng+Tu%2C+Yifan+Mai%2C+Kevin+Klyman%2C+Minzhou+Pan%2C+Ruoxi+Jia%2C+Dawn+Song%2C+Percy+Liang%2C+and+Bo+Li%2E+2024a%2E)
32. Air-bench 2024: A safety benchmark based on risk categories from regulations and policies. Preprint, arXiv:2407.17436. [find](https://scholar.google.com/scholar?q=Air%2Dbench+2024%3A+A+safety+benchmark+based+on+risk+categories+from+regulations+and+policies%2E+Preprint%2C+arXiv%3A2407%2E17436%2E)
33. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024b. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Jiatong+Yu%2C+Tianyu+Gao%2C+Yu+Meng%2C+Tanya+Goyal%2C+and+Danqi+Chen%2E+2024b%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+In+The+Twelfth+International+Conference+on+Lear)
34. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Thirty-seventh Conference on Neural Information Processing…

*Updated 31 October 2026*