---
title: "Large Language Models are Inconsistent and Biased Evaluators"
description: "The authors examine two properties of these judges, bias and consistency, in summary evaluation. 45 references, cited by 76 papers here."
authors: ["Rickard Stureborg", "Dimitris Alikaniotis", "Yoshi Suhara"]
year: 2024
venue: "arXiv:2405.01724"
source: "https://arxiv.org/abs/2405.01724"
doi: "10.48550/arxiv.2405.01724"
arxiv: "2405.01724"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/stureborg-2024-large-language-models-inconsistent-biased/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/stureborg-2024-large-language-models-inconsistent-biased/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Large Language Models are Inconsistent and Biased Evaluators

# Large Language Models are Inconsistent and Biased Evaluators

**Authors:** Rickard Stureborg, [Dimitris Alikaniotis](https://md.research.elamur.ai/free/en/authors/dimitris-alikaniotis/) and Yoshi Suhara · **Year:** 2024 · **Venue:** arXiv:2405.01724 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.01724) · [PDF](https://arxiv.org/pdf/2405.01724) · [DOI 10.48550/arxiv.2405.01724](https://doi.org/10.48550/arxiv.2405.01724) · [arXiv 2405.01724](https://arxiv.org/abs/2405.01724) · [OpenAlex](https://openalex.org/W4396777837)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [76](https://md.research.elamur.ai/free/en/papers/stureborg-2024-large-language-models-inconsistent-biased/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
LLM-based judges such as GPT-4 are now widely used to score generated text without references. However, earlier work mostly tuned them to match human ratings and paid little attention to how reliable they are. The authors examine two properties of these judges, bias and consistency, in summary evaluation. They use SummEval as a development set and RoSE as a held-out test set. Across more than 560,000 generated outputs from GPT-3.5 and GPT-4, they find three biases. The models favour low-perplexity text (familiarity bias), they distribute scores unevenly and favour round values, and their judgments on one attribute are anchored by ratings they have already given for other attributes. The judges are also unstable: repeated samples agree with each other less than human annotators do, and the models react to prompt changes that a human would consider irrelevant. From these findings the authors derive a set of configuration recipes. Their resulting evaluator significantly beats the previous state of the art on part of RoSE. The work highlights hidden risks of relying on LLM-as-a-judge metrics.

## Key points
- **Data and setup:**
  - SummEval provides expert scores for 16 system summaries of each of 100 CNN/Daily Mail articles. It covers Coherence, Consistency, Fluency and Relevance, and agreement with experts is measured with Kendall's τ.
  - RoSE spans CNN/Daily Mail, SAMSum and XSum, with recall-style annotations based on Atomic Content Units (ACU).
- **Models:**
  - The analysis uses gpt-3.5-turbo-0301 and gpt-4-0613.
  - The final RoSE comparison runs on GPT-4-Turbo against the authors' own re-implementation of G-Eval.
  - Perplexity is computed with text-davinci-003.
- **Familiarity bias:** Mean summary perplexity for GPT-4 Coherence ratings falls sharply as scores rise, from about 8.15 at score 2 to 5.34 at score 5. Expert ratings span a much narrower range (7.03 to 6.36). GPT-4's ratings also correlate negatively with source-document perplexity, while expert ratings show no such link.
- **Score granularity:** The authors compare several rating formats: 1–5, 1–5 with "strong/weak" word modifiers, GPA-style float values, 1–10, and 1–100. They also test averaging over multiple samples. LLMs often fail to use these finer scales as instructed.
- **Result and limitations:**
  - The recipe-based evaluator significantly outperforms the SOTA on the CNNDM partition of RoSE at 90% confidence. Gains on the other two domains are positive but not significant.
  - Only proprietary OpenAI models were tested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Duke University](https://md.research.elamur.ai/free/en/universities/duke-university/) — University, Durham, United States · 18 works · 14,944 citations
- [Nvidia](https://md.research.elamur.ai/free/en/companies/nvidia/) — Company, Santa Clara, United States · 16 works · 4,820 citations
- [Nvidia (United Kingdom)](https://md.research.elamur.ai/free/en/companies/nvidia-united-kingdom/) — Company, Reading, United Kingdom · 13 works · 4,550 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Rickard Stureborg, Dimitris Alikaniotis and Yoshi Suhara. 2024. Large Language Models are Inconsistent and Biased Evaluators. arXiv:2405.01724.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{stureborg2024large,
  title = {{Large Language Models are Inconsistent and Biased Evaluators}},
  author = {Rickard Stureborg, Dimitris Alikaniotis and Yoshi Suhara},
  year = {2024},
  doi = {10.48550/arxiv.2405.01724},
  eprint = {2405.01724},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.01724}
}
```

- [← previous FineSurE: Fine-grained Summarization Evaluation using LLMs](https://md.research.elamur.ai/free/en/papers/song-2024-finesure-fine-grained-summarization-evaluation/)
- [next → LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://md.research.elamur.ai/free/en/papers/wu-2024-longmemeval-benchmarking-chat-assistants-long/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

45 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E)
2. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
3. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. [Re-evaluating evaluation in text summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguist…
4. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. [find](https://scholar.google.com/scholar?q=Mingqi+Gao%2C+Jie+Ruan%2C+Renliang+Sun%2C+Xunjian+Yin%2C+Shiping+Yang%2C+and+Xiaojun+Wan%2E+2023%2E+Human%2Dlike+summarization+evaluation+with+chatgpt%2E)
5. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bogdan+Gliwa%2C+Iwona+Mochol%2C+Maciej+Biesek%2C+and+Aleksander+Wawer%2E+2019%2E+SAMSum+corpus%3A+A+human%2Dannotated+dialogue+dataset+for+abstractive+summarization%2E+In+Proceedings+of+the+2nd+Workshop+on+New+Fronti)
6. Hidehito Honda, Rina Kagawa, and Masaru Shirasuna. 2022. On the round number bias and wisdom of crowds in different response formats for numerical estimation. Scientific Reports, 12(1):8167. [find](https://scholar.google.com/scholar?q=Hidehito+Honda%2C+Rina+Kagawa%2C+and+Masaru+Shirasuna%2E+2022%2E+On+the+round+number+bias+and+wisdom+of+crowds+in+different+response+formats+for+numerical+estimation%2E+Scientific+Reports%2C+12%281%29%3A8167%2E)
7. Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. 2024. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers. [find](https://scholar.google.com/scholar?q=Hui+Huang%2C+Yingqi+Qu%2C+Jing+Liu%2C+Muyun+Yang%2C+and+Tiejun+Zhao%2E+2024%2E+An+empirical+study+of+llm%2Das%2Da%2Djudge+for+llm+evaluation%3A+Fine%2Dtuned+judge+models+are+task%2Dspecific+classifiers%2E)
8. Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+Nogueira+dos+santos%2C+Caglar+Gulcehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E)
9. Zdeněk Kasner and Ondřej Dušek. 2024. Beyond reference-based metrics: Analyzing behaviors of open llms on data-to-text generation. [find](https://scholar.google.com/scholar?q=Zdene%CC%8Ck+Kasner+and+Ondr%CC%8Cej+Du%C5%A1ek%2E+2024%2E+Beyond+reference%2Dbased+metrics%3A+Analyzing+behaviors+of+open+llms+on+data%2Dto%2Dtext+generation%2E)
10. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E)
11. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E)
12. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. [find](https://scholar.google.com/scholar?q=Ryan+Koo%2C+Minhwa+Lee%2C+Vipul+Raheja%2C+Jong+Inn+Park%2C+Zae+Myung+Kim%2C+and+Dongyeop+Kang%2E+2023%2E+Benchmarking+cognitive+biases+in+large+language+models+as+evaluators%2E)
13. Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, and Chongyang Tao. 2024. Leveraging large language models for nlg evaluation: A survey. [find](https://scholar.google.com/scholar?q=Zhen+Li%2C+Xiaohan+Xu%2C+Tao+Shen%2C+Can+Xu%2C+Jia%2DChen+Gu%2C+and+Chongyang+Tao%2E+2024%2E+Leveraging+large+language+models+for+nlg+evaluation%3A+A+survey%2E)
14. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
15. Yen-Ting Lin and Yun-Nung Chen. 2023. [Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models](https://md.research.elamur.ai/free/en/papers/lin-2023-llm-eval-unified-multi-dimensional/).
16. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023a. G-eval: Nlg evaluation using gpt-4 with better human alignment. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023a%2E+G%2Deval%3A+Nlg+evaluation+using+gpt%2D4+with+better+human+alignment%2E)
17. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: Nlg evaluation using gpt-4 with better human alignment. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023b%2E+G%2Deval%3A+Nlg+evaluation+using+gpt%2D4+with+better+human+alignment%2E)
18. Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. 2023c. LLMs as narcissistic evaluators: When ego inflates evaluation scores. [find](https://scholar.google.com/scholar?q=Yiqi+Liu%2C+Nafise+Sadat+Moosavi%2C+and+Chenghua+Lin%2E+2023c%2E+LLMs+as+narcissistic+evaluators%3A+When+ego+inflates+evaluation+scores%2E)
19. Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023d. [Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/). In Proceedings of the 61st Annual Meeting of the…
20. Annie Louis and Ani Nenkova. 2013. [Automatically assessing machine summary content without a gold standard](https://md.research.elamur.ai/free/en/papers/louis-2013-automatically-assessing-machine-summary-content/). Computational Linguistics, 39(2):267– 300.
21. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. [Chatgpt as a factual inconsistency evaluator for text summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/).
22. Tomoko Nemoto and David Beglar. 2014. Likert-scale questionnaires. In JALT 2013 conference proceedings, pages 1–8. [find](https://scholar.google.com/scholar?q=Tomoko+Nemoto+and+David+Beglar%2E+2014%2E+Likert%2Dscale+questionnaires%2E+In+JALT+2013+conference+proceedings%2C+pages+1%E2%80%938%2E)
23. Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, and Naoaki Okazaki. 2024. Likelihood-based mitigation of evaluation bias in large language models. [find](https://scholar.google.com/scholar?q=Masanari+Ohi%2C+Masahiro+Kaneko%2C+Ryuto+Koike%2C+Mengsay+Loem%2C+and+Naoaki+Okazaki%2E+2024%2E+Likelihood%2Dbased+mitigation+of+evaluation+bias+in+large+language+models%2E)
24. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. [find](https://scholar.google.com/scholar?q=Arjun+Panickssery%2C+Samuel+R%2E+Bowman%2C+and+Shi+Feng%2E+2024%2E+Llm+evaluators+recognize+and+favor+their+own+generations%2E)
25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
26. Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. [Are references really needed? unbabel-IST 2021 submission for the metrics shared task](https://md.research.elamur.ai/free/en/papers/rei-2021-are-references-really-needed-unbabel/). In Proceedings of the Sixth Conference on Machine Translation,…
27. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. [QuestEval: Summarization asks for fact-based evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594–6604, Online…
28. Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. [Answers unite! unsupervised metrics for reinforced summarization models](https://md.research.elamur.ai/free/en/papers/scialom-2019-answers-unite-unsupervised-metrics-reinforced/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La…
29. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
30. Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. [Large language models are not yet human-level evaluators for abstractive summarization](https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-models-are-not/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4215–4233, Singapore. Association for Computational Li…
31. Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Characterizing the confidence of large language model-based automatic evaluation metrics. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 76… [find](https://scholar.google.com/scholar?q=Rickard+Stureborg%2C+Dimitris+Alikaniotis%2C+and+Yoshi+Suhara%2E+2024%2E+Characterizing+the+confidence+of+large+language+model%2Dbased+automatic+evaluation+metrics%2E+In+Proceedings+of+the+18th+Conference+of+the+)
32. Manoj Thomas and Vicki Morwitz. 2009. Heuristics in numerical cognition: Implications for pricing. In Handbook of pricing research in marketing, pages 132–149. Edward Elgar Publishing. [find](https://scholar.google.com/scholar?q=Manoj+Thomas+and+Vicki+Morwitz%2E+2009%2E+Heuristics+in+numerical+cognition%3A+Implications+for+pricing%2E+In+Handbook+of+pricing+research+in+marketing%2C+pages+132%E2%80%93149%2E+Edward+Elgar+Publishing%2E)
33. Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124–1131. [find](https://scholar.google.com/scholar?q=Amos+Tversky+and+Daniel+Kahneman%2E+1974%2E+Judgment+under+uncertainty%3A+Heuristics+and+biases%2E+Science%2C+185%284157%29%3A1124%E2%80%931131%2E)
34. Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. 2020. [Fill in the BLANC: Human-free quality estimation of document summaries](https://md.research.elamur.ai/free/en/papers/vasilyev-2020-fill-blanc-human-free-quality/). In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 11–20, Online. Association for Computational Linguistics.
35. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/).
36. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Zefan+Cai%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2023b%2E+Large+language+models+are+not+fair+evaluators%2E)
37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
38. Minghao Wu and Alham Fikri Aji. 2023. [Style over substance: Evaluation biases for large language models](https://md.research.elamur.ai/free/en/papers/wu-2025-style-over-substance-evaluation-biases/).
39. Yunshu Wu, Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani, and Estevam Hruschka. 2023. Less is more for long document summary evaluation by llms. [find](https://scholar.google.com/scholar?q=Yunshu+Wu%2C+Hayate+Iso%2C+Pouya+Pezeshkpour%2C+Nikita+Bhutani%2C+and+Estevam+Hruschka%2E+2023%2E+Less+is+more+for+long+document+summary+evaluation+by+llms%2E)
40. Robert B Zajonc. 1968. Attitudinal effects of mere exposure. Journal of personality and social psychology, 9(2p2):1. [find](https://scholar.google.com/scholar?q=Robert+B+Zajonc%2E+1968%2E+Attitudinal+effects+of+mere+exposure%2E+Journal+of+personality+and+social+psychology%2C+9%282p2%29%3A1%2E)
41. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BertScore: Evaluating text generation with bert. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BertScore%3A+Evaluating+text+generation+with+bert%2E)
42. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. [MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International…
43. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023a. Large language models are not robust multiple choice selectors. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+2023a%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E)
44. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric%2E+P+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2023b%2E+Judging+llm%2Da)
45. Chloe Qinyu Zhu, Rickard Stureborg, and Bhuwan Dhingra. 2024. Hierarchical multi-label classification of online vaccine concerns. [find](https://scholar.google.com/scholar?q=Chloe+Qinyu+Zhu%2C+Rickard+Stureborg%2C+and+Bhuwan+Dhingra%2E+2024%2E+Hierarchical+multi%2Dlabel+classification+of+online+vaccine+concerns%2E)

*Updated 24 October 2026*