---
title: "ChatEval: Towards Better LLM-based Evaluators — Chan 2023"
description: "Judging the quality of generated text is costly when done by people."
authors: ["Chi-Min Chan", "Weize Chen", "Yusheng Su", "Jianxuan Yu", "Wei Xue", "Shanghang Zhang", "Jie Fu", "Zhiyuan Liu"]
year: 2023
venue: "arXiv:2308.07201"
source: "https://arxiv.org/abs/2308.07201"
doi: "10.48550/arxiv.2308.07201"
arxiv: "2308.07201"
licence: "arXiv non-exclusive distribution license"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

# ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

**Authors:** [Chi-Min Chan](https://md.research.elamur.ai/free/en/authors/chi-min-chan/), [Weize Chen](https://md.research.elamur.ai/free/en/authors/weize-chen/), [Yusheng Su](https://md.research.elamur.ai/free/en/authors/yusheng-su/), Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, [Zhiyuan Liu](https://md.research.elamur.ai/free/en/authors/zhiyuan-liu/) · **Year:** 2023 · **Venue:** arXiv:2308.07201 · **Licence:** [arXiv non-exclusive distribution license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.07201) · [PDF](https://arxiv.org/pdf/2308.07201) · [DOI 10.48550/arxiv.2308.07201](https://doi.org/10.48550/arxiv.2308.07201) · [arXiv 2308.07201](https://arxiv.org/abs/2308.07201) · [OpenAlex](https://openalex.org/W4385849309)

**Citations:** 54 citations (openalex-snapshot · 2026-10-14) · [297](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Judging the quality of generated text is costly when done by people. N-gram metrics such as BLEU, Rouge and METEOR are cheap but align poorly with human opinion on open-ended outputs. Using a single LLM as a judge narrows this gap but does not yet reach human-level quality. ChatEval responds by turning evaluation into a group discussion. Several LLM agents form a referee panel and debate candidate responses before giving a verdict, mirroring the common practice of having multiple annotators work together. Each agent gets its own persona, and the authors test three schemes for sharing discussion history. Rather than forcing consensus, final labels are obtained by majority vote or averaged scores. The panels were built from GPT-4 or ChatGPT and tested on FairEval (open-ended QA) and Topical-Chat (dialogue response generation). In both cases, the debating panels matched human judgments more closely than single-agent evaluators, in both accuracy and correlation. Assigning every agent the same role lowered performance, so persona diversity is essential. The released framework offers a scalable, more human-like route to automatic evaluation.

## Key points
- Benchmarks: FairEval uses 80 open-ended questions from Chiang et al. (2023), comparing Vicuna-13B and ChatGPT answers. Its human labels come from Wu et al. (2023), with three annotators and a majority vote. Topical-Chat covers dialogue response generation.
- Agents are homogeneous groups of either GPT-4 or ChatGPT (GPT-3.5-turbo), run at temperature 0. Mixed-model groups are left for future work.
- The three communication strategies are:
  - One-By-One: agents speak in a fixed order.
  - Simultaneous-Talk: agents respond in parallel, which removes order effects.
  - Simultaneous-Talk-with-Summarizer: an extra LLM condenses each round into the shared history.
- Role prompts are adapted from Wu et al. (2023): General Public, Critic, News Author, Psychologist and Scientist. Using identical role descriptions for all agents degrades results.
- Code is available at github.com/chanchimin/ChatEval and is built on the AgentVerse library.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

297 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 278 not yet.

- [AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data](https://md.research.elamur.ai/free/en/papers/alshaikh-2025-aratable-benchmarking-llms-reasoning-understanding/)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://md.research.elamur.ai/free/en/papers/chen-2023-agentverse-facilitating-multi-agent/)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://md.research.elamur.ai/free/en/papers/chen-2024-reconcile-round-table-conference-improves/)
- [M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/feng-2025-m-mad-multidimensional-multi-agent/)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://md.research.elamur.ai/free/en/papers/guo-2024-large-language-model-based/)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://md.research.elamur.ai/free/en/papers/kamoi-2024-when-can-llms-actually-correct/)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/)
- [MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/kwan-2024-mt-eval-multi-turn-capabilities/)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://md.research.elamur.ai/free/en/papers/li-2024-llms-judges-comprehensive-survey-llm/)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://md.research.elamur.ai/free/en/papers/lin-2023-unlocking-spell-base-llms-rethinking/)
- [CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation](https://md.research.elamur.ai/free/en/papers/ma-2026-clase-hybrid-method-chinese-legalese/)
- [GAIA: a benchmark for General AI Assistants](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/)
- [ChatDev: Communicative Agents for Software Development](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://md.research.elamur.ai/free/en/papers/tseng-2024-two-tales-persona-llms-survey/)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-sotopia-interactive-evaluation-social-intelligence/)
- [SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space](https://md.research.elamur.ai/free/en/papers/zinkovich-2026-sparta-evaluating-reasoning-segmentation-robustness/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chan2023chateval,
  title = {{ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate}},
  author = {Chan, Chi-Min and Chen, Weize and Su, Yusheng and Yu, Jianxuan and Xue, Wei and Zhang, Shanghang and Fu, Jie and Liu, Zhiyuan},
  year = {2023},
  doi = {10.48550/arxiv.2308.07201},
  eprint = {2308.07201},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.07201}
}
```

- [← previous ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://md.research.elamur.ai/free/en/papers/li-2023-chatdoctor-medical-chat-model-fine/)
- [next → ChatGPT as a Factual Inconsistency Evaluator for Text Summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

46 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with language-model-as-an-examiner. arXiv preprint arXiv:2306.04181, 2023. [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Jiahao+Ying%2C+Yixin+Cao%2C+Xin+Lv%2C+Yuze+He%2C+Xiaozhi+Wang%2C+Jifan+Yu%2C+Kaisheng+Zeng%2C+Yijia+Xiao%2C+Haozhe+Lyu%2C+et+al%2E+Benchmarking+foundation+models+with+language%2Dmodel%2Das%2Dan%2Dexaminer%2E+arXiv+prepr)
2. Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72, 2005. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+Meteor%3A+An+automatic+metric+for+mt+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proceedings+of+the+acl+workshop+on+intrinsic+and+extrinsic+evaluatio) [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
4. Chris Callison-Burch. Fast, cheap, and creative: Evaluating translation quality using amazon’s mechanical turk. In Proceedings of the 2009 conference on empirical methods in natural language processing, pp. 286–295, 2009. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2E+Fast%2C+cheap%2C+and+creative%3A+Evaluating+translation+quality+using+amazon%E2%80%99s+mechanical+turk%2E+In+Proceedings+of+the+2009+conference+on+empirical+methods+in+natural+language+processin) [our card](https://md.research.elamur.ai/free/en/papers/burch-2009-fast-cheap-creative-evaluating-translation/)
5. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey. CoRR, abs/2006.14799, 2020. URL https://arxiv.org/abs/2006.14799. [↗](https://arxiv.org/abs/2006.14799)
6. Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+arXiv+preprint+arXiv%3A2305%2E01937%2C+2023%2E)
7. Acc. (%) Kap. 0.32 0.36 0.30 0.34 Acc. (%) 0.28 Kap. Kap. 0.32 0.26 0.30 0.24 0.28 3 Discussion Turns (b) Acc. and Kap. vs Discussion Turns Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-… [find](https://scholar.google.com/scholar?q=Acc%2E+%28%25%29+Kap%2E+0%2E32+0%2E36+0%2E30+0%2E34+Acc%2E+%28%25%29+0%2E28+Kap%2E+Kap%2E+0%2E32+0%2E26+0%2E30+0%2E24+0%2E28+3+Discussion+Turns+%28b%29+Acc%2E+and+Kap%2E+vs+Discussion+Turns+Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2C+2018%2E)
9. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. [find](https://scholar.google.com/scholar?q=Yilun+Du%2C+Shuang+Li%2C+Antonio+Torralba%2C+Joshua+B+Tenenbaum%2C+and+Igor+Mordatch%2E+Improving+factuality+and+reasoning+in+language+models+through+multiagent+debate%2E+arXiv+preprint+arXiv%3A2305%2E14325%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/)
10. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+Gptscore%3A+Evaluate+as+you+desire%2E+arXiv+preprint+arXiv%3A2302%2E04166%2C+2023%2E)
11. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554, 2023. [find](https://scholar.google.com/scholar?q=Mingqi+Gao%2C+Jie+Ruan%2C+Renliang+Sun%2C+Xunjian+Yin%2C+Shiping+Yang%2C+and+Xiaojun+Wan%2E+Human%2Dlike+summarization+evaluation+with+chatgpt%2E+arXiv+preprint+arXiv%3A2304%2E02554%2C+2023%2E)
12. Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anushree Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. Topical-chat: Towards knowledge-grounded open-domain conversations. 2019. [find](https://scholar.google.com/scholar?q=Karthik+Gopalakrishnan%2C+Behnam+Hedayatnia%2C+Qinlang+Chen%2C+Anna+Gottardi%2C+Sanjeev+Kwatra%2C+Anushree+Venkatesh%2C+Raefer+Gabriel%2C+and+Dilek+Hakkani%2DTu%CC%88r%2E+Topical%2Dchat%3A+Towards+knowledge%2Dgrounded+open%2Ddomain)
13. Marzena Karpinska, Nader Akoury, and Mohit Iyyer. The perils of using mechanical turk to evaluate open-ended text generation. arXiv preprint arXiv:2109.06835, 2021. [find](https://scholar.google.com/scholar?q=Marzena+Karpinska%2C+Nader+Akoury%2C+and+Mohit+Iyyer%2E+The+perils+of+using+mechanical+turk+to+evaluate+open%2Dended+text+generation%2E+arXiv+preprint+arXiv%3A2109%2E06835%2C+2021%2E)
14. Grzegorz Kondrak. N-gram similarity and distance. In International symposium on string processing and information retrieval, pp. 115–126. Springer, 2005. [find](https://scholar.google.com/scholar?q=Grzegorz+Kondrak%2E+N%2Dgram+similarity+and+distance%2E+In+International+symposium+on+string+processing+and+information+retrieval%2C+pp%2E+115%E2%80%93126%2E+Springer%2C+2005%2E)
15. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for” mind” exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023a. [find](https://scholar.google.com/scholar?q=Guohao+Li%2C+Hasan+Abed+Al+Kader+Hammoud%2C+Hani+Itani%2C+Dmitrii+Khizbullin%2C+and+Bernard+Ghanem%2E+Camel%3A+Communicative+agents+for%E2%80%9D+mind%E2%80%9D+exploration+of+large+scale+language+model+society%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/)
16. Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02762, 2023b. [find](https://scholar.google.com/scholar?q=Ruosen+Li%2C+Teerth+Patel%2C+and+Xinya+Du%2E+Prd%3A+Peer+rank+and+discussion+improve+large+language+model+based+evaluations%2E+arXiv+preprint+arXiv%3A2307%2E02762%2C+2023b%2E)
17. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118, 2023. [find](https://scholar.google.com/scholar?q=Tian+Liang%2C+Zhiwei+He%2C+Wenxiang+Jiao%2C+Xing+Wang%2C+Yan+Wang%2C+Rui+Wang%2C+Yujiu+Yang%2C+Zhaopeng+Tu%2C+and+Shuming+Shi%2E+Encouraging+divergent+thinking+in+large+language+models+through+multi%2Dagent+debate%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/liang-2023-encouraging-divergent-thinking-large-language/)
18. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74–81, 2004. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+summarization+branches+out%2C+pp%2E+74%E2%80%9381%2C+2004%2E)
19. Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush Vosoughi. Training socially aligned language models in simulated human society. arXiv preprint arXiv:2305.16960, 2023a. [find](https://scholar.google.com/scholar?q=Ruibo+Liu%2C+Ruixin+Yang%2C+Chenyan+Jia%2C+Ge+Zhang%2C+Denny+Zhou%2C+Andrew+M+Dai%2C+Diyi+Yang%2C+and+Soroush+Vosoughi%2E+Training+socially+aligned+language+models+in+simulated+human+society%2E+arXiv+preprint+arXiv%3A230)
20. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023b. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+Gpteval%3A+Nlg+evaluation+using+gpt%2D4+with+better+human+alignment%2E+arXiv+preprint+arXiv%3A2303%2E16634%2C+2023b%2E)
21. Andrea I Luppi, Pedro AM Mediano, Fernando E Rosas, Negin Holland, Tim D Fryer, John T O’Brien, James B Rowe, David K Menon, Daniel Bor, and Emmanuel A Stamatakis. A synergistic core for human brain evolution and cognition. Nature Neuroscience, 25(6):771–782, 2022. [find](https://scholar.google.com/scholar?q=Andrea+I+Luppi%2C+Pedro+AM+Mediano%2C+Fernando+E+Rosas%2C+Negin+Holland%2C+Tim+D+Fryer%2C+John+T+O%E2%80%99Brien%2C+James+B+Rowe%2C+David+K+Menon%2C+Daniel+Bor%2C+and+Emmanuel+A+Stamatakis%2E+A+synergistic+core+for+human+brain+e)
22. Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large language models. arXiv preprint arXiv:2307.04738, 2023. — [arXiv:2307.04738](https://arxiv.org/abs/2307.04738) [find](https://scholar.google.com/scholar?q=Zhao+Mandi%2C+Shreeya+Jain%2C+and+Shuran+Song%2E+Roco%3A+Dialectic+multi%2Drobot+collaboration+with+large+language+models%2E+arXiv+preprint+arXiv%3A2307%2E04738%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307%2E04738%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E)
23. Mary L McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282, 2012. [find](https://scholar.google.com/scholar?q=Mary+L+McHugh%2E+Interrater+reliability%3A+the+kappa+statistic%2E+Biochemia+medica%2C+22%283%29%3A276%E2%80%93282%2C+2012%2E)
24. Shikib Mehri and Maxine Eskenazi. Usr: An unsupervised and reference free evaluation metric for dialog generation. arXiv preprint arXiv:2005.00456, 2020. [find](https://scholar.google.com/scholar?q=Shikib+Mehri+and+Maxine+Eskenazi%2E+Usr%3A+An+unsupervised+and+reference+free+evaluation+metric+for+dialog+generation%2E+arXiv+preprint+arXiv%3A2005%2E00456%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/)
25. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2241–2252, Copenhagen, Denmark, September 2017. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Jekaterina+Novikova%2C+Ondr%CC%8Cej+Dus%CC%8Cek%2C+Amanda+Cercas+Curry%2C+and+Verena+Rieser%2E+Why+we+need+new+evaluation+metrics+for+NLG%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+)
26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+wi)
27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+annual+meeting+of+the+Association+for+Computatio) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
28. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+C+O%E2%80%99Brien%2C+Carrie+J+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S+Bernstein%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+arXiv+preprint+arXiv%3A2304%2E0344)
29. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023. [find](https://scholar.google.com/scholar?q=Chen+Qian%2C+Xin+Cong%2C+Cheng+Yang%2C+Weize+Chen%2C+Yusheng+Su%2C+Juyuan+Xu%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Communicative+agents+for+software+development%2E+arXiv+preprint+arXiv%3A2307%2E07924%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/)
30. Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover’s distance as a metric for image retrieval. International journal of computer vision, 40:99–121, 2000. [find](https://scholar.google.com/scholar?q=Yossi+Rubner%2C+Carlo+Tomasi%2C+and+Leonidas+J+Guibas%2E+The+earth+mover%E2%80%99s+distance+as+a+metric+for+image+retrieval%2E+International+journal+of+computer+vision%2C+40%3A99%E2%80%93121%2C+2000%2E)
31. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Albert+Webson%2C+Colin+Raffel%2C+Stephen+H+Bach%2C+Lintang+Sutawika%2C+Zaid+Alyafeai%2C+Antoine+Chaffin%2C+Arnaud+Stiegler%2C+Teven+Le+Scao%2C+Arun+Raja%2C+et+al%2E+Multitask+prompted+training+enables+zero%2Ds) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
32. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696, 2020. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+P+Parikh%2E+Bleurt%3A+Learning+robust+metrics+for+text+generation%2E+arXiv+preprint+arXiv%3A2004%2E04696%2C+2020%2E)
33. Chenhui Shen, Liying Cheng, Yang You, and Lidong Bing. Are large language models good evaluators for abstractive summarization? arXiv preprint arXiv:2305.13091, 2023. [find](https://scholar.google.com/scholar?q=Chenhui+Shen%2C+Liying+Cheng%2C+Yang+You%2C+and+Lidong+Bing%2E+Are+large+language+models+good+evaluators+for+abstractive+summarization%3F+arXiv+preprint+arXiv%3A2305%2E13091%2C+2023%2E)
34. Amanda Stent, Matthew Marge, and Mohit Singhai. Evaluating evaluation methods for generation in the presence of variation. In International conference on intelligent text processing and computational linguistics, pp. 341–351. Springer, 2005. [find](https://scholar.google.com/scholar?q=Amanda+Stent%2C+Matthew+Marge%2C+and+Mohit+Singhai%2E+Evaluating+evaluation+methods+for+generation+in+the+presence+of+variation%2E+In+International+conference+on+intelligent+text+processing+and+computational+)
35. Chris Van Der Lee, Albert Gatt, Emiel Van Miltenburg, Sander Wubben, and Emiel Krahmer. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pp. 355–368, 2019. [find](https://scholar.google.com/scholar?q=Chris+Van+Der+Lee%2C+Albert+Gatt%2C+Emiel+Van+Miltenburg%2C+Sander+Wubben%2C+and+Emiel+Krahmer%2E+Best+practices+for+the+human+evaluation+of+automatically+generated+text%2E+In+Proceedings+of+the+12th+Internationa)
36. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048, 2023a. [find](https://scholar.google.com/scholar?q=Jiaan+Wang%2C+Yunlong+Liang%2C+Fandong+Meng%2C+Haoxiang+Shi%2C+Zhixu+Li%2C+Jinan+Xu%2C+Jianfeng+Qu%2C+and+Jie+Zhou%2E+Is+chatgpt+a+good+nlg+evaluator%3F+a+preliminary+study%2E+arXiv+preprint+arXiv%3A2303%2E04048%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/)
37. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023b. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023b%2E)
38. Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087, 2023c. [find](https://scholar.google.com/scholar?q=Yidong+Wang%2C+Zhuohao+Yu%2C+Zhengran+Zeng%2C+Linyi+Yang%2C+Cunxiang+Wang%2C+Hao+Chen%2C+Chaoya+Jiang%2C+Rui+Xie%2C+Jindong+Wang%2C+Xing+Xie%2C+et+al%2E+Pandalm%3A+An+automatic+evaluation+benchmark+for+llm+instruction+tuning) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-pandalm-automatic-evaluation-benchmark-llm/)
39. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300, 2023d. [find](https://scholar.google.com/scholar?q=Zhenhailong+Wang%2C+Shaoguang+Mao%2C+Wenshan+Wu%2C+Tao+Ge%2C+Furu+Wei%2C+and+Heng+Ji%2E+Unleashing+cognitive+synergy+in+large+language+models%3A+A+task%2Dsolving+agent+through+multi%2Dpersona+self%2Dcollaboration%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-unleashing-emergent-cognitive-synergy-large/)
40. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2109%2E01652)
41. Anita Williams Woolley, Christopher F Chabris, Alex Pentland, Nada Hashmi, and Thomas W Malone. Evidence for a collective intelligence factor in the performance of human groups. science, 330(6004):686–688, 2010. [find](https://scholar.google.com/scholar?q=Anita+Williams+Woolley%2C+Christopher+F+Chabris%2C+Alex+Pentland%2C+Nada+Hashmi%2C+and+Thomas+W+Malone%2E+Evidence+for+a+collective+intelligence+factor+in+the+performance+of+human+groups%2E+science%2C+330%286004%29%3A686)
42. Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang. Large language models are diverse role-players for summarization evaluation. arXiv preprint arXiv:2303.15078, 2023. [find](https://scholar.google.com/scholar?q=Ning+Wu%2C+Ming+Gong%2C+Linjun+Shou%2C+Shining+Liang%2C+and+Daxin+Jiang%2E+Large+language+models+are+diverse+role%2Dplayers+for+summarization+evaluation%2E+arXiv+preprint+arXiv%3A2303%2E15078%2C+2023%2E)
43. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2C+2019%2E)
44. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622, 2019. [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Maxime+Peyrard%2C+Fei+Liu%2C+Yang+Gao%2C+Christian+M+Meyer%2C+and+Steffen+Eger%2E+Moverscore%3A+Text+generation+evaluating+with+contextualized+embeddings+and+earth+mover+distance%2E+arXiv+preprint+arXiv%3A1) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/)
45. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+arXiv+p)
46. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards a unified multi-dimensional evaluator for text generation. arXiv preprint arXiv:2210.07197, 2022. [find](https://scholar.google.com/scholar?q=Ming+Zhong%2C+Yang+Liu%2C+Da+Yin%2C+Yuning+Mao%2C+Yizhu+Jiao%2C+Pengfei+Liu%2C+Chenguang+Zhu%2C+Heng+Ji%2C+and+Jiawei+Han%2E+Towards+a+unified+multi%2Ddimensional+evaluator+for+text+generation%2E+arXiv+preprint+arXiv%3A2210%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/)