---
title: "Human-like Summarization Evaluation with ChatGPT — Gao 2023"
description: "The authors test whether ChatGPT (gpt-3.5-turbo-0301, temperature 0) can stand in for human… 21 references, cited by 92 papers here."
authors: ["Mingqi Gao", "Jie Ruan", "Renliang Sun", "Xunjian Yin", "Shiping Yang", "Xiaojun Wan"]
year: 2023
venue: "arXiv:2304.02554"
source: "https://arxiv.org/abs/2304.02554"
doi: "10.48550/arxiv.2304.02554"
arxiv: "2304.02554"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gao-2023-human-like-summarization-evaluation-chatgpt/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gao-2023-human-like-summarization-evaluation-chatgpt/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Human-like Summarization Evaluation with ChatGPT

# Human-like Summarization Evaluation with ChatGPT

**Authors:** [Mingqi Gao](https://md.research.elamur.ai/free/en/authors/mingqi-gao/), [Jie Ruan](https://md.research.elamur.ai/free/en/authors/jie-ruan/), [Renliang Sun](https://md.research.elamur.ai/free/en/authors/renliang-sun/), [Xunjian Yin](https://md.research.elamur.ai/free/en/authors/xunjian-yin/), [Shiping Yang](https://md.research.elamur.ai/free/en/authors/shiping-yang/) and [Xiaojun Wan](https://md.research.elamur.ai/free/en/authors/xiaojun-wan/) · **Year:** 2023 · **Venue:** arXiv:2304.02554 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.02554) · [PDF](https://arxiv.org/pdf/2304.02554) · [DOI 10.48550/arxiv.2304.02554](https://doi.org/10.48550/arxiv.2304.02554) · [arXiv 2304.02554](https://arxiv.org/abs/2304.02554) · [OpenAlex](https://openalex.org/W4362679631)

**Citations:** 36 citations (OpenAlex, 2026-10-14) · [92](https://md.research.elamur.ai/free/en/papers/gao-2023-human-like-summarization-evaluation-chatgpt/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Automatic metrics for text summarization, such as ROUGE and embedding-based scores, often disagree with human judgment and offer little interpretability. Human evaluation is more reliable but slow and costly. The authors test whether ChatGPT (gpt-3.5-turbo-0301, temperature 0) can stand in for human annotators. Their prompts closely follow the original annotation instructions for four protocols:

- Likert-scale scoring on SummEval and Newsroom
- pairwise comparison on TLDR
- Pyramid-style SCU checking on REALSumm
- binary sentence-level factuality on QAGS

They compare its outputs with human labels and with ROUGE, BERTScore, MoverScore, BARTScore variants, FactCC and DAE.

ChatGPT handled every protocol with very few failed responses. On SummEval it correlated with humans better than all baseline metrics. On Newsroom, only two BARTScore variants beat it. In the other three protocols its advantage was modest, except on QAGS_XSUM. Results depended strongly on prompt wording. Evaluation was also far cheaper than paying annotators.

This matters because LLM-based evaluation is flexible, reproducible and inexpensive. It can also produce explanations alongside its scores.

## Key points

- On SummEval, ChatGPT's sample-level Spearman correlations were 0.435 for consistency, 0.433 for relevance, 0.419 for fluency and 0.561 for coherence. Its system-level correlation for relevance reached 0.901.
- Without dimension definitions in the prompt, ChatGPT's fluency and coherence scores were almost identical (correlation 0.960). Adding definitions lowered this to 0.843, meaning the model separated the two dimensions better.
- Invalid outputs, such as refusals or rewriting the summary, stayed at or below about 1%. The highest rate (0.0106) came with the prompt that added both definitions and step instructions.
- Scoring all of SummEval cost roughly 3.2 USD, compared with an estimated 60 USD for a human annotator. The authors put human evaluation at about 10–20 times the cost.
- The explanations ChatGPT gave generally matched its scores. However, the authors note that whether those explanations are factually correct still needs to be verified.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 243 works

## Cite

Cite as: Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang and Xiaojun Wan. 2023. Human-like Summarization Evaluation with ChatGPT. arXiv:2304.02554.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gao2023human,
  title = {{Human-like Summarization Evaluation with ChatGPT}},
  author = {Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang and Xiaojun Wan},
  year = {2023},
  doi = {10.48550/arxiv.2304.02554},
  eprint = {2304.02554},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.02554}
}
```

- [← previous ExpertPrompting: Instructing Large Language Models to be Distinguished Experts](https://md.research.elamur.ai/free/en/papers/xu-2023-expertprompting-instructing-large-language-models/)
- [next → Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://md.research.elamur.ai/free/en/papers/golchin-2023-time-travel-llms-tracing-data/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

21 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anya Belz, Anastasia Shimorina, Shubham Agarwal, and Ehud Reiter. 2021. The ReproGen shared task on reproducibility of human evaluations in NLG: Overview and results. In Proceedings of the 14th International Conference on Natural Language Generation, pages 249–258, Aberdeen, Scotland, UK. Associatio… [find](https://scholar.google.com/scholar?q=Anya+Belz%2C+Anastasia+Shimorina%2C+Shubham+Agarwal%2C+and+Ehud+Reiter%2E+2021%2E+The+ReproGen+shared+task+on+reproducibility+of+human+evaluations+in+NLG%3A+Overview+and+results%2E+In+Proceedings+of+the+14th+Intern)
2. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. [Re-evaluating evaluation in text summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguist…
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Esin Durmus, He He, and Mona Diab. 2020. [FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055– 5070, Online. Association for Computational Lingu…
5. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R%2E+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+SummEval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+for)
6. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. [Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa…
7. Yunjie Ji, Yan Gong, Yiping Peng, Chao Ni, Peiyan Sun, Dongyu Pan, Baochang Ma, and Xiangang Li. 2023. Exploring chatgpt’s ability to rank content: A preliminary study on consistency with human preferences. Computing Research Repository, arXiv:2303.07610. — [arXiv:2303.07610](https://arxiv.org/abs/2303.07610) [find](https://scholar.google.com/scholar?q=Yunjie+Ji%2C+Yan+Gong%2C+Yiping+Peng%2C+Chao+Ni%2C+Peiyan+Sun%2C+Dongyu+Pan%2C+Baochang+Ma%2C+and+Xiangang+Li%2E+2023%2E+Exploring+chatgpt%E2%80%99s+ability+to+rank+content%3A+A+preliminary+study+on+consistency+with+human+prefer)
8. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. Computing Research Repository, arXiv:2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+Computing+Research+Repository%2C+arXiv%3A2302%2E14520%2E)
9. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. [Evaluating the factual consistency of abstractive text summarization](https://md.research.elamur.ai/free/en/papers/kryscinski-2020-evaluating-factual-consistency-abstractive-text/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Computational…
10. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
11. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [Gpteval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/).
12. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. [Chatgpt as a factual inconsistency evaluator for abstractive text summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/). Computing Research Repository, arXiv:2303.15621.
13. Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145–152, Boston, Massachusetts… [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Rebecca+Passonneau%2E+2004%2E+Evaluating+content+selection+in+summarization%3A+The+pyramid+method%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+North+American+Chapter+of)
14. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
15. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. Learning to summarize from human feedback. Computing Research Repository, arXiv:2009.01325. Version 3. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeff+Wu%2C+Daniel+M%2E+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+Christiano%2E+2022%2E+Learning+to+summarize+from+human+feedback%2E+Computing+Research+R)
16. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comput)
17. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). Computing Research Repository, arXiv:2303.04048.
18. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). Computing Research Repository, arXiv:2106.11520. Version 2.
19. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Computing Research Repository, arXiv:1904.09675. Version 3. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+Computing+Research+Repository%2C+arXiv%3A1904%2E09675%2E+Version+3%2E)
20. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2023. [Benchmarking large language models for news summarization](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/). Computing Research Repository, arXiv:2301.13848.
21. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. [MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International…

*Updated 2 November 2026*