---
title: "G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment"
description: "The authors, from Microsoft Azure AI, propose G-Eval, an approach in which a large language… 37 references, cited by 1,096 papers here."
authors: ["Yang Liu", "Dan Iter", "Yichong Xu", "Shuohang Wang", "Ruochen Xu", "Chenguang Zhu"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.153/"
doi: "10.18653/v1/2023.emnlp-main.153"
arxiv: "2303.16634"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment

# G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment

**Authors:** [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/), [Dan Iter](https://md.research.elamur.ai/free/en/authors/dan-iter/), [Yichong Xu](https://md.research.elamur.ai/free/en/authors/yichong-xu/), [Shuohang Wang](https://md.research.elamur.ai/free/en/authors/shuohang-wang/), [Ruochen Xu](https://md.research.elamur.ai/free/en/authors/ruochen-xu/) and [Chenguang Zhu](https://md.research.elamur.ai/free/en/authors/chenguang-zhu/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.153/) · [PDF](https://aclanthology.org/2023.emnlp-main.153.pdf) · [DOI 10.18653/v1/2023.emnlp-main.153](https://doi.org/10.18653/v1/2023.emnlp-main.153) · [arXiv 2303.16634](https://arxiv.org/abs/2303.16634) · [OpenAlex](https://openalex.org/W4389519254)

**Citations:** 904 citations (OpenAlex, 2026-10-14) · [1096](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Automatically evaluating the quality of texts generated by NLG systems is difficult. Classical metrics such as BLEU and ROUGE correlate poorly with human judgments and also require reference texts. The authors, from Microsoft Azure AI, propose G-Eval, an approach in which a large language model evaluates text without a reference. The model receives a task description and a criterion, for example the coherence of a summary. First, it generates its own step-by-step evaluation plan (chain-of-thought). Then, following this plan, it fills in a short "form" with a score from 1 to 5. The final score is computed as the sum of the scores weighted by their probabilities. This makes the scores continuous, with fewer ties and less skew toward a single score. The method was tested on three meta-evaluation benchmarks for summarization and dialogue response generation. G-Eval based on GPT-4 agrees with humans noticeably better than previous metrics. The authors also point out a risk: LLM evaluators tend to prefer texts written by LLMs. If such scores are used as a reward during training, models may end up reinforcing themselves.

## Key points
- On SummEval, G-Eval-4 achieves an average Spearman correlation of 0.514 (Kendall-Tau 0.418). For comparison: UniEval scores 0.474, GPTScore 0.417, BERTScore 0.225, and ROUGE-1 0.192. The GPT-3.5 version (text-davinci-003) achieves only 0.401.
- Ablation for GPT-4: without probability weighting, the correlation drops to 0.502; without automatic CoT, to 0.500; without the description (Description), to 0.479.
- Three benchmarks were used:
  - SummEval, based on CNN/DailyMail, with the aspects fluency, coherence, consistency, relevance;
  - Topical-Chat with the aspects naturalness, coherence, engagingness, groundedness;
  - QAGS for hallucination detection.
- GPT-4 did not return token probabilities, so they were estimated by sampling: 20 generations with temperature 1. For GPT-3.5, temperature was set to 0.
- Limitations: the method depends on closed, paid LLMs, and updates to their versions harm reproducibility. The code is available at github.com/nlpyang/geval.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 214 works

💡 Concepts (2): [💡 Text summarization](https://md.research.elamur.ai/free/en/concepts/text-summarization/) · 49 works · [💡 Large language models](https://md.research.elamur.ai/free/en/concepts/large-language-models/) · 48 works

## Cite

Cite as: Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2023g,
  title = {{G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment}},
  author = {Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu and Chenguang Zhu},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.153},
  eprint = {2303.16634},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.153}
}
```

- [← previous Toolformer: Language Models Can Teach Themselves to Use Tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/)
- [next → Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)

## References

37 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. [Meteor: An automatic metric for mt evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
2. Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. [Factual error correction for abstractive summarization models](https://md.research.elamur.ai/free/en/papers/cao-2020-factual-error-correction-abstractive-summarization/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251–6258.
3. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In thirty-second AAAI conference on artificial intelligence. [find](https://scholar.google.com/scholar?q=Ziqiang+Cao%2C+Furu+Wei%2C+Wenjie+Li%2C+and+Sujian+Li%2E+2018%2E+Faithful+to+the+original%3A+Fact+aware+neural+abstractive+summarization%2E+In+thirty%2Dsecond+AAAI+conference+on+artificial+intelligence%2E)
4. Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith. 2019. [Sentence mover’s similarity: Automatic evaluation for multi-sentence texts](https://md.research.elamur.ai/free/en/papers/clark-2019-sentence-mover-s-similarity-automatic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2748–2760.
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
6. Esin Durmus, He He, and Mona Diab. 2020. [Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055– 5070.
7. Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar R Zaiane. 2019. Evaluating coherence in dialogue systems using entailment. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Shor… [find](https://scholar.google.com/scholar?q=Nouha+Dziri%2C+Ehsan+Kamalloo%2C+Kory+Mathewson%2C+and+Osmar+R+Zaiane%2E+2019%2E+Evaluating+coherence+in+dialogue+systems+using+entailment%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of)
8. Alexander R Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+for)
9. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E+arXiv+preprint+arXiv%3A2302%2E04166%2E)
10. Sarik Ghazarian, Johnny Wei, Aram Galstyan, and Nanyun Peng. 2019. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 82–89, Minneapolis, Minnesota. Asso… [find](https://scholar.google.com/scholar?q=Sarik+Ghazarian%2C+Johnny+Wei%2C+Aram+Galstyan%2C+and+Nanyun+Peng%2E+2019%2E+Better+automatic+evaluation+of+open%2Ddomain+dialogue+systems+with+contextualized+embeddings%2E+In+Proceedings+of+the+Workshop+on+Methods)
11. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). Advances in neural information processing systems, 28.
12. Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, and Noah A. Smith. 2022. Bidimensional leaderboards: Generate and evaluate language hand in hand. In Proceedings of the 2022 Conference of the North American Chapter of the Association for C… [find](https://scholar.google.com/scholar?q=Jungo+Kasai%2C+Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Lavinia+Dunagan%2C+Jacob+Morrison%2C+Alexander+Fabbri%2C+Yejin+Choi%2C+and+Noah+A%2E+Smith%2E+2022%2E+Bidimensional+leaderboards%3A+Generate+and+evaluate+language+hand+i)
13. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+arXiv+preprint+arXiv%3A2302%2E14520%2E)
14. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. [Evaluating the factual consistency of abstractive text summarization](https://md.research.elamur.ai/free/en/papers/kryscinski-2020-evaluating-factual-consistency-abstractive-text/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346.
15. Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning, pages 957–966. PMLR. [find](https://scholar.google.com/scholar?q=Matt+Kusner%2C+Yu+Sun%2C+Nicholas+Kolkin%2C+and+Kilian+Weinberger%2E+2015%2E+From+word+embeddings+to+document+distances%2E+In+International+conference+on+machine+learning%2C+pages+957%E2%80%93966%2E+PMLR%2E)
16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
17. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
18. Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023. [Error analysis prompting enables human-like translation evaluation in large language models](https://md.research.elamur.ai/free/en/papers/lu-2024-error-analysis-prompting-enables-human/): A case study on chatgpt. arXiv preprint arXiv:2303.13809.
19. Shikib Mehri and Maxine Eskenazi. 2020. [USR: An unsupervised and reference free evaluation metric for dialog generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707, Online. Association for Computational Linguistics.
20. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807.
21. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21:1– 67.
24. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529–558. [find](https://scholar.google.com/scholar?q=Ehud+Reiter+and+Anja+Belz%2E+2009%2E+An+investigation+into+the+validity+of+some+metrics+for+automatically+evaluating+natural+language+generation+systems%2E+Computational+Linguistics%2C+35%284%29%3A529%E2%80%93558%2E)
25. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2023b%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2E)
26. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 28.
27. Zheng Ye, Liucun Lu, Lishan Huang, Liang Lin, and Xiaodan Liang. 2021. Towards quantifiable dialogue coherence evaluation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:… [find](https://scholar.google.com/scholar?q=Zheng+Ye%2C+Liucun+Lu%2C+Lishan+Huang%2C+Liang+Lin%2C+and+Xiaodan+Liang%2E+2021%2E+Towards+quantifiable+dialogue+coherence+evaluation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computationa)
28. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). Advances in Neural Information Processing Systems, 34.
29. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E)
30. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2023. [Benchmarking large language models for news summarization](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/).
31. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. [Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International…
32. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In Proceedings of the 6th international conference on Computational Linguistics and Intelligent Text Processing, pages 341–351. [find](https://scholar.google.com/scholar?q=Amanda+Stent%2C+Matthew+Marge%2C+and+Mohit+Singhai%2E+2005%2E+Evaluating+evaluation+methods+for+generation+in+the+presence+of+variation%2E+In+Proceedings+of+the+6th+international+conference+on+Computational+Lin)
33. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/).
34. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
35. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multidimensional evaluator for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2023– 2038, Abu Dhabi, U… [find](https://scholar.google.com/scholar?q=Ming+Zhong%2C+Yang+Liu%2C+Da+Yin%2C+Yuning+Mao%2C+Yizhu+Jiao%2C+Pengfei+Liu%2C+Chenguang+Zhu%2C+Heng+Ji%2C+and+Jiawei+Han%2E+2022%2E+Towards+a+unified+multidimensional+evaluator+for+text+generation%2E+In+Proceedings+of+the)
36. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comput)
37. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). arXiv preprint arXiv:2303.04048.

*Updated 31 October 2026*