---
title: "ChatGPT as a Factual Inconsistency Evaluator for — Luo 2023"
description: "Summarizers built on pre-trained language models often add content the source document does not support."
authors: ["Zheheng Luo", "Qianqian Xie", "Sophia Ananiadou"]
year: 2023
venue: "arXiv:2303.15621"
source: "https://arxiv.org/abs/2303.15621"
doi: "10.48550/arxiv.2303.15621"
arxiv: "2303.15621"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / ChatGPT as a Factual Inconsistency Evaluator for Text Summarization

# ChatGPT as a Factual Inconsistency Evaluator for Text Summarization

**Authors:** Zheheng Luo, [Qianqian Xie](https://md.research.elamur.ai/free/en/authors/qianqian-xie/), [Sophia Ananiadou](https://md.research.elamur.ai/free/en/authors/sophia-ananiadou/) · **Year:** 2023 · **Venue:** arXiv:2303.15621 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.15621) · [PDF](https://arxiv.org/pdf/2303.15621) · [DOI 10.48550/arxiv.2303.15621](https://doi.org/10.48550/arxiv.2303.15621) · [arXiv 2303.15621](https://arxiv.org/abs/2303.15621) · [OpenAlex](https://openalex.org/W4361230777)

**Citations:** 50 citations (openalex-snapshot · 2026-10-14) · [110](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Summarizers built on pre-trained language models often add content the source document does not support. Dedicated factuality metrics, whether NLI-, QA- or IE-based or weakly supervised like FactCC, are costly to run, need labelled data or multi-model pipelines, and agree only moderately with human judges. The authors test whether ChatGPT, prompted zero-shot with no task-specific training, can act as such a judge. They define three tasks of increasing granularity. The first is binary entailment inference, a yes/no decision on whether a summary is consistent. The second is choosing the consistent summary from a pair. The third is assigning a numeric consistency score. Each task uses a plain zero-shot prompt and a zero-shot chain-of-thought variant. The evaluation covers the six SUMMAC datasets plus existing data for ranking and rating, and compares ChatGPT with specialised metrics. ChatGPT beats earlier state-of-the-art methods on most datasets, and chain-of-thought prompting often helps. Manual error analysis, however, reveals reliance on surface word overlap, faulty reasoning and drift from instructions. These are important caveats before LLMs are trusted as evaluators.

## Key points
- Three evaluation settings are tested: entailment inference (EI), summary ranking and consistency rating. All use zero-shot and zero-shot CoT prompting, with no few-shot examples, partly because of ChatGPT API costs.
- The data are the SUMMAC benchmark (six standardized datasets, e.g. CoGenSumm) plus sets such as SummEval and FRANK (including its XSum portion). ChatGPT outperforms SOTA metrics on 6 of 9 datasets.
- ChatGPT is compared against factuality-specific metrics such as FactCC, DAE and SummaC. Earlier LLM-as-evaluator work had only compared against general metrics like ROUGE and BERTScore.
- Typical errors:
  - Accepting a summary that silently drops a quantifier ("half of") because of high word overlap with the source.
  - Unsupported inferences, as in the Matteo Renzi bill example.
  - Under CoT prompting, stating the verdict before the reasoning.
- ChatGPT sometimes ignores the rating rubric. On SummEval it gave 1 point to a short but fully consistent summary for poor coverage, while experts rated it 5/5. Comparing other LLMs such as GPT-3.5 and GPT-4 is left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

110 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 99 not yet.

- [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)
- [Evaluating Large Language Models for Health-related Queries with Presuppositions](https://md.research.elamur.ai/free/en/papers/kaur-2024-evaluating-large-language-models-health/)
- [Multi-Session Client-Centered Treatment Outcome Evaluation in Psychotherapy](https://md.research.elamur.ai/free/en/papers/na-2026-multi-session-client-centered-treatment/)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)
- [Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors](https://md.research.elamur.ai/free/en/papers/tang-2023-understanding-factual-errors-summarization-errors/)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://md.research.elamur.ai/free/en/papers/tang-2024-minicheck-efficient-fact-checking-llms/)
- [QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing](https://md.research.elamur.ai/free/en/papers/wu-2023-qudeval-evaluation-questions-under-discussion/)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-sotopia-interactive-evaluation-social-intelligence/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{luo2023chatgpt,
  title = {{ChatGPT as a Factual Inconsistency Evaluator for Text Summarization}},
  author = {Luo, Zheheng and Xie, Qianqian and Ananiadou, Sophia},
  year = {2023},
  doi = {10.48550/arxiv.2303.15621},
  eprint = {2303.15621},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.15621}
}
```

- [← previous ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://md.research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/)
- [next → ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

50 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. [find](https://scholar.google.com/scholar?q=Yejin+Bang%2C+Samuel+Cahyawijaya%2C+Nayeon+Lee%2C+Wenliang+Dai%2C+Dan+Su%2C+Bryan+Wilie%2C+Holy+Lovenia%2C+Ziwei+Ji%2C+Tiezheng+Yu%2C+Willy+Chung%2C+et+al%2E+2023%2E+A+multitask%2C+multilingual%2C+multimodal+evaluation+of+chatgp) [our card](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
2. Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. 2010. The balanced accuracy and its posterior distribution. In 2010 20th international conference on pattern recognition, pages 3121–3124. IEEE. [find](https://scholar.google.com/scholar?q=Kay+Henning+Brodersen%2C+Cheng+Soon+Ong%2C+Klaas+Enno+Stephan%2C+and+Joachim+M+Buhmann%2E+2010%2E+The+balanced+accuracy+and+its+posterior+distribution%2E+In+2010+20th+international+conference+on+pattern+recogniti)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear)
4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+et+al%2E+2022%2E+Palm%3A+Scaling+language+mode)
5. Esin Durmus, He He, and Mona Diab. 2020a. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055–5070, Online. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Esin+Durmus%2C+He+He%2C+and+Mona+Diab%2E+2020a%2E+FEQA%3A+A+question+answering+evaluation+framework+for+faithfulness+assessment+in+abstractive+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Ass) [our card](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/)
6. Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+for+)
7. Tobias Falke, Leonardo F R Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019b. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. [find](https://scholar.google.com/scholar?q=Tobias+Falke%2C+Leonardo+F+R+Ribeiro%2C+Prasetya+Ajie+Utama%2C+Ido+Dagan%2C+and+Iryna+Gurevych%2E+2019b%2E+Ranking+generated+summaries+by+correctness%3A+An+interesting+but+challenging+application+for+natural+langua) [our card](https://md.research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/)
8. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. ArXiv, abs/2302.04166. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E+ArXiv%2C+abs%2F2302%2E04166%2E)
9. Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 166–175. [find](https://scholar.google.com/scholar?q=Ben+Goodrich%2C+Vinay+Rao%2C+Peter+J+Liu%2C+and+Mohammad+Saleh%2E+2019%2E+Assessing+the+factual+accuracy+of+generated+text%2E+In+Proceedings+of+the+25th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+)
10. Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. pages 3592–3603. Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Tanya+Goyal+and+Greg+Durrett%2E+2020%2E+Evaluating+factuality+in+generation+with+dependency%2Dlevel+entailment%2E+pages+3592%E2%80%933603%2E+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2020-evaluating-factuality-generation-dependency-level/)
11. Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang, Jun Xie, and Yue Zhang. 2020. What have we achieved on text summarization? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 446–469, Online. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Dandan+Huang%2C+Leyang+Cui%2C+Sen+Yang%2C+Guangsheng+Bao%2C+Kun+Wang%2C+Jun+Xie%2C+and+Yue+Zhang%2E+2020%2E+What+have+we+achieved+on+text+summarization%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+N)
12. Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021. The factual inconsistency problem in abstractive text summarization: A survey. arXiv preprint arXiv:2104.14839. — [arXiv:2104.14839](https://arxiv.org/abs/2104.14839) [find](https://scholar.google.com/scholar?q=Yichong+Huang%2C+Xiachong+Feng%2C+Xiaocheng+Feng%2C+and+Bing+Qin%2E+2021%2E+The+factual+inconsistency+problem+in+abstractive+text+summarization%3A+A+survey%2E+arXiv+preprint+arXiv%3A2104%2E14839%2E+%E2%80%94+%5BarXiv%3A2104%2E14839%5D%28h)
13. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745. [find](https://scholar.google.com/scholar?q=Wenxiang+Jiao%2C+Wenxuan+Wang%2C+Jen%2Dtse+Huang%2C+Xing+Wang%2C+and+Zhaopeng+Tu%2E+2023%2E+Is+chatgpt+a+good+translator%3F+a+preliminary+study%2E+arXiv+preprint+arXiv%3A2301%2E08745%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiao-2023-chatgpt-good-translator-yes-gpt/)
14. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. [find](https://scholar.google.com/scholar?q=Maurice+G+Kendall%2E+1938%2E+A+new+measure+of+rank+correlation%2E+Biometrika%2C+30%281%2F2%29%3A81%E2%80%9393%2E)
15. Tom Kocmi and Christian Federmann. 2023a. Large language models are state-of-the-art evaluators of translation quality. ArXiv, abs/2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023a%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+ArXiv%2C+abs%2F2302%2E14520%2E)
16. Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. ArXiv, abs/2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023b%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+ArXiv%2C+abs%2F2302%2E14520%2E)
17. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916. [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+Shane+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+2022%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+ArXiv%2C+abs%2F2205%2E11916%2E) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
18. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. pages 9332–9346. Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2020%2E+Evaluating+the+factual+consistency+of+abstractive+text+summarization%2E+pages+9332%E2%80%939346%2E+Association+for+Computational+Lingu)
19. Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A. Hearst. 2021. Keep it simple: Unsupervised simplification of multi-paragraph text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pr… [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+Bennett%2C+and+Marti+A%2E+Hearst%2E+2021%2E+Keep+it+simple%3A+Unsupervised+simplification+of+multi%2Dparagraph+text%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Associat)
20. Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+N+Bennett%2C+and+Marti+A+Hearst%2E+2022%2E+Summac%3A+Re%2Dvisiting+nli%2Dbased+models+for+inconsistency+detection+in+summarization%2E+Transactions+of+the+Association+for+Comput)
21. Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, and Kartik Talamadupula. 2021. Looking beyond sentence-level natural language inference for question answering and text summarization. In Proceedings of the 2021 Conference of the North American Chapter of the… [find](https://scholar.google.com/scholar?q=Anshuman+Mishra%2C+Dhruvesh+Patel%2C+Aparna+Vijayakumar%2C+Xiang+Lorraine+Li%2C+Pavan+Kapanipathi%2C+and+Kartik+Talamadupula%2E+2021%2E+Looking+beyond+sentence%2Dlevel+natural+language+inference+for+question+answerin)
22. Mavuto M Mukaka. 2012. A guide to appropriate use of correlation coefficient in medical research. Malawi medical journal, 24(3):69–71. [find](https://scholar.google.com/scholar?q=Mavuto+M+Mukaka%2E+2012%2E+A+guide+to+appropriate+use+of+correlation+coefficient+in+medical+research%2E+Malawi+medical+journal%2C+24%283%29%3A69%E2%80%9371%2E)
23. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Caglar+Gulcehre%2C+Bing+Xiang%2C+et+al%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2E)
24. Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. Entity-level factual consistency of abstractive text summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computation… [find](https://scholar.google.com/scholar?q=Feng+Nan%2C+Ramesh+Nallapati%2C+Zhiguo+Wang%2C+Cicero+Nogueira+dos+Santos%2C+Henghui+Zhu%2C+Dejiao+Zhang%2C+Kathleen+McKeown%2C+and+Bing+Xiang%2E+2021%2E+Entity%2Dlevel+factual+consistency+of+abstractive+text+summarizati) [our card](https://md.research.elamur.ai/free/en/papers/nan-2021-entity-level-factual-consistency-abstractive/)
25. OpenAI. 2022. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
26. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020%2E+BART%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+l) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
27. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeff+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+L%2E+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+John+Schulman%2C+Jacob+Hilton%2C+Fraser+Kelton%2C+Luke+E%2E+Mille)
28. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
29. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. [find](https://scholar.google.com/scholar?q=Artidoro+Pagnoni%2C+Vidhisha+Balachandran%2C+and+Yulia+Tsvetkov%2E+2021%2E+Understanding+factuality+in+abstractive+summarization+with+frank%3A+A+benchmark+for+factuality+metrics%2E) [our card](https://md.research.elamur.ai/free/en/papers/pagnoni-2021-understanding-factuality-abstractive-summarization-frank/)
30. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Assoc… [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+2019%2E+Text+summarization+with+pretrained+encoders%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+International+Join)
31. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476. [find](https://scholar.google.com/scholar?q=Chengwei+Qin%2C+Aston+Zhang%2C+Zhuosheng+Zhang%2C+Jiaao+Chen%2C+Michihiro+Yasunaga%2C+and+Diyi+Yang%2E+2023%2E+Is+chatgpt+a+general%2Dpurpose+natural+language+processing+task+solver%3F+arXiv+preprint+arXiv%3A2302%2E06476%2E) [our card](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/)
32. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+A)
33. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T McDonald. 2020. On faithfulness and factuality in abstractive summarization. ArXiv, abs/2005.00661. [find](https://scholar.google.com/scholar?q=Joshua+Maynez%2C+Shashi+Narayan%2C+Bernd+Bohnet%2C+and+Ryan+T+McDonald%2E+2020%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+ArXiv%2C+abs%2F2005%2E00661%2E)
34. Yiwei Qin, Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2022. T5score: Discriminative fine-tuning of generative evaluation metrics. arXiv preprint arXiv:2212.05726. [find](https://scholar.google.com/scholar?q=Yiwei+Qin%2C+Weizhe+Yuan%2C+Graham+Neubig%2C+and+Pengfei+Liu%2E+2022%2E+T5score%3A+Discriminative+fine%2Dtuning+of+generative+evaluation+metrics%2E+arXiv+preprint+arXiv%3A2212%2E05726%2E)
35. Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick, and Suzana Ili’c et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. ArXiv, abs/2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Elizabeth%2DJane+Pavlick%2C+and+Suzana+Ili%E2%80%99c+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model%2E+ArXiv%2C+abs%2F2211%2E05100%2E)
36. Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021. Questeval: Summarization asks for fact-based evaluation. [find](https://scholar.google.com/scholar?q=Thomas+Scialom%2C+Paul%2DAlexis+Dray%2C+Patrick+Gallinari%2C+Sylvain+Lamprier%2C+Benjamin+Piwowarski%2C+Jacopo+Staiano%2C+and+Alex+Wang%2E+2021%2E+Questeval%3A+Summarization+asks+for+fact%2Dbased+evaluation%2E) [our card](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/)
37. Mayank Soni and Vincent P. Wade. 2023. Comparing abstractive summaries generated by chatgpt to real summaries through blinded reviewers and text classification algorithms. [find](https://scholar.google.com/scholar?q=Mayank+Soni+and+Vincent+P%2E+Wade%2E+2023%2E+Comparing+abstractive+summaries+generated+by+chatgpt+to+real+summaries+through+blinded+reviewers+and+text+classification+algorithms%2E)
38. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
39. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comput)
40. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. ArXiv, abs/2302.10198. — [arXiv:2302.10198](https://arxiv.org/abs/2302.10198) [find](https://scholar.google.com/scholar?q=Qihuang+Zhong%2C+Liang+Ding%2C+Juhua+Liu%2C+Bo+Du%2C+and+Dacheng+Tao%2E+2023%2E+Can+chatgpt+understand+too%3F+a+comparative+study+on+chatgpt+and+fine%2Dtuned+bert%2E+ArXiv%2C+abs%2F2302%2E10198%2E+%E2%80%94+%5BarXiv%3A2302%2E10198%5D%28https%3A%2F%2F)
41. Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Cross-lingual summarization via chatgpt. arXiv preprint arXiv:2302.14229. — [arXiv:2302.14229](https://arxiv.org/abs/2302.14229) [find](https://scholar.google.com/scholar?q=Jiaan+Wang%2C+Yunlong+Liang%2C+Fandong+Meng%2C+Zhixu+Li%2C+Jianfeng+Qu%2C+and+Jie+Zhou%2E+2023a%2E+Cross%2Dlingual+summarization+via+chatgpt%2E+arXiv+preprint+arXiv%3A2302%2E14229%2E+%E2%80%94+%5BarXiv%3A2302%2E14229%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
42. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023b. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048. [find](https://scholar.google.com/scholar?q=Jiaan+Wang%2C+Yunlong+Liang%2C+Fandong+Meng%2C+Haoxiang+Shi%2C+Zhixu+Li%2C+Jinan+Xu%2C+Jianfeng+Qu%2C+and+Jie+Zhou%2E+2023b%2E+Is+chatgpt+a+good+nlg+evaluator%3F+a+preliminary+study%2E+arXiv+preprint+arXiv%3A2303%2E04048%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/)
43. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023c. Is chatgpt a good nlg evaluator? a preliminary study. ArXiv, abs/2303.04048. [find](https://scholar.google.com/scholar?q=Jiaan+Wang%2C+Yunlong+Liang%2C+Fandong+Meng%2C+Haoxiang+Shi%2C+Zhixu+Li%2C+Jinan+Xu%2C+Jianfeng+Qu%2C+and+Jie+Zhou%2E+2023c%2E+Is+chatgpt+a+good+nlg+evaluator%3F+a+preliminary+study%2E+ArXiv%2C+abs%2F2303%2E04048%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/)
44. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Ed+Huai+hsin+Chi%2C+Quoc+Le%2C+and+Denny+Zhou%2E+2022%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+ArXiv%2C+abs%2F2201%2E11903%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
45. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
46. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, and Sophia Ananiadou. 2023a. On the evaluations of chatgpt and emotion-enhanced prompting for mental health analysis. arXiv preprint arXiv:2304.03347. — [arXiv:2304.03347](https://arxiv.org/abs/2304.03347) [find](https://scholar.google.com/scholar?q=Kailai+Yang%2C+Shaoxiong+Ji%2C+Tianlin+Zhang%2C+Qianqian+Xie%2C+and+Sophia+Ananiadou%2E+2023a%2E+On+the+evaluations+of+chatgpt+and+emotion%2Denhanced+prompting+for+mental+health+analysis%2E+arXiv+preprint+arXiv%3A2304%2E)
47. Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023b. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081. [find](https://scholar.google.com/scholar?q=Xianjun+Yang%2C+Yan+Li%2C+Xinlu+Zhang%2C+Haifeng+Chen%2C+and+Wei+Cheng%2E+2023b%2E+Exploring+the+limits+of+chatgpt+for+query+or+aspect%2Dbased+text+summarization%2E+arXiv+preprint+arXiv%3A2302%2E08081%2E)
48. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263–27277. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Graham+Neubig%2C+and+Pengfei+Liu%2E+2021%2E+Bartscore%3A+Evaluating+generated+text+as+text+generation%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A27263%E2%80%9327277%2E) [our card](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/)
49. Jerrold H Zar. 2005. Spearman rank correlation. Encyclopedia of biostatistics, 7. [find](https://scholar.google.com/scholar?q=Jerrold+H+Zar%2E+2005%2E+Spearman+rank+correlation%2E+Encyclopedia+of+biostatistics%2C+7%2E)
50. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328–11339. PMLR. [find](https://scholar.google.com/scholar?q=Jingqing+Zhang%2C+Yao+Zhao%2C+Mohammad+Saleh%2C+and+Peter+Liu%2E+2020%2E+Pegasus%3A+Pre%2Dtraining+with+extracted+gap%2Dsentences+for+abstractive+summarization%2E+In+International+Conference+on+Machine+Learning%2C+pages+) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)