---
title: "FineSurE: Fine-grained Summarization Evaluation using LLMs"
description: "Evaluating summaries automatically remains difficult. 7.2 points. 49 references, cited by 60 papers here."
authors: ["Hwanjun Song", "Hang Su", "Igor Shalyminov", "Jason Cai", "Saab Mansour"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.51/"
doi: "10.18653/v1/2024.acl-long.51"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/song-2024-finesure-fine-grained-summarization-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/song-2024-finesure-fine-grained-summarization-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / FineSurE: Fine-grained Summarization Evaluation using LLMs

# FineSurE: Fine-grained Summarization Evaluation using LLMs

**Authors:** [Hwanjun Song](https://md.research.elamur.ai/free/en/authors/hwanjun-song/), [Hang Su](https://md.research.elamur.ai/free/en/authors/hang-su/), [Igor Shalyminov](https://md.research.elamur.ai/free/en/authors/igor-shalyminov/), Jason Cai and [Saab Mansour](https://md.research.elamur.ai/free/en/authors/saab-mansour/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.51/) · [PDF](https://aclanthology.org/2024.acl-long.51.pdf) · [DOI 10.18653/v1/2024.acl-long.51](https://doi.org/10.18653/v1/2024.acl-long.51) · [OpenAlex](https://openalex.org/W4402667059)

**Citations:** 31 citations (OpenAlex, 2026-10-14) · [60](https://md.research.elamur.ai/free/en/papers/song-2024-finesure-fine-grained-summarization-evaluation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Evaluating summaries automatically remains difficult. ROUGE-style overlap metrics track human judgment poorly, and recent LLM judges such as G-Eval assign a single Likert score per summary, which hides where and how a summary fails. FineSurE addresses this by having an LLM run two fine-grained tasks. In fact checking, the LLM labels every summary sentence as correct or as containing one of seven error types (one extrinsic, six intrinsic). In keyfact alignment, it matches each keyfact to the summary sentences that express it; keyfacts can come from human annotators or be generated by an LLM. The outcomes become percentage scores for faithfulness, completeness and conciseness, the latter two proposed as clearer replacements for the loosely defined coherence and relevance. The authors test several open-source and proprietary LLM backbones and compare FineSurE with similarity-, NLI-, QA- and LLM-based evaluators. It shows higher agreement with humans, with the clearest gains on completeness and conciseness. This yields interpretable, sentence-level diagnostics for benchmarking summarizers, although validation so far covers only news data.

## Key points
- Scores are simple ratios: the share of factually correct sentences (faithfulness), the share of keyfacts covered (completeness), and the share of sentences tied to keyfacts (conciseness).
- Among backbones, proprietary models generally agree more closely with humans, and GPT-4-omni ranks highest. On error localization, Llama3-70B-Inst reaches 49.4% average accuracy, 7.2 points above GPT-4-turbo.
- For open-source backbones, larger models and instruction tuning both help; for example, Mixtral-8x7b-Inst clearly beats Mixtral-8x7b.
- As a case study, the authors scored summaries from eight LLMs on 100 CNNDM articles that also appear in REALSumm, which supplies human-written keyfacts. The best model differs by dimension: GPT-3.5-turbo for faithfulness, GPT-4-turbo for completeness and GPT-4-omni for conciseness. Llama3-70B-Inst comes close to the proprietary models.
- Code is released at github.com/DISL-Lab/FineSurE-ACL24. Experiments cover only the news domain, and prompts may need adjustment for other domains such as medicine.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 70 works · 39,977 citations
- [Korea Advanced Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/korea-advanced-institute-of-science-and-technology/) — University, Daejeon, South Korea · 17 works · 2,673 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 159 works

## Cite

Cite as: Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai and Saab Mansour. 2024. FineSurE: Fine-grained Summarization Evaluation using LLMs. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{song2024finesure,
  title = {{FineSurE: Fine-grained Summarization Evaluation using LLMs}},
  author = {Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai and Saab Mansour},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.51},
  url = {https://doi.org/10.18653/v1/2024.acl-long.51}
}
```

- [← previous Do-Not-Answer: Evaluating Safeguards in LLMs](https://md.research.elamur.ai/free/en/papers/wang-2024-do-not-answer-evaluating-safeguards/)
- [next → Large Language Models are Inconsistent and Biased Evaluators](https://md.research.elamur.ai/free/en/papers/stureborg-2024-large-language-models-inconsistent-biased/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

49 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for mt evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In ACLW.
3. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. [Re-evaluating evaluation in text summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/). In EMNLP.
4. Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. 2010. The balanced accuracy and its posterior distribution. In ICPR. [find](https://scholar.google.com/scholar?q=Kay+Henning+Brodersen%2C+Cheng+Soon+Ong%2C+Klaas+Enno+Stephan%2C+and+Joachim+M+Buhmann%2E+2010%2E+The+balanced+accuracy+and+its+posterior+distribution%2E+In+ICPR%2E)
5. Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. arXiv preprint arXiv:2304.00723. — [arXiv:2304.00723](https://arxiv.org/abs/2304.00723) [find](https://scholar.google.com/scholar?q=Yi+Chen%2C+Rui+Wang%2C+Haiyun+Jiang%2C+Shuming+Shi%2C+and+Ruifeng+Xu%2E+2023%2E+Exploring+the+use+of+large+language+models+for+reference%2Dfree+text+quality+evaluation%3A+A+preliminary+empirical+study%2E+arXiv+preprint)
6. Som Gupta and Sanjai Kumar Gupta. 2019. Abstractive summarization: An overview of the state of the art. Expert Systems with Applications, 121:49–65. [find](https://scholar.google.com/scholar?q=Som+Gupta+and+Sanjai+Kumar+Gupta%2E+2019%2E+Abstractive+summarization%3A+An+overview+of+the+state+of+the+art%2E+Expert+Systems+with+Applications%2C+121%3A49%E2%80%9365%2E)
7. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In NeurIPS.
8. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. [TRUE: Re-evaluating factual consistency evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/). In NAACL.
9. Yiran Chen, Pengfei Liu, and Xipeng Qiu. 2021. Are factuality checkers reliable? adversarial meta-evaluation of factuality in summarization. In EMNLP. [find](https://scholar.google.com/scholar?q=Yiran+Chen%2C+Pengfei+Liu%2C+and+Xipeng+Qiu%2E+2021%2E+Are+factuality+checkers+reliable%3F+adversarial+meta%2Devaluation+of+factuality+in+summarization%2E+In+EMNLP%2E)
10. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38.
11. Alexander R Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+for)
12. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+2024%2E+Mixtral+o)
13. Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. [QAFactEval: Improved qa-based factual consistency evaluation for summarization](https://md.research.elamur.ai/free/en/papers/fabbri-2022-qafacteval-improved-qa-based-factual/). In NAACL.
14. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In EMNLP. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2020%2E+Evaluating+the+factual+consistency+of+abstractive+text+summarization%2E+In+EMNLP%2E)
15. Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. 2023. FactKB: Generalizable factuality evaluation using language models enhanced with factual knowledge. In EMNLP. [find](https://scholar.google.com/scholar?q=Shangbin+Feng%2C+Vidhisha+Balachandran%2C+Yuyang+Bai%2C+and+Yulia+Tsvetkov%2E+2023%2E+FactKB%3A+Generalizable+factuality+evaluation+using+language+models+enhanced+with+factual+knowledge%2E+In+EMNLP%2E)
16. Philippe Laban, Wojciech Kryściński, Divyansh Agarwal, Alexander Richard Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. SummEdits: Measuring llm ability at factual reasoning through the lens of summarization. In EMNLP. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Divyansh+Agarwal%2C+Alexander+Richard+Fabbri%2C+Caiming+Xiong%2C+Shafiq+Joty%2C+and+Chien%2DSheng+Wu%2E+2023%2E+SummEdits%3A+Measuring+llm+ability+at+factual+reasoning+through+t)
17. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. GPTscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+GPTscore%3A+Evaluate+as+you+desire%2E+arXiv+preprint+arXiv%3A2302%2E04166%2E)
18. Xiyan Fu and Anette Frank. 2023. SETI: Systematicity evaluation of textual inference. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4101–4114, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiyan+Fu+and+Anette+Frank%2E+2023%2E+SETI%3A+Systematicity+evaluation+of+textual+inference%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+4101%E2%80%934114%2C+Toronto%2C+Canada%2E+Associat)
19. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554. [find](https://scholar.google.com/scholar?q=Mingqi+Gao%2C+Jie+Ruan%2C+Renliang+Sun%2C+Xunjian+Yin%2C+Shiping+Yang%2C+and+Xiaojun+Wan%2E+2023%2E+Human%2Dlike+summarization+evaluation+with+chatgpt%2E+arXiv+preprint+arXiv%3A2304%2E02554%2E)
20. Mingqi Gao and Xiaojun Wan. 2022. Dialsummeval: Revisiting summarization evaluation for dialogues. In NAACL. [find](https://scholar.google.com/scholar?q=Mingqi+Gao+and+Xiaojun+Wan%2E+2022%2E+Dialsummeval%3A+Revisiting+summarization+evaluation+for+dialogues%2E+In+NAACL%2E)
21. John Glover, Federico Fancellu, Vasudevan Jagannathan, Matthew R Gormley, and Thomas Schaaf. 2022. Revisiting text decomposition methods for nli-based factuality scoring of summaries. In GEM. [find](https://scholar.google.com/scholar?q=John+Glover%2C+Federico+Fancellu%2C+Vasudevan+Jagannathan%2C+Matthew+R+Gormley%2C+and+Thomas+Schaaf%2E+2022%2E+Revisiting+text+decomposition+methods+for+nli%2Dbased+factuality+scoring+of+summaries%2E+In+GEM%2E)
22. Tanya Goyal and Greg Durrett. 2020. [Evaluating factuality in generation with dependency-level entailment](https://md.research.elamur.ai/free/en/papers/goyal-2020-evaluating-factuality-generation-dependency-level/). In EMNLP.
23. Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. SummaC: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+N+Bennett%2C+and+Marti+A+Hearst%2E+2022%2E+SummaC%3A+Re%2Dvisiting+nli%2Dbased+models+for+inconsistency+detection+in+summarization%2E+Transactions+of+the+Association+for+Comput)
24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In ACL.
25. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out.
26. Yen-Ting Lin and Yun-Nung Chen. 2023. [Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models](https://md.research.elamur.ai/free/en/papers/lin-2023-llm-eval-unified-multi-dimensional/). arXiv preprint arXiv:2305.13711.
27. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023%2E+G%2DEval%3A+NLG+evaluation+using+gpt%2D4+with+better+human+alignment%2E+arXiv+preprint+arXiv%3A2303%2E16634%2E)
28. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In ACL. [find](https://scholar.google.com/scholar?q=Joshua+Maynez%2C+Shashi+Narayan%2C+Bernd+Bohnet%2C+and+Ryan+McDonald%2E+2020%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+In+ACL%2E)
29. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In ACL. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+ACL%2E)
30. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. [FActScore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). In EMNLP.
31. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). arXiv preprint arXiv:2303.04048.
32. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808.08745. [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+ArXiv%2C+abs%2F1808%2E08745%2E)
33. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. [Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics](https://md.research.elamur.ai/free/en/papers/pagnoni-2021-understanding-factuality-abstractive-summarization-frank/). In NAACL.
34. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [BLEU: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL.
35. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076. — [arXiv:2310.10076](https://arxiv.org/abs/2310.10076) [find](https://scholar.google.com/scholar?q=Keita+Saito%2C+Akifumi+Wachi%2C+Koki+Wataoka%2C+and+Youhei+Akimoto%2E+2023%2E+Verbosity+bias+in+preference+labeling+by+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E10076%2E+%E2%80%94+%5BarXiv%3A2310%2E10076%5D%28https%3A%2F%2Farxiv%2E)
36. Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021. [QuestEval: Summarization asks for fact-based evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/). In EMNLP.
37. Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. [Large language models are not yet human-level evaluators for abstractive summarization](https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-models-are-not/). In EMNLP.
38. Yuchen Shen and Xiaojun Wan. 2023. Opinsummeval: Revisiting automated evaluation for opinion summarization. arXiv preprint arXiv:2310.18122. — [arXiv:2310.18122](https://arxiv.org/abs/2310.18122) [find](https://scholar.google.com/scholar?q=Yuchen+Shen+and+Xiaojun+Wan%2E+2023%2E+Opinsummeval%3A+Revisiting+automated+evaluation+for+opinion+summarization%2E+arXiv+preprint+arXiv%3A2310%2E18122%2E+%E2%80%94+%5BarXiv%3A2310%2E18122%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E18122%29)
39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In NeurIPS.
40. Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. 2023. Towards better chain-of-thought prompting strategies: A survey. arXiv preprint arXiv:2310.04959. — [arXiv:2310.04959](https://arxiv.org/abs/2310.04959) [find](https://scholar.google.com/scholar?q=Zihan+Yu%2C+Liang+He%2C+Zhen+Wu%2C+Xinyu+Dai%2C+and+Jiajun+Chen%2E+2023%2E+Towards+better+chain%2Dof%2Dthought+prompting+strategies%3A+A+survey%2E+arXiv+preprint+arXiv%3A2310%2E04959%2E+%E2%80%94+%5BarXiv%3A2310%2E04959%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
41. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [BARTScore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). In NeurIPS.
42. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. [Instruction tuning for large language models: A survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/). arXiv preprint arXiv:2308.10792.
43. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with bert. In ICLR. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+BERTScore%3A+Evaluating+text+generation+with+bert%2E+In+ICLR%2E)
44. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. [MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/). In EMNLP.
45. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. [Towards a unified multi-dimensional evaluator for text generation](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/). In EMNLP.
46. Xiaoming Shi, Jie Xu, Jinru Ding, Jiali Pang, Sichen Liu, Shuqing Luo, Xingwei Peng, Lu Lu, Haihong Yang, Mingtao Hu, et al. 2023. Llm-mini-cex: Automatic evaluation of large language model for diagnostic conversation. arXiv preprint arXiv:2308.07635. — [arXiv:2308.07635](https://arxiv.org/abs/2308.07635) [find](https://scholar.google.com/scholar?q=Xiaoming+Shi%2C+Jie+Xu%2C+Jinru+Ding%2C+Jiali+Pang%2C+Sichen+Liu%2C+Shuqing+Luo%2C+Xingwei+Peng%2C+Lu+Lu%2C+Haihong+Yang%2C+Mingtao+Hu%2C+et+al%2E+2023%2E+Llm%2Dmini%2Dcex%3A+Automatic+evaluation+of+large+language+model+for+diagno)
47. Hwanjun Song, Igor Shalyminov, Hang Su, Singh Siffi, Kaisheng Yao, and Saab Mansour. 2023. Enhancing abstractiveness of summarization models through calibrated distillation. In EMNLP. [find](https://scholar.google.com/scholar?q=Hwanjun+Song%2C+Igor+Shalyminov%2C+Hang+Su%2C+Singh+Siffi%2C+Kaisheng+Yao%2C+and+Saab+Mansour%2E+2023%2E+Enhancing+abstractiveness+of+summarization+models+through+calibrated+distillation%2E+In+EMNLP%2E)
48. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
49. PA Utama, J Bambrick, NS Moosavi, and I Gurevych. 2022. Falsesum: generating document-level nli examples for recognizing factual inconsistency in summarization. In NAACL. [find](https://scholar.google.com/scholar?q=PA+Utama%2C+J+Bambrick%2C+NS+Moosavi%2C+and+I+Gurevych%2E+2022%2E+Falsesum%3A+generating+document%2Dlevel+nli+examples+for+recognizing+factual+inconsistency+in+summarization%2E+In+NAACL%2E)

*Updated 24 October 2026*