---
title: "Large Language Models are Not Yet Human-Level Evaluators for"
description: "As a stopgap, the authors propose checking how well RTS and MCQ agree before trusting LLM scores. 59 references, cited by 70 papers here."
authors: ["Chenhui Shen", "Liying Cheng", "Xuan-Phi Nguyen", "Yang You", "Lidong Bing"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: EMNLP 2023"
source: "https://aclanthology.org/2023.findings-emnlp.278/"
doi: "10.18653/v1/2023.findings-emnlp.278"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shen-2023-large-language-models-are-not/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shen-2023-large-language-models-are-not/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization

# Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization

**Authors:** [Chenhui Shen](https://md.research.elamur.ai/free/en/authors/chenhui-shen/), [Liying Cheng](https://md.research.elamur.ai/free/en/authors/liying-cheng/), [Xuan-Phi Nguyen](https://md.research.elamur.ai/free/en/authors/xuan-phi-nguyen/), [Yang You](https://md.research.elamur.ai/free/en/authors/yang-you/) and [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/) · **Year:** 2023 · **Venue:** [Findings of the Association for Computational Linguistics: EMNLP 2023](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/findings/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-emnlp.278/) · [PDF](https://aclanthology.org/2023.findings-emnlp.278.pdf) · [DOI 10.18653/v1/2023.findings-emnlp.278](https://doi.org/10.18653/v1/2023.findings-emnlp.278) · [OpenAlex](https://openalex.org/W4389520163)

**Citations:** 49 citations (OpenAlex, 2026-10-14) · [70](https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-models-are-not/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This paper asks whether LLMs such as ChatGPT and GPT-4 can take over from human annotators when judging abstractive summaries. Human evaluation is costly and slow, and n-gram or embedding metrics capture quality poorly. The authors test zero-shot LLM judges on the human-annotated SummEval benchmark, across the coherence, consistency, fluency and relevance dimensions. They use three prompting styles: reason-then-score (RTS), multiple-choice scoring (MCQ) and head-to-head (H2H) preference. They also introduce a meta-correlation measure that checks whether an evaluator's agreement with humans shifts as the evaluated systems get better. The LLM judges beat metrics like ROUGE and BARTScore, but they are not reliable replacements for people. Their human alignment varies by system and by dimension, they have trouble separating systems of similar quality, and their correlation with humans drops for stronger summarizers. This matters because newer, better systems may be misjudged. As a stopgap, the authors propose checking how well RTS and MCQ agree before trusting LLM scores.

## Key points
- **Data:** SummEval, with 12 abstractive systems and 100 summaries per system (1,200 in total). Each summary is scored on a 1–5 Likert scale, and the reference is the mean of three expert ratings.
- **Methods:** RTS (a short justification, then a score), MCQ (choosing option A–E, each with a fixed description) and H2H pairwise comparison. The main ChatGPT snapshot was gpt-3.5-turbo-0301, which has since been retired.
- **Main finding:** Meta-correlation is negative, meaning the LLM judges agree less with humans as summary quality improves.
- **Proposed check:** The per-system correlation between RTS and MCQ scores (R_i) acts as a reliability signal. It tracks human alignment for consistency, fluency and coherence, but it does not work for relevance.
- **Cost:** Evaluating 100 summaries with ChatGPT costs about 2 USD and takes about 2 minutes, compared with roughly 10 hours for human annotators. Code and LLM outputs are available at github.com/DAMO-NLP-SG/LLM_summeval.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 84 works · 32,204 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 64 works · 14,113 citations
- [Alibaba DAMO Academy](https://md.research.elamur.ai/free/en/companies/alibaba-damo-academy/) — Company, Hangzhou, China · 28 works · 6,115 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 159 works

## Cite

Cite as: Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You and Lidong Bing. 2023. Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shen2023large,
  title = {{Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization}},
  author = {Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You and Lidong Bing},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.278},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.278}
}
```

- [← previous Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering](https://md.research.elamur.ai/free/en/papers/baek-2023-knowledge-augmented-language-model-prompting/)
- [next → UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://md.research.elamur.ai/free/en/papers/masry-2023-unichart-universal-vision-language-pretrained/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

59 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lidong Bing, Piji Li, Yi Liao, Wai Lam, Weiwei Guo, and Rebecca Passonneau. 2015. Abstractive multi-document summarization via phrase selection and merging. In Proceedings of ACL-IJCNLP. [find](https://scholar.google.com/scholar?q=Lidong+Bing%2C+Piji+Li%2C+Yi+Liao%2C+Wai+Lam%2C+Weiwei+Guo%2C+and+Rebecca+Passonneau%2E+2015%2E+Abstractive+multi%2Ddocument+summarization+via+phrase+selection+and+merging%2E+In+Proceedings+of+ACL%2DIJCNLP%2E)
2. Moye Chen, Wei Li, Jiachen Liu, Xinyan Xiao, Hua Wu, and Haifeng Wang. 2021. Sgsum: Transforming multi-document summarization into sub-graph selection. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Moye+Chen%2C+Wei+Li%2C+Jiachen+Liu%2C+Xinyan+Xiao%2C+Hua+Wu%2C+and+Haifeng+Wang%2E+2021%2E+Sgsum%3A+Transforming+multi%2Ddocument+summarization+into+sub%2Dgraph+selection%2E+In+Proceedings+of+EMNLP%2E)
3. Liying Cheng, Xingxuan Li, and Lidong Bing. 2023. Is gpt-4 a good data analyst? arXiv preprint arXiv:2305.15038. [find](https://scholar.google.com/scholar?q=Liying+Cheng%2C+Xingxuan+Li%2C+and+Lidong+Bing%2E+2023%2E+Is+gpt%2D4+a+good+data+analyst%3F+arXiv+preprint+arXiv%3A2305%2E15038%2E)
4. Liying Cheng, Dekun Wu, Lidong Bing, Yan Zhang, Zhanming Jie, Wei Lu, and Luo Si. 2020. Ent-desc: Entity description generation by exploring knowledge graph. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Liying+Cheng%2C+Dekun+Wu%2C+Lidong+Bing%2C+Yan+Zhang%2C+Zhanming+Jie%2C+Wei+Lu%2C+and+Luo+Si%2E+2020%2E+Ent%2Ddesc%3A+Entity+description+generation+by+exploring+knowledge+graph%2E+In+Proceedings+of+EMNLP%2E)
5. Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloé Clavel. 2022. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In Proceedings of Coling. [find](https://scholar.google.com/scholar?q=Cyril+Chhun%2C+Pierre+Colombo%2C+Fabian+M%2E+Suchanek%2C+and+Chlo%C3%A9+Clavel%2E+2022%2E+Of+human+criteria+and+automatic+metrics%3A+A+benchmark+of+the+evaluation+of+story+generation%2E+In+Proceedings+of+Coling%2E)
6. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Proceedings+of+ACL%2E)
7. Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. Noise reduction in speech processing. [find](https://scholar.google.com/scholar?q=Israel+Cohen%2C+Yiteng+Huang%2C+Jingdong+Chen%2C+Jacob+Benesty%2C+Jacob+Benesty%2C+Jingdong+Chen%2C+Yiteng+Huang%2C+and+Israel+Cohen%2E+2009%2E+Pearson+correlation+coefficient%2E+Noise+reduction+in+speech+processing%2E)
8. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Banditsum: Extractive summarization as a contextual bandit. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Yue+Dong%2C+Yikang+Shen%2C+Eric+Crawford%2C+Herke+van+Hoof%2C+and+Jackie+Chi+Kit+Cheung%2E+2018%2E+Banditsum%3A+Extractive+summarization+as+a+contextual+bandit%2E+In+Proceedings+of+EMNLP%2E)
9. Ori Ernst, Avi Caciularu, Ori Shapira, Ramakanth Pasunuru, Mohit Bansal, Jacob Goldberger, and Ido Dagan. 2022. Proposition-level clustering for multi-document summarization. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Ori+Ernst%2C+Avi+Caciularu%2C+Ori+Shapira%2C+Ramakanth+Pasunuru%2C+Mohit+Bansal%2C+Jacob+Goldberger%2C+and+Ido+Dagan%2E+2022%2E+Proposition%2Dlevel+clustering+for+multi%2Ddocument+summarization%2E+In+Proceedings+of+NAACL%2E)
10. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of ACL.
11. Piji Li, Wai Lam, Lidong Bing, and Zihao Wang. 2017. Deep recurrent generative decoder for abstractive text summarization. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Piji+Li%2C+Wai+Lam%2C+Lidong+Bing%2C+and+Zihao+Wang%2E+2017%2E+Deep+recurrent+generative+decoder+for+abstractive+text+summarization%2E+In+Proceedings+of+EMNLP%2E)
12. Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. [Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model](https://md.research.elamur.ai/free/en/papers/fabbri-2019-multi-news-large-scale-multi/). In Proceedings of ACL.
13. Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020. Leveraging graph to improve abstractive multi-document summarization. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Wei+Li%2C+Xinyan+Xiao%2C+Jiachen+Liu%2C+Hua+Wu%2C+Haifeng+Wang%2C+and+Junping+Du%2E+2020%2E+Leveraging+graph+to+improve+abstractive+multi%2Ddocument+summarization%2E+In+Proceedings+of+ACL%2E)
14. Alexander R Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. TACL. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+TACL%2E)
15. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out.
16. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+In+Proceedings+of+ACL%2E)
17. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. [Generating wikipedia by summarizing long sequences](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/). In ICLR.
18. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+2022%2E+Pal%3A+Program%2Daided+language+models%2E+In+Proceedings+of+ICML%2E)
19. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv 2303.16634. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023%2E+G%2Deval%3A+Nlg+evaluation+using+gpt%2D4+with+better+human+alignment%2E+arXiv+2303%2E16634%2E)
20. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554. [find](https://scholar.google.com/scholar?q=Mingqi+Gao%2C+Jie+Ruan%2C+Renliang+Sun%2C+Xunjian+Yin%2C+Shiping+Yang%2C+and+Xiaojun+Wan%2E+2023%2E+Human%2Dlike+summarization+evaluation+with+chatgpt%2E+arXiv+preprint+arXiv%3A2304%2E02554%2E)
21. Shen Gao, Xiuying Chen, Piji Li, Zhaochun Ren, Lidong Bing, Dongyan Zhao, and Rui Yan. 2019. Abstractive text summarization by incorporating reader comments. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Shen+Gao%2C+Xiuying+Chen%2C+Piji+Li%2C+Zhaochun+Ren%2C+Lidong+Bing%2C+Dongyan+Zhao%2C+and+Rui+Yan%2E+2019%2E+Abstractive+text+summarization+by+incorporating+reader+comments%2E+In+Proceedings+of+AAAI%2E)
22. Jian Guan and Minlie Huang. 2020. UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Jian+Guan+and+Minlie+Huang%2E+2020%2E+UNION%3A+An+Unreferenced+Metric+for+Evaluating+Open%2Dended+Story+Generation%2E+In+Proceedings+of+EMNLP%2E)
23. Junxian He, Wojciech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2022. [CTRLsum: Towards generic controllable text summarization](https://md.research.elamur.ai/free/en/papers/he-2022-ctrlsum-generic-controllable-text-summarization/). In Proceedings of EMNLP.
24. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). NeurIPS.
25. MG Kendall. 1938. A new measure of rank correlation. Biometrika. [find](https://scholar.google.com/scholar?q=MG+Kendall%2E+1938%2E+A+new+measure+of+rank+correlation%2E+Biometrika%2E)
26. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). NeurIPS.
27. Wojciech Kryściński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Romain+Paulus%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2018%2E+Improving+abstraction+in+text+summarization%2E+In+Proceedings+of+EMNLP%2E)
28. Yang Liu and Mirella Lapata. 2019. [Hierarchical transformers for multi-document summarization](https://md.research.elamur.ai/free/en/papers/liu-2019-hierarchical-transformers-multi-document-summarization/). In Proceedings of ACL.
29. Yixin Liu, Ansong Ni, Linyong Nan, Budhaditya Deb, Chenguang Zhu, Ahmed H Awadallah, and Dragomir Radev. 2022. Leveraging locality in abstractive text summarization. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Yixin+Liu%2C+Ansong+Ni%2C+Linyong+Nan%2C+Budhaditya+Deb%2C+Chenguang+Zhu%2C+Ahmed+H+Awadallah%2C+and+Dragomir+Radev%2E+2022%2E+Leveraging+locality+in+abstractive+text+summarization%2E+In+Proceedings+of+EMNLP%2E)
30. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. [Chatgpt as a factual inconsistency evaluator for abstractive text summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/). arXiv preprint arXiv:2303.15621.
31. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. [Faithful chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/lyu-2023-faithful-chain-thought-reasoning/). arXiv preprint arXiv:2301.13379.
32. Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023a. Zero-shot listwise document reranking with a large language model. arXiv preprint arXiv:2305.02156. [find](https://scholar.google.com/scholar?q=Xueguang+Ma%2C+Xinyu+Zhang%2C+Ronak+Pradeep%2C+and+Jimmy+Lin%2E+2023a%2E+Zero%2Dshot+listwise+document+reranking+with+a+large+language+model%2E+arXiv+preprint+arXiv%3A2305%2E02156%2E)
33. Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023b. [Large language model is not a good few-shot information extractor, but a good reranker for hard samples!](https://md.research.elamur.ai/free/en/papers/ma-2023-large-language-model-is-not/) arXiv preprint arXiv:2303.08559.
34. Ani Nenkova. 2006. Summarization evaluation for text and speech: issues and approaches. In Ninth International Conference on Spoken Language Processing. [find](https://scholar.google.com/scholar?q=Ani+Nenkova%2E+2006%2E+Summarization+evaluation+for+text+and+speech%3A+issues+and+approaches%2E+In+Ninth+International+Conference+on+Spoken+Language+Processing%2E)
35. Ani Nenkova and Kathleen McKeown. 2012. A survey of text summarization techniques. Mining text data. [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Kathleen+McKeown%2E+2012%2E+A+survey+of+text+summarization+techniques%2E+Mining+text+data%2E)
36. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint.
37. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). NeurIPS.
38. Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. 2021. [Challenges in detoxifying language models](https://md.research.elamur.ai/free/en/papers/welbl-2021-challenges-detoxifying-language-models/). In Findings of EMNLP.
39. Karolina Owczarzak, John Conroy, Hoa Trang Dang, and Ani Nenkova. 2012. An assessment of the accuracy of automatic evaluation in summarization. In Proceedings of workshop on evaluation metrics and system comparison for automatic summarization. [find](https://scholar.google.com/scholar?q=Karolina+Owczarzak%2C+John+Conroy%2C+Hoa+Trang+Dang%2C+and+Ani+Nenkova%2E+2012%2E+An+assessment+of+the+accuracy+of+automatic+evaluation+in+summarization%2E+In+Proceedings+of+workshop+on+evaluation+metrics+and+sys)
40. Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang. 2023. Large language models are diverse role-players for summarization evaluation. arXiv preprint arXiv:2303.15078. [find](https://scholar.google.com/scholar?q=Ning+Wu%2C+Ming+Gong%2C+Linjun+Shou%2C+Shining+Liang%2C+and+Daxin+Jiang%2E+2023%2E+Large+language+models+are+diverse+role%2Dplayers+for+summarization+evaluation%2E+arXiv+preprint+arXiv%3A2303%2E15078%2E)
41. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. [Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics](https://md.research.elamur.ai/free/en/papers/pagnoni-2021-understanding-factuality-abstractive-summarization-frank/). In Proceedings of NAACL-HLT.
42. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of ACL.
43. Swarnadeep Saha, Shiyue Zhang, Peter Hase, and Mohit Bansal. 2022. Summarization programs: Interpretable abstractive summarization with neural modular trees. In ICLR. [find](https://scholar.google.com/scholar?q=Swarnadeep+Saha%2C+Shiyue+Zhang%2C+Peter+Hase%2C+and+Mohit+Bansal%2E+2022%2E+Summarization+programs%3A+Interpretable+abstractive+summarization+with+neural+modular+trees%2E+In+ICLR%2E)
44. Chenhui Shen, Liying Cheng, Lidong Bing, Yang You, and Luo Si. 2022a. Sentbs: Sentence-level beam search for controllable summarization. EMNLP. [find](https://scholar.google.com/scholar?q=Chenhui+Shen%2C+Liying+Cheng%2C+Lidong+Bing%2C+Yang+You%2C+and+Luo+Si%2E+2022a%2E+Sentbs%3A+Sentence%2Dlevel+beam+search+for+controllable+summarization%2E+EMNLP%2E)
45. Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Lidong Bing, and Yang You. 2023. A hierarchical encoding-decoding scheme for abstractive multi-document summarization. In Findings of EMNLP. [find](https://scholar.google.com/scholar?q=Chenhui+Shen%2C+Liying+Cheng%2C+Xuan%2DPhi+Nguyen%2C+Lidong+Bing%2C+and+Yang+You%2E+2023%2E+A+hierarchical+encoding%2Ddecoding+scheme+for+abstractive+multi%2Ddocument+summarization%2E+In+Findings+of+EMNLP%2E)
46. Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. [Primera: Pyramid-based masked sentence pre-training for multi-document summarization](https://md.research.elamur.ai/free/en/papers/xiao-2022-primera-pyramid-based-masked-sentence/). In Proceedings of ACL.
47. Jiacheng Xu and Greg Durrett. 2021. Dissecting generation modes for abstractive summarization models via ablation and attribution. In Proceedings of ACL-IJCNLP. [find](https://scholar.google.com/scholar?q=Jiacheng+Xu+and+Greg+Durrett%2E+2021%2E+Dissecting+generation+modes+for+abstractive+summarization+models+via+ablation+and+attribution%2E+In+Proceedings+of+ACL%2DIJCNLP%2E)
48. Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Mingfeng Xue, Boxing Chen, and Jun Xie. 2023. Tailor: A soft-prompt-based approach to attribute-based controlled text generation. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Kexin+Yang%2C+Dayiheng+Liu%2C+Wenqiang+Lei%2C+Baosong+Yang%2C+Mingfeng+Xue%2C+Boxing+Chen%2C+and+Jun+Xie%2E+2023%2E+Tailor%3A+A+soft%2Dprompt%2Dbased+approach+to+attribute%2Dbased+controlled+text+generation%2E+In+Proceedings+o)
49. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). NeurIPS.
50. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020a. [Pegasus: Pre-training with extracted gap-sentences for abstractive summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/). In Proceedings of ICML.
51. Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2022b. Mred: A meta-review dataset for structure-controllable text generation. Findings of ACL. [find](https://scholar.google.com/scholar?q=Chenhui+Shen%2C+Liying+Cheng%2C+Ran+Zhou%2C+Lidong+Bing%2C+Yang+You%2C+and+Luo+Si%2E+2022b%2E+Mred%3A+A+meta%2Dreview+dataset+for+structure%2Dcontrollable+text+generation%2E+Findings+of+ACL%2E)
52. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020b. Bertscore: Evaluating text generation with bert. In ICLR. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2020b%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+ICLR%2E)
53. C Spearman. 1987. The proof and measurement of association between two things. American Journal of Psychology. [find](https://scholar.google.com/scholar?q=C+Spearman%2E+1987%2E+The+proof+and+measurement+of+association+between+two+things%2E+American+Journal+of+Psychology%2E)
54. Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. [Neural document summarization by jointly learning to score and select sentences](https://md.research.elamur.ai/free/en/papers/zhou-2018-neural-document-summarization-jointly-learning/). In Proceedings of ACL.
55. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
56. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2019%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+ar)
57. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). arXiv preprint arXiv:2303.04048.
58. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2023b. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In ICLR.
59. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). NeurIPS.

*Updated 24 October 2026*