--- title: "Large Language Models are Diverse Role-Players for" description: "The authors propose DRPE (Diverse Role-Player for Summarization Generation Evaluation), which… 2 references, cited by 35 papers here." authors: ["Ning Wu", "Ming Gong", "Linjun Shou", "Shining Liang", "Daxin Jiang"] year: 2023 venue: "arXiv:2303.15078" source: "https://arxiv.org/abs/2303.15078" doi: "10.48550/arxiv.2303.15078" arxiv: "2303.15078" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wu-2023-large-language-models-diverse-role/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2023-large-language-models-diverse-role/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Large Language Models are Diverse Role-Players for Summarization Evaluation # Large Language Models are Diverse Role-Players for Summarization Evaluation **Authors:** [Ning Wu](https://md.research.elamur.ai/free/en/authors/ning-wu/), [Ming Gong](https://md.research.elamur.ai/free/en/authors/ming-gong/), [Linjun Shou](https://md.research.elamur.ai/free/en/authors/linjun-shou/), Shining Liang and [Daxin Jiang](https://md.research.elamur.ai/free/en/authors/daxin-jiang/) · **Year:** 2023 · **Venue:** [arXiv:2303.15078](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/) ([Lecture notes in computer science 2023](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/2023/)) · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2303.15078) · [PDF](https://arxiv.org/pdf/2303.15078) · [DOI 10.48550/arxiv.2303.15078](https://doi.org/10.48550/arxiv.2303.15078) · [arXiv 2303.15078](https://arxiv.org/abs/2303.15078) · [OpenAlex](https://openalex.org/W4387428151) **Citations:** 51 citations (OpenAlex, 2026-10-14) · [35](https://md.research.elamur.ai/free/en/papers/wu-2023-large-language-models-diverse-role/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Metrics such as BLEU, ROUGE and BERTScore measure surface or embedding similarity to a reference. They often diverge from human judgement, especially for fluent LLM-written summaries. Two outputs can score alike on these metrics yet differ clearly in coherence or interest to readers. The authors propose DRPE (Diverse Role-Player for Summarization Generation Evaluation), which asks an LLM to compare two candidate summaries from several viewpoints. Objective qualities such as fluency, consistency and grammar are handled by manually written "judger" roles. Subjective qualities such as interestingness and usefulness are handled by reader personas that the LLM generates from each article. Similar roles are merged by clustering. All roles then vote within one batched prompt, and an aggregator combines the votes into a final verdict. On three human-annotated summarization datasets, DRPE agrees with human annotators better than zero-shot LLM prompting and standard metrics. This suggests that simulating varied audiences is a practical route to more human-like automatic evaluation. ## Key points - Evaluation is pairwise: the LLM picks the better of two summaries rather than assigning an absolute score, since absolute scores from LLMs tend to be unstable. - Three static roles are defined for summarization: - General Public - Critic, who checks wording and clarity - News Author, who checks consistency with the source - Dynamic reader profiles are produced with two prompt variants: - Coarse-grained prompting groups readers by intent. - Fine-grained prompting groups readers by how familiar they are with the event. - Experiments use CNN2022, SummEval and BBC2022. In the ablation table, the average result drops from 0.761 (full DRPE) to 0.683 when static roles are removed. Removing dynamic roles, clustering or batching also lowers performance. - About four subjective roles work best; using six brings no further gain. Batch prompting greatly reduces inference tokens with little loss in quality. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 325 works · 142,931 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 232 works ## Cite Cite as: Ning Wu, Ming Gong, Linjun Shou, Shining Liang and Daxin Jiang. 2023. Large Language Models are Diverse Role-Players for Summarization Evaluation. arXiv:2303.15078. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{wu2023large, title = {{Large Language Models are Diverse Role-Players for Summarization Evaluation}}, author = {Ning Wu, Ming Gong, Linjun Shou, Shining Liang and Daxin Jiang}, year = {2023}, doi = {10.48550/arxiv.2303.15078}, eprint = {2303.15078}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2303.15078} } ``` - [← previous A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://md.research.elamur.ai/free/en/papers/he-2023-survey-large-language-models-healthcare/) - [next → Mathematics, word problems, common sense, and artificial intelligence](https://md.research.elamur.ai/free/en/papers/davis-2023-mathematics-word-problems-common-sense/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) ## References 2 references, 2 of them with a card of ours. ↗ leads to the work. 1. 1. Banerjee, S., Lavie, A.: [Meteor: An automatic metric for mt evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. pp. 65–72 (2005) 2. N. Wu et al. 2. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems 33, 1877–1901 (2020) 3. Fabbri, A.R., Kryściński, W., McCann… *Updated 31 October 2026*