---
title: "Evaluating Professional Acceptability of LLM-Generated"
description: "They then used it to create PsycSumEval, a dataset of 460 summaries that four LLMs produced for 115 mental-health… 51 references."
authors: ["Paul Thompson", "Artemis Boulogeorgou", "Fotini Kaponi", "Efstathia Soufleri", "Sophia Ananiadou"]
year: 2026
venue: "Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026"
source: "https://aclanthology.org/2026.cl4health-1.18/"
doi: "10.63317/4exjqeuaw9ti"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/thompson-2026-evaluating-professional-acceptability-llm-generated/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/thompson-2026-evaluating-professional-acceptability-llm-generated/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / Evaluating Professional Acceptability of LLM-Generated Systematic Review Summaries in Healthcare: Psychiatrists' Perspectives

# Evaluating Professional Acceptability of LLM-Generated Systematic Review Summaries in Healthcare: Psychiatrists' Perspectives

**Authors:** Paul Thompson, Artemis Boulogeorgou, Fotini Kaponi, [Efstathia Soufleri](https://md.research.elamur.ai/free/en/authors/efstathia-soufleri/) and [Sophia Ananiadou](https://md.research.elamur.ai/free/en/authors/sophia-ananiadou/) · **Year:** 2026 · **Venue:** [Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/cl4health/) ([Workshop on Patient-Oriented Language Processing @ LREC 2026](https://md.research.elamur.ai/free/en/venues/cl4health/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.cl4health-1.18/) · [PDF](https://aclanthology.org/2026.cl4health-1.18.pdf) · [DOI 10.63317/4exjqeuaw9ti](https://doi.org/10.63317/4exjqeuaw9ti) · [OpenAlex](https://openalex.org/W7166515177)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Cochrane systematic review abstracts have grown long (often approaching 1000 words), so busy clinicians struggle to appraise them quickly. The authors ask whether LLMs can shorten these abstracts while keeping them accurate and acceptable to professionals. They built an evaluation framework based on PRISMA-A and Cochrane handbook requirements. They then used it to create PsycSumEval, a dataset of 460 summaries that four LLMs produced for 115 mental-health Cochrane abstracts. Two psychiatrists rated every summary on nine content dimensions. For each dimension they gave an ordinal score. When a score fell short, they also noted the fix needed (add, remove or modify), marked the faulty summary sentences, linked them to the source abstract, and wrote a free-text rationale. The resource shows how experts judge compressed medical evidence and where those judgments agree or diverge. Its structured scores and rationales could help train explainable LLM judges. This matters because standard overlap metrics and current LLM-as-judge methods fail to reliably catch clinically important errors in medical summaries.

## Key points
- **PsycSumEval**: 115 Cochrane mental-health abstracts × 4 LLMs = 460 summaries. GPT-4o is among the models, as shown in the worked example. Two psychiatrists evaluated the summaries.
- **Nine dimensions**: the four PICO elements (Population, Intervention, Comparator, Outcome), plus Methodology, Main Findings and Conclusions, among others. All are derived from PRISMA-A and Cochrane guidelines.
- **Annotation per dimension**: up to four pieces of information, namely
  - an ordinal score,
  - a corrective action (add/remove/modify),
  - the problematic sentences, traced to the source abstract,
  - a rationale.
- **Motivation**: the median length of Cochrane abstracts rose from 353 words (2000) to 838 words (2022) after PRISMA-A was introduced. Metrics such as ROUGE, BLEU, METEOR and BERTScore cannot verify compliance with clinically detailed criteria.
- **Example of errors**: in the GPT-4o summary, typical problems were omitting the number of included studies and adding future-work recommendations that the source abstract does not support.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Manchester](https://md.research.elamur.ai/free/en/universities/university-of-manchester/) — University, Manchester, United Kingdom · 87 works · 44,938 citations

📚 Shelves (6): [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 234 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 231 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 209 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 241 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 184 works · [🎯 Statistics, Probability and Uncertainty](https://md.research.elamur.ai/free/en/shelves/decision-sciences/statistics-probability-and-uncertainty/) · 61 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Paul Thompson, Artemis Boulogeorgou, Fotini Kaponi, Efstathia Soufleri and Sophia Ananiadou. 2026. Evaluating Professional Acceptability of LLM-Generated Systematic Review Summaries in Healthcare: Psychiatrists' Perspectives. Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{thompson2026evaluating,
  title = {{Evaluating Professional Acceptability of LLM-Generated Systematic Review Summaries in Healthcare: Psychiatrists' Perspectives}},
  author = {Paul Thompson, Artemis Boulogeorgou, Fotini Kaponi, Efstathia Soufleri and Sophia Ananiadou},
  booktitle = {Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026},
  year = {2026},
  doi = {10.63317/4exjqeuaw9ti},
  url = {https://doi.org/10.63317/4exjqeuaw9ti}
}
```

- [← previous Evaluating Perspectival Biases in Cross-Modal Retrieval](https://md.research.elamur.ai/free/en/papers/saengsukhiran-2026-evaluating-perspectival-biases-cross-modal/)
- [next → Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation](https://md.research.elamur.ai/free/en/papers/arcan-2026-evaluating-prompt-scope-demonstration-similarity/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/)

## References

51 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gerd Antes, David Tovey, and Nancy Owens. 2022. Cochrane: Trusted evidence. informed decisions. better health. Systematic Reviews in Health Research: Meta-Analysis in Context, pages 413–423. [find](https://scholar.google.com/scholar?q=Gerd+Antes%2C+David+Tovey%2C+and+Nancy+Owens%2E+2022%2E+Cochrane%3A+Trusted+evidence%2E+informed+decisions%2E+better+health%2E+Systematic+Reviews+in+Health+Research%3A+Meta%2DAnalysis+in+Context%2C+pages+413%E2%80%93423%2E)
2. Felipe Arias-Russi, Carolina Salazar-Lara, and Rubén Manrique. 2025a. Bridging the gap in health literacy: Harnessing the power of large language models to generate plain language summaries from biomedical texts. In Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Healt… [find](https://scholar.google.com/scholar?q=Felipe+Arias%2DRussi%2C+Carolina+Salazar%2DLara%2C+and+Rub%C3%A9n+Manrique%2E+2025a%2E+Bridging+the+gap+in+health+literacy%3A+Harnessing+the+power+of+large+language+models+to+generate+plain+language+summaries+from+biome)
3. Felipe Arias-Russi, Carolina Salazar Lara, and Ruben Manrique. 2025b. A multi-agent framework with diagnostic feedback for iterative plain language summary generation from cochrane medical abstracts. In Proceedings of the Fourth Workshop on Text Simplification, Accessibility and Readability (TSAR 20… [find](https://scholar.google.com/scholar?q=Felipe+Arias%2DRussi%2C+Carolina+Salazar+Lara%2C+and+Ruben+Manrique%2E+2025b%2E+A+multi%2Dagent+framework+with+diagnostic+feedback+for+iterative+plain+language+summary+generation+from+cochrane+medical+abstracts%2E+)
4. Howard Balshem, Mark Helfand, Holger J. Schünemann, Andrew D. Oxman, Regina Kunz, Jan Brozek, Gunn E. Vist, Yngve Falck-Ytter, Joerg Meerpohl, Susan Norris, and Gordon H. Guyatt. 2011. GRADE guidelines: 3. Rating the quality of evidence. Journal of Clinical Epidemiology, 64(4):401–406. [find](https://scholar.google.com/scholar?q=Howard+Balshem%2C+Mark+Helfand%2C+Holger+J%2E+Sch%C3%BCnemann%2C+Andrew+D%2E+Oxman%2C+Regina+Kunz%2C+Jan+Brozek%2C+Gunn+E%2E+Vist%2C+Yngve+Falck%2DYtter%2C+Joerg+Meerpohl%2C+Susan+Norris%2C+and+Gordon+H%2E+Guyatt%2E+2011%2E+GRADE+guideline)
5. Lydie Bednarczyk, Daniel Reichenpfader, Christophe Gaudet-Blavignac, Amon Kenna Ette, Jamil Zaghir, Yuanyuan Zheng, Adel Bensahla, Mina Bjelogrlic, and Christian Lovis. 2025. Scientific evidence for clinical text summarization using large language models: scoping review. Journal of Medical Internet… [find](https://scholar.google.com/scholar?q=Lydie+Bednarczyk%2C+Daniel+Reichenpfader%2C+Christophe+Gaudet%2DBlavignac%2C+Amon+Kenna+Ette%2C+Jamil+Zaghir%2C+Yuanyuan+Zheng%2C+Adel+Bensahla%2C+Mina+Bjelogrlic%2C+and+Christian+Lovis%2E+2025%2E+Scientific+evidence+for+c)
6. Elaine M Beller, Paul P Glasziou, Douglas G Altman, Sally Hopewell, Hilda Bastian, Iain Chalmers, Peter C Gøtzsche, Toby Lasserson, David Tovey, and PRISMA for Abstracts Group. 2013. PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS medicine, 10(4):e1001419… [find](https://scholar.google.com/scholar?q=Elaine+M+Beller%2C+Paul+P+Glasziou%2C+Douglas+G+Altman%2C+Sally+Hopewell%2C+Hilda+Bastian%2C+Iain+Chalmers%2C+Peter+C+G%C3%B8tzsche%2C+Toby+Lasserson%2C+David+Tovey%2C+and+PRISMA+for+Abstracts+Group%2E+2013%2E+PRISMA+for+abstra)
7. Elaine M Beller, Paul P Glasziou, Sally Hopewell, and Douglas G Altman. 2011. Reporting of effect direction and size in abstracts of systematic reviews. Jama, 306(18):1981–1982. [find](https://scholar.google.com/scholar?q=Elaine+M+Beller%2C+Paul+P+Glasziou%2C+Sally+Hopewell%2C+and+Douglas+G+Altman%2E+2011%2E+Reporting+of+effect+direction+and+size+in+abstracts+of+systematic+reviews%2E+Jama%2C+306%2818%29%3A1981%E2%80%931982%2E)
8. BE Belsher, E Beech, D Evatt, DJ Smolenski, MT Shea, JL Otto, CS Rosen, and PP Schnurr. Present‐centered therapy (pct) for 2019. post‐traumatic stress disorder (ptsd) in adults. Cochrane Database of Systematic Reviews, (11). [find](https://scholar.google.com/scholar?q=BE+Belsher%2C+E+Beech%2C+D+Evatt%2C+DJ+Smolenski%2C+MT+Shea%2C+JL+Otto%2C+CS+Rosen%2C+and+PP+Schnurr%2E+Present%E2%80%90centered+therapy+%28pct%29+for+2019%2E+post%E2%80%90traumatic+stress+disorder+%28ptsd%29+in+adults%2E+Cochrane+Database+of+S)
9. Iain Chalmers and Douglas G Altman. 1995. Systematic reviews. BMJ Publishing London. [find](https://scholar.google.com/scholar?q=Iain+Chalmers+and+Douglas+G+Altman%2E+1995%2E+Systematic+reviews%2E+BMJ+Publishing+London%2E)
10. Bohao Chu, Meijie Li, Sameh Frihat, Chengyu Gu, Georg Lodde, Elisabeth Livingstone, and Norbert Fuhr. 2025. TracSum: A new benchmark for aspect-based summarization with sentence-level traceability in medical domain. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Proce… [find](https://scholar.google.com/scholar?q=Bohao+Chu%2C+Meijie+Li%2C+Sameh+Frihat%2C+Chengyu+Gu%2C+Georg+Lodde%2C+Elisabeth+Livingstone%2C+and+Norbert+Fuhr%2E+2025%2E+TracSum%3A+A+new+benchmark+for+aspect%2Dbased+summarization+with+sentence%2Dlevel+traceability+in+)
11. Declan Devane, Johanna Pope, Paula Byrne, Evan Forde, Isabel O’Byrne, Steven Woloshin, Eileen Culloty, Darren Dahly, Ingeborg Hess Elgersma, Heather Munthe-Kaas, Conor Judge, Martin O’Donnell, Finn Krewer, Sandra Galvin, Nikita N. Burke, Theresa Tierney, KM Saif-Ur-Rahman, Tom Conway, and James Thom… [find](https://scholar.google.com/scholar?q=Declan+Devane%2C+Johanna+Pope%2C+Paula+Byrne%2C+Evan+Forde%2C+Isabel+O%E2%80%99Byrne%2C+Steven+Woloshin%2C+Eileen+Culloty%2C+Darren+Dahly%2C+Ingeborg+Hess+Elgersma%2C+Heather+Munthe%2DKaas%2C+Conor+Judge%2C+Martin+O%E2%80%99Donnell%2C+Finn+Kr)
12. Kay Dickersin, Roberta Scherer, and Carol Lefebvre. 1994. Identifying relevant studies for systematic reviews. BMJ, 309(6964):1286–1291. [find](https://scholar.google.com/scholar?q=Kay+Dickersin%2C+Roberta+Scherer%2C+and+Carol+Lefebvre%2E+1994%2E+Identifying+relevant+studies+for+systematic+reviews%2E+BMJ%2C+309%286964%29%3A1286%E2%80%931291%2E)
13. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2024%2E+Length%2Dcontrolled+AlpacaEval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2E)
14. Zelalem Gero, Chandan Singh, Yiqing Xie, Sheng Zhang, Praveen Subramanian, Paul Vozila, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2024. Attribute structuring improves LLM-based evaluation of clinical text summaries. arXiv preprint arXiv:2403.01002. [find](https://scholar.google.com/scholar?q=Zelalem+Gero%2C+Chandan+Singh%2C+Yiqing+Xie%2C+Sheng+Zhang%2C+Praveen+Subramanian%2C+Paul+Vozila%2C+Tristan+Naumann%2C+Jianfeng+Gao%2C+and+Hoifung+Poon%2E+2024%2E+Attribute+structuring+improves+LLM%2Dbased+evaluation+of+cl)
15. Jasmin Helbach, Falk Hoffmann, Dawid Pieper, and Katharina Allers. 2023a. Reporting according to the preferred reporting items for systematic reviews and meta-analyses for abstracts (PRISMA-A) depends on abstract length. Journal of Clinical Epidemiology, 154:167–177. [find](https://scholar.google.com/scholar?q=Jasmin+Helbach%2C+Falk+Hoffmann%2C+Dawid+Pieper%2C+and+Katharina+Allers%2E+2023a%2E+Reporting+according+to+the+preferred+reporting+items+for+systematic+reviews+and+meta%2Danalyses+for+abstracts+%28PRISMA%2DA%29+depends)
16. Jasmin Helbach, Kathrin Wandscher, Dawid Pieper, and Falk Hoffmann. 2023b. Cochrane review abstracts are getting longer, but this has no large impact on the reporting quality. Journal of Evidence-Based Medicine, 16(3):294–302. [find](https://scholar.google.com/scholar?q=Jasmin+Helbach%2C+Kathrin+Wandscher%2C+Dawid+Pieper%2C+and+Falk+Hoffmann%2E+2023b%2E+Cochrane+review+abstracts+are+getting+longer%2C+but+this+has+no+large+impact+on+the+reporting+quality%2E+Journal+of+Evidence%2DBase)
17. JPT Higgins, J Thomas, J Chandler, M Cumpston, T Li, MJ Page, and VA Welch. 2024. Cochrane Handbook for Systematic Reviews of Interventions version 6.5 (updated August 2024). Cochrane. [find](https://scholar.google.com/scholar?q=JPT+Higgins%2C+J+Thomas%2C+J+Chandler%2C+M+Cumpston%2C+T+Li%2C+MJ+Page%2C+and+VA+Welch%2E+2024%2E+Cochrane+Handbook+for+Systematic+Reviews+of+Interventions+version+6%2E5+%28updated+August+2024%29%2E+Cochrane%2E)
18. Zhenyu Huang, Xianlai Chen, Yunbo Wang, Jincai Huang, and Xing Zhao. 2025. A survey on biomedical automatic text summarization with large language models. Information Processing & Management, 62(5):104216. [find](https://scholar.google.com/scholar?q=Zhenyu+Huang%2C+Xianlai+Chen%2C+Yunbo+Wang%2C+Jincai+Huang%2C+and+Xing+Zhao%2E+2025%2E+A+survey+on+biomedical+automatic+text+summarization+with+large+language+models%2E+Information+Processing+%26+Management%2C+62%285%29%3A10)
19. Sagar Jose, Khanh T.P Nguyen, Kamal Medjaher, Ryad Zemouri, Mélanie Lévesque, and Antoine Tahan. 2024. Advancing multimodal diagnostics: Integrating industrial textual data and domain knowledge with large language models. Expert Systems with Applications, 255:124603. [find](https://scholar.google.com/scholar?q=Sagar+Jose%2C+Khanh+T%2EP+Nguyen%2C+Kamal+Medjaher%2C+Ryad+Zemouri%2C+M%C3%A9lanie+L%C3%A9vesque%2C+and+Antoine+Tahan%2E+2024%2E+Advancing+multimodal+diagnostics%3A+Integrating+industrial+textual+data+and+domain+knowledge+with+l)
20. Sebastian Joseph, Lily Chen, Jan Trienes, Hannah Göke, Monika Coers, Wei Xu, Byron Wallace, and Junyi Jessy Li. 2024. FactPICO: Factuality evaluation for plain language summarization of medical evidence. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu… [find](https://scholar.google.com/scholar?q=Sebastian+Joseph%2C+Lily+Chen%2C+Jan+Trienes%2C+Hannah+G%C3%B6ke%2C+Monika+Coers%2C+Wei+Xu%2C+Byron+Wallace%2C+and+Junyi+Jessy+Li%2E+2024%2E+FactPICO%3A+Factuality+evaluation+for+plain+language+summarization+of+medical+eviden)
21. Hita Kambhamettu, Jamie Flores, and Andrew Head. 2025. Traceable texts and their effects: A study of summary-source links in ai-generated summaries. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–7. [find](https://scholar.google.com/scholar?q=Hita+Kambhamettu%2C+Jamie+Flores%2C+and+Andrew+Head%2E+2025%2E+Traceable+texts+and+their+effects%3A+A+study+of+summary%2Dsource+links+in+ai%2Dgenerated+summaries%2E+In+Proceedings+of+the+Extended+Abstracts+of+the+CHI)
22. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). In The Twelfth International Conference on Learning Representations.
23. Mahira Kirmani, Gagandeep Kour, Mudasir Mohd, Nasrullah Sheikh, Dawood Ashraf Khan, Zahid Maqbool, Mohsin Altaf Wani, and Abid Hussain Wani. 2024. Biomedical semantic text summarizer. BMC Bioinformatics, 25(1):152. [find](https://scholar.google.com/scholar?q=Mahira+Kirmani%2C+Gagandeep+Kour%2C+Mudasir+Mohd%2C+Nasrullah+Sheikh%2C+Dawood+Ashraf+Khan%2C+Zahid+Maqbool%2C+Mohsin+Altaf+Wani%2C+and+Abid+Hussain+Wani%2E+2024%2E+Biomedical+semantic+text+summarizer%2E+BMC+Bioinformati)
24. Alon Lavie and Abhaya Agarwal. 2007. ME-TEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228– 231, Prague, Czech Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alon+Lavie+and+Abhaya+Agarwal%2E+2007%2E+ME%2DTEOR%3A+An+automatic+metric+for+MT+evaluation+with+high+levels+of+correlation+with+human+judgments%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+T)
25. Alessandro Liberati, Douglas G Altman, Jennifer Tetzlaff, Cynthia Mulrow, Peter C Gøtzsche, John PA Ioannidis, Mike Clarke, Philip J Devereaux, Jos Kleijnen, and David Moher. 2009. [find](https://scholar.google.com/scholar?q=Alessandro+Liberati%2C+Douglas+G+Altman%2C+Jennifer+Tetzlaff%2C+Cynthia+Mulrow%2C+Peter+C+G%C3%B8tzsche%2C+John+PA+Ioannidis%2C+Mike+Clarke%2C+Philip+J+Devereaux%2C+Jos+Kleijnen%2C+and+David+Moher%2E+2009%2E)
26. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ, 339. [find](https://scholar.google.com/scholar?q=The+PRISMA+statement+for+reporting+systematic+reviews+and+meta%2Danalyses+of+studies+that+evaluate+healthcare+interventions%3A+explanation+and+elaboration%2E+BMJ%2C+339%2E)
27. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
28. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2024. Factual consistency evaluation of summarization in the era of large language models. Expert Systems with Applications, 254:124456. [find](https://scholar.google.com/scholar?q=Zheheng+Luo%2C+Qianqian+Xie%2C+and+Sophia+Ananiadou%2E+2024%2E+Factual+consistency+evaluation+of+summarization+in+the+era+of+large+language+models%2E+Expert+Systems+with+Applications%2C+254%3A124456%2E)
29. Katarina Maticic, Marina Krnic Martinic, and Livia Puljak. 2019. Assessment of reporting quality of abstracts of systematic reviews with meta-analysis using PRISMA-A and discordance in assessments between raters without prior experience. BMC Medical Research Methodology, 19(1):32. [find](https://scholar.google.com/scholar?q=Katarina+Maticic%2C+Marina+Krnic+Martinic%2C+and+Livia+Puljak%2E+2019%2E+Assessment+of+reporting+quality+of+abstracts+of+systematic+reviews+with+meta%2Danalysis+using+PRISMA%2DA+and+discordance+in+assessments+bet)
30. Sally Morton, Alfred Berg, Laura Levit, and Jill Eden. 2011. Finding what works in health care: standards for systematic reviews. National Academies Press. [find](https://scholar.google.com/scholar?q=Sally+Morton%2C+Alfred+Berg%2C+Laura+Levit%2C+and+Jill+Eden%2E+2011%2E+Finding+what+works+in+health+care%3A+standards+for+systematic+reviews%2E+National+Academies+Press%2E)
31. Ian G Needleman. 2002. A guide to systematic reviews. Journal of clinical periodontology, 29:6– 9. [find](https://scholar.google.com/scholar?q=Ian+G+Needleman%2E+2002%2E+A+guide+to+systematic+reviews%2E+Journal+of+clinical+periodontology%2C+29%3A6%E2%80%93+9%2E)
32. Matthew J Page, Joanne E McKenzie, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, et al. 2021a. [find](https://scholar.google.com/scholar?q=Matthew+J+Page%2C+Joanne+E+McKenzie%2C+Patrick+M+Bossuyt%2C+Isabelle+Boutron%2C+Tammy+C+Hoffmann%2C+Cynthia+D+Mulrow%2C+Larissa+Shamseer%2C+Jennifer+M+Tetzlaff%2C+Elie+A+Akl%2C+Sue+E+Brennan%2C+et+al%2E+2021a%2E)
33. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372. [find](https://scholar.google.com/scholar?q=The+PRISMA+2020+statement%3A+an+updated+guideline+for+reporting+systematic+reviews%2E+BMJ%2C+372%2E)
34. W Scott Richardson, Mark C Wilson, Jim Nishikawa, and Robert S Hayward. 1995. The well-built clinical question: a key to evidence-based decisions. ACP Journal Club, 123(3):A12–13. [find](https://scholar.google.com/scholar?q=W+Scott+Richardson%2C+Mark+C+Wilson%2C+Jim+Nishikawa%2C+and+Robert+S+Hayward%2E+1995%2E+The+well%2Dbuilt+clinical+question%3A+a+key+to+evidence%2Dbased+decisions%2E+ACP+Journal+Club%2C+123%283%29%3AA12%E2%80%9313%2E)
35. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. [Self-critiquing models for assisting human evaluators](https://md.research.elamur.ai/free/en/papers/saunders-2022-self-critiquing-models-assisting-human/). arXiv preprint arXiv:2206.05802.
36. Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F Rousseau, et al. 2023. Evaluating large language models on medical evidence summarization. NPJ Digital Medicine, 6(1):158. [find](https://scholar.google.com/scholar?q=Liyan+Tang%2C+Zhaoyi+Sun%2C+Betina+Idnay%2C+Jordan+G+Nestor%2C+Ali+Soroush%2C+Pierre+A+Elias%2C+Ziyang+Xu%2C+Ying+Ding%2C+Greg+Durrett%2C+Justin+F+Rousseau%2C+et+al%2E+2023%2E+Evaluating+large+language+models+on+medical+evid)
37. Amy Y Tsou and Jonathan R Treadwell. 2016. Quality and clarity in systematic review abstracts: an empirical study. Research synthesis methods, 7(4):447–458. [find](https://scholar.google.com/scholar?q=Amy+Y+Tsou+and+Jonathan+R+Treadwell%2E+2016%2E+Quality+and+clarity+in+systematic+review+abstracts%3A+an+empirical+study%2E+Research+synthesis+methods%2C+7%284%29%3A447%E2%80%93458%2E)
38. Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, et al. 2024. [Adapted large language models can outperform medical experts in clinical text summarization](https://md.research.elamur.ai/free/en/papers/veen-2023-adapted-large-language-models-outperform/). Nature Medicine,…
39. Yuriy Vasilev, Irina Raznitsyna, Anastasia Pamova, Tikhon Burtsev, Tatiana Bobrovskaya, Pavel Kosov, Anton Vladzymyrskyy, Olga Omelyan-skaya, and Kirill Arzamasov. 2025. Evaluating medical text summaries using automatic evaluation metrics and LLM-as-a-judge approach: A pilot study. Diagnostics, 16(1… [find](https://scholar.google.com/scholar?q=Yuriy+Vasilev%2C+Irina+Raznitsyna%2C+Anastasia+Pamova%2C+Tikhon+Burtsev%2C+Tatiana+Bobrovskaya%2C+Pavel+Kosov%2C+Anton+Vladzymyrskyy%2C+Olga+Omelyan%2Dskaya%2C+and+Kirill+Arzamasov%2E+2025%2E+Evaluating+medical+text+summar)
40. Matthew J Page, David Moher, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, et al. 2021b. [find](https://scholar.google.com/scholar?q=Matthew+J+Page%2C+David+Moher%2C+Patrick+M+Bossuyt%2C+Isabelle+Boutron%2C+Tammy+C+Hoffmann%2C+Cynthia+D+Mulrow%2C+Larissa+Shamseer%2C+Jennifer+M+Tetzlaff%2C+Elie+A+Akl%2C+Sue+E+Brennan%2C+et+al%2E+2021b%2E)
41. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, 372. [find](https://scholar.google.com/scholar?q=PRISMA+2020+explanation+and+elaboration%3A+updated+guidance+and+exemplars+for+reporting+systematic+reviews%2E+BMJ%2C+372%2E)
42. Yuriy Vasilev, Irina Raznitsyna, Anastasia Pamova, Tikhon Burtsev, Tatiana Bobrovskaya, Pavel Kosov, Anton Vladzymyrskyy, Olga Omelyan-skaya, and Kirill Arzamasov. 2026. Evaluating medical text summaries using automatic evaluation metrics and llm-as-a-judge approach: A pilot study. Diagnostics, 16(1… [find](https://scholar.google.com/scholar?q=Yuriy+Vasilev%2C+Irina+Raznitsyna%2C+Anastasia+Pamova%2C+Tikhon+Burtsev%2C+Tatiana+Bobrovskaya%2C+Pavel+Kosov%2C+Anton+Vladzymyrskyy%2C+Olga+Omelyan%2Dskaya%2C+and+Kirill+Arzamasov%2E+2026%2E+Evaluating+medical+text+summar)
43. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
44. Qianqian Xie, Zheheng Luo, Benyou Wang, and Sophia Ananiadou. 2023. A survey for biomedical text summarization: From pre-trained to large language models. arXiv preprint arXiv:2304.08763. — [arXiv:2304.08763](https://arxiv.org/abs/2304.08763) [find](https://scholar.google.com/scholar?q=Qianqian+Xie%2C+Zheheng+Luo%2C+Benyou+Wang%2C+and+Sophia+Ananiadou%2E+2023%2E+A+survey+for+biomedical+text+summarization%3A+From+pre%2Dtrained+to+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E08763%2E+%E2%80%94+%5BarXiv%3A230)
45. Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. 2024. Large language models meet NLP: A survey. arXiv preprint arXiv:2405.12819. — [arXiv:2405.12819](https://arxiv.org/abs/2405.12819) [find](https://scholar.google.com/scholar?q=Libo+Qin%2C+Qiguang+Chen%2C+Xiachong+Feng%2C+Yang+Wu%2C+Yongheng+Zhang%2C+Yinghui+Li%2C+Min+Li%2C+Wanxiang+Che%2C+and+Philip+S+Yu%2E+2024%2E+Large+language+models+meet+NLP%3A+A+survey%2E+arXiv+preprint+arXiv%3A2405%2E12819%2E+%E2%80%94+%5Ba)
46. Yiqing Xie, Sheng Zhang, Hao Cheng, Pengfei Liu, Zelalem Gero, Cliff Wong, Tristan Naumann, Hoifung Poon, and Carolyn Rose. 2024. DocLens: Multi-aspect fine-grained evaluation for medical text generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol… [find](https://scholar.google.com/scholar?q=Yiqing+Xie%2C+Sheng+Zhang%2C+Hao+Cheng%2C+Pengfei+Liu%2C+Zelalem+Gero%2C+Cliff+Wong%2C+Tristan+Naumann%2C+Hoifung+Poon%2C+and+Carolyn+Rose%2E+2024%2E+DocLens%3A+Multi%2Daspect+fine%2Dgrained+evaluation+for+medical+text+generat)
47. Fangyi Yu. 2025. When AIs judge AIs: The rise of agent-as-a-judge evaluation for LLMs. arXiv preprint arXiv:2508.02994. — [arXiv:2508.02994](https://arxiv.org/abs/2508.02994) [find](https://scholar.google.com/scholar?q=Fangyi+Yu%2E+2025%2E+When+AIs+judge+AIs%3A+The+rise+of+agent%2Das%2Da%2Djudge+evaluation+for+LLMs%2E+arXiv+preprint+arXiv%3A2508%2E02994%2E+%E2%80%94+%5BarXiv%3A2508%2E02994%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2508%2E02994%29)
48. Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina Idnay, Yiming Luo, Elizabeth Park, Jordan G Nestor, Matthew E Spotnitz, Ali Soroush, et al. 2024. Closing the gap between open source and commercial large language models for medical evidence summarization. NPJ Digital Medicine, 7(1):239. [find](https://scholar.google.com/scholar?q=Gongbo+Zhang%2C+Qiao+Jin%2C+Yiliang+Zhou%2C+Song+Wang%2C+Betina+Idnay%2C+Yiming+Luo%2C+Elizabeth+Park%2C+Jordan+G+Nestor%2C+Matthew+E+Spotnitz%2C+Ali+Soroush%2C+et+al%2E+2024%2E+Closing+the+gap+between+open+source+and+commer)
49. Haopeng Zhang, Philip S Yu, and Jiawei Zhang. 2025. A systematic survey of text summarization: From statistical methods to large language models. ACM Computing Surveys, 57(11):1–41. [find](https://scholar.google.com/scholar?q=Haopeng+Zhang%2C+Philip+S+Yu%2C+and+Jiawei+Zhang%2E+2025%2E+A+systematic+survey+of+text+summarization%3A+From+statistical+methods+to+large+language+models%2E+ACM+Computing+Surveys%2C+57%2811%29%3A1%E2%80%9341%2E)
50. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+BERTScore%3A+Evaluating+text+generation+with+BERT%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E)
51. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Proceedings of the 37th International Conference on Neu…

*Updated 1 November 2026*