---
title: "DeVisE — paper by Tagliabue et al. (2026) — summary and ideas"
description: "The authors build admission notes from MIMIC-IV ICU discharge summaries in two versions, the raw text and a minimal… 46 references."
authors: ["Camila Zurdo Tagliabue", "Heloisa Oss Boll", "Aykut Erdem", "Erkut Erdem", "Iacer Calixto"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.338/"
doi: "10.18653/v1/2026.findings-eacl.338"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tagliabue-2026-devise-behavioral-testing-medical-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tagliabue-2026-devise-behavioral-testing-medical-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / DeVisE: Towards the Behavioral Testing of Medical Large Language Models

# DeVisE: Towards the Behavioral Testing of Medical Large Language Models

**Authors:** Camila Zurdo Tagliabue, Heloisa Oss Boll, Aykut Erdem, Erkut Erdem and [Iacer Calixto](https://md.research.elamur.ai/free/en/authors/iacer-calixto/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.338/) · [PDF](https://aclanthology.org/2026.findings-eacl.338.pdf) · [DOI 10.18653/v1/2026.findings-eacl.338](https://doi.org/10.18653/v1/2026.findings-eacl.338) · [OpenAlex](https://openalex.org/W7140101800)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Clinical decision-support evaluations of large language models usually report task accuracy. They do not show whether a model reasons about clinical variables or relies on spurious cues. DeVisE (Demographics and Vital signs Evaluation) addresses this with behavioral testing: it edits one patient attribute at a time and checks whether the model responds in a clinically sensible way. The authors build admission notes from MIMIC-IV ICU discharge summaries in two versions, the raw text and a minimal template containing only the target variables. They then create single-variable counterfactuals for demographics (age, gender, ethnicity) and six vital signs. Eight LLMs are tested zero-shot in two ways: how perplexity changes on the edited notes, and how predicted ICU mortality and length-of-stay change. Vital-sign edits generally move predictions in the medically expected, monotonic direction, but models differ widely in how sensitive and stable they are. Demographic edits cause small but systematic shifts. These shifts suggest bias and are invisible to standard metrics, which makes counterfactual audits important before clinical deployment.

## Key points
- The benchmark contains 1,001 raw and 1,001 template-based validated admission notes from the MIMIC-IV test set, each paired with counterfactuals. The patients are 45% female, have a mean age of 64±17, and 31% are non-white. Vital signs were extracted with LLaMA-3.3-70B-Instruct using few-shot prompting.
- The eight models evaluated are Phi4, Meditron3-Phi4, OpenBioLLM, LLaMA-3.3-Instruct, DeepSeek-R1-Distill, Qwen-2.5-Instruct, GPT-OSS, and GPT-4.1-mini, covering medical, general-purpose, and reasoning-oriented models.
- Template notes raise the rate of correctly directed predictions from 67.1% to 87.5% for mortality and from 66.5% to 79.9% for LoS. Prediction flips also become more frequent, for example 38.1% for LoS with Meditron3-Phi4-14B.
- Medical models barely react to edits in raw notes. General models are highly sensitive but often not directionally consistent. DeepSeek-R1-Distill-70B makes modest, well-calibrated shifts.
- Of the demographic variables, age causes the largest shifts. Gender and race effects are smaller but consistent, and they produce flips near decision boundaries, which raises fairness concerns.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Amsterdam](https://md.research.elamur.ai/free/en/universities/university-of-amsterdam/) — University, Amsterdam, Netherlands · 151 works · 174,506 citations
- [Koç University](https://md.research.elamur.ai/free/en/universities/koc-university/) — University, Istanbul, Türkiye · 3 works · 414 citations
- [Hacettepe University](https://md.research.elamur.ai/free/en/universities/hacettepe-university/) — University, Ankara, Türkiye · 2 works · 8 citations
- [Amsterdam University Medical Centers](https://md.research.elamur.ai/free/en/hospitals/amsterdam-university-medical-centers/) — Hospital, Amsterdam, Netherlands · 4 works
- [Amsterdam Public Health](https://md.research.elamur.ai/free/en/labs/amsterdam-public-health/) — Research institute, Amsterdam, Netherlands · 2 works

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 275 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 214 works

## Cite

Cite as: Camila Zurdo Tagliabue, Heloisa Oss Boll, Aykut Erdem, Erkut Erdem and Iacer Calixto. 2026. DeVisE: Towards the Behavioral Testing of Medical Large Language Models. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tagliabue2026devise,
  title = {{DeVisE: Towards the Behavioral Testing of Medical Large Language Models}},
  author = {Camila Zurdo Tagliabue, Heloisa Oss Boll, Aykut Erdem, Erkut Erdem and Iacer Calixto},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.338},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.338}
}
```

- [← previous Development of an Arabic Grammar Instructional Medium Based on Illustrated Stories in Elementary School](https://md.research.elamur.ai/free/en/papers/solihah-2026-development-arabic-grammar-instructional-medium/)
- [next → DevLake at LoResMT 2026: The Impact of Pre-training and Model Scale on Russian-Bashkir Low-Resource Translation](https://md.research.elamur.ai/free/en/papers/tyurin-2026-devlake-loresmt-2026-impact-pre/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/)

## References

46 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. 2024. [Phi-4 technical report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/). arXiv preprint arXiv:2412.08905.
2. M. Aguiar, P. Zweigenbaum, and N. Naderi. 2024. Seme at semeval-2024 task 2: Comparing masked and generative language models on natural language inference for clinical trials. arXiv preprint arXiv:2404.03977. [find](https://scholar.google.com/scholar?q=M%2E+Aguiar%2C+P%2E+Zweigenbaum%2C+and+N%2E+Naderi%2E+2024%2E+Seme+at+semeval%2D2024+task+2%3A+Comparing+masked+and+generative+language+models+on+natural+language+inference+for+clinical+trials%2E+arXiv+preprint+arXiv%3A240)
3. Khalid Alghatani, Nariman Ammar, Abdelmounaam Rezgui, Arash Shaban-Nejad, and 1 others. 2021. Predicting intensive care unit length of stay and mortality using patient vital signs: machine learning model development and validation. JMIR medical informatics, 9(5):e21347. [find](https://scholar.google.com/scholar?q=Khalid+Alghatani%2C+Nariman+Ammar%2C+Abdelmounaam+Rezgui%2C+Arash+Shaban%2DNejad%2C+and+1+others%2E+2021%2E+Predicting+intensive+care+unit+length+of+stay+and+mortality+using+patient+vital+signs%3A+machine+learning+mo)
4. D. Altinok. 2024. D-nlp at semeval-2024 task 2: Evaluating clinical inference capabilities of large language models. arXiv preprint arXiv:2405.04170. [find](https://scholar.google.com/scholar?q=D%2E+Altinok%2E+2024%2E+D%2Dnlp+at+semeval%2D2024+task+2%3A+Evaluating+clinical+inference+capabilities+of+large+language+models%2E+arXiv+preprint+arXiv%3A2405%2E04170%2E)
5. American Heart Association. 2024. Understanding blood pressure readings and heart rate guidelines. [https://www.heart.org/](https://www.heart.org/) en/health-topics/high-blood-pressure/ understanding-blood-pressure-readings. [↗](https://www.heart.org/)
6. Malaikannan Sankarasubbu Ankit Pal. 2024. Openbiollms: Advancing open-source large language models for healthcare and life sciences. [https://huggingface.co/aaditya/](https://huggingface.co/aaditya/) OpenBioLLM-Llama3-70B. [↗](https://huggingface.co/aaditya/)
7. Sadra Bakhshandeh. 2023. Benchmarking medical large language models. Nature Reviews Bioengineering, 1(8):543–543. [find](https://scholar.google.com/scholar?q=Sadra+Bakhshandeh%2E+2023%2E+Benchmarking+medical+large+language+models%2E+Nature+Reviews+Bioengineering%2C+1%288%29%3A543%E2%80%93543%2E)
8. Boris Beizer and J Wiley. 1996. Black box testing: Techniques for functional testing of software and systems. iEEE Software, 13(5):98. [find](https://scholar.google.com/scholar?q=Boris+Beizer+and+J+Wiley%2E+1996%2E+Black+box+testing%3A+Techniques+for+functional+testing+of+software+and+systems%2E+iEEE+Software%2C+13%285%29%3A98%2E)
9. Bart GJ Candel, Renée Duijzer, Menno I Gaakeer, Ewoud Ter Avest, Özcan Sir, Heleen Lameijer, Roger Hessels, Resi Reijnen, Erik W van Zwet, Evert de Jonge, and 1 others. 2022. The association between vital signs and clinical outcomes in emergency department patients of different age categories. Emerg… [find](https://scholar.google.com/scholar?q=Bart+GJ+Candel%2C+Ren%C3%A9e+Duijzer%2C+Menno+I+Gaakeer%2C+Ewoud+Ter+Avest%2C+%C3%96zcan+Sir%2C+Heleen+Lameijer%2C+Roger+Hessels%2C+Resi+Reijnen%2C+Erik+W+van+Zwet%2C+Evert+de+Jonge%2C+and+1+others%2E+2022%2E+The+association+between+v)
10. Leo Anthony Celi, Jacqueline Cellini, Marie-Laure Charpignon, Edward Christopher Dee, Franck Dernoncourt, Rene Eber, William Greig Mitchell, Lama Moukheiber, Julian Schirmer, Julia Situ, and 1 others. 2022. Sources of bias in artificial intelligence that perpetuate healthcare disparities—a global re… [find](https://scholar.google.com/scholar?q=Leo+Anthony+Celi%2C+Jacqueline+Cellini%2C+Marie%2DLaure+Charpignon%2C+Edward+Christopher+Dee%2C+Franck+Dernoncourt%2C+Rene+Eber%2C+William+Greig+Mitchell%2C+Lama+Moukheiber%2C+Julian+Schirmer%2C+Julia+Situ%2C+and+1+others%2E)
11. Candice L Downey, W Tahir, R Randell, JM Brown, and DG Jayne. 2017. Strengths and limitations of early warning scores: a systematic review and narrative synthesis. International journal of nursing studies, 76:106–119. [find](https://scholar.google.com/scholar?q=Candice+L+Downey%2C+W+Tahir%2C+R+Randell%2C+JM+Brown%2C+and+DG+Jayne%2E+2017%2E+Strengths+and+limitations+of+early+warning+scores%3A+a+systematic+review+and+narrative+synthesis%2E+International+journal+of+nursing+stu)
12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
13. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. [Domain-specific language model pretraining for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/). ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23.
14. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
15. Lars Hempel, Sina Sadeghi, and Toralf Kirsten. 2023. Prediction of intensive care unit length of stay in the mimic-iv dataset. Applied Sciences, 13(12):6930. [find](https://scholar.google.com/scholar?q=Lars+Hempel%2C+Sina+Sadeghi%2C+and+Toralf+Kirsten%2E+2023%2E+Prediction+of+intensive+care+unit+length+of+stay+in+the+mimic%2Div+dataset%2E+Applied+Sciences%2C+13%2812%29%3A6930%2E)
16. Svetlana Herasevich, Kirill Lipatov, Yuliya Pinevich, Heidi Lindroth, Aysun Tekin, Vitaly Herasevich, Brian W Pickering, and Amelia K Barwise. 2022. The impact of health information technology for early detection of patient deterioration on mortality and length of stay in the hospital acute care set… [find](https://scholar.google.com/scholar?q=Svetlana+Herasevich%2C+Kirill+Lipatov%2C+Yuliya+Pinevich%2C+Heidi+Lindroth%2C+Aysun+Tekin%2C+Vitaly+Herasevich%2C+Brian+W+Pickering%2C+and+Amelia+K+Barwise%2E+2022%2E+The+impact+of+health+information+technology+for+ear)
17. Infectious Diseases Society of America. 2003. Fever and neutropenia clinical practice guidelines. [https://www.idsociety.org/](https://www.idsociety.org/) practice-guideline/febrile-neutropenia/. [↗](https://www.idsociety.org/)
18. Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. 2023a. MIMIC-IV-Note: Deidentified free-text clinical notes. [find](https://scholar.google.com/scholar?q=Alistair+Johnson%2C+Tom+Pollard%2C+Steven+Horng%2C+Leo+Anthony+Celi%2C+and+Roger+Mark%2E+2023a%2E+MIMIC%2DIV%2DNote%3A+Deidentified+free%2Dtext+clinical+notes%2E)
19. Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, and 1 others. 2023b. [Mimic-iv, a freely accessible electronic health record dataset](https://md.research.elamur.ai/free/en/papers/johnson-2023-mimic-iv-freely-accessible-electronic/). Scientific data, 10(1):1.
20. M. Jullien, M. Valentino, and A. Freitas. 2024. Semeval-2024 task 2: Safe biomedical natural language inference for clinical trials. arXiv preprint arXiv:2404.04963. [find](https://scholar.google.com/scholar?q=M%2E+Jullien%2C+M%2E+Valentino%2C+and+A%2E+Freitas%2E+2024%2E+Semeval%2D2024+task+2%3A+Safe+biomedical+natural+language+inference+for+clinical+trials%2E+arXiv+preprint+arXiv%3A2404%2E04963%2E)
21. Ilker Kesen, Andrea Pedrotti, Mustafa Dogan, Michele Cafagna, Emre Can Acikgoz, Letitia Parcalabescu, Iacer Calixto, Anette Frank, Albert Gatt, Aykut Erdem, and Erkut Erdem. 2024. Vilma: A zero-shot benchmark for linguistic and temporal grounding in video-language models. In International Conference… [find](https://scholar.google.com/scholar?q=Ilker+Kesen%2C+Andrea+Pedrotti%2C+Mustafa+Dogan%2C+Michele+Cafagna%2C+Emre+Can+Acikgoz%2C+Letitia+Parcalabescu%2C+Iacer+Calixto%2C+Anette+Frank%2C+Albert+Gatt%2C+Aykut+Erdem%2C+and+Erkut+Erdem%2E+2024%2E+Vilma%3A+A+zero%2Dshot+b)
22. V. Kougia, A. Sedova, A. Stephan, K. Zaporojets, and B. Roth. 2024. Analysing zero-shot temporal relation extraction on clinical notes using temporal consistency. arXiv preprint arXiv:2406.11486. [find](https://scholar.google.com/scholar?q=V%2E+Kougia%2C+A%2E+Sedova%2C+A%2E+Stephan%2C+K%2E+Zaporojets%2C+and+B%2E+Roth%2E+2024%2E+Analysing+zero%2Dshot+temporal+relation+extraction+on+clinical+notes+using+temporal+consistency%2E+arXiv+preprint+arXiv%3A2406%2E11486%2E)
23. Joseph Lee, Tianqi Shang, Jae Young Baik, Duy Duong-Tran, Shu Yang, Lingyao Li, and Li Shen. 2025. Investigating llms in clinical triage: Promising capabilities, persistent intersectional biases. arXiv preprint arXiv:2504.16273. — [arXiv:2504.16273](https://arxiv.org/abs/2504.16273) [find](https://scholar.google.com/scholar?q=Joseph+Lee%2C+Tianqi+Shang%2C+Jae+Young+Baik%2C+Duy+Duong%2DTran%2C+Shu+Yang%2C+Lingyao+Li%2C+and+Li+Shen%2E+2025%2E+Investigating+llms+in+clinical+triage%3A+Promising+capabilities%2C+persistent+intersectional+biases%2E+arXi)
24. Wendy L Macias-Konstantopoulos, Kimberly A Collins, Rosemarie Diaz, Herbert C Duber, Courtney D Edwards, Antony P Hsu, Megan L Ranney, Ralph J Riviello, Zachary S Wettstein, and Carolyn J Sachs. 2023. Race, healthcare, and health disparities: A critical review and recommendations for advancing healt… [find](https://scholar.google.com/scholar?q=Wendy+L+Macias%2DKonstantopoulos%2C+Kimberly+A+Collins%2C+Rosemarie+Diaz%2C+Herbert+C+Duber%2C+Courtney+D+Edwards%2C+Antony+P+Hsu%2C+Megan+L+Ranney%2C+Ralph+J+Riviello%2C+Zachary+S+Wettstein%2C+and+Carolyn+J+Sachs%2E+2023%2E)
25. D. MacPhail, D. Harbecke, L. Raithel, and S. Möller. 2024. Evaluating the robustness of adverse drug event classification models using templates. arXiv preprint arXiv:2407.02432. [find](https://scholar.google.com/scholar?q=D%2E+MacPhail%2C+D%2E+Harbecke%2C+L%2E+Raithel%2C+and+S%2E+M%C3%B6ller%2E+2024%2E+Evaluating+the+robustness+of+adverse+drug+event+classification+models+using+templates%2E+arXiv+preprint+arXiv%3A2407%2E02432%2E)
26. Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, and 1 others. 2023. Towards accurate differential diagnosis with large language models. arXiv preprint arXiv:2312.00164. — [arXiv:2312.00164](https://arxiv.org/abs/2312.00164) [find](https://scholar.google.com/scholar?q=Daniel+McDuff%2C+Mike+Schaekermann%2C+Tao+Tu%2C+Anil+Palepu%2C+Amy+Wang%2C+Jake+Garrison%2C+Karan+Singhal%2C+Yash+Sharma%2C+Shekoofeh+Azizi%2C+Kavita+Kulkarni%2C+and+1+others%2E+2023%2E+Towards+accurate+differential+diagnosi)
27. OpenAI. 2025. [gpt-oss-120b & gpt-oss-20b model card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/). Preprint, arXiv:2508.10925.
28. OpenAI(gpt-4.1-mini). 2025. Gpt-4.1 mini. [https://platform.openai.com/docs/models/](https://platform.openai.com/docs/models/) gpt-4.1-mini. Large language model. [↗](https://platform.openai.com/docs/models/)
29. OpenMeditron. 2025. Meditron-3-phi4-14b. [https://huggingface.co/OpenMeditron/](https://huggingface.co/OpenMeditron/) Meditron3-Phi4-14B. Accessed: 2025-05-13. [↗](https://huggingface.co/OpenMeditron/)
30. Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. 2022. VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Letitia+Parcalabescu%2C+Michele+Cafagna%2C+Lilitta+Muradjan%2C+Anette+Frank%2C+Iacer+Calixto%2C+and+Albert+Gatt%2E+2022%2E+VALSE%3A+A+task%2Dindependent+benchmark+for+vision+and+language+models+centered+on+linguistic+p)
31. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118. [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Tongshuang+Wu%2C+Carlos+Guestrin%2C+and+Sameer+Singh%2E+2020%2E+Beyond+accuracy%3A+Behavioral+testing+of+nlp+models+with+checklist%2E+arXiv+preprint+arXiv%3A2005%2E04118%2E)
32. Tom Röhr, Alexei Figueroa, Jens-Michalis Papaioannou, Conor Fallon, Keno Bressem, Wolfgang Nejdl, and Alexander Löser. 2024. Revisiting clinical outcome prediction for mimic-iv. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 208–217. [find](https://scholar.google.com/scholar?q=Tom+R%C3%B6hr%2C+Alexei+Figueroa%2C+Jens%2DMichalis+Papaioannou%2C+Conor+Fallon%2C+Keno+Bressem%2C+Wolfgang+Nejdl%2C+and+Alexander+L%C3%B6ser%2E+2024%2E+Revisiting+clinical+outcome+prediction+for+mimic%2Div%2E+In+Proceedings+of+the+)
33. Royal College of Physicians. 2017. National early warning score (news) 2. [https://www.rcplondon.ac.uk/projects/outputs/](https://www.rcplondon.ac.uk/projects/outputs/) national-early-warning-score-news-2. [↗](https://www.rcplondon.ac.uk/projects/outputs/)
34. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, and 1 others. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1–8. [find](https://scholar.google.com/scholar?q=Karan+Singhal%2C+Tao+Tu%2C+Juraj+Gottweis%2C+Rory+Sayres%2C+Ellery+Wulczyn%2C+Mohamed+Amin%2C+Le+Hou%2C+Kevin+Clark%2C+Stephen+R+Pfohl%2C+Heather+Cole%2DLewis%2C+and+1+others%2E+2025%2E+Toward+expert%2Dlevel+medical+question+ans)
35. Y. Zhao, H. Wang, Y. Liu, W. Suhuang, X. Wu, and Y. Zheng. 2024. Can llms replace clinical doctors? exploring bias in disease diagnosis by large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13914– 13935. [find](https://scholar.google.com/scholar?q=Y%2E+Zhao%2C+H%2E+Wang%2C+Y%2E+Liu%2C+W%2E+Suhuang%2C+X%2E+Wu%2C+and+Y%2E+Zheng%2E+2024%2E+Can+llms+replace+clinical+doctors%3F+exploring+bias+in+disease+diagnosis+by+large+language+models%2E+In+Findings+of+the+Association+for+Com)
36. Society of Critical Care Medicine. 2015. Targeted temperature management after cardiac arrest. [https://www.sccm.org/blog/](https://www.sccm.org/blog/) A Appendix concise-critical-appraisal-temperature-management-after-cardiac-arrest. [↗](https://www.sccm.org/blog/)
37. Society of Critical Care Medicine. 2021. Surviving sepsis campaign: International guidelines for the management of sepsis and septic shock 2021. [https://www.sccm.org/SurvivingSepsisCampaign/](https://www.sccm.org/SurvivingSepsisCampaign/) Guidelines/Adult-Patients. [↗](https://www.sccm.org/SurvivingSepsisCampaign/)
38. Qwen Team. 2024. Qwen2.5: A party of foundation models. with shortness of breath. (. . . ) [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E+with+shortness+of+breath%2E+%28%2E+%2E+%2E+%29)
39. B. Van Aken, S. Herrmann, and A. Löser. 2021a. What do you see in this patient? behavioral testing of clinical nlp models. arXiv preprint arXiv:2111.15512. [find](https://scholar.google.com/scholar?q=B%2E+Van+Aken%2C+S%2E+Herrmann%2C+and+A%2E+L%C3%B6ser%2E+2021a%2E+What+do+you+see+in+this+patient%3F+behavioral+testing+of+clinical+nlp+models%2E+arXiv+preprint+arXiv%3A2111%2E15512%2E)
40. Betty Van Aken, Jens-Michalis Papaioannou, Manuel Mayrdorfer, Klemens Budde, Felix A Gers, and Alexander Loeser. 2021b. Clinical outcome prediction from admission notes using self-supervised knowledge integration. arXiv preprint arXiv:2102.04110. [find](https://scholar.google.com/scholar?q=Betty+Van+Aken%2C+Jens%2DMichalis+Papaioannou%2C+Manuel+Mayrdorfer%2C+Klemens+Budde%2C+Felix+A+Gers%2C+and+Alexander+Loeser%2E+2021b%2E+Clinical+outcome+prediction+from+admission+notes+using+self%2Dsupervised+knowledge)
41. Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, and 1 others. 2024. [Adapted large language models can outperform medical experts in clinical text summarization](https://md.research.elamur.ai/free/en/papers/veen-2023-adapted-large-language-models-outperform/). Nature med…
42. World Health Organization. 2016. Oxygen therapy for children: A manual for health workers. [https://www.who.int/publications/i/](https://www.who.int/publications/i/) item/9789241549554. [↗](https://www.who.int/publications/i/)
43. Jie Xu, Lu Lu, Sen Yang, Bilin Liang, Xinwei Peng, Jiali Pang, Jinru Ding, Xiaoming Shi, Lingrui Yang, Huan Song, and 1 others. 2023. Medgpteval: A dataset and benchmark to evaluate responses of large language models in medicine. arXiv preprint arXiv:2305.07340. — [arXiv:2305.07340](https://arxiv.org/abs/2305.07340) [find](https://scholar.google.com/scholar?q=Jie+Xu%2C+Lu+Lu%2C+Sen+Yang%2C+Bilin+Liang%2C+Xinwei+Peng%2C+Jiali+Pang%2C+Jinru+Ding%2C+Xiaoming+Shi%2C+Lingrui+Yang%2C+Huan+Song%2C+and+1+others%2E+2023%2E+Medgpteval%3A+A+dataset+and+benchmark+to+evaluate+responses+of+large)
44. Zonghai Yao, Zihao Zhang, Chaolong Tang, Xingyu Bian, Youxia Zhao, Zhichao Yang, Junda Wang, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, and 1 others. 2024. Medqa-cs: Benchmarking large language models clinical skills using an ai-sce framework. arXiv preprint arXiv:2410.01553. — [arXiv:2410.01553](https://arxiv.org/abs/2410.01553) [find](https://scholar.google.com/scholar?q=Zonghai+Yao%2C+Zihao+Zhang%2C+Chaolong+Tang%2C+Xingyu+Bian%2C+Youxia+Zhao%2C+Zhichao+Yang%2C+Junda+Wang%2C+Huixue+Zhou%2C+Won+Seok+Jang%2C+Feiyun+Ouyang%2C+and+1+others%2E+2024%2E+Medqa%2Dcs%3A+Benchmarking+large+language+models)
45. T. Zack, E. Lehman, M. Suzgun, J. A. Rodriguez, L. A. Celi, J. Gichoya, and E. Alsentzer. 2024. Assessing the potential of gpt-4 to perpetuate racial and gender biases in health care: a model evaluation study. The Lancet Digital Health, 6(1):e12–e22. [find](https://scholar.google.com/scholar?q=T%2E+Zack%2C+E%2E+Lehman%2C+M%2E+Suzgun%2C+J%2E+A%2E+Rodriguez%2C+L%2E+A%2E+Celi%2C+J%2E+Gichoya%2C+and+E%2E+Alsentzer%2E+2024%2E+Assessing+the+potential+of+gpt%2D4+to+perpetuate+racial+and+gender+biases+in+health+care%3A+a+model+evaluati)
46. Raw Clinical Note Examples Correct Discharge Note (Raw). PRESENT ILLNESS: The patient is a ___ year-old female with a history of NSCLC (stage IV) who presents MEDICAL HISTORY: CAD s/p MI ___, s/p CABG ___, Hypertension, Dyslipidemia, CVA: small left posterior frontal infarct in ___, Macular Degenera… [find](https://scholar.google.com/scholar?q=Raw+Clinical+Note+Examples+Correct+Discharge+Note+%28Raw%29%2E+PRESENT+ILLNESS%3A+The+patient+is+a+%5F%5F%5F+year%2Dold+female+with+a+history+of+NSCLC+%28stage+IV%29+who+presents+MEDICAL+HISTORY%3A+CAD+s%2Fp+MI+%5F%5F%5F%2C+s%2Fp+CABG)

*Updated 1 November 2026*