---
title: "HealthBench: Evaluating Large Language Models — Arora 2025"
description: "Most medical benchmarks for LLMs use multiple-choice or short-answer questions."
authors: ["Rahul K. Arora", "Jason Wei", "Rebecca Soskin Hicks", "Preston Bowman", "Joaquin Quinonero-Candela", "Foivos Tsimpourlas", "Michael Sharman", "Meghan Shah", "Andrea Vallone", "Alex Beutel", "Johannes Heidecke", "Karan Singhal"]
year: 2025
venue: "arXiv:2505.08775"
source: "https://arxiv.org/abs/2505.08775"
doi: "10.48550/arxiv.2505.08775"
arxiv: "2505.08775"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/arora-2025-healthbench-evaluating-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/arora-2025-healthbench-evaluating-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / HealthBench: Evaluating Large Language Models Towards Improved Human Health

# HealthBench: Evaluating Large Language Models Towards Improved Human Health

**Authors:** [Rahul K. Arora](https://md.research.elamur.ai/free/en/authors/rahul-k-arora/), [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/), Rebecca Soskin Hicks, Preston Bowman, Joaquin Quinonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, [Alex Beutel](https://md.research.elamur.ai/free/en/authors/alex-beutel/), [Johannes Heidecke](https://md.research.elamur.ai/free/en/authors/johannes-heidecke/), [Karan Singhal](https://md.research.elamur.ai/free/en/authors/karan-singhal/) · **Year:** 2025 · **Venue:** arXiv:2505.08775 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2505.08775) · [PDF](https://arxiv.org/pdf/2505.08775) · [DOI 10.48550/arxiv.2505.08775](https://doi.org/10.48550/arxiv.2505.08775) · [arXiv 2505.08775](https://arxiv.org/abs/2505.08775) · [OpenAlex](https://openalex.org/W6891853064)

**Citations:** 27 citations (openalex-snapshot · 2026-10-14) · [88](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Most medical benchmarks for LLMs use multiple-choice or short-answer questions. These capture little of how people actually seek health help, are rarely checked against expert judgment, and are often close to saturated. HealthBench, from OpenAI, addresses this with 5,000 realistic single- and multi-turn health dialogues involving lay users and clinicians, across many languages, regions and personas. The model must reply to the final user message. Each dialogue has its own rubric, written by physicians, that awards or deducts points for specific content and behaviors. A model-based grader applies these rubrics, and its verdicts agree with physicians about as often as physicians agree with one another. Scores have risen steeply: from 16% for GPT-3.5 Turbo to 32% for GPT-4o and 60% for o3. Recent models outperform physicians writing responses without help. Two variants are included: a physician-consensus subset and a hard subset that remains far from solved. The benchmark gives the field an open, extensible and unsaturated standard for judging whether models are safe and useful in health settings.

## Key points
- **Scale and authorship:** 262 physicians from 26 specialties, with practice experience in 60 countries, wrote 48,562 unique rubric criteria. Each criterion carries a weight from −10 to +10, and a typical example has 11 criteria.
- **Structure and scoring:** The benchmark is organized into seven themes (e.g., emergency referrals, global health, context seeking) and five axes (e.g., accuracy, completeness, communication, instruction following). Per-example scores are averaged and clipped to [0, 1].
- **Model results:** GPT-4.1 nano beats GPT-4o at about 25× lower cost. The weakest areas are worst-case reliability and asking users for missing context.
- **Variants:**
  - **HealthBench Consensus** contains 3,671 examples scored only on 34 physician-validated criteria, which appear 8,053 times in total.
  - **HealthBench Hard** contains 1,000 difficult examples, and no evaluated model exceeds 32% on it.
- **Physician baselines and release:** Physicians could improve responses from September 2024 models but not from April 2025 models. Data and code are available in OpenAI's simple-evals repository.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

88 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 85 not yet.

- [gpt-oss-120b & gpt-oss-20b Model Card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/)
- [Humanity's Last Exam](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/)
- [OpenAI GPT-5 System Card](https://md.research.elamur.ai/free/en/papers/singh-2025-openai-gpt-5-system-card/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{arora2025healthbench,
  title = {{HealthBench: Evaluating Large Language Models Towards Improved Human Health}},
  author = {Arora, Rahul K. and Wei, Jason and Hicks, Rebecca Soskin and Bowman, Preston and Quinonero-Candela, Joaquin and Tsimpourlas, Foivos and Sharman, Michael and Shah, Meghan and Vallone, Andrea and Beutel, Alex and Heidecke, Johannes and Singhal, Karan},
  year = {2025},
  doi = {10.48550/arxiv.2505.08775},
  eprint = {2505.08775},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.08775}
}
```

- [← previous Head-initial and head-Final coordinate structures in two annotation schemes of dependency grammar](https://md.research.elamur.ai/free/en/papers/ud-2025-osborne-head-initial-head-final-coordinate/)
- [next → HeBID: Detecting Social Identities in Hebrew-language Political Text](https://md.research.elamur.ai/free/en/papers/findings-2025-lan-hebid-detecting-social-identities-hebrew/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

51 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. W. Ayers, A. Poliak, M. Dredze, E. C. Leas, Z. Zhu, J. B. Kelley, D. J. Faix, A. M. Goodman, C. A. Longhurst, M. Hogarth, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA internal medicine, 183(6):589–596, 20… [find](https://scholar.google.com/scholar?q=J%2E+W%2E+Ayers%2C+A%2E+Poliak%2C+M%2E+Dredze%2C+E%2E+C%2E+Leas%2C+Z%2E+Zhu%2C+J%2E+B%2E+Kelley%2C+D%2E+J%2E+Faix%2C+A%2E+M%2E+Goodman%2C+C%2E+A%2E+Longhurst%2C+M%2E+Hogarth%2C+et+al%2E+Comparing+physician+and+artificial+intelligence+chatbot+responses+to)
2. H. P. Baker, E. Dwyer, S. Kalidoss, and et al. Chatgpt’s ability to assist with clinical documentation: A randomized controlled trial. Journal of the American Academy of Orthopaedic Surgeons, 32(3):123–129, 2024. doi: 10.5435/JAAOS-D-23-00474. — [doi:10.5435/jaaos-d-23-00474](https://doi.org/10.5435/jaaos-d-23-00474) [find](https://scholar.google.com/scholar?q=H%2E+P%2E+Baker%2C+E%2E+Dwyer%2C+S%2E+Kalidoss%2C+and+et+al%2E+Chatgpt%E2%80%99s+ability+to+assist+with+clinical+documentation%3A+A+randomized+controlled+trial%2E+Journal+of+the+American+Academy+of+Orthopaedic+Surgeons%2C+32%283%29%3A12)
3. A. L. Beam and I. S. Kohane. Big data and machine learning in health care. JAMA, 319(13):1317–1318, 2018. doi: 10.1001/jama.2017.18391. — [doi:10.1001/jama.2017.18391](https://doi.org/10.1001/jama.2017.18391) [find](https://scholar.google.com/scholar?q=A%2E+L%2E+Beam+and+I%2E+S%2E+Kohane%2E+Big+data+and+machine+learning+in+health+care%2E+JAMA%2C+319%2813%29%3A1317%E2%80%931318%2C+2018%2E+doi%3A+10%2E1001%2Fjama%2E2017%2E18391%2E+%E2%80%94+%5Bdoi%3A10%2E1001%2Fjama%2E2017%2E18391%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1001%2Fjama%2E201)
4. G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669, 2024. [find](https://scholar.google.com/scholar?q=G%2E+H%2E+Chen%2C+S%2E+Chen%2C+Z%2E+Liu%2C+F%2E+Jiang%2C+and+B%2E+Wang%2E+Humans+or+llms+as+the+judge%3F+a+study+on+judgement+biases%2E+arXiv+preprint+arXiv%3A2402%2E10669%2C+2024%2E)
5. S. Clémençon, I. Colin, and A. Bellet. Scaling-up empirical risk minimization: Optimization of incomplete u-statistics. Journal of Machine Learning Research, 17(76):1–36, 2016. URL [http://jmlr.org/papers/](http://jmlr.org/papers/) v17/15-012.html. [↗](http://jmlr.org/papers/)
6. J. Clusmann, F. R. Kolbinger, H. S. Muti, Z. I. Carrero, J.-N. Eckardt, et al. The future landscape of large language models in medicine. Communications Medicine, 3:141, 2023. doi: 10.1038/s43856-023-00370-1. — [doi:10.1038/s43856-023-00370-1](https://doi.org/10.1038/s43856-023-00370-1) [find](https://scholar.google.com/scholar?q=J%2E+Clusmann%2C+F%2E+R%2E+Kolbinger%2C+H%2E+S%2E+Muti%2C+Z%2E+I%2E+Carrero%2C+J%2E%2DN%2E+Eckardt%2C+et+al%2E+The+future+landscape+of+large+language+models+in+medicine%2E+Communications+Medicine%2C+3%3A141%2C+2023%2E+doi%3A+10%2E1038%2Fs43856%2D023%2D)
7. J. Cosentino, A. Belyaeva, X. Liu, N. A. Furlotte, Z. Yang, C. Lee, E. Schenck, Y. Patel, J. Cui, L. D. Schneider, et al. Towards a personal health large language model. arXiv preprint arXiv:2406.06474, 2024. — [arXiv:2406.06474](https://arxiv.org/abs/2406.06474) [find](https://scholar.google.com/scholar?q=J%2E+Cosentino%2C+A%2E+Belyaeva%2C+X%2E+Liu%2C+N%2E+A%2E+Furlotte%2C+Z%2E+Yang%2C+C%2E+Lee%2C+E%2E+Schenck%2C+Y%2E+Patel%2C+J%2E+Cui%2C+L%2E+D%2E+Schneider%2C+et+al%2E+Towards+a+personal+health+large+language+model%2E+arXiv+preprint+arXiv%3A2406%2E0647)
8. D. Dash, R. Thapa, J. M. Banda, A. Swaminathan, M. Cheatham, M. Kashyap, N. Kotecha, J. H. Chen, S. Gombar, L. Downing, et al. Evaluation of gpt-3.5 and gpt-4 for supporting real-world information needs in healthcare delivery. arXiv preprint arXiv:2304.13714, 2023. [find](https://scholar.google.com/scholar?q=D%2E+Dash%2C+R%2E+Thapa%2C+J%2E+M%2E+Banda%2C+A%2E+Swaminathan%2C+M%2E+Cheatham%2C+M%2E+Kashyap%2C+N%2E+Kotecha%2C+J%2E+H%2E+Chen%2C+S%2E+Gombar%2C+L%2E+Downing%2C+et+al%2E+Evaluation+of+gpt%2D3%2E5+and+gpt%2D4+for+supporting+real%2Dworld+information+nee)
9. A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):115–118, 2017. doi: 10.1038/nature21056. — [doi:10.1038/nature21056](https://doi.org/10.1038/nature21056) [find](https://scholar.google.com/scholar?q=A%2E+Esteva%2C+B%2E+Kuprel%2C+R%2E+A%2E+Novoa%2C+J%2E+Ko%2C+S%2E+M%2E+Swetter%2C+H%2E+M%2E+Blau%2C+and+S%2E+Thrun%2E+Dermatologist%2Dlevel+classification+of+skin+cancer+with+deep+neural+networks%2E+Nature%2C+542%287639%29%3A115%E2%80%93118%2C+2017%2E+doi%3A+10)
10. D. Fast, L. C. Adams, F. Busch, C. Fallon, M. Huppertz, R. Siepmann, P. Prucker, N. Bayerl, D. Truhn, M. Makowski, et al. Autonomous medical evaluation for guideline adherence of large language models. NPJ Digital Medicine, 7(1):1–14, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Fast%2C+L%2E+C%2E+Adams%2C+F%2E+Busch%2C+C%2E+Fallon%2C+M%2E+Huppertz%2C+R%2E+Siepmann%2C+P%2E+Prucker%2C+N%2E+Bayerl%2C+D%2E+Truhn%2C+M%2E+Makowski%2C+et+al%2E+Autonomous+medical+evaluation+for+guideline+adherence+of+large+language+models)
11. S. L. Fleming, A. Lozano, W. J. Haberkorn, J. A. Jindal, E. Reis, R. Thapa, L. Blankemeier, J. Z. Genkins, E. Steinberg, A. Nayak, et al. Medalign: A clinician-generated dataset for instruction following with electronic medical records. In Proceedings of the AAAI Conference on Artificial Intelligenc… [find](https://scholar.google.com/scholar?q=S%2E+L%2E+Fleming%2C+A%2E+Lozano%2C+W%2E+J%2E+Haberkorn%2C+J%2E+A%2E+Jindal%2C+E%2E+Reis%2C+R%2E+Thapa%2C+L%2E+Blankemeier%2C+J%2E+Z%2E+Genkins%2C+E%2E+Steinberg%2C+A%2E+Nayak%2C+et+al%2E+Medalign%3A+A+clinician%2Dgenerated+dataset+for+instruction+follow)
12. A. Gilson, C. W. Safranek, T. Huang, V. Socrates, L. Chi, R. A. Taylor, D. Chartash, et al. How does chatgpt perform on the united states medical licensing examination (usmle)? the implications of large language models for medical education and knowledge assessment. JMIR medical education, 9(1):e453… [find](https://scholar.google.com/scholar?q=A%2E+Gilson%2C+C%2E+W%2E+Safranek%2C+T%2E+Huang%2C+V%2E+Socrates%2C+L%2E+Chi%2C+R%2E+A%2E+Taylor%2C+D%2E+Chartash%2C+et+al%2E+How+does+chatgpt+perform+on+the+united+states+medical+licensing+examination+%28usmle%29%3F+the+implications+of+lar)
13. E. Goh, R. Gallo, J. Hom, E. Strong, Y. Weng, H. Kerman, J. A. Cool, Z. Kanjee, A. S. Parsons, N. Ahuja, et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Network Open, 7(10):e2440969–e2440969, 2024. [find](https://scholar.google.com/scholar?q=E%2E+Goh%2C+R%2E+Gallo%2C+J%2E+Hom%2C+E%2E+Strong%2C+Y%2E+Weng%2C+H%2E+Kerman%2C+J%2E+A%2E+Cool%2C+Z%2E+Kanjee%2C+A%2E+S%2E+Parsons%2C+N%2E+Ahuja%2C+et+al%2E+Large+language+model+influence+on+diagnostic+reasoning%3A+a+randomized+clinical+trial%2E+JAM)
14. V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, S. Venugopalan, K. Widner, T. Madams, J. Cuadros, R. Kim, R. Raman, P. C. Nelson, J. L. Mega, and D. R. Webster. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photo… — [doi:10.1001/jama.2016.17216](https://doi.org/10.1001/jama.2016.17216) [find](https://scholar.google.com/scholar?q=V%2E+Gulshan%2C+L%2E+Peng%2C+M%2E+Coram%2C+M%2E+C%2E+Stumpe%2C+D%2E+Wu%2C+A%2E+Narayanaswamy%2C+S%2E+Venugopalan%2C+K%2E+Widner%2C+T%2E+Madams%2C+J%2E+Cuadros%2C+R%2E+Kim%2C+R%2E+Raman%2C+P%2E+C%2E+Nelson%2C+J%2E+L%2E+Mega%2C+and+D%2E+R%2E+Webster%2E+Development+and+v)
15. V. Hartman, X. Zhang, R. Poddar, and et al. Developing and evaluating large language model–generated emergency medicine handoff notes. JAMA Network Open, 7(12):e2448723, 2024. doi: 10.1001/ jamanetworkopen.2024.48723. [find](https://scholar.google.com/scholar?q=V%2E+Hartman%2C+X%2E+Zhang%2C+R%2E+Poddar%2C+and+et+al%2E+Developing+and+evaluating+large+language+model%E2%80%93generated+emergency+medicine+handoff+notes%2E+JAMA+Network+Open%2C+7%2812%29%3Ae2448723%2C+2024%2E+doi%3A+10%2E1001%2F+jamanetwor)
16. M. V. Heinz, N. Jacobson, E. Wright, and et al. Randomized trial of a generative ai chatbot for mental health treatment. NEJM AI, 1(2):e2400802, 2025. doi: 10.1056/AIoa2400802. — [doi:10.1056/aioa2400802](https://doi.org/10.1056/aioa2400802) [find](https://scholar.google.com/scholar?q=M%2E+V%2E+Heinz%2C+N%2E+Jacobson%2C+E%2E+Wright%2C+and+et+al%2E+Randomized+trial+of+a+generative+ai+chatbot+for+mental+health+treatment%2E+NEJM+AI%2C+1%282%29%3Ae2400802%2C+2025%2E+doi%3A+10%2E1056%2FAIoa2400802%2E+%E2%80%94+%5Bdoi%3A10%2E1056%2Faioa2400)
17. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Hurst%2C+A%2E+Lerer%2C+A%2E+P%2E+Goucher%2C+A%2E+Perelman%2C+A%2E+Ramesh%2C+A%2E+Clark%2C+A%2E+Ostrow%2C+A%2E+Welihinda%2C+A%2E+Hayes%2C+A%2E+Radford%2C+et+al%2E+Gpt%2D4o+system+card%2E+arXiv+preprint+arXiv%3A2410%2E21276%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/)
18. S. Jain, A. Agrawal, A. Saporta, S. Q. Truong, D. N. Duong, T. Bui, P. Chambon, Y. Zhang, M. P. Lungren, A. Y. Ng, et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463, 2021. — [arXiv:2106.14463](https://arxiv.org/abs/2106.14463) [find](https://scholar.google.com/scholar?q=S%2E+Jain%2C+A%2E+Agrawal%2C+A%2E+Saporta%2C+S%2E+Q%2E+Truong%2C+D%2E+N%2E+Duong%2C+T%2E+Bui%2C+P%2E+Chambon%2C+Y%2E+Zhang%2C+M%2E+P%2E+Lungren%2C+A%2E+Y%2E+Ng%2C+et+al%2E+Radgraph%3A+Extracting+clinical+entities+and+relations+from+radiology+reports%2E+a)
19. D. Jin et al. What disease does this patient have? a large-scale open-domain medical qa dataset. arXiv preprint arXiv:2010.06081, 2021. — [arXiv:2010.06081](https://arxiv.org/abs/2010.06081) [find](https://scholar.google.com/scholar?q=D%2E+Jin+et+al%2E+What+disease+does+this+patient+have%3F+a+large%2Dscale+open%2Ddomain+medical+qa+dataset%2E+arXiv+preprint+arXiv%3A2010%2E06081%2C+2021%2E+%E2%80%94+%5BarXiv%3A2010%2E06081%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2010%2E06081%29)
20. Q. Jin, D. Dhingra, and W. Cohen. Pubmedqa: A dataset for biomedical research question answering. In EMNLP, 2019. [find](https://scholar.google.com/scholar?q=Q%2E+Jin%2C+D%2E+Dhingra%2C+and+W%2E+Cohen%2E+Pubmedqa%3A+A+dataset+for+biomedical+research+question+answering%2E+In+EMNLP%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/)
21. Z. Kanjee, B. Crowe, and A. Rodman. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. Jama, 330(1):78–80, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Kanjee%2C+B%2E+Crowe%2C+and+A%2E+Rodman%2E+Accuracy+of+a+generative+artificial+intelligence+model+in+a+complex+diagnostic+challenge%2E+Jama%2C+330%281%29%3A78%E2%80%9380%2C+2023%2E)
22. P. Lee, S. Bubeck, and J. Petro. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. New England Journal of Medicine, 388:1233–1239, 2023. doi: 10.1056/NEJMsr2214184. — [doi:10.1056/nejmsr2214184](https://doi.org/10.1056/nejmsr2214184) [find](https://scholar.google.com/scholar?q=P%2E+Lee%2C+S%2E+Bubeck%2C+and+J%2E+Petro%2E+Benefits%2C+limits%2C+and+risks+of+GPT%2D4+as+an+AI+chatbot+for+medicine%2E+New+England+Journal+of+Medicine%2C+388%3A1233%E2%80%931239%2C+2023%2E+doi%3A+10%2E1056%2FNEJMsr2214184%2E+%E2%80%94+%5Bdoi%3A10%2E1056%2Fne)
23. C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28564, 2023. [find](https://scholar.google.com/scholar?q=C%2E+Li%2C+C%2E+Wong%2C+S%2E+Zhang%2C+N%2E+Usuyama%2C+H%2E+Liu%2C+J%2E+Yang%2C+T%2E+Naumann%2C+H%2E+Poon%2C+and+J%2E+Gao%2E+Llava%2Dmed%3A+Training+a+large+language%2Dand%2Dvision+assistant+for+biomedicine+in+one+day%2E+Advances+in+Neural+Informa) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-llava-med-training-large-language/)
24. B. Y. Lin, Y. Deng, K. Chandu, A. Ravichander, V. Pyatkin, N. Dziri, R. Le Bras, and Y. Choi. Wildbench: Benchmarking LLMs with challenging tasks from real users in the wild. In Proceedings of the 13 th International Conference on Learning Representations (ICLR 2025), 2025. URL [https://openreview.n…](https://openreview.n…) [↗](https://openreview.n…)
25. X. Liu, D. McDuff, G. Kovacs, I. Galatzer-Levy, J. Sunshine, et al. Large language models are few-shot health learners. arXiv preprint arXiv:2305.15525, 2023. URL https://arxiv.org/abs/2305.15525. — [arXiv:2305.15525](https://arxiv.org/abs/2305.15525) [↗](https://arxiv.org/abs/2305.15525)
26. R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409, 2022. [find](https://scholar.google.com/scholar?q=R%2E+Luo%2C+L%2E+Sun%2C+Y%2E+Xia%2C+T%2E+Qin%2C+S%2E+Zhang%2C+H%2E+Poon%2C+and+T%2E%2DY%2E+Liu%2E+Biogpt%3A+generative+pre%2Dtrained+transformer+for+biomedical+text+generation+and+mining%2E+Briefings+in+bioinformatics%2C+23%286%29%3Abbac409%2C+2022)
27. D. McDuff, M. Schaekermann, T. Tu, and et al. Towards accurate differential diagnosis with large language models. Nature, 626:102–118, 2025. doi: 10.1038/s41586-025-08869-4. — [doi:10.1038/s41586-025-08869-4](https://doi.org/10.1038/s41586-025-08869-4) [find](https://scholar.google.com/scholar?q=D%2E+McDuff%2C+M%2E+Schaekermann%2C+T%2E+Tu%2C+and+et+al%2E+Towards+accurate+differential+diagnosis+with+large+language+models%2E+Nature%2C+626%3A102%E2%80%93118%2C+2025%2E+doi%3A+10%2E1038%2Fs41586%2D025%2D08869%2D4%2E+%E2%80%94+%5Bdoi%3A10%2E1038%2Fs41586%2D025%2D)
28. M. Moor, L. von Rueden, S. Adler, W. Ping, H. Valentin, et al. Med-flamingo: A multimodal medical few-shot learner. In Proceedings of the 3rd Machine Learning for Health Symposium, pages 353–367. PMLR, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Moor%2C+L%2E+von+Rueden%2C+S%2E+Adler%2C+W%2E+Ping%2C+H%2E+Valentin%2C+et+al%2E+Med%2Dflamingo%3A+A+multimodal+medical+few%2Dshot+learner%2E+In+Proceedings+of+the+3rd+Machine+Learning+for+Health+Symposium%2C+pages+353%E2%80%93367%2E+PMLR)
29. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2022… [find](https://scholar.google.com/scholar?q=R%2E+Nakano%2C+J%2E+Hilton%2C+S%2E+Balaji%2C+J%2E+Wu%2C+L%2E+Ouyang%2C+C%2E+Kim%2C+C%2E+Hesse%2C+S%2E+Jain%2C+V%2E+Kosaraju%2C+W%2E+Saunders%2C+X%2E+Jiang%2C+K%2E+Cobbe%2C+T%2E+Eloundou%2C+G%2E+Krueger%2C+K%2E+Button%2C+M%2E+Knight%2C+B%2E+Chess%2C+and+J%2E+Schulman%2E+We) [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
30. H. Nori, N. King, S. McKinney, D. Carignan, and E. Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023. doi: 10.48550/ARXIV.2303.13375. [find](https://scholar.google.com/scholar?q=H%2E+Nori%2C+N%2E+King%2C+S%2E+McKinney%2C+D%2E+Carignan%2C+and+E%2E+Horvitz%2E+Capabilities+of+gpt%2D4+on+medical+challenge+problems%2E+arXiv+preprint+arXiv%3A2303%2E13375%2C+2023%2E+doi%3A+10%2E48550%2FARXIV%2E2303%2E13375%2E) [our card](https://md.research.elamur.ai/free/en/papers/nori-2023-capabilities-gpt-4-medical-challenge/)
31. J. A. Omiye, H. Gui, S. J. Rezaei, J. Zou, and R. Daneshjou. Large language models in medicine: the potentials and pitfalls: a narrative review. Annals of Internal Medicine, 177(2):210–220, 2024. doi: 10.7326/M23-2772. — [doi:10.7326/m23-2772](https://doi.org/10.7326/m23-2772) [find](https://scholar.google.com/scholar?q=J%2E+A%2E+Omiye%2C+H%2E+Gui%2C+S%2E+J%2E+Rezaei%2C+J%2E+Zou%2C+and+R%2E+Daneshjou%2E+Large+language+models+in+medicine%3A+the+potentials+and+pitfalls%3A+a+narrative+review%2E+Annals+of+Internal+Medicine%2C+177%282%29%3A210%E2%80%93220%2C+2024%2E+doi%3A)
32. S. Pal et al. Medmcqa: A large-scale multi-subject multi-choice dataset for the medical domain. In EMNLP, 2022. [find](https://scholar.google.com/scholar?q=S%2E+Pal+et+al%2E+Medmcqa%3A+A+large%2Dscale+multi%2Dsubject+multi%2Dchoice+dataset+for+the+medical+domain%2E+In+EMNLP%2C+2022%2E)
33. S. R. Pfohl, H. Cole-Lewis, R. Sayres, D. Neal, M. Asiedu, A. Dieng, N. Tomasev, Q. M. Rashid, S. Azizi, N. Rostamzadeh, et al. A toolbox for surfacing health equity harms and biases in large language models. Nature Medicine, 30(12):3590–3600, 2024. [find](https://scholar.google.com/scholar?q=S%2E+R%2E+Pfohl%2C+H%2E+Cole%2DLewis%2C+R%2E+Sayres%2C+D%2E+Neal%2C+M%2E+Asiedu%2C+A%2E+Dieng%2C+N%2E+Tomasev%2C+Q%2E+M%2E+Rashid%2C+S%2E+Azizi%2C+N%2E+Rostamzadeh%2C+et+al%2E+A+toolbox+for+surfacing+health+equity+harms+and+biases+in+large+language)
34. K. Saab, T. Tu, W.-H. Weng, R. Tanno, D. Stutz, E. Wulczyn, F. Zhang, T. Strother, C. Park, E. Vedadi, et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416, 2024. [find](https://scholar.google.com/scholar?q=K%2E+Saab%2C+T%2E+Tu%2C+W%2E%2DH%2E+Weng%2C+R%2E+Tanno%2C+D%2E+Stutz%2C+E%2E+Wulczyn%2C+F%2E+Zhang%2C+T%2E+Strother%2C+C%2E+Park%2C+E%2E+Vedadi%2C+et+al%2E+Capabilities+of+gemini+models+in+medicine%2E+arXiv+preprint+arXiv%3A2404%2E18416%2C+2024%2E)
35. Scale AI. VISTA: Visual–language understanding leaderboard. [https://scale.com/leaderboard/visual_](https://scale.com/leaderboard/visual_) language_understanding, 2025. Accessed 3 May 2025. [↗](https://scale.com/leaderboard/visual_)
36. N. Shah, M. Pfeffer, and P. Liang. Holistic evaluation of large language models for medical applications, 2025. URL [https://hai.stanford.edu/news/](https://hai.stanford.edu/news/) holistic-evaluation-of-large-language-models-for-medical-applications. [↗](https://hai.stanford.edu/news/)
37. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023. [find](https://scholar.google.com/scholar?q=K%2E+Singhal%2C+S%2E+Azizi%2C+T%2E+Tu%2C+S%2E+S%2E+Mahdavi%2C+J%2E+Wei%2C+H%2E+W%2E+Chung%2C+N%2E+Scales%2C+A%2E+Tanwani%2C+H%2E+Cole%2DLewis%2C+S%2E+Pfohl%2C+et+al%2E+Large+language+models+encode+clinical+knowledge%2E+Nature%2C+620%287972%29%3A172%E2%80%93180%2C+2023) [our card](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)
38. K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1–8, 2025. [find](https://scholar.google.com/scholar?q=K%2E+Singhal%2C+T%2E+Tu%2C+J%2E+Gottweis%2C+R%2E+Sayres%2C+E%2E+Wulczyn%2C+M%2E+Amin%2C+L%2E+Hou%2C+K%2E+Clark%2C+S%2E+R%2E+Pfohl%2C+H%2E+Cole%2DLewis%2C+et+al%2E+Toward+expert%2Dlevel+medical+question+answering+with+large+language+models%2E+Nature+M)
39. V. Sirdeshmukh, K. Deshpande, J. Mols, L. Jin, E.-Y. Cardona, D. Lee, J. Kritz, W. Primack, S. Yue, and C. Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier LLMs. arXiv preprint arXiv:2501.17399, 2025. URL https://arxiv.org/abs/2501.17399. [↗](https://arxiv.org/abs/2501.17399)
40. G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025. [find](https://scholar.google.com/scholar?q=G%2E+Starace%2C+O%2E+Jaffe%2C+D%2E+Sherburn%2C+J%2E+Aung%2C+J%2E+S%2E+Chan%2C+L%2E+Maksin%2C+R%2E+Dias%2C+E%2E+Mays%2C+B%2E+Kinsella%2C+W%2E+Thompson%2C+et+al%2E+Paperbench%3A+Evaluating+ai%E2%80%99s+ability+to+replicate+ai+research%2E+arXiv+preprint+arXiv)
41. S. Takagi, T. Watari, A. Erabi, K. Sakaguchi, et al. Performance of gpt-3.5 and gpt-4 on the japanese medical licensing examination: comparison study. JMIR medical education, 9(1):e48002, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Takagi%2C+T%2E+Watari%2C+A%2E+Erabi%2C+K%2E+Sakaguchi%2C+et+al%2E+Performance+of+gpt%2D3%2E5+and+gpt%2D4+on+the+japanese+medical+licensing+examination%3A+comparison+study%2E+JMIR+medical+education%2C+9%281%29%3Ae48002%2C+2023%2E)
42. R. Tanno, D. G. Barrett, A. Sellergren, S. Ghaisas, S. Dathathri, A. See, J. Welbl, C. Lau, T. Tu, S. Azizi, et al. Collaboration between clinicians and vision–language models in radiology report generation. Nature Medicine, 31(2):599–608, 2025. [find](https://scholar.google.com/scholar?q=R%2E+Tanno%2C+D%2E+G%2E+Barrett%2C+A%2E+Sellergren%2C+S%2E+Ghaisas%2C+S%2E+Dathathri%2C+A%2E+See%2C+J%2E+Welbl%2C+C%2E+Lau%2C+T%2E+Tu%2C+S%2E+Azizi%2C+et+al%2E+Collaboration+between+clinicians+and+vision%E2%80%93language+models+in+radiology+report+gene)
43. A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. [find](https://scholar.google.com/scholar?q=A%2E+J%2E+Thirunavukarasu%2C+D%2E+S%2E+J%2E+Ting%2C+K%2E+Elangovan%2C+L%2E+Gutierrez%2C+T%2E+F%2E+Tan%2C+and+D%2E+S%2E+W%2E+Ting%2E+Large+language+models+in+medicine%2E+Nature+medicine%2C+29%288%29%3A1930%E2%80%931940%2C+2023%2E)
44. E. J. Topol. High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25(1):44–56, 2019. doi: 10.1038/s41591-018-0300-7. — [doi:10.1038/s41591-018-0300-7](https://doi.org/10.1038/s41591-018-0300-7) [find](https://scholar.google.com/scholar?q=E%2E+J%2E+Topol%2E+High%2Dperformance+medicine%3A+The+convergence+of+human+and+artificial+intelligence%2E+Nature+Medicine%2C+25%281%29%3A44%E2%80%9356%2C+2019%2E+doi%3A+10%2E1038%2Fs41591%2D018%2D0300%2D7%2E+%E2%80%94+%5Bdoi%3A10%2E1038%2Fs41591%2D018%2D0300%2D7%5D%28http)
45. T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P.-C. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena, et al. Towards generalist biomedical ai. Nejm Ai, 1(3):AIoa2300138, 2024. [find](https://scholar.google.com/scholar?q=T%2E+Tu%2C+S%2E+Azizi%2C+D%2E+Driess%2C+M%2E+Schaekermann%2C+M%2E+Amin%2C+P%2E%2DC%2E+Chang%2C+A%2E+Carroll%2C+C%2E+Lau%2C+R%2E+Tanno%2C+I%2E+Ktena%2C+et+al%2E+Towards+generalist+biomedical+ai%2E+Nejm+Ai%2C+1%283%29%3AAIoa2300138%2C+2024%2E)
46. T. Tu, M. Schaekermann, A. Palepu, K. Saab, J. Freyberg, R. Tanno, A. Wang, B. Li, M. Amin, Y. Cheng, et al. Towards conversational diagnostic artificial intelligence. Nature, pages 1–9, 2025. [find](https://scholar.google.com/scholar?q=T%2E+Tu%2C+M%2E+Schaekermann%2C+A%2E+Palepu%2C+K%2E+Saab%2C+J%2E+Freyberg%2C+R%2E+Tanno%2C+A%2E+Wang%2C+B%2E+Li%2C+M%2E+Amin%2C+Y%2E+Cheng%2C+et+al%2E+Towards+conversational+diagnostic+artificial+intelligence%2E+Nature%2C+pages+1%E2%80%939%2C+2025%2E)
47. C. Y. K. Williams, C. R. Subramanian, S. S. Ali, and et al. Physician- and large language model-generated hospital discharge summaries: A blinded, comparative quality and safety study. medRxiv, 2024. doi: 10.1101/2024.09.29.24314562. — [doi:10.1101/2024.09.29.24314562](https://doi.org/10.1101/2024.09.29.24314562) [find](https://scholar.google.com/scholar?q=C%2E+Y%2E+K%2E+Williams%2C+C%2E+R%2E+Subramanian%2C+S%2E+S%2E+Ali%2C+and+et+al%2E+Physician%2D+and+large+language+model%2Dgenerated+hospital+discharge+summaries%3A+A+blinded%2C+comparative+quality+and+safety+study%2E+medRxiv%2C+2024%2E+)
48. J. Ye, Y. Wang, Y. Huang, D. Chen, Q. Zhang, N. Moniz, T. Gao, W. Geyer, C. Huang, P.-Y. Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Ye%2C+Y%2E+Wang%2C+Y%2E+Huang%2C+D%2E+Chen%2C+Q%2E+Zhang%2C+N%2E+Moniz%2C+T%2E+Gao%2C+W%2E+Geyer%2C+C%2E+Huang%2C+P%2E%2DY%2E+Chen%2C+et+al%2E+Justice+or+prejudice%3F+quantifying+biases+in+llm%2Das%2Da%2Djudge%2E+arXiv+preprint+arXiv%3A2410%2E02736%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/ye-2024-justice-prejudice-quantifying-biases-llm/)
49. C. Zakka, N. Kiani, A. Wong, J. Shi, C. Raffel, et al. Almanac — retrieval-augmented language models for clinical medicine. NEJM AI, 2024. doi: 10.1056/AIoa2300068. — [doi:10.1056/aioa2300068](https://doi.org/10.1056/aioa2300068) [find](https://scholar.google.com/scholar?q=C%2E+Zakka%2C+N%2E+Kiani%2C+A%2E+Wong%2C+J%2E+Shi%2C+C%2E+Raffel%2C+et+al%2E+Almanac+%E2%80%94+retrieval%2Daugmented+language+models+for+clinical+medicine%2E+NEJM+AI%2C+2024%2E+doi%3A+10%2E1056%2FAIoa2300068%2E+%E2%80%94+%5Bdoi%3A10%2E1056%2Faioa2300068%5D%28https%3A%2F)
50. W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2309.11998, 2024. doi: 10.48550/arXiv.2309.11998. [find](https://scholar.google.com/scholar?q=W%2E+Zhao%2C+X%2E+Ren%2C+J%2E+Hessel%2C+C%2E+Cardie%2C+Y%2E+Choi%2C+and+Y%2E+Deng%2E+Wildchat%3A+1m+chatgpt+interaction+logs+in+the+wild%2E+arXiv+preprint+arXiv%3A2309%2E11998%2C+2024%2E+doi%3A+10%2E48550%2FarXiv%2E2309%2E11998%2E) [our card](https://md.research.elamur.ai/free/en/papers/zheng-2023-lmsys-chat-1m-large-scale/)
51. L. Zheng, W.-L. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2405.01470, 2024. doi: 10.48550/arXiv.2405.01470. [find](https://scholar.google.com/scholar?q=L%2E+Zheng%2C+W%2E%2DL%2E+Chiang%2C+Y%2E+Sheng%2C+T%2E+Li%2C+S%2E+Zhuang%2C+Z%2E+Wu%2C+Y%2E+Zhuang%2C+Z%2E+Li%2C+Z%2E+Lin%2C+E%2E+P%2E+Xing%2C+J%2E+E%2E+Gonzalez%2C+I%2E+Stoica%2C+and+H%2E+Zhang%2E+Lmsys%2Dchat%2D1m%3A+A+large%2Dscale+real%2Dworld+llm+conversation+datas) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2024-wildchat-1m-chatgpt-interaction-logs/)