---
title: "Benchmarking Large Language Models on Answering and"
description: "The authors build two English multiple-choice datasets with expert-written explanations. 73 references, cited by 50 papers here."
authors: ["Hanjie Chen", "Zhouxiang Fang", "Yash Singla", "Mark Dredze"]
year: 2025
venue: "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.naacl-long.182/"
doi: "10.18653/v1/2025.naacl-long.182"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-benchmarking-large-language-models-answering/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-benchmarking-large-language-models-answering/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions

# Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions

**Authors:** [Hanjie Chen](https://md.research.elamur.ai/free/en/authors/hanjie-chen/), Zhouxiang Fang, Yash Singla and [Mark Dredze](https://md.research.elamur.ai/free/en/authors/mark-dredze/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2025](https://md.research.elamur.ai/free/en/venues/naacl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.naacl-long.182/) · [PDF](https://aclanthology.org/2025.naacl-long.182.pdf) · [DOI 10.18653/v1/2025.naacl-long.182](https://doi.org/10.18653/v1/2025.naacl-long.182) · [OpenAlex](https://openalex.org/W4411119840)

**Citations:** 25 citations (OpenAlex, 2026-10-14) · [50](https://md.research.elamur.ai/free/en/papers/chen-2025-benchmarking-large-language-models-answering/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Medical LLMs score well on licensing-exam benchmarks, but those questions mostly test textbook knowledge. They do not reflect messy real clinical cases, and they rarely include reference rationales for checking model reasoning. The authors build two English multiple-choice datasets with expert-written explanations. JAMA Clinical Challenge is drawn from difficult, atypical cases published by JAMA. Medbullets consists of recent USMLE Step 2/3-style questions shared publicly on X. Images were removed so that text-only LLMs could be tested.

Seven models were benchmarked under several prompting setups, including direct answering, chain-of-thought, and post-hoc explanation of a given answer. The models were GPT-3.5, GPT-4, PaLM 2, Llama 2, Llama 3, MedAlpaca and Meerkat. Accuracy on both datasets falls below that on earlier benchmarks such as MedQA, and in-context learning or prompting gives only small gains.

Generated explanations were assessed with automatic metrics and with human raters. The two kinds of scores agree poorly, which exposes a gap in evaluation methods for explainable clinical QA.

## Key points
- JAMA Clinical Challenge has 1524 cases from July 2013 to October 2023, covering 13 medical domains, each with 4 options. Inputs average 371 tokens and explanations 632.
- Medbullets has 308 Step 2/3 questions from April 2022 to December 2023, with 5 options. A 4-option variant was also built for comparison, giving Medbullets-5 and Medbullets-4. None of these questions overlap with MedQA.
- Medbullets inputs average 163 tokens, compared with 137 for MedQA. Its explanations average 413 tokens.
- In human evaluation on Medbullets-5 explanations produced with XY*→R prompting, GPT-4 outperformed PaLM 2 on all three rated properties. Raters were recruited via Prolific, with 3 annotations per item.
- Automatic metrics such as G-Eval showed weak or no Pearson correlation with human scores. Data and code are available at github.com/HanjieChen/ChallengeClinicalQA.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 187 works · 92,194 citations
- [Rice University](https://md.research.elamur.ai/free/en/universities/rice-university/) — University, Houston, United States · 5 works · 1,679 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 80 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 53 works

## Cite

Cite as: Hanjie Chen, Zhouxiang Fang, Yash Singla and Mark Dredze. 2025. Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2025benchmarking,
  title = {{Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions}},
  author = {Hanjie Chen, Zhouxiang Fang, Yash Singla and Mark Dredze},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.182},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.182}
}
```

- [← previous AudioBench: A Universal Benchmark for Audio Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2025-audiobench-universal-benchmark-audio-large/)
- [next → In-Context Learning with Long-Context Models: An In-Depth Exploration](https://md.research.elamur.ai/free/en/papers/bertsch-2025-context-learning-long-context-models/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/)

## References

73 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman. 2017. Overview of the medical question answering task at trec 2017 liveqa. In Text Retrieval Conference. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha%2C+Eugene+Agichtein%2C+Yuval+Pinter%2C+and+Dina+Demner%2DFushman%2E+2017%2E+Overview+of+the+medical+question+answering+task+at+trec+2017+liveqa%2E+In+Text+Retrieval+Conference%2E)
2. Asma Ben Abacha and Dina Demner-Fushman. 2019. A question-entailment approach to question answering. BMC Bioinformatics, 20. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha+and+Dina+Demner%2DFushman%2E+2019%2E+A+question%2Dentailment+approach+to+question+answering%2E+BMC+Bioinformatics%2C+20%2E)
3. Asma Ben Abacha, Yassine Mrabet, Mark E. Sharp, Travis R. Goodwin, Sonya E. Shooshan, and Dina Demner-Fushman. 2019. Bridging the gap between consumers’ medication questions and trusted answers. Studies in health technology and informatics, 264:25–29. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha%2C+Yassine+Mrabet%2C+Mark+E%2E+Sharp%2C+Travis+R%2E+Goodwin%2C+Sonya+E%2E+Shooshan%2C+and+Dina+Demner%2DFushman%2E+2019%2E+Bridging+the+gap+between+consumers%E2%80%99+medication+questions+and+trusted+answers%2E+Studi)
4. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. [The falcon series of open language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/). arXiv preprint arXiv:2311.16867.
5. Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. 2024. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards.… [find](https://scholar.google.com/scholar?q=Norah+Alzahrani%2C+Hisham+Alyahya%2C+Yazeed+Alnumay%2C+Sultan+AlRashed%2C+Shaykhah+Alsubaie%2C+Yousef+Almushayqih%2C+Faisal+Mirza%2C+Nouf+Alotaibi%2C+Nora+Al%2DTwairesh%2C+Areeb+Alowisheq%2C+M+Saiful+Bari%2C+and+Haidar+Khan%2E)
6. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403.
7. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E)
8. Joseph Barile, Alex Margolis, Grace Cason, Rachel Kim, Saia Kalash, Alexis Tchaconas, and Ruth Mi-lanaik. 2024. Diagnostic accuracy of a large language model in pediatric case studies. JAMA Pediatrics, 178(3):313–315. [find](https://scholar.google.com/scholar?q=Joseph+Barile%2C+Alex+Margolis%2C+Grace+Cason%2C+Rachel+Kim%2C+Saia+Kalash%2C+Alexis+Tchaconas%2C+and+Ruth+Mi%2Dlanaik%2E+2024%2E+Diagnostic+accuracy+of+a+large+language+model+in+pediatric+case+studies%2E+JAMA+Pediatrics)
9. E. M. BENNETT, R. ALPERT, and A. C. GOLDSTEIN. 1954. Communications through limited-response questioning. Public Opinion Quarterly, 18(3):303– 308. [find](https://scholar.google.com/scholar?q=E%2E+M%2E+BENNETT%2C+R%2E+ALPERT%2C+and+A%2E+C%2E+GOLDSTEIN%2E+1954%2E+Communications+through+limited%2Dresponse+questioning%2E+Public+Opinion+Quarterly%2C+18%283%29%3A303%E2%80%93+308%2E)
10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
11. Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2022. Meta-learning via language model in-context tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 719–730, Dublin, Ireland. Association for Computational L… [find](https://scholar.google.com/scholar?q=Yanda+Chen%2C+Ruiqi+Zhong%2C+Sheng+Zha%2C+George+Karypis%2C+and+He+He%2E+2022%2E+Meta%2Dlearning+via+language+model+in%2Dcontext+tuning%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+)
12. Zeming Chen, Alejandro Hern’andez Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Kopf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne… — [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/)
13. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, E… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
14. Tirth Dave, Sai Anirudh Athaluri, and Satyam Singh. 2023. Chatgpt in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Frontiers in Artificial Intelligence, 6:1169595. [find](https://scholar.google.com/scholar?q=Tirth+Dave%2C+Sai+Anirudh+Athaluri%2C+and+Satyam+Singh%2E+2023%2E+Chatgpt+in+medicine%3A+an+overview+of+its+applications%2C+advantages%2C+limitations%2C+future+prospects%2C+and+ethical+considerations%2E+Frontiers+in+Arti)
15. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. [Investigating data contamination in modern benchmarks for large language models](https://md.research.elamur.ai/free/en/papers/deng-2024-investigating-data-contamination-modern-benchmarks/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
16. Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021. [Compression, transduction, and creation: A unified framework for evaluating natural language generation](https://md.research.elamur.ai/free/en/papers/deng-2021-compression-transduction-creation-unified-framework/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580–7605, Online and Pu…
17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
18. Alexander V. Eriksen, Sören Möller, and Jesper Ryg. 2024. Use of gpt-4 to diagnose complex clinical cases. NEJM AI, 1(1):AIp2300031. [find](https://scholar.google.com/scholar?q=Alexander+V%2E+Eriksen%2C+S%C3%B6ren+M%C3%B6ller%2C+and+Jesper+Ryg%2E+2024%2E+Use+of+gpt%2D4+to+diagnose+complex+clinical+cases%2E+NEJM+AI%2C+1%281%29%3AAIp2300031%2E)
19. Scott L. Fleming, Alejandro Lozano, William J. Haberkorn, Jenelle A. Jindal, Eduardo Pontes Reis, Rahul Thapa, Louis Blankemeier, Julian Z. Genkins, Ethan Steinberg, Ashwin Nayak, Birju S. Patel, Chia-Chun Chiang, Alison Callahan, Zepeng Huo, Sergios Gatidis, Scott J. Adams, Oluseyi Fayanju, Shreya… [find](https://scholar.google.com/scholar?q=Scott+L%2E+Fleming%2C+Alejandro+Lozano%2C+William+J%2E+Haberkorn%2C+Jenelle+A%2E+Jindal%2C+Eduardo+Pontes+Reis%2C+Rahul+Thapa%2C+Louis+Blankemeier%2C+Julian+Z%2E+Genkins%2C+Ethan+Steinberg%2C+Ashwin+Nayak%2C+Birju+S%2E+Patel%2C+Chia)
20. Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. 2023. [Medalpaca–an open-source collection of medical conversational ai models and training data](https://md.research.elamur.ai/free/en/papers/han-2023-medalpaca-open-source-collection-medical/). arXiv preprint arXiv:2304.08247.
21. Emily Harris. 2023. Large language models answer medical questions accurately, but can’t match clinicians’ knowledge. JAMA, 330(9):792–794. [find](https://scholar.google.com/scholar?q=Emily+Harris%2E+2023%2E+Large+language+models+answer+medical+questions+accurately%2C+but+can%E2%80%99t+match+clinicians%E2%80%99+knowledge%2E+JAMA%2C+330%289%29%3A792%E2%80%93794%2E)
22. Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. 2009. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer. [find](https://scholar.google.com/scholar?q=Trevor+Hastie%2C+Robert+Tibshirani%2C+Jerome+H+Friedman%2C+and+Jerome+H+Friedman%2E+2009%2E+The+elements+of+statistical+learning%3A+data+mining%2C+inference%2C+and+prediction%2C+volume+2%2E+Springer%2E)
23. Zexue He, Yu Wang, An Yan, Yao Liu, Eric Chang, Amilcare Gentili, Julian McAuley, and Chun-Nan Hsu. 2023. MedEval: A multi-level, multi-task, and multi-domain medical benchmark for language model evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p… [find](https://scholar.google.com/scholar?q=Zexue+He%2C+Yu+Wang%2C+An+Yan%2C+Yao+Liu%2C+Eric+Chang%2C+Amilcare+Gentili%2C+Julian+McAuley%2C+and+Chun%2DNan+Hsu%2E+2023%2E+MedEval%3A+A+multi%2Dlevel%2C+multi%2Dtask%2C+and+multi%2Ddomain+medical+benchmark+for+language+model+eval)
24. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
25. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). Advances in neural information processing systems, 28.
26. J. Edward Hu, Abhinav Singh, Nils Holzenberger, Matt Post, and Benjamin Van Durme. 2019. Large-scale, diverse, paraphrastic bitexts via sampling and clustering. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 44–54, Hong Kong, China. Association for Co… [find](https://scholar.google.com/scholar?q=J%2E+Edward+Hu%2C+Abhinav+Singh%2C+Nils+Holzenberger%2C+Matt+Post%2C+and+Benjamin+Van+Durme%2E+2019%2E+Large%2Dscale%2C+diverse%2C+paraphrastic+bitexts+via+sampling+and+clustering%2E+In+Proceedings+of+the+23rd+Conference+o)
27. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421.
28. Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, Justin M Cheung, Robert Chen, Ronald M Summers, Justin F Rousseau, Peiyun Ni, Marc J Landsman, et al. 2024. Hidden flaws behind expert-level accuracy of gpt-4 vision in medicine. arXiv preprint arXiv:2401.08396. [find](https://scholar.google.com/scholar?q=Qiao+Jin%2C+Fangyuan+Chen%2C+Yiliang+Zhou%2C+Ziyang+Xu%2C+Justin+M+Cheung%2C+Robert+Chen%2C+Ronald+M+Summers%2C+Justin+F+Rousseau%2C+Peiyun+Ni%2C+Marc+J+Landsman%2C+et+al%2E+2024%2E+Hidden+flaws+behind+expert%2Dlevel+accuracy+)
29. Zahir Kanjee, Byron Crowe, and Adam Rodman. 2023. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. JAMA. [find](https://scholar.google.com/scholar?q=Zahir+Kanjee%2C+Byron+Crowe%2C+and+Adam+Rodman%2E+2023%2E+Accuracy+of+a+generative+artificial+intelligence+model+in+a+complex+diagnostic+challenge%2E+JAMA%2E)
30. Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi, and Jaewoo Kang. 2024a. Small language models learn enhanced reasoning skills from medical textbooks. arXiv preprint arXiv:2404.00376. [find](https://scholar.google.com/scholar?q=Hyunjae+Kim%2C+Hyeon+Hwang%2C+Jiwoo+Lee%2C+Sihyeon+Park%2C+Dain+Kim%2C+Taewhoo+Lee%2C+Chanwoong+Yoon%2C+Jiwoong+Sohn%2C+Donghee+Choi%2C+and+Jaewoo+Kang%2E+2024a%2E+Small+language+models+learn+enhanced+reasoning+skills+from)
31. Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. 2024b. MedExQA: Medical question answering benchmark with multiple explanations. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 167–181, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yunsoo+Kim%2C+Jinge+Wu%2C+Yusuf+Abdulle%2C+and+Honghan+Wu%2E+2024b%2E+MedExQA%3A+Medical+question+answering+benchmark+with+multiple+explanations%2E+In+Proceedings+of+the+23rd+Workshop+on+Biomedical+Natural+Language)
32. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199– 22213.
33. Sunjun Kweon, Junu Kim, Jiyoun Kim, Sujeong Im, Eunbyeol Cho, Seongsu Bae, Jungwoo Oh, Gyubok Lee, Jong Hak Moon, Seng Chan You, Seungjin Baek, Chang Hoon Han, Yoon Bin Jung, Yohan Jo, and Edward Choi. 2024. Publicly shareable clinical large language model built on synthetic clinical notes. In Findi… [find](https://scholar.google.com/scholar?q=Sunjun+Kweon%2C+Junu+Kim%2C+Jiyoun+Kim%2C+Sujeong+Im%2C+Eunbyeol+Cho%2C+Seongsu+Bae%2C+Jungwoo+Oh%2C+Gyubok+Lee%2C+Jong+Hak+Moon%2C+Seng+Chan+You%2C+Seungjin+Baek%2C+Chang+Hoon+Han%2C+Yoon+Bin+Jung%2C+Yohan+Jo%2C+and+Edward+Choi)
34. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
35. Dongfang Li, Jindi Yu, Baotian Hu, Zhenran Xu, and Min Zhang. 2023a. ExplainCPE: A free-text explanation benchmark of Chinese pharmacist examination. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1922–1940, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dongfang+Li%2C+Jindi+Yu%2C+Baotian+Hu%2C+Zhenran+Xu%2C+and+Min+Zhang%2E+2023a%2E+ExplainCPE%3A+A+free%2Dtext+explanation+benchmark+of+Chinese+pharmacist+examination%2E+In+Findings+of+the+Association+for+Computational+L)
36. Ron Li, Andre Kumar, and Jonathan H Chen. 2023b. How chatbots and large language model artificial intelligence systems will reshape modern medicine: Fountain of creativity or pandora’s box? JAMA Internal Medicine. [find](https://scholar.google.com/scholar?q=Ron+Li%2C+Andre+Kumar%2C+and+Jonathan+H+Chen%2E+2023b%2E+How+chatbots+and+large+language+model+artificial+intelligence+systems+will+reshape+modern+medicine%3A+Fountain+of+creativity+or+pandora%E2%80%99s+box%3F+JAMA+Inter)
37. Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. 2022. Can large language models reason about medical questions? arXiv preprint arXiv:2207.08143. [find](https://scholar.google.com/scholar?q=Valentin+Li%C3%A9vin%2C+Christoffer+Egeberg+Hother%2C+and+Ole+Winther%2E+2022%2E+Can+large+language+models+reason+about+medical+questions%3F+arXiv+preprint+arXiv%3A2207%2E08143%2E)
38. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
39. Fenglin Liu, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang, Xianfeng Tang, Chen Luo, Ming Zeng, Haoming Jiang, Yifan Gao, Priyanka Nigam, Sreyashi Nag, Bing Yin, Yining Hua, Xuan Zhou, Omid Rohanian, Anshul Thakur, Lei Clifton, and David A. Clifton. 2024. Large language models are poor clinical… [find](https://scholar.google.com/scholar?q=Fenglin+Liu%2C+Zheng+Li%2C+Hongjian+Zhou%2C+Qingyu+Yin%2C+Jingfeng+Yang%2C+Xianfeng+Tang%2C+Chen+Luo%2C+Ming+Zeng%2C+Haoming+Jiang%2C+Yifan+Gao%2C+Priyanka+Nigam%2C+Sreyashi+Nag%2C+Bing+Yin%2C+Yining+Hua%2C+Xuan+Zhou%2C+Omid+Rohan)
40. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023%2E+G%2Deval%3A+NLG+evaluation+using+gpt%2D4+with+better+human+alignment%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+M)
41. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
42. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157–165, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Inbal+Magar+and+Roy+Schwartz%2E+2022%2E+Data+contamination%3A+From+memorization+to+exploitation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+P)
43. Itay Manes, Naama Ronn, David Cohen, Ran Ilan Ber, Zehavi Horowitz-Kugler, and Gabriel Stanovsky. 2024. K-qa: A real-world medical q&a benchmark. arXiv preprint arXiv:2401.14493. [find](https://scholar.google.com/scholar?q=Itay+Manes%2C+Naama+Ronn%2C+David+Cohen%2C+Ran+Ilan+Ber%2C+Zehavi+Horowitz%2DKugler%2C+and+Gabriel+Stanovsky%2E+2024%2E+K%2Dqa%3A+A+real%2Dworld+medical+q%26a+benchmark%2E+arXiv+preprint+arXiv%3A2401%2E14493%2E)
44. Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. [find](https://scholar.google.com/scholar?q=Meta%2E+2024%2E+Introducing+meta+llama+3%3A+The+most+capable+openly+available+llm+to+date%2E)
45. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab…
46. Mihir Parmar, Aakanksha Naik, Himanshu Gupta, Disha Agrawal, and Chitta Baral. 2023. Longbox: Evaluating transformers on long-sequence clinical tasks. arXiv preprint arXiv:2311.09564. — [arXiv:2311.09564](https://arxiv.org/abs/2311.09564) [find](https://scholar.google.com/scholar?q=Mihir+Parmar%2C+Aakanksha+Naik%2C+Himanshu+Gupta%2C+Disha+Agrawal%2C+and+Chitta+Baral%2E+2023%2E+Longbox%3A+Evaluating+transformers+on+long%2Dsequence+clinical+tasks%2E+arXiv+preprint+arXiv%3A2311%2E09564%2E+%E2%80%94+%5BarXiv%3A2311%2E09)
47. Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. 2024. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416. [find](https://scholar.google.com/scholar?q=Khaled+Saab%2C+Tao+Tu%2C+Wei%2DHung+Weng%2C+Ryutaro+Tanno%2C+David+Stutz%2C+Ellery+Wulczyn%2C+Fan+Zhang%2C+Tim+Strother%2C+Chunjong+Park%2C+Elahe+Vedadi%2C+et+al%2E+2024%2E+Capabilities+of+gemini+models+in+medicine%2E+arXiv+prep)
48. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. [NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singap…
49. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023a. [Capabilities of gpt-4 on medical challenge problems](https://md.research.elamur.ai/free/en/papers/nori-2023-capabilities-gpt-4-medical-challenge/). arXiv preprint arXiv:2303.13375.
50. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
51. Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. 2023b. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452. [find](https://scholar.google.com/scholar?q=Harsha+Nori%2C+Yin+Tat+Lee%2C+Sheng+Zhang%2C+Dean+Carignan%2C+Richard+Edgar%2C+Nicolo+Fusi%2C+Nicholas+King%2C+Jonathan+Larson%2C+Yuanzhi+Li%2C+Weishung+Liu%2C+et+al%2E+2023b%2E+Can+generalist+foundation+models+outcompete+sp)
52. Yat-Fung Shea, Cynthia Min Yao Lee, Whitney Chin Tung Ip, Dik Wai Anderson Luk, and Stephanie Sze Wing Wong. 2023. Use of gpt-4 to analyze medical records of patients with extensive investigations and delayed diagnosis. JAMA Network Open, 6(8):e2325000–e2325000. [find](https://scholar.google.com/scholar?q=Yat%2DFung+Shea%2C+Cynthia+Min+Yao+Lee%2C+Whitney+Chin+Tung+Ip%2C+Dik+Wai+Anderson+Luk%2C+and+Stephanie+Sze+Wing+Wong%2E+2023%2E+Use+of+gpt%2D4+to+analyze+medical+records+of+patients+with+extensive+investigations+and)
53. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
54. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022. [Large language models encode clinical knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/). arXiv preprint arXiv:2212.13138.
55. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
56. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. [Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/). In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 24…
57. Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. [emrqa: A large corpus for question answering on electronic medical records](https://md.research.elamur.ai/free/en/papers/pampari-2018-emrqa-large-corpus-question-answering/). arXiv preprint arXiv:1809.00732.
58. Cecilia Panigutti, Andrea Beretta, Fosca Giannotti, and Dino Pedreschi. 2022. Understanding the impact of explanations on advice-taking: a user study for ai-based clinical decision support systems. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY… [find](https://scholar.google.com/scholar?q=Cecilia+Panigutti%2C+Andrea+Beretta%2C+Fosca+Giannotti%2C+and+Dino+Pedreschi%2E+2022%2E+Understanding+the+impact+of+explanations+on+advice%2Dtaking%3A+a+user+study+for+ai%2Dbased+clinical+decision+support+systems%2E+In)
59. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023. [Towards expert-level medical question answering with large language models](https://md.research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/). arXiv preprint arXiv:2305.09617.
60. Sarvesh Soni, Meghana Gudala, Atieh Pajouhi, and Kirk Roberts. 2022. RadQA: A question answering dataset to improve comprehension of radiology reports. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6250– 6259, Marseille, France. European Language Resources Asso… [find](https://scholar.google.com/scholar?q=Sarvesh+Soni%2C+Meghana+Gudala%2C+Atieh+Pajouhi%2C+and+Kirk+Roberts%2E+2022%2E+RadQA%3A+A+question+answering+dataset+to+improve+comprehension+of+radiology+reports%2E+In+Proceedings+of+the+Thirteenth+Language+Resour)
61. Eric Strong, Alicia DiGiammarino, Yingjie Weng, Andre Kumar, Poonam Hosamani, Jason Hom, and Jonathan H. Chen. 2023. Chatbot vs medical student performance on free-response clinical reasoning examinations. JAMA Internal Medicine, 183(9):1028– 1030. [find](https://scholar.google.com/scholar?q=Eric+Strong%2C+Alicia+DiGiammarino%2C+Yingjie+Weng%2C+Andre+Kumar%2C+Poonam+Hosamani%2C+Jason+Hom%2C+and+Jonathan+H%2E+Chen%2E+2023%2E+Chatbot+vs+medical+student+performance+on+free%2Dresponse+clinical+reasoning+examinat)
62. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
63. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
64. Guanchu Wang, Junhao Ran, Ruixiang Tang, Chia-Yuan Chang, Yu-Neng Chuang, Zirui Liu, Vladimir Braverman, Zhandong Liu, and Xia Hu. 2024. Assessing and enhancing large language models in rare disease question-answering. arXiv preprint arXiv:2408.08422. — [arXiv:2408.08422](https://arxiv.org/abs/2408.08422) [find](https://scholar.google.com/scholar?q=Guanchu+Wang%2C+Junhao+Ran%2C+Ruixiang+Tang%2C+Chia%2DYuan+Chang%2C+Yu%2DNeng+Chuang%2C+Zirui+Liu%2C+Vladimir+Braverman%2C+Zhandong+Liu%2C+and+Xia+Hu%2E+2024%2E+Assessing+and+enhancing+large+language+models+in+rare+disease+q)
65. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
66. Yubo Wang, Xueguang Ma, and Wenhu Chen. 2023. Augmenting black-box llms with medical textbooks for clinical question answering. arXiv preprint arXiv:2309.02233. — [arXiv:2309.02233](https://arxiv.org/abs/2309.02233) [find](https://scholar.google.com/scholar?q=Yubo+Wang%2C+Xueguang+Ma%2C+and+Wenhu+Chen%2E+2023%2E+Augmenting+black%2Dbox+llms+with+medical+textbooks+for+clinical+question+answering%2E+arXiv+preprint+arXiv%3A2309%2E02233%2E+%E2%80%94+%5BarXiv%3A2309%2E02233%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
67. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, R…
68. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [Bartscore: evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA. Curran Associates Inc.
69. Xiang Yue, Xinliang Frederick Zhang, Ziyu Yao, Simon M. Lin, and Huan Sun. 2020. Cliniqg4qa: Generating diverse questions for domain adaptation of clinical question answering. 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 580–587. [find](https://scholar.google.com/scholar?q=Xiang+Yue%2C+Xinliang+Frederick+Zhang%2C+Ziyu+Yao%2C+Simon+M%2E+Lin%2C+and+Huan+Sun%2E+2020%2E+Cliniqg4qa%3A+Generating+diverse+questions+for+domain+adaptation+of+clinical+question+answering%2E+2021+IEEE+International+)
70. Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, Xiang Wan, Benyou Wang, and Haizhou Li. 2023a. [HuatuoGPT, towards taming language model to be a doctor](https://md.research.elamur.ai/free/en/papers/zhang-2023-huatuogpt-taming-language-model-be/). In Findings of the Association for Computational Linguistics: EMNLP…
71. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E)
72. Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. 2023b. Alpacare: Instruction-tuned large language models for medical application. arXiv preprint arXiv:2310.14558. — [arXiv:2310.14558](https://arxiv.org/abs/2310.14558) [find](https://scholar.google.com/scholar?q=Xinlu+Zhang%2C+Chenxin+Tian%2C+Xianjun+Yang%2C+Lichang+Chen%2C+Zekun+Li%2C+and+Linda+Ruth+Petzold%2E+2023b%2E+Alpacare%3A+Instruction%2Dtuned+large+language+models+for+medical+application%2E+arXiv+preprint+arXiv%3A2310%2E145)
73. Domain A in /M in A exp /M exp Ophthalmology 395/564 593/810 Dermatology 339/522 648/854 General 335/575 711/1080 Pathology 382/503 626/876 Surgery 328/610 496/837 Radiology 367/674 629/785 Oncology 393/728 670/1369 Cardiology 370/574 603/876 Diagnostic 331/611 833/1196 Pediatrics 351/525 659/875 Ne… [find](https://scholar.google.com/scholar?q=Domain+A+in+%2FM+in+A+exp+%2FM+exp+Ophthalmology+395%2F564+593%2F810+Dermatology+339%2F522+648%2F854+General+335%2F575+711%2F1080+Pathology+382%2F503+626%2F876+Surgery+328%2F610+496%2F837+Radiology+367%2F674+629%2F785+Oncology+3)

*Updated 24 October 2026*