--- title: "Towards Expert-Level Medical Question — Singhal 2023" description: "Answering medical questions as well as a doctor has long been a major goal for AI." authors: ["Karan Singhal", "Tao Tu", "Juraj Gottweis", "Rory Sayres", "Ellery Wulczyn", "Le Hou", "Kevin Clark", "Stephen Pfohl", "Heather Cole-Lewis", "Darlene Neal", "Mike Schaekermann", "Amy Wang", "Mohamed Amin", "Sami Lachgar", "Philip Mansfield", "Sushant Prakash", "Bradley Green", "Ewa Dominowska", "Blaise Aguera y Arcas", "Nenad Tomasev", "Yun Liu", "Renee Wong", "Christopher Semturs", "S. Sara Mahdavi", "Joelle Barral", "Dale Webster", "Greg S. Corrado", "Yossi Matias", "Shekoofeh Azizi", "Alan Karthikesalingam", "Vivek Natarajan"] year: 2023 venue: "arXiv:2305.09617" source: "https://arxiv.org/abs/2305.09617" doi: "10.48550/arxiv.2305.09617" arxiv: "2305.09617" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Towards Expert-Level Medical Question Answering with Large Language Models # Towards Expert-Level Medical Question Answering with Large Language Models **Authors:** [Karan Singhal](https://md.research.elamur.ai/free/en/authors/karan-singhal/), [Tao Tu](https://md.research.elamur.ai/free/en/authors/tao-tu/), [Juraj Gottweis](https://md.research.elamur.ai/free/en/authors/juraj-gottweis/), Rory Sayres, Ellery Wulczyn, [Le Hou](https://md.research.elamur.ai/free/en/authors/le-hou/), [Kevin Clark](https://md.research.elamur.ai/free/en/authors/kevin-clark/), [Stephen Pfohl](https://md.research.elamur.ai/free/en/authors/stephen-pfohl/), [Heather Cole-Lewis](https://md.research.elamur.ai/free/en/authors/heather-cole-lewis/), Darlene Neal, [Mike Schaekermann](https://md.research.elamur.ai/free/en/authors/mike-schaekermann/), Amy Wang et al. (31 authors) · **Year:** 2023 · **Venue:** arXiv:2305.09617 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2305.09617) · [PDF](https://arxiv.org/pdf/2305.09617) · [DOI 10.48550/arxiv.2305.09617](https://doi.org/10.48550/arxiv.2305.09617) · [arXiv 2305.09617](https://arxiv.org/abs/2305.09617) · [OpenAlex](https://openalex.org/W4377009978) **Citations:** 319 citations (openalex-snapshot · 2026-10-14) · [183](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Answering medical questions as well as a doctor has long been a major goal for AI. The earlier Med-PaLM passed the USMLE-style threshold, but its answers still fell short of clinicians' answers. This Google Research/DeepMind paper presents Med-PaLM 2, which combines three things: the stronger PaLM 2 base model, further tuning on medical question-answering data, and a new prompting technique called ensemble refinement. With ensemble refinement, the model first generates several reasoning chains and then uses them together to produce a final, improved answer. The model is tested on the MultiMedQA suite, which has both multiple-choice and long-form questions, and on two new adversarial question sets that probe safety, bias and health equity. Med-PaLM 2 scores 86.5% on MedQA, a new state of the art, and is close to or above the best prior results on MedMCQA, PubMedQA and MMLU clinical topics. In head-to-head comparisons, physicians often preferred its long-form answers over answers written by other physicians. The work shows that a general-purpose LLM, aligned to medicine and evaluated by humans in detail, is approaching expert level. The authors stress that it still needs validation in real clinical settings. ## Key points - **MedQA (USMLE-style, 1273 questions):** 86.5% accuracy, more than 19 points above Med-PaLM's 67.2%. For comparison, the paper cites 86.1% for GPT-4-base and 60.2% for GPT 3.5. - **Comparison with physician answers:** physicians ranked answers to 1066 consumer health questions drawn from HealthSearchQA, LiveQA and MedicationQA. They preferred Med-PaLM 2 over physician-written answers on 8 of 9 clinical-utility axes (p \< 0.001). For example, raters judged its answers closer to medical consensus 72.9% of the time. - **New adversarial datasets:** Adversarial (General) has 58 questions and Adversarial (Health equity) has 182, for 240 in total. They cover topics such as drug use, mental health, misinformation, racial bias in clinical calculators, and access to care. Med-PaLM 2 beat Med-PaLM on every axis, and 90.6% of its answers were rated low risk of harm, against 79.4% for Med-PaLM. - **Multiple-choice benchmarks used:** MedQA, MedMCQA (4183 questions from Indian medical entrance exams), PubMedQA (500) and six MMLU medical subsets. - **Limitations the authors acknowledge:** - The rating rubric has not been formally validated. - Model answers are longer than physician answers, which may inflate their ratings. - Physicians wrote only one answer per question and were given no clinical context. - The adversarial sets cover only part of the safety and equity issues. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 183 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 166 not yet. - [Llemma: An Open Language Model For Mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/) - [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/) - [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/) - [Lawyer LLaMA Technical Report](https://md.research.elamur.ai/free/en/papers/huang-2023-lawyer-llama-technical-report/) - [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/) - [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/) - [Evaluating Large Language Models for Health-related Queries with Presuppositions](https://md.research.elamur.ai/free/en/papers/kaur-2024-evaluating-large-language-models-health/) - [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://md.research.elamur.ai/free/en/papers/labrak-2024-biomistral-collection-open-source-pretrained/) - [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/) - [Orca 2: Teaching Small Language Models How to Reason](https://md.research.elamur.ai/free/en/papers/mitra-2023-orca-2-teaching-small-language/) - [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/) - [Med-HALT: Medical Domain Hallucination Test for Large Language Models](https://md.research.elamur.ai/free/en/papers/pal-2023-med-halt-medical-domain-hallucination/) - [Faithful Medical Dialogue Generation Using Homo-Heterogeneous Exemplar-based In-Context Knowledge Grounding](https://md.research.elamur.ai/free/en/papers/priya-2026-faithful-medical-dialogue-generation-homo/) - [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://md.research.elamur.ai/free/en/papers/tang-2024-medagents-large-language-models-as/) - [Benchmarking Retrieval-Augmented Generation for Medicine](https://md.research.elamur.ai/free/en/papers/xiong-2024-benchmarking-retrieval-augmented-generation-medicine/) - [HuatuoGPT, Towards Taming Language Model to Be a Doctor](https://md.research.elamur.ai/free/en/papers/zhang-2023-huatuogpt-taming-language-model-be/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{singhal2023expert, title = {{Towards Expert-Level Medical Question Answering with Large Language Models}}, author = {Singhal, Karan and Tu, Tao and Gottweis, Juraj and Sayres, Rory and Wulczyn, Ellery and Hou, Le and Clark, Kevin and Pfohl, Stephen and Cole-Lewis, Heather and Neal, Darlene and Schaekermann, Mike and Wang, Amy and Amin, Mohamed and Lachgar, Sami and Mansfield, Philip and Prakash, Sushant and Green, Bradley and Dominowska, Ewa and Arcas, Blaise Aguera y and Tomasev, Nenad and Liu, Yun and Wong, Renee and Semturs, Christopher and Mahdavi, S. Sara and Barral, Joelle and Webster, Dale and Corrado, Greg S. and Matias, Yossi and Azizi, Shekoofeh and Karthikesalingam, Alan and Natarajan, Vivek}, year = {2023}, doi = {10.48550/arxiv.2305.09617}, eprint = {2305.09617}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2305.09617} } ``` - [← previous Towards Effective Disambiguation for Machine Translation with Large Language Models](https://md.research.elamur.ai/free/en/papers/iyer-2023-effective-disambiguation-machine-translation-large/) - [next → Towards General Text Embeddings with Multi-stage Contrastive Learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) ## References 3 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. Large Language Models Encode Clinical Knowl… [find](https://scholar.google.com/scholar?q=1%2E+2%2E+3%2E+4%2E+5%2E+6%2E+7%2E+8%2E+9%2E+10%2E+11%2E+12%2E+13%2E+14%2E+15%2E+16%2E+17%2E+18%2E+19%2E+20%2E+21%2E+22%2E+23%2E+24%2E+25%2E+26%2E+27%2E+28%2E+29%2E+30%2E+31%2E+32%2E+33%2E+34%2E+Singhal%2C+K%2E%2C+Azizi%2C+S%2E%2C+Tu%2C+T%2E%2C+Mahdavi%2C+S%2E+S%2E%2C+Wei%2C+J%2E%2C+Chung%2C+H%2E+W%2E%2C+Sc) [our card](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/) 2. JAMA Internal Medicine (2023). Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E. & Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022). Sun, Z., Wang, X., Tay, Y., Yang, Y. & Zhou, D. Recitation-Augmented Language Models. arXiv pr… [find](https://scholar.google.com/scholar?q=JAMA+Internal+Medicine+%282023%29%2E+Wang%2C+X%2E%2C+Wei%2C+J%2E%2C+Schuurmans%2C+D%2E%2C+Le%2C+Q%2E%2C+Chi%2C+E%2E+%26+Zhou%2C+D%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+arXiv+preprint+arXiv%3A2203%2E11171+%282) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/) 3. Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., et al. New creatinine-and cystatin C–based equations to estimate GFR without race. New England Journal of Medicine 385, 1737–1749 (2021). Eneanya, N. D., Boulware, L… [find](https://scholar.google.com/scholar?q=Inker%2C+L%2E+A%2E%2C+Eneanya%2C+N%2E+D%2E%2C+Coresh%2C+J%2E%2C+Tighiouart%2C+H%2E%2C+Wang%2C+D%2E%2C+Sang%2C+Y%2E%2C+Crews%2C+D%2E+C%2E%2C+Doria%2C+A%2E%2C+Estrella%2C+M%2E+M%2E%2C+Froissart%2C+M%2E%2C+et+al%2E+New+creatinine%2Dand+cystatin+C%E2%80%93based+equations+to+estimate+) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)