--- title: "Large Language Models Encode Clinical — Singhal 2022" description: "Medical applications of large language models require very high reliability, yet their clinical knowledge has mostly been tested with narrow automatic…" authors: ["Karan Singhal", "Shekoofeh Azizi", "Tao Tu", "S. Sara Mahdavi", "Jason Wei", "Hyung Won Chung", "Nathan Scales", "Ajay Tanwani", "Heather Cole-Lewis", "Stephen Pfohl", "Perry Payne", "Martin Seneviratne", "Paul Gamble", "Chris Kelly", "Nathaneal Scharli", "Aakanksha Chowdhery", "Philip Mansfield", "Blaise Aguera y Arcas", "Dale Webster", "Greg S. Corrado", "Yossi Matias", "Katherine Chou", "Juraj Gottweis", "Nenad Tomasev", "Yun Liu", "Alvin Rajkomar", "Joelle Barral", "Christopher Semturs", "Alan Karthikesalingam", "Vivek Natarajan"] year: 2022 venue: "arXiv:2212.13138" source: "https://arxiv.org/abs/2212.13138" doi: "10.48550/arxiv.2212.13138" arxiv: "2212.13138" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Large Language Models Encode Clinical Knowledge # Large Language Models Encode Clinical Knowledge **Authors:** [Karan Singhal](https://md.research.elamur.ai/free/en/authors/karan-singhal/), Shekoofeh Azizi, [Tao Tu](https://md.research.elamur.ai/free/en/authors/tao-tu/), [S. Sara Mahdavi](https://md.research.elamur.ai/free/en/authors/s-sara-mahdavi/), [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/), [Hyung Won Chung](https://md.research.elamur.ai/free/en/authors/hyung-won-chung/), [Nathan Scales](https://md.research.elamur.ai/free/en/authors/nathan-scales/), Ajay Tanwani, [Heather Cole-Lewis](https://md.research.elamur.ai/free/en/authors/heather-cole-lewis/), [Stephen Pfohl](https://md.research.elamur.ai/free/en/authors/stephen-pfohl/), Perry Payne, Martin Seneviratne et al. (30 authors) · **Year:** 2022 · **Venue:** arXiv:2212.13138 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2212.13138) · [PDF](https://arxiv.org/pdf/2212.13138) · [DOI 10.48550/arxiv.2212.13138](https://doi.org/10.48550/arxiv.2212.13138) · [arXiv 2212.13138](https://arxiv.org/abs/2212.13138) · [OpenAlex](https://openalex.org/W4384071683) **Citations:** 3967 citations (openalex-snapshot · 2026-10-14) · [496](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Medical applications of large language models require very high reliability, yet their clinical knowledge has mostly been tested with narrow automatic metrics on a few benchmarks. The authors, from Google Research and DeepMind, built MultiMedQA. It combines six public question-answering sets covering licensing exams, biomedical research and lay health questions, plus a new collection, HealthSearchQA, drawn from health queries people often type into search engines. They also designed a pilot protocol in which physicians and non-experts rate long-form answers for factual grounding, possible harm, reasoning quality, completeness, bias and usefulness. They tested PaLM (540B) and its instruction-tuned version Flan-PaLM using few-shot, chain-of-thought and self-consistency prompting. Flan-PaLM set new best scores on every multiple-choice part of the benchmark. Human review, however, revealed serious weaknesses in its free-text answers. To address this, the authors proposed instruction prompt tuning, a low-cost alignment method that uses a small number of clinician-written examples. The resulting model, Med-PaLM, comes close to clinicians on several criteria but still falls short overall. The study's value lies in providing both a broad benchmark and an evaluation template for safer medical LLMs. ## Key points - MultiMedQA consists of seven datasets: - six existing ones: MedQA (USMLE), MedMCQA, PubMedQA, LiveQA, MedicationQA and MMLU clinical topics; - one new one: HealthSearchQA, with 3,375 consumer medical questions. - Flan-PaLM with combined prompting reached 67.6% on MedQA, more than 17 points above the previous state of the art. It also scored 57.6% on MedMCQA and 79.0% on PubMedQA. - Clinicians rated 61.9% of Flan-PaLM's long answers as consistent with scientific consensus. The figure was 92.6% for Med-PaLM and 92.9% for answers written by clinicians. - Potentially harmful answers fell from 29.7% (Flan-PaLM) to 5.8% (Med-PaLM), compared with 6.5% for clinicians. - Accuracy improves with model scale and with instruction prompt tuning. The authors stress that fairness, bias and safety evaluations remain incomplete, and that the models are not yet suitable for clinical deployment. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 496 papers of our library cite it (of the 140673 whose bibliographies we read); 34 of them have a page here, 462 not yet. - [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://md.research.elamur.ai/free/en/papers/allen-2023-zhu-physics-language-models-part-3/) - [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://md.research.elamur.ai/free/en/papers/arora-2025-healthbench-evaluating-large-language-models/) - [Llemma: An Open Language Model For Mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/) - [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://md.research.elamur.ai/free/en/papers/bianchi-2023-safety-tuned-llamas-lessons-improving/) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/) - [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) - [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/) - [Multilingual Jailbreak Challenges in Large Language Models](https://md.research.elamur.ai/free/en/papers/deng-2023-multilingual-jailbreak-challenges-large-language/) - [How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition](https://md.research.elamur.ai/free/en/papers/dong-2024-how-abilities-large-language-models/) - [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/) - [Lawyer LLaMA Technical Report](https://md.research.elamur.ai/free/en/papers/huang-2023-lawyer-llama-technical-report/) - [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/) - [Evaluating Large Language Models for Health-related Queries with Presuppositions](https://md.research.elamur.ai/free/en/papers/kaur-2024-evaluating-large-language-models-health/) - [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://md.research.elamur.ai/free/en/papers/labrak-2024-biomistral-collection-open-source-pretrained/) - [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/) - [Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus](https://md.research.elamur.ai/free/en/papers/mannion-2026-is-biomedical-specialization-still-worth/) - [Capabilities of GPT-4 on Medical Challenge Problems](https://md.research.elamur.ai/free/en/papers/nori-2023-capabilities-gpt-4-medical-challenge/) - [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/) - [Med-HALT: Medical Domain Hallucination Test for Large Language Models](https://md.research.elamur.ai/free/en/papers/pal-2023-med-halt-medical-domain-hallucination/) - [Humanity's Last Exam](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/) - [Faithful Medical Dialogue Generation Using Homo-Heterogeneous Exemplar-based In-Context Knowledge Grounding](https://md.research.elamur.ai/free/en/papers/priya-2026-faithful-medical-dialogue-generation-homo/) - [DeepICD-R1: Medical Reasoning through Hierarchical Rewards and Unsupervised Distillation](https://md.research.elamur.ai/free/en/papers/rohr-2026-deepicd-r1-medical-reasoning-through/) - [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://md.research.elamur.ai/free/en/papers/sainz-2023-gollie-annotation-guidelines-improve-zero/) - [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/) - [Towards Expert-Level Medical Question Answering with Large Language Models](https://md.research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/) - [Addressing the Ecological Fallacy in Larger LMs with Human Context](https://md.research.elamur.ai/free/en/papers/soni-2026-addressing-ecological-fallacy-larger-lms/) - [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://md.research.elamur.ai/free/en/papers/tang-2024-medagents-large-language-models-as/) - [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://md.research.elamur.ai/free/en/papers/tseng-2024-two-tales-persona-llms-survey/) - [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/) - [BloombergGPT: A Large Language Model for Finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/) - [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://md.research.elamur.ai/free/en/papers/wu-2023-pmc-llama-towards-building-open/) - [Benchmarking Retrieval-Augmented Generation for Medicine](https://md.research.elamur.ai/free/en/papers/xiong-2024-benchmarking-retrieval-augmented-generation-medicine/) - [Knowledge Conflicts for LLMs: A Survey](https://md.research.elamur.ai/free/en/papers/xu-2024-knowledge-conflicts-llms-survey/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{singhal2022large, title = {{Large Language Models Encode Clinical Knowledge}}, author = {Singhal, Karan and Azizi, Shekoofeh and Tu, Tao and Mahdavi, S. Sara and Wei, Jason and Chung, Hyung Won and Scales, Nathan and Tanwani, Ajay and Cole-Lewis, Heather and Pfohl, Stephen and Payne, Perry and Seneviratne, Martin and Gamble, Paul and Kelly, Chris and Scharli, Nathaneal and Chowdhery, Aakanksha and Mansfield, Philip and Arcas, Blaise Aguera y and Webster, Dale and Corrado, Greg S. and Matias, Yossi and Chou, Katherine and Gottweis, Juraj and Tomasev, Nenad and Liu, Yun and Rajkomar, Alvin and Barral, Joelle and Semturs, Christopher and Karthikesalingam, Alan and Natarajan, Vivek}, year = {2022}, doi = {10.48550/arxiv.2212.13138}, eprint = {2212.13138}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2212.13138} } ``` - [← previous Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) - [next → Large Language Models Struggle to Learn Long-Tail Knowledge](https://md.research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) ## References 8 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27. 28. 29. 30. 31. 32. Abacha, A. B., Agichtein, E., Pinter, Y. & Demner-Fushman, D. Overview of the medical question answering task at TREC 2017 LiveQA. in TREC (2017), 1–12. Abacha, A. B., Mrabet, Y., Sharp, M., Goodwin… [find](https://scholar.google.com/scholar?q=1%2E+2%2E+3%2E+4%2E+5%2E+6%2E+7%2E+8%2E+9%2E+10%2E11%2E12%2E13%2E14%2E15%2E16%2E17%2E18%2E19%2E20%2E21%2E22%2E23%2E24%2E25%2E26%2E27%2E+28%2E+29%2E+30%2E+31%2E+32%2E+Abacha%2C+A%2E+B%2E%2C+Agichtein%2C+E%2E%2C+Pinter%2C+Y%2E+%26+Demner%2DFushman%2C+D%2E+Overview+of+the+medical+question+ans) 2. Mar. 2021. https : //doi.org/10.5281/zenodo.5297715. Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R. & Young, S. L. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Frontiers in public health 6, 149 (2018). Bolton, E… [find](https://scholar.google.com/scholar?q=Mar%2E+2021%2E+https+%3A+%2F%2Fdoi%2Eorg%2F10%2E5281%2Fzenodo%2E5297715%2E+Boateng%2C+G%2E+O%2E%2C+Neilands%2C+T%2E+B%2E%2C+Frongillo%2C+E%2E+A%2E%2C+Melgar%2DQui%C3%B1onez%2C+H%2E+R%2E+%26+Young%2C+S%2E+L%2E+Best+practices+for+developing+and+validating+scales+for+he) [our card](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) 3. World Health Organization (2021). Han, X., Zhao, W., Ding, N., Liu, Z. & Sun, M. Ptr: Prompt tuning with rules for text classification. [find](https://scholar.google.com/scholar?q=World+Health+Organization+%282021%29%2E+Han%2C+X%2E%2C+Zhao%2C+W%2E%2C+Ding%2C+N%2E%2C+Liu%2C+Z%2E+%26+Sun%2C+M%2E+Ptr%3A+Prompt+tuning+with+rules+for+text+classification%2E) 4. AI Open (2022). Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. & Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020). Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendric… [find](https://scholar.google.com/scholar?q=AI+Open+%282022%29%2E+Hendrycks%2C+D%2E%2C+Burns%2C+C%2E%2C+Basart%2C+S%2E%2C+Zou%2C+A%2E%2C+Mazeika%2C+M%2E%2C+Song%2C+D%2E+%26+Steinhardt%2C+J%2E+Measuring+massive+multitask+language+understanding%2E+arXiv+preprint+arXiv%3A2009%2E03300+%282020%29%2E+Hoffma) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/) 5. Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H. & Szolovits, P. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11, 6421 (2021). Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W. & Lu, X. PubMedQA: A dataset for biomedic… [find](https://scholar.google.com/scholar?q=Jin%2C+D%2E%2C+Pan%2C+E%2E%2C+Oufattole%2C+N%2E%2C+Weng%2C+W%2E%2DH%2E%2C+Fang%2C+H%2E+%26+Szolovits%2C+P%2E+What+disease+does+this+patient+have%3F+a+large%2Dscale+open+domain+question+answering+dataset+from+medical+exams%2E+Applied+Sciences+11) [our card](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/) 6. The Lancet Digital Health (2022). Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017). Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H. & Liu, T.-Y. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brie… [find](https://scholar.google.com/scholar?q=The+Lancet+Digital+Health+%282022%29%2E+Loshchilov%2C+I%2E+%26+Hutter%2C+F%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101+%282017%29%2E+Luo%2C+R%2E%2C+Sun%2C+L%2E%2C+Xia%2C+Y%2E%2C+Qin%2C+T%2E%2C+Zhang%2C+S%2E%2C+Poon%2C+H%2E+%26+L) 7. Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation in Proceedings of the 40th annual meeting of the Association for Computational Linguistics (2002), 311–318. Prabhakaran, V., Hutchinson, B. & Mitchell, M. Perturbation sensitivity analysis… [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+Roukos%2C+S%2E%2C+Ward%2C+T%2E+%26+Zhu%2C+W%2E%2DJ%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation+in+Proceedings+of+the+40th+annual+meeting+of+the+Association+for+Computational+Linguistic) 8. Zhang, H., Lu, A. X., Abdalla, M., McDermott, M. & Ghassemi, M. Hurtful words: quantifying biases in clinical contextual word embeddings in proceedings of the ACM Conference on Health, Inference, and Learning (2020), 110–120. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan,… [find](https://scholar.google.com/scholar?q=Zhang%2C+H%2E%2C+Lu%2C+A%2E+X%2E%2C+Abdalla%2C+M%2E%2C+McDermott%2C+M%2E+%26+Ghassemi%2C+M%2E+Hurtful+words%3A+quantifying+biases+in+clinical+contextual+word+embeddings+in+proceedings+of+the+ACM+Conference+on+Health%2C+Inference%2C+and+)