---
title: "Performance of ChatGPT on USMLE — Kung 2023"
description: "Kung and colleagues test whether a general-purpose chatbot, ChatGPT (built on GPT3.5), can handle the clinical reasoning required by the United States…"
authors: ["Tiffany H. Kung", "Morgan Cheatham", "Arielle Medenilla", "Czarina Sillos", "Lorie De Leon", "Camille Elepaño", "Maria Madriaga", "Rimel Aggabao", "Giezel Diaz-Candido", "James Maningo", "Victor Tseng"]
year: 2023
venue: "PLOS Digital Health"
source: "https://doi.org/10.1371/journal.pdig.0000198"
doi: "10.1371/journal.pdig.0000198"
licence: "CC BY 4.0 (publisher, via OpenAlex)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models

# Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models

**Authors:** Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, Victor Tseng · **Year:** 2023 · **Venue:** PLOS Digital Health · **Licence:** [CC BY 4.0 (publisher, via OpenAlex)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1371/journal.pdig.0000198) · [PDF](https://journals.plos.org/digitalhealth/article/file?id=10.1371/journal.pdig.0000198&type=printable) · [DOI 10.1371/journal.pdig.0000198](https://doi.org/10.1371/journal.pdig.0000198) · [OpenAlex](https://openalex.org/W4319662928)

**Citations:** 4065 citations (OpenAlex, 2026-10-14) · [89](https://md.research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary

Kung and colleagues test whether a general-purpose chatbot, ChatGPT (built on GPT3.5), can handle the clinical reasoning required by the United States Medical Licensing Examination (USMLE) without any medical fine-tuning. They used the official June 2022 public sample set (USMLE-2022), which postdates the model's training data. They dropped items that relied on images or graphs and kept 350 text-only questions spanning Step 1, Step 2CK and Step 3. Each question was presented in three ways: open-ended with the options removed, standard multiple choice, and multiple choice with a required rationale for every option. Two blinded physicians graded the outputs on Accuracy, Concordance and Insight (ACI), and a third physician resolved disagreements. ChatGPT scored at or close to the roughly 60% pass mark on all three exams, and its explanations were mostly coherent and often insightful. The authors present this as a first milestone for AI in medicine. They argue that LLMs could support medical learners and generate exam content, and might eventually aid clinical decisions.

## Key points

- **Dataset:** 376 public USMLE-2022 sample items were collected. After removing image-based questions, 350 remained: Step 1: 119, Step 2CK: 102, Step 3: 122. The authors spot-checked that no answers were indexed online before 1 January 2022, the model's data cutoff.
- **Prompt formats:**
  - OE: open-ended, with answer options removed.
  - MC-NJ: the original multiple-choice question, with no justification requested.
  - MC-J: multiple choice with a forced justification for each option.

  Each item was run in a fresh chat session to avoid memory carry-over. Wording was varied across 8 human encoders, and a post hoc 2-way ANOVA checked for encoder effects.
- **Grading protocol:** Responses were scored on Accuracy, Concordance and Insight (ACI) by two blinded physicians. 21 items (6.2%) went to a third adjudicator, and interrater agreement was measured with Cohen's κ.
- **Headline result:** Performance was at or near the ~60% passing threshold on every Step, with no specialized training. Note that the excerpt supplied for this summary omitted the per-Step accuracy figures.
- **Limitations noted by the authors:** The sample is small, human grading is subjective, and there is no breakdown by subject or failure mode. Real-world educational trials are still needed.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Institutions

- [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) — University, Cambridge, United States · 64 works · 35 670 citations
- [Brown University](https://md.research.elamur.ai/free/en/universities/brown-university/) — University, Providence, United States · 40 works · 21 721 citations
- [Massachusetts General Hospital](https://md.research.elamur.ai/free/en/hospitals/massachusetts-general-hospital/) — Hospital, Boston, United States · 3 works · 5 392 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kung2023performance,
  title = {{Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models}},
  author = {Kung, Tiffany H. and Cheatham, Morgan and Medenilla, Arielle and Sillos, Czarina and Leon, Lorie De and Elepaño, Camille and Madriaga, Maria and Aggabao, Rimel and Diaz-Candido, Giezel and Maningo, James and Tseng, Victor},
  booktitle = {PLOS Digital Health},
  year = {2023},
  doi = {10.1371/journal.pdig.0000198},
  url = {https://doi.org/10.1371/journal.pdig.0000198}
}
```

- [← previous LLaMA: Open and Efficient Foundation Language Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
- [next → A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)

## References

24 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2016. https://doi.org/10.1109/cvpr.2016.308 [↗](https://doi.org/10.1109/cvpr.2016.308)
2. Zhang W, Feng Y, Meng F, You D, Liu Q. Bridging the gap between training and inference for neural machine translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA, USA: Association for Computational Linguistics; 2019. https://doi.org/10. 1… [find](https://scholar.google.com/scholar?q=Zhang+W%2C+Feng+Y%2C+Meng+F%2C+You+D%2C+Liu+Q%2E+Bridging+the+gap+between+training+and+inference+for+neural+machine+translation%2E+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Lingu) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-bridging-gap-between-training-inference/)
3. Bhatia Y, Bajpayee A, Raghuvanshi D, Mittal H. Image captioning using Google’s inception-resnet-v2 and recurrent neural network. 2019 Twelfth International Conference on Contemporary Computing (IC3). IEEE; 2019. https://doi.org/10.1109/ic3.2019.8844921 — [doi:10.1109/ic3.2019.8844921](https://doi.org/10.1109/ic3.2019.8844921) [↗](https://doi.org/10.1109/ic3.2019.8844921)
4. McDermott MBA, Wang S, Marinsek N, Ranganath R, Foschini L, Ghassemi M. Reproducibility in machine learning for health research: Still a ways to go. Sci Transl Med. 2021;13. https://doi.org/10. 1126/scitranslmed.abb1655 PMID: 33762434 [find](https://scholar.google.com/scholar?q=McDermott+MBA%2C+Wang+S%2C+Marinsek+N%2C+Ranganath+R%2C+Foschini+L%2C+Ghassemi+M%2E+Reproducibility+in+machine+learning+for+health+research%3A+Still+a+ways+to+go%2E+Sci+Transl+Med%2E+2021%3B13%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E+1126%2Fs)
5. Chen P-HC, Liu Y, Peng L. How to develop machine learning models for healthcare. Nat Mater. 2019; 18: 410–414. https://doi.org/10.1038/s41563-019-0345-0 PMID: 31000806 — [doi:10.1038/s41563-019-0345-0](https://doi.org/10.1038/s41563-019-0345-0) [↗](https://doi.org/10.1038/s41563-019-0345-0)
6. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016; 316: 2402. https://doi.org/10.1001/jama.2016.17216 PMID: 27898976 — [doi:10.1001/jama.2016.17216](https://doi.org/10.1001/jama.2016.17216) [↗](https://doi.org/10.1001/jama.2016.17216)
7. Nagpal K, Foote D, Liu Y, Chen P-HC, Wulczyn E, Tan F, et al. Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. NPJ Digit Med. 2019; 2: 48. https://doi.org/10.1038/s41746-019-0112-2 PMID: 31304394 — [doi:10.1038/s41746-019-0112-2](https://doi.org/10.1038/s41746-019-0112-2) [↗](https://doi.org/10.1038/s41746-019-0112-2)
8. Liu Y, Jain A, Eng C, Way DH, Lee K, Bui P, et al. A deep learning system for differential diagnosis of skin diseases. Nat Med. 2020; 26: 900–908. https://doi.org/10.1038/s41591-020-0842-3 PMID: 32424212 9. [cited 26 Jan 2023]. Available: https://openai.com/blog/chatgpt/ — [doi:10.1038/s41591-020-0842-3](https://doi.org/10.1038/s41591-020-0842-3) [↗](https://doi.org/10.1038/s41591-020-0842-3)
9. Performance data. [cited 26 Jan 2023]. Available: [https://www.usmle.org/performance-data](https://www.usmle.org/performance-data) [↗](https://www.usmle.org/performance-data)
10. Burk-Rafel J, Santen SA, Purkiss J. Study Behaviors and USMLE Step 1 Performance: Implications of a Student Self-Directed Parallel Curriculum. Acad Med. 2017; 92: S67–S74. https://doi.org/10.1097/ ACM.0000000000001916 PMID: 29065026 [find](https://scholar.google.com/scholar?q=Burk%2DRafel+J%2C+Santen+SA%2C+Purkiss+J%2E+Study+Behaviors+and+USMLE+Step+1+Performance%3A+Implications+of+a+Student+Self%2DDirected+Parallel+Curriculum%2E+Acad+Med%2E+2017%3B+92%3A+S67%E2%80%93S74%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1097%2F+ACM)
11. Liévin V, Hother CE, Winther O. Can large language models reason about medical questions? arXiv [cs.CL]. 2022. Available: http://arxiv.org/abs/2207.08143 — [arXiv:2207.08143](https://arxiv.org/abs/2207.08143) [↗](http://arxiv.org/abs/2207.08143)
12. Jin D, Pan E, Oufattole N, Weng W-H, Fang H, Szolovits P. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. arXiv [cs.CL]. 2020. Available: http://arxiv.org/abs/2009.13081 [↗](http://arxiv.org/abs/2009.13081) [our card](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/)
13. Stanford CRFM. [cited 18 Jan 2023]. Available: [https://crfm.stanford.edu/2022/12/15/pubmedgpt.html](https://crfm.stanford.edu/2022/12/15/pubmedgpt.html) [↗](https://crfm.stanford.edu/2022/12/15/pubmedgpt.html)
14. Densen P. Challenges and opportunities facing medical education. Trans Am Clin Climatol Assoc. 2011; 122: 48–58. Available: [https://www.ncbi.nlm.nih.gov/pubmed/21686208](https://www.ncbi.nlm.nih.gov/pubmed/21686208) PMID: 21686208 [↗](https://www.ncbi.nlm.nih.gov/pubmed/21686208)
15. Prasad V, Vandross A, Toomey C, Cheung M, Rho J, Quinn S, et al. A decade of reversal: an analysis of 146 contradicted medical practices. Mayo Clin Proc. 2013; 88: 790–798. https://doi.org/10.1016/j. mayocp.2013.05.012 PMID: 23871230 — [doi:10.1016/j](https://doi.org/10.1016/j) [find](https://scholar.google.com/scholar?q=Prasad+V%2C+Vandross+A%2C+Toomey+C%2C+Cheung+M%2C+Rho+J%2C+Quinn+S%2C+et+al%2E+A+decade+of+reversal%3A+an+analysis+of+146+contradicted+medical+practices%2E+Mayo+Clin+Proc%2E+2013%3B+88%3A+790%E2%80%93798%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1016%2Fj%2E+)
16. Herrera-Perez D, Haslam A, Crain T, Gill J, Livingston C, Kaestner V, et al. A comprehensive review of randomized clinical trials in three medical journals reveals 396 medical reversals. Elife. 2019; 8. https://doi.org/10.7554/eLife.45183 PMID: 31182188 — [doi:10.7554/elife.45183](https://doi.org/10.7554/elife.45183) [↗](https://doi.org/10.7554/eLife.45183)
17. Abou-Hanna JJ, Owens ST, Kinnucan JA, Mian SI, Kolars JC. Resuscitating the Socratic Method: Student and Faculty Perspectives on Posing Probing Questions During Clinical Teaching. Acad Med. 2021; 96: 113–117. https://doi.org/10.1097/ACM.0000000000003580 PMID: 33394663 — [doi:10.1097/acm.0000000000003580](https://doi.org/10.1097/acm.0000000000003580) [↗](https://doi.org/10.1097/ACM.0000000000003580)
18. Plana D, Shung DL, Grimshaw AA, Saraf A, Sung JJY, Kann BH. Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review. JAMA Netw Open. 2022; 5: e2233946. https://doi.org/10.1001/jamanetworkopen.2022.33946 PMID: 36173632 — [doi:10.1001/jamanetworkopen.2022.33946](https://doi.org/10.1001/jamanetworkopen.2022.33946) [↗](https://doi.org/10.1001/jamanetworkopen.2022.33946)
19. Kan HJ, Kharrazi H, Chang H-Y, Bodycombe D, Lemke K, Weiner JP. Exploring the use of machine learning for risk adjustment: A comparison of standard and penalized linear regression models in predicting health care costs in older adults. PLoS One. 2019; 14: e0213258. https://doi.org/10.1371/ journal.p… [find](https://scholar.google.com/scholar?q=Kan+HJ%2C+Kharrazi+H%2C+Chang+H%2DY%2C+Bodycombe+D%2C+Lemke+K%2C+Weiner+JP%2E+Exploring+the+use+of+machine+learning+for+risk+adjustment%3A+A+comparison+of+standard+and+penalized+linear+regression+models+in+predicting)
20. Delahanty RJ, Kaufman D, Jones SS. Development and Evaluation of an Automated Machine Learning Algorithm for In-Hospital Mortality Risk Adjustment Among Critical Care Patients. Crit Care Med. 2018; 46: e481–e488. https://doi.org/10.1097/CCM.0000000000003011 PMID: 29419557 — [doi:10.1097/ccm.0000000000003011](https://doi.org/10.1097/ccm.0000000000003011) [↗](https://doi.org/10.1097/CCM.0000000000003011)
21. Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med. 2022; 28: 924–933. https://doi.org/10.1038/s41591-022-01772-9 PMID: 35585198 — [doi:10.1038/s41591-022-01772-9](https://doi.org/10.1038/s41591-022-01772-9) [↗](https://doi.org/10.1038/s41591-022-01772-9)
22. Garcia-Vidal C, Sanjuan G, Puerta-Alcalde P, Moreno-Garcı́a E, Soriano A. Artificial intelligence to support clinical decision-making processes. EBioMedicine. 2019; 46: 27–29. https://doi.org/10.1016/j. ebiom.2019.07.019 PMID: 31303500 — [doi:10.1016/j](https://doi.org/10.1016/j) [find](https://scholar.google.com/scholar?q=Garcia%2DVidal+C%2C+Sanjuan+G%2C+Puerta%2DAlcalde+P%2C+Moreno%2DGarc%C4%B1%CC%81a+E%2C+Soriano+A%2E+Artificial+intelligence+to+support+clinical+decision%2Dmaking+processes%2E+EBioMedicine%2E+2019%3B+46%3A+27%E2%80%9329%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1016%2F)
23. Bala S, Keniston A, Burden M. Patient Perception of Plain-Language Medical Notes Generated Using Artificial Intelligence Software: Pilot Mixed-Methods Study. JMIR Form Res. 2020; 4: e16670. https://doi.org/10.2196/16670 PMID: 32442148 — [doi:10.2196/16670](https://doi.org/10.2196/16670) [↗](https://doi.org/10.2196/16670)
24. Milne-Ives M, de Cock C, Lim E, Shehadeh MH, de Pennington N, Mole G, et al. The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review. J Med Internet Res. 2020; 22: e20346. https://doi.org/10.2196/20346 PMID: 33090118 — [doi:10.2196/20346](https://doi.org/10.2196/20346) [↗](https://doi.org/10.2196/20346)