---
title: "MedXpertQA — paper by Zuo et al. (2025) — summary and ideas"
description: "Popular medical QA benchmarks such as MedQA and MMLU (Medical) no longer challenge frontier… 48 references, cited by 62 papers here."
authors: ["Yuxin Zuo", "Shang Qu", "Yifei Li", "Zhangren Chen", "Xuekai Zhu", "Ermo Hua", "Kaiyan Zhang", "Ning Ding", "Bowen Zhou"]
year: 2025
venue: "arXiv:2501.18362"
source: "https://arxiv.org/abs/2501.18362"
doi: "10.48550/arxiv.2501.18362"
arxiv: "2501.18362"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zuo-2025-medxpertqa-benchmarking-expert-level-medical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zuo-2025-medxpertqa-benchmarking-expert-level-medical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding

# MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding

**Authors:** [Yuxin Zuo](https://md.research.elamur.ai/free/en/authors/yuxin-zuo/), [Shang Qu](https://md.research.elamur.ai/free/en/authors/shang-qu/), [Yifei Li](https://md.research.elamur.ai/free/en/authors/yifei-li/), Zhangren Chen, [Xuekai Zhu](https://md.research.elamur.ai/free/en/authors/xuekai-zhu/), [Ermo Hua](https://md.research.elamur.ai/free/en/authors/ermo-hua/), [Kaiyan Zhang](https://md.research.elamur.ai/free/en/authors/kaiyan-zhang/), [Ning Ding](https://md.research.elamur.ai/free/en/authors/ning-ding/) and [Bowen Zhou](https://md.research.elamur.ai/free/en/authors/bowen-zhou/) · **Year:** 2025 · **Venue:** arXiv:2501.18362 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2501.18362) · [PDF](https://arxiv.org/pdf/2501.18362) · [DOI 10.48550/arxiv.2501.18362](https://doi.org/10.48550/arxiv.2501.18362) · [arXiv 2501.18362](https://arxiv.org/abs/2501.18362) · [OpenAlex](https://openalex.org/W4407012395)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/zuo-2025-medxpertqa-benchmarking-expert-level-medical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Popular medical QA benchmarks such as MedQA and MMLU (Medical) no longer challenge frontier models. o1-preview reportedly reaches 96% on MedQA-USMLE. Multimodal sets like VQA-Rad or PMC-VQA mostly contain simple caption-derived questions without clinical context. MedXpertQA addresses both gaps. This multiple-choice benchmark contains 4,460 questions collected from USMLE, COMLEX-USA, 17 American specialty board exams, and collections with many images, for example the NEJM Image Challenges. It has a text-only subset (Text) and a multimodal subset (MM) whose items combine images with patient histories and test results.

The authors built it in several stages:
- filtering questions by difficulty using human response statistics and repeated sampling from AI models;
- removing near-duplicates;
- having proprietary LLMs rewrite questions and add distractors to reduce training-data leakage;
- having licensed physicians review the result.

Across 18 evaluated LLMs and LMMs, even leading systems perform poorly, especially on a dedicated Reasoning subset. This makes the benchmark a useful testbed for o1-style reasoning models beyond math and code.

## Key points
- Scale and coverage: 4,460 questions across 17 specialties, 11 body systems and 3 task categories. MM contains 2,005 questions with 2,839 images, including radiology, pathology, ECGs, photos, diagrams and tables.
- Text items have 10 answer options and MM items have 5, after weak distractors were removed. Each subset has a 5-question few-shot development split plus a test set.
- Difficulty filtering combined an adaptive Brier score threshold based on human answer data with 14 sampled votes from 8 AI models per question.
- Inference-time scaling was tested on three model pairs: DeepSeek-R1 vs DeepSeek-V3, QwQ-32B-Preview vs Qwen2.5-32B-Instruct, and QVQ-72B-Preview vs Qwen2-VL-72B. The reasoning-focused models showed consistent gains on the Reasoning split but uneven results on the Understanding split.
- A per-system analysis of GPT-4o found it relatively strong on Integumentary questions and weaker on Cardiovascular ones. Data and leaderboard are available via GitHub (TsinghuaC3I/MedXpertQA).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 152 works · 13,166 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 263 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 260 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 275 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 214 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding and Bowen Zhou. 2025. MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding. arXiv:2501.18362.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zuo2025medxpertqa,
  title = {{MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding}},
  author = {Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding and Bowen Zhou},
  year = {2025},
  doi = {10.48550/arxiv.2501.18362},
  eprint = {2501.18362},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.18362}
}
```

- [← previous MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation](https://md.research.elamur.ai/free/en/papers/chang-2025-main-rag-multi-agent-filtering/)
- [next → Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset](https://md.research.elamur.ai/free/en/papers/su-2025-nemotron-cc-transforming-common-crawl/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

48 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Google. Gemini 2.0 flash. [https://gemini.google.com](https://gemini.google.com), 2024. Accessed: 12/2024. [↗](https://gemini.google.com)
3. Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024. URL [https://api.semanticscholar.org/CorpusID:268232499](https://api.semanticscholar.org/CorpusID:268232499). [↗](https://api.semanticscholar.org/CorpusID:268232499)
4. Ben Abacha, A., Hasan, S. A., Datla, V. V., Demner-Fushman, D., and Müller, H. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019, 2019. [find](https://scholar.google.com/scholar?q=Ben+Abacha%2C+A%2E%2C+Hasan%2C+S%2E+A%2E%2C+Datla%2C+V%2E+V%2E%2C+Demner%2DFushman%2C+D%2E%2C+and+Mu%CC%88ller%2C+H%2E+Vqa%2Dmed%3A+Overview+of+the+medical+visual+question+answering+task+at+imageclef+2019%2E+In+Proceedings+of+CLEF+%28Conference+an)
5. Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H. [Domain-specific language model pretraining for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/), 2020.
6. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025.
7. He, X., Zhang, Y., Mou, L., Xing, E., and Xie, P. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020. — [arXiv:2003.10286](https://arxiv.org/abs/2003.10286) [find](https://scholar.google.com/scholar?q=He%2C+X%2E%2C+Zhang%2C+Y%2E%2C+Mou%2C+L%2E%2C+Xing%2C+E%2E%2C+and+Xie%2C+P%2E+Pathvqa%3A+30000%2B+questions+for+medical+visual+question+answering%2E+arXiv+preprint+arXiv%3A2003%2E10286%2C+2020%2E+%E2%80%94+%5BarXiv%3A2003%2E10286%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F200)
8. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
9. Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H., Fan, Q., Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo, Q., He, C., Hu, Y., Huang, T., Jiang, T., Jiao, P., Jin, Z., Lei, Z., Li, J., Li, J., Li, L., Li, S., Li, W.,… [find](https://scholar.google.com/scholar?q=Cai%2C+Z%2E%2C+Cao%2C+M%2E%2C+Chen%2C+H%2E%2C+Chen%2C+K%2E%2C+Chen%2C+K%2E%2C+Chen%2C+X%2E%2C+Chen%2C+X%2E%2C+Chen%2C+Z%2E%2C+Chen%2C+Z%2E%2C+Chu%2C+P%2E%2C+Dong%2C+X%2E%2C+Duan%2C+H%2E%2C+Fan%2C+Q%2E%2C+Fei%2C+Z%2E%2C+Gao%2C+Y%2E%2C+Ge%2C+J%2E%2C+Gu%2C+C%2E%2C+Gu%2C+Y%2E%2C+Gui%2C+T%2E%2C+Guo%2C+A%2E%2C+Guo%2C+Q%2E%2C+He%2C+C)
10. Hu, Y., Li, T., Lu, Q., Shao, W., He, J., Qiao, Y., and Luo, P. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22170–22183, 2024. [find](https://scholar.google.com/scholar?q=Hu%2C+Y%2E%2C+Li%2C+T%2E%2C+Lu%2C+Q%2E%2C+Shao%2C+W%2E%2C+He%2C+J%2E%2C+Qiao%2C+Y%2E%2C+and+Luo%2C+P%2E+Omnimedvqa%3A+A+new+large%2Dscale+comprehensive+evaluation+benchmark+for+medical+lvlm%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer)
11. Huang, Z., Geng, G., Hua, S., Huang, Z., Zou, H., Zhang, S., Liu, P., and Zhang, X. O1 replication journey–part 3: Inference-time scaling for medical reasoning. arXiv preprint arXiv:2501.06458, 2025. — [arXiv:2501.06458](https://arxiv.org/abs/2501.06458) [find](https://scholar.google.com/scholar?q=Huang%2C+Z%2E%2C+Geng%2C+G%2E%2C+Hua%2C+S%2E%2C+Huang%2C+Z%2E%2C+Zou%2C+H%2E%2C+Zhang%2C+S%2E%2C+Liu%2C+P%2E%2C+and+Zhang%2C+X%2E+O1+replication+journey%E2%80%93part+3%3A+Inference%2Dtime+scaling+for+medical+reasoning%2E+arXiv+preprint+arXiv%3A2501%2E06458%2C+2025%2E+)
12. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276, 2024. medical challenge problems and beyond. arXiv preprint arXiv:2411.03590, 2024.
13. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). arXiv preprint arXiv:2412.16720, 2024.
14. OpenAI. Gpt-o3-mini. [https://openai.com/](https://openai.com/) index/openai-o3-mini/, 2025. 2025-01-31. [↗](https://openai.com/)
15. Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pp. 248–260. PMLR, 2022. — [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/)
16. Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, P. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421, 2021.
17. Patel, B., Gheihman, G., Katz, J. T., Begin, A. S., and Solomon, S. R. Navigating uncertainty in clinical practice: A structured approach. Journal of General Internal Medicine, pp. 1–8, 2024. [find](https://scholar.google.com/scholar?q=Patel%2C+B%2E%2C+Gheihman%2C+G%2E%2C+Katz%2C+J%2E+T%2E%2C+Begin%2C+A%2E+S%2E%2C+and+Solomon%2C+S%2E+R%2E+Navigating+uncertainty+in+clinical+practice%3A+A+structured+approach%2E+Journal+of+General+Internal+Medicine%2C+pp%2E+1%E2%80%938%2C+2024%2E)
18. Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., and Lu, X. [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). arXiv preprint arXiv:1909.06146, 2019.
19. Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C. B. C., Shaaban, M., Ling, J., Shi, S., et al. Humanity’s last exam. arXiv preprint arXiv:2501.[14249, 2025](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/).
20. Jin, Q., Kim, W., Chen, Q., Comeau, D. C., Yeganova, L., Wilbur, W. J., and Lu, Z. Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics, 39(11):btad651, 2023. [find](https://scholar.google.com/scholar?q=Jin%2C+Q%2E%2C+Kim%2C+W%2E%2C+Chen%2C+Q%2E%2C+Comeau%2C+D%2E+C%2E%2C+Yeganova%2C+L%2E%2C+Wilbur%2C+W%2E+J%2E%2C+and+Lu%2C+Z%2E+Medcpt%3A+Contrastive+pre%2Dtrained+transformers+with+large%2Dscale+pubmed+search+logs+for+zero%2Dshot+biomedical+information)
21. Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL [https://qwenlm.github.io/blog/qwen2.5/](https://qwenlm.github.io/blog/qwen2.5/). [↗](https://qwenlm.github.io/blog/qwen2.5/)
22. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35: 22199–22213, 2022.
23. Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. [Capabilities of gemini models in medicine](https://md.research.elamur.ai/free/en/papers/saab-2024-capabilities-gemini-models-medicine/). arXiv preprint arXiv:2404.18416, 2024.
24. Lau, J. J., Gayen, S., Ben Abacha, A., and Demner-Fushman, D. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018. [find](https://scholar.google.com/scholar?q=Lau%2C+J%2E+J%2E%2C+Gayen%2C+S%2E%2C+Ben+Abacha%2C+A%2E%2C+and+Demner%2DFushman%2C+D%2E+A+dataset+of+clinically+generated+visual+questions+and+answers+about+radiology+images%2E+Scientific+data%2C+5%281%29%3A1%E2%80%9310%2C+2018%2E)
25. Seller, R. H. and Symons, A. B. Differential Diagnosis of Common Complaints E-Book. Elsevier Health Sciences, 2011. [find](https://scholar.google.com/scholar?q=Seller%2C+R%2E+H%2E+and+Symons%2C+A%2E+B%2E+Differential+Diagnosis+of+Common+Complaints+E%2DBook%2E+Elsevier+Health+Sciences%2C+2011%2E)
26. Liachovitzky, C. Human Anatomy and Physiology Preparatory Course. 2015. [find](https://scholar.google.com/scholar?q=Liachovitzky%2C+C%2E+Human+Anatomy+and+Physiology+Preparatory+Course%2E+2015%2E)
27. Shin, H. S. Reasoning processes in clinical reasoning: from the perspective of cognitive psychology. Korean journal of medical education, 31(4):299, 2019. [find](https://scholar.google.com/scholar?q=Shin%2C+H%2E+S%2E+Reasoning+processes+in+clinical+reasoning%3A+from+the+perspective+of+cognitive+psychology%2E+Korean+journal+of+medical+education%2C+31%284%29%3A299%2C+2019%2E)
28. Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., et al. [Rho-1: Not all tokens are what you need](https://md.research.elamur.ai/free/en/papers/lin-2024-rho-1-all-tokens-need/). arXiv preprint arXiv:2404.07965, 2024.
29. Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530, 2024.
30. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseekv3 technical report. arXiv preprint arXiv:2412.19437, 2024a. [find](https://scholar.google.com/scholar?q=Liu%2C+A%2E%2C+Feng%2C+B%2E%2C+Xue%2C+B%2E%2C+Wang%2C+B%2E%2C+Wu%2C+B%2E%2C+Lu%2C+C%2E%2C+Zhao%2C+C%2E%2C+Deng%2C+C%2E%2C+Zhang%2C+C%2E%2C+Ruan%2C+C%2E%2C+et+al%2E+Deepseekv3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437%2C+2024a%2E)
31. Team, Q. Qvq: To see the world with wisdom, December 2024a. URL [https://qwenlm.github.io/](https://qwenlm.github.io/) blog/qvq-72b-preview/. [↗](https://qwenlm.github.io/)
32. Liu, B., Zhan, L.-M., Xu, L., Ma, L., Yang, Y., and Wu, X.-M. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1650–1654. [find](https://scholar.google.com/scholar?q=Liu%2C+B%2E%2C+Zhan%2C+L%2E%2DM%2E%2C+Xu%2C+L%2E%2C+Ma%2C+L%2E%2C+Yang%2C+Y%2E%2C+and+Wu%2C+X%2E%2DM%2E+Slake%3A+A+semantically%2Dlabeled+knowledge%2Denhanced+dataset+for+medical+visual+question+answering%2E+In+2021+IEEE+18th+International+Symposium+)
33. Team, Q. Qwq: Reflect deeply on the boundaries of the unknown, November 2024b. URL [https://qwenlm.github.io/blog/qwq-32b-preview/](https://qwenlm.github.io/blog/qwq-32b-preview/). [↗](https://qwenlm.github.io/blog/qwq-32b-preview/)
34. Team, Q. Qwen2.5-vl, January 2025. URL [https://qwenlm.github.io/blog/qwen2.5-vl/](https://qwenlm.github.io/blog/qwen2.5-vl/). [↗](https://qwenlm.github.io/blog/qwen2.5-vl/)
35. Liu, H., Li, C., Wu, Q., and Lee, Y. J. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36, 2024b.
36. Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.[12191, 2…](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
37. Nori, H., Usuyama, N., King, N., McKinney, S. M., Fernandes, X., Zhang, S., and Horvitz, E. From medprompt to o1: Exploration of run-time strategies for Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging… — [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)
38. Zhu, K., Zheng, Y., and Chan, K. C. G. Weighted brier score–an overall summary measure for risk prediction models with clinical utility consideration. arXiv preprint arXiv:2408.01626, 2024. — [arXiv:2408.01626](https://arxiv.org/abs/2408.01626) [find](https://scholar.google.com/scholar?q=Zhu%2C+K%2E%2C+Zheng%2C+Y%2E%2C+and+Chan%2C+K%2E+C%2E+G%2E+Weighted+brier+score%E2%80%93an+overall+summary+measure+for+risk+prediction+models+with+clinical+utility+consideration%2E+arXiv+preprint+arXiv%3A2408%2E01626%2C+2024%2E+%E2%80%94+%5BarXiv%3A2)
39. Wu, J., Deng, W., Li, X., Liu, S., Mi, T., Peng, Y., Xu, Z., Liu, Y., Cho, H., Choi, C.-I., et al. Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs. arXiv preprint arXiv:2504.00993, 2025. — [arXiv:2504.00993](https://arxiv.org/abs/2504.00993) [find](https://scholar.google.com/scholar?q=Wu%2C+J%2E%2C+Deng%2C+W%2E%2C+Li%2C+X%2E%2C+Liu%2C+S%2E%2C+Mi%2C+T%2E%2C+Peng%2C+Y%2E%2C+Xu%2C+Z%2E%2C+Liu%2C+Y%2E%2C+Cho%2C+H%2E%2C+Choi%2C+C%2E%2DI%2E%2C+et+al%2E+Medreason%3A+Eliciting+factual+medical+reasoning+steps+in+llms+via+knowledge+graphs%2E+arXiv+preprint+arX)
40. Wu, S., Peng, Z., Du, X., Zheng, T., Liu, M., Wu, J., Ma, J., Li, Y., Yang, J., Zhou, W., et al. A comparative study on reasoning patterns of openai’s o1 model. arXiv preprint arXiv:2410.13639, 2024. — [arXiv:2410.13639](https://arxiv.org/abs/2410.13639) [find](https://scholar.google.com/scholar?q=Wu%2C+S%2E%2C+Peng%2C+Z%2E%2C+Du%2C+X%2E%2C+Zheng%2C+T%2E%2C+Liu%2C+M%2E%2C+Wu%2C+J%2E%2C+Ma%2C+J%2E%2C+Li%2C+Y%2E%2C+Yang%2C+J%2E%2C+Zhou%2C+W%2E%2C+et+al%2E+A+comparative+study+on+reasoning+patterns+of+openai%E2%80%99s+o1+model%2E+arXiv+preprint+arXiv%3A2410%2E13639%2C+2024%2E+)
41. Xie, Y., Wu, J., Tu, H., Yang, S., Zhao, B., Zong, Y., Jin, Q., Xie, C., and Zhou, Y. A preliminary study of o1 in medicine: Are we closer to an ai doctor? arXiv preprint arXiv:2409.15277, 2024. — [arXiv:2409.15277](https://arxiv.org/abs/2409.15277) [find](https://scholar.google.com/scholar?q=Xie%2C+Y%2E%2C+Wu%2C+J%2E%2C+Tu%2C+H%2E%2C+Yang%2C+S%2E%2C+Zhao%2C+B%2E%2C+Zong%2C+Y%2E%2C+Jin%2C+Q%2E%2C+Xie%2C+C%2E%2C+and+Zhou%2C+Y%2E+A+preliminary+study+of+o1+in+medicine%3A+Are+we+closer+to+an+ai+doctor%3F+arXiv+preprint+arXiv%3A2409%2E15277%2C+2024%2E+%E2%80%94+%5Bar)
42. Xu, R., Wang, Z., Fan, R.-Z., and Liu, P. [Benchmarking benchmark leakage in large language models](https://md.research.elamur.ai/free/en/papers/xu-2024-benchmarking-benchmark-leakage-large-language/). arXiv preprint arXiv:2404.18824, 2024.
43. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. [find](https://scholar.google.com/scholar?q=Yang%2C+A%2E%2C+Yang%2C+B%2E%2C+Zhang%2C+B%2E%2C+Hui%2C+B%2E%2C+Zheng%2C+B%2E%2C+Yu%2C+B%2E%2C+Li%2C+C%2E%2C+Liu%2C+D%2E%2C+Huang%2C+F%2E%2C+Wei%2C+H%2E%2C+et+al%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2C+2024%2E)
44. Yazdani, S. and Hoseini Abardeh, M. Five decades of research and theorization on clinical reasoning: a critical review. Advances in medical education and practice, pp. 703–716, 2019. [find](https://scholar.google.com/scholar?q=Yazdani%2C+S%2E+and+Hoseini+Abardeh%2C+M%2E+Five+decades+of+research+and+theorization+on+clinical+reasoning%3A+a+critical+review%2E+Advances+in+medical+education+and+practice%2C+pp%2E+703%E2%80%93716%2C+2019%2E)
45. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556…
46. Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S., Sun, Y., Yu, B., Zhang, G., Sun, H., et al. [Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark](https://md.research.elamur.ai/free/en/papers/yue-2024-mmmu-pro-more-robust-multi/). arXiv preprint arXiv:2409.02813, 2024b.
47. Zhang, K., Zeng, S., Hua, E., Ding, N., Chen, Z.-R., Ma, Z., Li, H., Cui, G., Qi, B., Zhu, X., et al. Ultramedical: Building specialized generalists in biomedicine. Advances in Neural Information Processing Systems, 37: 26045–26081, 2024. [find](https://scholar.google.com/scholar?q=Zhang%2C+K%2E%2C+Zeng%2C+S%2E%2C+Hua%2C+E%2E%2C+Ding%2C+N%2E%2C+Chen%2C+Z%2E%2DR%2E%2C+Ma%2C+Z%2E%2C+Li%2C+H%2E%2C+Cui%2C+G%2E%2C+Qi%2C+B%2E%2C+Zhu%2C+X%2E%2C+et+al%2E+Ultramedical%3A+Building+specialized+generalists+in+biomedicine%2E+Advances+in+Neural+Information+Proc)
48. Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., and Xie, W. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023. — [arXiv:2305.10415](https://arxiv.org/abs/2305.10415) [find](https://scholar.google.com/scholar?q=Zhang%2C+X%2E%2C+Wu%2C+C%2E%2C+Zhao%2C+Z%2E%2C+Lin%2C+W%2E%2C+Zhang%2C+Y%2E%2C+Wang%2C+Y%2E%2C+and+Xie%2C+W%2E+Pmc%2Dvqa%3A+Visual+instruction+tuning+for+medical+visual+question+answering%2E+arXiv+preprint+arXiv%3A2305%2E10415%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E10)

*Updated 2 November 2026*