---
title: "Reasoning or Knowledge: Stratified Evaluation of Biomedical LLMs"
description: "Popular biomedical QA benchmarks such as MedQA, MedMCQA and PubMedQA mix… 81% accuracy. 47 references, cited by 1 paper here."
authors: ["Rahul Thapa", "Qingyang Wu", "Kevin Wu", "Harrison G Zhang", "Angela Zhang", "Eric Wu", "Haotian Ye", "James Zou"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.111/"
doi: "10.18653/v1/2026.eacl-long.111"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/thapa-2026-reasoning-or-knowledge-stratified-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/thapa-2026-reasoning-or-knowledge-stratified-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/16/) / Reasoning or Knowledge: Stratified Evaluation of Biomedical LLMs

# Reasoning or Knowledge: Stratified Evaluation of Biomedical LLMs

**Authors:** [Rahul Thapa](https://md.research.elamur.ai/free/en/authors/rahul-thapa/), [Qingyang Wu](https://md.research.elamur.ai/free/en/authors/qingyang-wu/), [Kevin Wu](https://md.research.elamur.ai/free/en/authors/kevin-wu/), Harrison G Zhang, Angela Zhang, [Eric Wu](https://md.research.elamur.ai/free/en/authors/eric-wu/), [Haotian Ye](https://md.research.elamur.ai/free/en/authors/haotian-ye/) and [James Zou](https://md.research.elamur.ai/free/en/authors/james-zou/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.111/) · [PDF](https://aclanthology.org/2026.eacl-long.111.pdf) · [DOI 10.18653/v1/2026.eacl-long.111](https://doi.org/10.18653/v1/2026.eacl-long.111) · [OpenAlex](https://openalex.org/W7140073993)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary

Popular biomedical QA benchmarks such as MedQA, MedMCQA and PubMedQA mix two kinds of questions: some need multi-step clinical inference, and others only need a remembered fact. Because of this, higher scores may reflect better memorization rather than better reasoning. The authors address this by fine-tuning a PubMedBERT classifier on the knowledge/reasoning labels in MedXpertQA. They validate it against gold labels and two medical experts, then use it to split 11 benchmarks into knowledge-heavy and reasoning-heavy subsets.

Under this split, roughly two thirds of questions turn out to be recall-oriented. Every model tested, biomedical or general-domain, scores lower on the reasoning subset. The authors also stress-test models by prepending misleading or doubt-inducing statements. Biomedical reasoning models collapse under this pressure, especially on reasoning-heavy items.

Finally, the authors train BioMed-R1 models with SFT on reasoning-focused data enriched with adversarial traces, followed by GRPO. These models gain in both accuracy and robustness. The work matters because it gives a more honest picture of what "medical reasoning" LLMs can actually do, and it suggests better training curricula.

## Key points

- The classifier reaches 81% accuracy on a balanced held-out test set and roughly human-level agreement with experts (about 79–81%). Across the 11 benchmarks, only 32.8% of questions are classified as reasoning-heavy.
- The accuracy gap between subsets is consistent:
  - HuatuoGPT-o1 scores 56.9% on knowledge-heavy questions versus 44.8% on reasoning-heavy ones.
  - o4-mini scores 77.0% versus 68.9%.
- In the adversarial setting, MedReason falls from 50.4% to 24.4%, while larger general-domain models hold up better.
- The training data extends the m23k dataset with 7,627 reasoning-heavy questions selected by the classifier from MedQA-USMLE, MedMCQA and HeadQA, plus a small set of adversarial traces.
- Two models were trained:
  - BioMed-R1-8B (based on Llama-3.1-8B-Instruct) is the strongest among biomedical LLMs of similar size.
  - BioMed-R1-32B (based on Qwen2.5-32B-Instruct) outperforms MedGemma-27B and approaches HuatuoGPT-o1-70B.
- Code and data are available at github.com/zou-group/BioMed-R1.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations
- [University of California, San Francisco](https://md.research.elamur.ai/free/en/universities/university-of-california-san-francisco/) — University, San Francisco, United States · 4 works · 9,908 citations

📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 234 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 231 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 241 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 184 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Rahul Thapa, Qingyang Wu, Kevin Wu, Harrison G Zhang, Angela Zhang, Eric Wu, Haotian Ye and James Zou. 2026. Reasoning or Knowledge: Stratified Evaluation of Biomedical LLMs. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{thapa2026reasoning,
  title = {{Reasoning or Knowledge: Stratified Evaluation of Biomedical LLMs}},
  author = {Rahul Thapa, Qingyang Wu, Kevin Wu, Harrison G Zhang, Angela Zhang, Eric Wu, Haotian Ye and James Zou},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.111},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.111}
}
```

- [← previous Reasoning Fails Where Step Flow Breaks](https://md.research.elamur.ai/free/en/papers/xu-2026-reasoning-fails-where-step-flow/)
- [next → Reasoning Over Space: Enabling Geographic Reasoning for LLM-Based Generative Next POI Recommendation](https://md.research.elamur.ai/free/en/papers/lv-2026-reasoning-over-space-enabling-geographic/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/16/)

## References

47 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Howard S Barrows and Paul J Feltovich. 1987. The clinical reasoning process. Medical education, 21(2):86– 91. [find](https://scholar.google.com/scholar?q=Howard+S+Barrows+and+Paul+J+Feltovich%2E+1987%2E+The+clinical+reasoning+process%2E+Medical+education%2C+21%282%29%3A86%E2%80%93+91%2E)
2. Patricia Benner, Ronda G Hughes, and Molly Sutphen. 2008. Clinical reasoning, decisionmaking, and action: Thinking critically and clinically. Patient safety and quality: An evidence-based handbook for nurses. [find](https://scholar.google.com/scholar?q=Patricia+Benner%2C+Ronda+G+Hughes%2C+and+Molly+Sutphen%2E+2008%2E+Clinical+reasoning%2C+decisionmaking%2C+and+action%3A+Thinking+critically+and+clinically%2E+Patient+safety+and+quality%3A+An+evidence%2Dbased+handbook+for)
3. Blake Bullwinkel, Amanda Minnich, Shiven Chawla, Gary Lopez, Martin Pouliot, Whitney Maxwell, Joris de Gruyter, Katherine Pratt, Saphir Qi, Nina Chikanov, and 1 others. 2025. Lessons from red teaming 100 generative ai products. arXiv preprint arXiv:2501.07238. — [arXiv:2501.07238](https://arxiv.org/abs/2501.07238) [find](https://scholar.google.com/scholar?q=Blake+Bullwinkel%2C+Amanda+Minnich%2C+Shiven+Chawla%2C+Gary+Lopez%2C+Martin+Pouliot%2C+Whitney+Maxwell%2C+Joris+de+Gruyter%2C+Katherine+Pratt%2C+Saphir+Qi%2C+Nina+Chikanov%2C+and+1+others%2E+2025%2E+Lessons+from+red+teaming+)
4. Crystal T Chang, Hodan Farah, Haiwen Gui, Shawheen Justin Rezaei, Charbel Bou-Khalil, Ye-Jean Park, Akshay Swaminathan, Jesutofunmi A Omiye, Akaash Kolluri, Akash Chaurasia, and 1 others. 2025. Red teaming chatgpt in medicine to yield real-world insights on model behavior. npj Digital Medicine, 8(1)… [find](https://scholar.google.com/scholar?q=Crystal+T+Chang%2C+Hodan+Farah%2C+Haiwen+Gui%2C+Shawheen+Justin+Rezaei%2C+Charbel+Bou%2DKhalil%2C+Ye%2DJean+Park%2C+Akshay+Swaminathan%2C+Jesutofunmi+A+Omiye%2C+Akaash+Kolluri%2C+Akash+Chaurasia%2C+and+1+others%2E+2025%2E+Red+te)
5. Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. 2024a. [Benchmarking large language models on answering and explaining challenging medical questions](https://md.research.elamur.ai/free/en/papers/chen-2025-benchmarking-large-language-models-answering/). arXiv preprint arXiv:2402.18060, 10:13.
6. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024b. Huatuogpt-o1, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925. [find](https://scholar.google.com/scholar?q=Junying+Chen%2C+Zhenyang+Cai%2C+Ke+Ji%2C+Xidong+Wang%2C+Wanlong+Liu%2C+Rongsheng+Wang%2C+Jianye+Hou%2C+and+Benyou+Wang%2E+2024b%2E+Huatuogpt%2Do1%2C+towards+medical+complex+reasoning+with+llms%2E+arXiv+preprint+arXiv%3A2412%2E18)
7. Pat Croskerry. 2009. A universal model of diagnostic reasoning. Academic medicine, 84(8):1022–1028. [find](https://scholar.google.com/scholar?q=Pat+Croskerry%2E+2009%2E+A+universal+model+of+diagnostic+reasoning%2E+Academic+medicine%2C+84%288%29%3A1022%E2%80%931028%2E)
8. Arthur S Elstein, Lee S Shulman, and Sarah A Sprafka. 1978. Medical problem solving: An analysis of clinical reasoning. Harvard University Press. [find](https://scholar.google.com/scholar?q=Arthur+S+Elstein%2C+Lee+S+Shulman%2C+and+Sarah+A+Sprafka%2E+1978%2E+Medical+problem+solving%3A+An+analysis+of+clinical+reasoning%2E+Harvard+University+Press%2E)
9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
10. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. [Domain-specific language model pretraining for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/). ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23.
11. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
12. Tianyu Han, Sven Nebelung, Firas Khader, Tianci Wang, Gustav Müller-Franzes, Christiane Kuhl, Sebastian Försch, Jens Kleesiek, Christoph Haarburger, Keno K Bressem, and 1 others. 2024. Medical large language models are susceptible to targeted misinformation attacks. NPJ digital medicine, 7(1):288. [find](https://scholar.google.com/scholar?q=Tianyu+Han%2C+Sven+Nebelung%2C+Firas+Khader%2C+Tianci+Wang%2C+Gustav+M%C3%BCller%2DFranzes%2C+Christiane+Kuhl%2C+Sebastian+F%C3%B6rsch%2C+Jens+Kleesiek%2C+Christoph+Haarburger%2C+Keno+K+Bressem%2C+and+1+others%2E+2024%2E+Medical+large+l)
13. Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. 2025. m1: Unleash the potential of test-time scaling for medical reasoning with large language models. arXiv preprint arXiv:2504.00869. — [arXiv:2504.00869](https://arxiv.org/abs/2504.00869) [find](https://scholar.google.com/scholar?q=Xiaoke+Huang%2C+Juncheng+Wu%2C+Hui+Liu%2C+Xianfeng+Tang%2C+and+Yuyin+Zhou%2E+2025%2E+m1%3A+Unleash+the+potential+of+test%2Dtime+scaling+for+medical+reasoning+with+large+language+models%2E+arXiv+preprint+arXiv%3A2504%2E0086)
14. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276.
15. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). arXiv preprint arXiv:2412.16720.
16. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421.
17. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). arXiv preprint arXiv:1909.06146.
18. Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170. [find](https://scholar.google.com/scholar?q=Anastasia+Krithara%2C+Anastasios+Nentidis%2C+Konstantinos+Bougiatiotis%2C+and+Georgios+Paliouras%2E+2023%2E+Bioasq%2Dqa%3A+A+manually+curated+corpus+for+biomedical+question+answering%2E+Scientific+Data%2C+10%281%29%3A170%2E)
19. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, and 1 others. 2024. [Training language models to self-correct via reinforcement learning](https://md.research.elamur.ai/free/en/papers/kumar-2024-training-language-models-self-correct/). arXiv preprint arXiv:2409.12917.
20. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
21. Stuart Lubarsky, Valérie Dory, Marie-Claude Audétat, Eugène Custers, and Bernard Charlin. 2015. Using script theory to cultivate illness script formation and clinical reasoning in health professions education. Canadian Medical Education Journal, 6(2):e61. [find](https://scholar.google.com/scholar?q=Stuart+Lubarsky%2C+Val%C3%A9rie+Dory%2C+Marie%2DClaude+Aud%C3%A9tat%2C+Eug%C3%A8ne+Custers%2C+and+Bernard+Charlin%2E+2015%2E+Using+script+theory+to+cultivate+illness+script+formation+and+clinical+reasoning+in+health+professions+e)
22. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+and+1+others%2E+2023%2E+Self%2Drefine%3A+Iterative+refinement+with)
23. James A Marcum. 2012. An integrated model of clinical reasoning: dual-process theory of cognition and metacognition. Journal of evaluation in clinical practice, 18(5):954–961. [find](https://scholar.google.com/scholar?q=James+A+Marcum%2E+2012%2E+An+integrated+model+of+clinical+reasoning%3A+dual%2Dprocess+theory+of+cognition+and+metacognition%2E+Journal+of+evaluation+in+clinical+practice%2C+18%285%29%3A954%E2%80%93961%2E)
24. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. [Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/). In Conference on health, inference, and learning, pages 248–260. PMLR.
25. Vimla L Patel and Guy J Groen. 1986. Knowledge based solution strategies in medical reasoning. Cognitive science, 10(1):91–116. [find](https://scholar.google.com/scholar?q=Vimla+L+Patel+and+Guy+J+Groen%2E+1986%2E+Knowledge+based+solution+strategies+in+medical+reasoning%2E+Cognitive+science%2C+10%281%29%3A91%E2%80%93116%2E)
26. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, and 1 others. 2025. [Humanity’s last exam](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/). arXiv preprint arXiv:2501.14249.
27. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling.
28. Roger C Schank and Robert P Abelson. 2013. Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology press. [find](https://scholar.google.com/scholar?q=Roger+C+Schank+and+Robert+P+Abelson%2E+2013%2E+Scripts%2C+plans%2C+goals%2C+and+understanding%3A+An+inquiry+into+human+knowledge+structures%2E+Psychology+press%2E)
29. Ian A Scott. 2009. Errors in clinical reasoning: causes and remedial strategies. BMj, 338. [find](https://scholar.google.com/scholar?q=Ian+A+Scott%2E+2009%2E+Errors+in+clinical+reasoning%3A+causes+and+remedial+strategies%2E+BMj%2C+338%2E)
30. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, and 1 others. 2025. Medgemma technical report. arXiv preprint arXiv:2507.05201. [find](https://scholar.google.com/scholar?q=Andrew+Sellergren%2C+Sahar+Kazemzadeh%2C+Tiam+Jaroensri%2C+Atilla+Kiraly%2C+Madeleine+Traverse%2C+Timo+Kohlberger%2C+Shawn+Xu%2C+Fayaz+Jamil%2C+C%C3%ADan+Hughes%2C+Charles+Lau%2C+and+1+others%2E+2025%2E+Medgemma+technical+report%2E)
31. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. [Hybridflow: A flexible and efficient rlhf framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). arXiv preprint arXiv:2409.19256.
32. Hyoung Seok Shin. 2019. Reasoning processes in clinical reasoning: from the perspective of cognitive psychology. Korean journal of medical education, 31(4):299. [find](https://scholar.google.com/scholar?q=Hyoung+Seok+Shin%2E+2019%2E+Reasoning+processes+in+clinical+reasoning%3A+from+the+perspective+of+cognitive+psychology%2E+Korean+journal+of+medical+education%2C+31%284%29%3A299%2E)
33. Erik Stolper, Paul Van Royen, Edmund Jack, Jeroen Uleman, and Marcel Olde Rikkert. 2021. Embracing complexity with systems thinking in general practitioners’ clinical reasoning helps handling uncertainty. Journal of Evaluation in Clinical Practice, 27(5):1175–1181. [find](https://scholar.google.com/scholar?q=Erik+Stolper%2C+Paul+Van+Royen%2C+Edmund+Jack%2C+Jeroen+Uleman%2C+and+Marcel+Olde+Rikkert%2E+2021%2E+Embracing+complexity+with+systems+thinking+in+general+practitioners%E2%80%99+clinical+reasoning+helps+handling+uncertai)
34. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. — [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/)
35. David Vilares and Carlos Gómez-Rodríguez. 2019. [Head-qa: A healthcare dataset for complex reasoning](https://md.research.elamur.ai/free/en/papers/vilares-2019-head-qa-healthcare-dataset-complex/). arXiv preprint arXiv:1906.04701.
36. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). In The Thirty-eight Conference on Neural Information Processing S…
37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824– 24837.
38. Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. [Generating sequences by learning to self-correct](https://md.research.elamur.ai/free/en/papers/welleck-2022-generating-sequences-learning-self-correct/). arXiv preprint arXiv:2211.00053.
39. Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, and 1 others. 2025a. Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs. arXiv preprint arXiv:2504.00993. — [arXiv:2504.00993](https://arxiv.org/abs/2504.00993) [find](https://scholar.google.com/scholar?q=Juncheng+Wu%2C+Wenlong+Deng%2C+Xingxuan+Li%2C+Sheng+Liu%2C+Taomian+Mi%2C+Yifan+Peng%2C+Ziyang+Xu%2C+Yi+Liu%2C+Hyunjin+Cho%2C+Chang%2DIn+Choi%2C+and+1+others%2E+2025a%2E+Medreason%3A+Eliciting+factual+medical+reasoning+steps+in+l)
40. Kevin Wu, Eric Wu, Rahul Thapa, Kevin Wei, Angela Zhang, Arvind Suresh, Jacqueline J Tao, Min Woo Sun, Alejandro Lozano, and James Zou. 2025b. Medcasereasoning: Evaluating and learning diagnostic reasoning from clinical case reports. arXiv preprint arXiv:2505.11733. — [arXiv:2505.11733](https://arxiv.org/abs/2505.11733) [find](https://scholar.google.com/scholar?q=Kevin+Wu%2C+Eric+Wu%2C+Rahul+Thapa%2C+Kevin+Wei%2C+Angela+Zhang%2C+Arvind+Suresh%2C+Jacqueline+J+Tao%2C+Min+Woo+Sun%2C+Alejandro+Lozano%2C+and+James+Zou%2E+2025b%2E+Medcasereasoning%3A+Evaluating+and+learning+diagnostic+reas)
41. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+and+1+others%2E+2024%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115)
42. Shahram Yazdani and Maryam Hoseini Abardeh. 2019. Five decades of research and theorization on clinical reasoning: a critical review. Advances in medical education and practice, pages 703–716. [find](https://scholar.google.com/scholar?q=Shahram+Yazdani+and+Maryam+Hoseini+Abardeh%2E+2019%2E+Five+decades+of+research+and+theorization+on+clinical+reasoning%3A+a+critical+review%2E+Advances+in+medical+education+and+practice%2C+pages+703%E2%80%93716%2E)
43. Yiming Zhang, Jianfeng Chi, Hailey Nguyen, Kartikeya Upasani, Daniel M Bikel, Jason Weston, and Eric Michael Smith. 2024. Backtracking improves generation safety. arXiv preprint arXiv:2409.14586. — [arXiv:2409.14586](https://arxiv.org/abs/2409.14586) [find](https://scholar.google.com/scholar?q=Yiming+Zhang%2C+Jianfeng+Chi%2C+Hailey+Nguyen%2C+Kartikeya+Upasani%2C+Daniel+M+Bikel%2C+Jason+Weston%2C+and+Eric+Michael+Smith%2E+2024%2E+Backtracking+improves+generation+safety%2E+arXiv+preprint+arXiv%3A2409%2E14586%2E+%E2%80%94+%5Ba)
44. Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. 2025. Easyr1: An efficient, scalable, multi-modality rl training framework. [find](https://scholar.google.com/scholar?q=Yaowei+Zheng%2C+Junting+Lu%2C+Shenzhi+Wang%2C+Zhangchi+Feng%2C+Dongdong+Kuang%2C+and+Yuwen+Xiong%2E+2025%2E+Easyr1%3A+An+efficient%2C+scalable%2C+multi%2Dmodality+rl+training+framework%2E)
45. Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, and 1 others. 2024. [Evaluation of openai o1: Opportunities and challenges of agi](https://md.research.elamur.ai/free/en/papers/zhong-2024-evaluation-openai-o1-opportunities-challenges/). arXiv preprint arXiv:2409.18486.
46. Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. arXiv preprint arXiv:2302.13439. [find](https://scholar.google.com/scholar?q=Kaitlyn+Zhou%2C+Dan+Jurafsky%2C+and+Tatsunori+Hashimoto%2E+2023%2E+Navigating+the+grey+area%3A+How+expressions+of+uncertainty+and+overconfidence+affect+language+models%2E+arXiv+preprint+arXiv%3A2302%2E13439%2E)
47. Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362. [find](https://scholar.google.com/scholar?q=Yuxin+Zuo%2C+Shang+Qu%2C+Yifei+Li%2C+Zhangren+Chen%2C+Xuekai+Zhu%2C+Ermo+Hua%2C+Kaiyan+Zhang%2C+Ning+Ding%2C+and+Bowen+Zhou%2E+2025%2E+Medxpertqa%3A+Benchmarking+expert%2Dlevel+medical+reasoning+and+understanding%2E+arXiv+prep)

*Updated 31 October 2026*