---
title: "From Scores to Steps — paper by Wang et al. (2025)"
description: "Finally, they propose MedRaC, a training-free agentic pipeline. 58 references, cited by 4 papers here."
authors: ["Benlu Wang", "Iris Xia", "Yifan Zhang", "Junda Wang", "Feiyun Ouyang", "Shuo Han", "Arman Cohan", "Hong Yu", "Zonghai Yao"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.548/"
doi: "10.18653/v1/2025.emnlp-main.548"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2025-scores-steps-diagnosing-improving-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2025-scores-steps-diagnosing-improving-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations

# From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations

**Authors:** [Benlu Wang](https://md.research.elamur.ai/free/en/authors/benlu-wang/), Iris Xia, [Yifan Zhang](https://md.research.elamur.ai/free/en/authors/yifan-zhang/), [Junda Wang](https://md.research.elamur.ai/free/en/authors/junda-wang/), [Feiyun Ouyang](https://md.research.elamur.ai/free/en/authors/feiyun-ouyang/), [Shuo Han](https://md.research.elamur.ai/free/en/authors/shuo-han/), [Arman Cohan](https://md.research.elamur.ai/free/en/authors/arman-cohan/), [Hong Yu](https://md.research.elamur.ai/free/en/authors/hong-yu/) and [Zonghai Yao](https://md.research.elamur.ai/free/en/authors/zonghai-yao/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.548/) · [PDF](https://aclanthology.org/2025.emnlp-main.548.pdf) · [DOI 10.18653/v1/2025.emnlp-main.548](https://doi.org/10.18653/v1/2025.emnlp-main.548) · [OpenAlex](https://openalex.org/W4416036814)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/wang-2025-scores-steps-diagnosing-improving-llm/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs are increasingly considered for clinical decision support, yet their skill at medical calculations, such as GFR or cardiovascular risk scores, is poorly measured. MedCalc-Bench, the main benchmark, only checks whether the final number lands within a loose ±5% window. That hides wrong formulas, misread patient variables and faulty arithmetic. The authors first clean and restructure MedCalc-Bench. They then score model outputs in four sequential stages: formula choice, variable extraction, arithmetic and final answer. Each stage is judged by an LLM, and the numerical tolerances follow MDCalc's rules. An automatic error-attribution component labels where and why each failure happens, and two medical experts confirmed that it agrees with their judgment. Finally, they propose MedRaC, a training-free agentic pipeline. It retrieves the correct formula through retrieval-augmented generation and delegates computation to executed Python code. Under the stricter protocol, GPT-4o's accuracy falls from 62.7% to 43.6%, while MedRaC substantially raises the accuracy of several LLMs. The work argues that safety-critical domains need evaluation of intermediate reasoning steps, not just final-answer scores.

## Key points
- The base resource is MedCalc-Bench: 55 MDCalc calculators and 1,047 patient vignettes. The authors fixed corrupted items and built a reference library of formulas for all 55 calculators.
- Their metrics include strict end-to-end correctness, which requires all four steps to pass. They also report the conditional correctness of each step and how often each step is the first point of failure.
- Numerical checking replaces the ±5% margin with MDCalc-style precision (±0.005 at two decimals). This drops GPT-4o from 62.7% to 43.6%.
- MedRaC needs no fine-tuning and yields accuracy gains of 16.35% to 53.19%, depending on the model. Ablations show that both the retrieval and the code-execution components reduce errors.
- Formula retrieval scales well when the bank grows from 55 to 785 formulas:
  - all three OpenAI embedding models reach 100% top-2 accuracy;
  - top-1 accuracy stays above 96%.
- Code and data are available on GitHub (Super-Billy/EMNLP-2025-MedRaC) under the Apache-2.0 license. All experiments are English-only.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) — University, Amherst Center, United States · 106 works · 66,692 citations
- [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) — University, New Haven, United States · 95 works · 12,765 citations
- [University of Massachusetts Lowell](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-lowell/) — University, Lowell, United States · 15 works · 3,942 citations
- [Center for Health Optimization and Implementation Research](https://md.research.elamur.ai/free/en/labs/center-for-health-optimization-and-implementation-research/) — Research institute, Bedford, United States · 3 works · 14 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 241 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 184 works

## Cite

Cite as: Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu and Zonghai Yao. 2025. From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2025scores,
  title = {{From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations}},
  author = {Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu and Zonghai Yao},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.548},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.548}
}
```

- [← previous Feature Engineering is not Dead: A Step Towards State of the Art for Arabic Automated Essay Scoring](https://md.research.elamur.ai/free/en/papers/sayed-2025-feature-engineering-is-not-dead/)
- [next → GATEAU: Selecting Influential Samples for Long Context Alignment](https://md.research.elamur.ai/free/en/papers/si-2025-gateau-selecting-influential-samples-long/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/)

## References

58 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2025. About us - mdcalc. [https://www.mdcalc.com/](https://www.mdcalc.com/) about-us. Accessed: 2025-09-19. [↗](https://www.mdcalc.com/)
2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
3. Lameck Mbangula Amugongo, Pietro Mascheroni, Steven Brooks, Stefan Doering, and Jan Seidel. 2025. Retrieval augmented generation for large language models in healthcare: A systematic review. PLOS Digital Health, 4(6):e0000877. [find](https://scholar.google.com/scholar?q=Lameck+Mbangula+Amugongo%2C+Pietro+Mascheroni%2C+Steven+Brooks%2C+Stefan+Doering%2C+and+Jan+Seidel%2E+2025%2E+Retrieval+augmented+generation+for+large+language+models+in+healthcare%3A+A+systematic+review%2E+PLOS+Digi)
4. Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, et al. 2025. [Healthbench: Evaluating large language models towards improved human health](https://md.research.elamur.ai/free/en/papers/arora-2025-healthbench-evaluating-large-language-models/). arXiv preprint arXiv:2505.08775.
5. John W Ayers, Adam Poliak, Mark Dredze, Eric C Leas, Zechariah Zhu, Jessica B Kelley, Dennis J Faix, Aaron M Goodman, Christopher A Longhurst, Michael Hogarth, et al. 2023. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. J… [find](https://scholar.google.com/scholar?q=John+W+Ayers%2C+Adam+Poliak%2C+Mark+Dredze%2C+Eric+C+Leas%2C+Zechariah+Zhu%2C+Jessica+B+Kelley%2C+Dennis+J+Faix%2C+Aaron+M+Goodman%2C+Christopher+A+Longhurst%2C+Michael+Hogarth%2C+et+al%2E+2023%2E+Comparing+physician+and+art)
6. Nathan Brake and Thomas Schaaf. 2024. Comparing two model designs for clinical note generation; is an llm a useful evaluator of consistency? arXiv preprint arXiv:2404.06503. [find](https://scholar.google.com/scholar?q=Nathan+Brake+and+Thomas+Schaaf%2E+2024%2E+Comparing+two+model+designs+for+clinical+note+generation%3B+is+an+llm+a+useful+evaluator+of+consistency%3F+arXiv+preprint+arXiv%3A2404%2E06503%2E)
7. Hong Chen, Duc Vo, Hiroya Takamura, Yusuke Miyao, and Hideki Nakayama. 2022a. Storyer: Automatic story evaluation via ranking, rating and reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1739–1753. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hong+Chen%2C+Duc+Vo%2C+Hiroya+Takamura%2C+Yusuke+Miyao%2C+and+Hideki+Nakayama%2E+2022a%2E+Storyer%3A+Automatic+story+evaluation+via+ranking%2C+rating+and+reasoning%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+)
8. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022b. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W+Cohen%2E+2022b%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E+arXiv+preprint+arXiv%3A2211%2E12588%2E)
9. Yun-Wei Chu, Kai Zhang, Christopher Malon, and Martin Renqiang Min. 2025. Reducing hallucinations of medical multimodal large language models with visual retrieval-augmented generation. [find](https://scholar.google.com/scholar?q=Yun%2DWei+Chu%2C+Kai+Zhang%2C+Christopher+Malon%2C+and+Martin+Renqiang+Min%2E+2025%2E+Reducing+hallucinations+of+medical+multimodal+large+language+models+with+visual+retrieval%2Daugmented+generation%2E)
10. Philip Chung, Akshay Swaminathan, Alex J Goodell, Yeasul Kim, S Momsen Reincke, Lichy Han, Ben Deverett, Mohammad Amin Sadeghi, Abdel-Badih Ariss, Marc Ghanem, et al. 2025. Verifact: Verifying facts in llm-generated clinical text with electronic health records. arXiv preprint arXiv:2501.16672. — [arXiv:2501.16672](https://arxiv.org/abs/2501.16672) [find](https://scholar.google.com/scholar?q=Philip+Chung%2C+Akshay+Swaminathan%2C+Alex+J+Goodell%2C+Yeasul+Kim%2C+S+Momsen+Reincke%2C+Lichy+Han%2C+Ben+Deverett%2C+Mohammad+Amin+Sadeghi%2C+Abdel%2DBadih+Ariss%2C+Marc+Ghanem%2C+et+al%2E+2025%2E+Verifact%3A+Verifying+facts+i)
11. Donald W Cockcroft and Henry Gault. 1976. Prediction of creatinine clearance from serum creatinine. Nephron, 16(1):31–41. [find](https://scholar.google.com/scholar?q=Donald+W+Cockcroft+and+Henry+Gault%2E+1976%2E+Prediction+of+creatinine+clearance+from+serum+creatinine%2E+Nephron%2C+16%281%29%3A31%E2%80%9341%2E)
12. Emma Croxford, Yanjun Gao, Elliot First, Nicholas Pellegrino, Miranda Schnier, John Caskey, Madeline Oguss, Graham Wills, Guanhua Chen, Dmitriy Dligach, et al. 2025. Automating evaluation of ai text generation in healthcare with a large language model (llm)-as-a-judge. medRxiv, pages 2025–04. [find](https://scholar.google.com/scholar?q=Emma+Croxford%2C+Yanjun+Gao%2C+Elliot+First%2C+Nicholas+Pellegrino%2C+Miranda+Schnier%2C+John+Caskey%2C+Madeline+Oguss%2C+Graham+Wills%2C+Guanhua+Chen%2C+Dmitriy+Dligach%2C+et+al%2E+2025%2E+Automating+evaluation+of+ai+text+g)
13. Hannah Decker, Karen Trang, Joel Ramirez, Alexis Colley, Logan Pierce, Melissa Coleman, Tasce Bon-giovanni, Genevieve B Melton, and Elizabeth Wick. 2023. Large language model- based chatbot vs surgeon-generated informed consent documentation for common procedures. JAMA network open, 6(10):e2336997–e… [find](https://scholar.google.com/scholar?q=Hannah+Decker%2C+Karen+Trang%2C+Joel+Ramirez%2C+Alexis+Colley%2C+Logan+Pierce%2C+Melissa+Coleman%2C+Tasce+Bon%2Dgiovanni%2C+Genevieve+B+Melton%2C+and+Elizabeth+Wick%2E+2023%2E+Large+language+model%2D+based+chatbot+vs+surgeon)
14. Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. pages 6556–6576. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See+Kiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2024%2E+Gptscore%3A+Evaluate+as+you+desire%2E+pages+6556%E2%80%936576%2E)
15. Brian F Gage, Amy D Waterman, William Shannon, Michael Boechler, Michael W Rich, and Martha J Radford. 2001. Validation of clinical classification schemes for predicting stroke: results from the national registry of atrial fibrillation. Jama, 285(22):2864–2870. [find](https://scholar.google.com/scholar?q=Brian+F+Gage%2C+Amy+D+Waterman%2C+William+Shannon%2C+Michael+Boechler%2C+Michael+W+Rich%2C+and+Martha+J+Radford%2E+2001%2E+Validation+of+clinical+classification+schemes+for+predicting+stroke%3A+results+from+the+natio)
16. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+2023%2E+Pal%3A+Program%2Daided+language+models%2E+In+International+Conference+on+Machine+Learning%2C+page)
17. Alex J Goodell, Simon N Chu, Dara Rouholiman, and Larry F Chu. 2025. Large language model agents can use tools to perform clinical calculations. npj Digital Medicine, 8(1):163. [find](https://scholar.google.com/scholar?q=Alex+J+Goodell%2C+Simon+N+Chu%2C+Dara+Rouholiman%2C+and+Larry+F+Chu%2E+2025%2E+Large+language+model+agents+can+use+tools+to+perform+clinical+calculations%2E+npj+Digital+Medicine%2C+8%281%29%3A163%2E)
18. Rachel S Goodman, J Randall Patrinely, Cosby A Stone Jr, Eli Zimmerman, Rebecca R Donald, Sam S Chang, Sean T Berkowitz, Avni P Finn, Eiman Jahangir, Elizabeth A Scoville, et al. 2023. Accuracy and reliability of chatbot responses to physician questions. JAMA network open, 6(10):e2336483. [find](https://scholar.google.com/scholar?q=Rachel+S+Goodman%2C+J+Randall+Patrinely%2C+Cosby+A+Stone+Jr%2C+Eli+Zimmerman%2C+Rebecca+R+Donald%2C+Sam+S+Chang%2C+Sean+T+Berkowitz%2C+Avni+P+Finn%2C+Eiman+Jahangir%2C+Elizabeth+A+Scoville%2C+et+al%2E+2023%2E+Accuracy+and+re)
19. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. The Innovation. [find](https://scholar.google.com/scholar?q=Jiawei+Gu%2C+Xuhui+Jiang%2C+Zhichao+Shi%2C+Hexiang+Tan%2C+Xuehao+Zhai%2C+Chengjin+Xu%2C+Wei+Li%2C+Yinghan+Shen%2C+Shengjie+Ma%2C+Honghao+Liu%2C+et+al%2E+2024%2E+A+survey+on+llm%2Das%2Da%2Djudge%2E+The+Innovation%2E)
20. Collaborative Initiative. 2010. 2010 rheumatoid arthritis classification criteria. Arthritis & Rheumatism, 62(9):2569–2581. [find](https://scholar.google.com/scholar?q=Collaborative+Initiative%2E+2010%2E+2010+rheumatoid+arthritis+classification+criteria%2E+Arthritis+%26+Rheumatism%2C+62%289%29%3A2569%E2%80%932581%2E)
21. Minbyul Jeong, Jiwoong Sohn, Mujeen Sung, and Jaewoo Kang. 2024. [Improving medical reasoning through retrieval and self-reflection with retrieval-augmented large language models](https://md.research.elamur.ai/free/en/papers/jeong-2024-improving-medical-reasoning-through-retrieval/). Bioinformatics, 40(Supplement_1):i119–i129.
22. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421.
23. Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, Justin M Cheung, Robert Chen, Ronald M Summers, Justin F Rousseau, Peiyun Ni, Marc J Landsman, et al. 2024. Hidden flaws behind expert-level accuracy of multimodal gpt-4 vision in medicine. NPJ Digital Medicine, 7(1):190. [find](https://scholar.google.com/scholar?q=Qiao+Jin%2C+Fangyuan+Chen%2C+Yiliang+Zhou%2C+Ziyang+Xu%2C+Justin+M+Cheung%2C+Robert+Chen%2C+Ronald+M+Summers%2C+Justin+F+Rousseau%2C+Peiyun+Ni%2C+Marc+J+Landsman%2C+et+al%2E+2024%2E+Hidden+flaws+behind+expert%2Dlevel+accuracy+)
24. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language proce…
25. Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, et al. 2023. [Critiquellm: Scaling llm-as-critic for effective and explainable evaluation of large language model generation](https://md.research.elamur.ai/free/en/papers/ke-2023-critiquellm-informative-critique-generation/). corr, abs/2311.18702. detection for generative large…
26. Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina Applebaum, Zain Anwar, Maame Sarfo-Gyamfi, Conrad Safranek, Abid Anwar, Andrew Zhang, et al. 2024. Medcalc-bench: Evaluating large language models for medical calculations. Advances in Neural Information Processing Systems, 37:84730– 847… [find](https://scholar.google.com/scholar?q=Nikhil+Khandekar%2C+Qiao+Jin%2C+Guangzhi+Xiong%2C+Soren+Dunn%2C+Serina+Applebaum%2C+Zain+Anwar%2C+Maame+Sarfo%2DGyamfi%2C+Conrad+Safranek%2C+Abid+Anwar%2C+Andrew+Zhang%2C+et+al%2E+2024%2E+Medcalc%2Dbench%3A+Evaluating+large+langua)
27. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+arXiv+preprint+arXiv%3A2302%2E14520%2E)
28. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. [Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models](https://md.research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/). PLoS digital h…
29. Jinu Lee and Julia Hockenmaier. 2025. Evaluating step-by-step reasoning traces: A survey. arXiv preprint arXiv:2502.12289. [find](https://scholar.google.com/scholar?q=Jinu+Lee+and+Julia+Hockenmaier%2E+2025%2E+Evaluating+step%2Dby%2Dstep+reasoning+traces%3A+A+survey%2E+arXiv+preprint+arXiv%3A2502%2E12289%2E)
30. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2024a. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594. [find](https://scholar.google.com/scholar?q=Dawei+Li%2C+Bohan+Jiang%2C+Liangjie+Huang%2C+Alimohammad+Beigi%2C+Chengshuai+Zhao%2C+Zhen+Tan%2C+Amrita+Bhattacharjee%2C+Yuxuan+Jiang%2C+Canyu+Chen%2C+Tianhao+Wu%2C+et+al%2E+2024a%2E+From+generation+to+judgment%3A+Opportunitie)
31. Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen, Emma Pierson, Pang Wei W Koh, and Yulia Tsvetkov. 2024b. [Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning](https://md.research.elamur.ai/free/en/papers/li-2024-mediq-question-asking-llms-benchmark/). Advances in Neural Information Processing Systems, 37:28858–28888.
32. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). In The Twelfth International Conference on Learning Representations.
33. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634.
34. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+2023%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Df)
35. Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. [Augmented language models: a survey](https://md.research.elamur.ai/free/en/papers/mialon-2023-augmented-language-models-survey/). Transactions on Machine Learning Re…
36. Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. 2023. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452. [find](https://scholar.google.com/scholar?q=Harsha+Nori%2C+Yin+Tat+Lee%2C+Sheng+Zhang%2C+Dean+Carignan%2C+Richard+Edgar%2C+Nicolo+Fusi%2C+Nicholas+King%2C+Jonathan+Larson%2C+Yuanzhi+Li%2C+Weishung+Liu%2C+et+al%2E+2023%2E+Can+generalist+foundation+models+outcompete+spe)
37. Harsha Nori, Naoto Usuyama, Nicholas King, Scott Mayer McKinney, Xavier Fernandes, Sheng Zhang, and Eric Horvitz. 2024. From medprompt to o1: Exploration of run-time strategies for medical challenge problems and beyond. arXiv preprint arXiv:2411.03590. — [arXiv:2411.03590](https://arxiv.org/abs/2411.03590) [find](https://scholar.google.com/scholar?q=Harsha+Nori%2C+Naoto+Usuyama%2C+Nicholas+King%2C+Scott+Mayer+McKinney%2C+Xavier+Fernandes%2C+Sheng+Zhang%2C+and+Eric+Horvitz%2E+2024%2E+From+medprompt+to+o1%3A+Exploration+of+run%2Dtime+strategies+for+medical+challenge+p)
38. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. [Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/). In Conference on health, inference, and learning, pages 248–260. PMLR.
39. Chantal Shaib, Millicent Li, Sebastian Joseph, Iain Marshall, Junyi Jessy Li, and Byron C Wallace. 2023. Summarizing, simplifying, and synthesizing medical evidence using gpt-3 (with varying success). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume… [find](https://scholar.google.com/scholar?q=Chantal+Shaib%2C+Millicent+Li%2C+Sebastian+Joseph%2C+Iain+Marshall%2C+Junyi+Jessy+Li%2C+and+Byron+C+Wallace%2E+2023%2E+Summarizing%2C+simplifying%2C+and+synthesizing+medical+evidence+using+gpt%2D3+%28with+varying+success%29%2E)
40. Chengyu Shen, Zhen Hao Wong, Runming He, Hao Liang, Meiyi Qiang, Zimo Meng, Zhengyang Zhao, Bohan Zeng, Zhengzhou Zhu, Bin Cui, et al. 2025. Let’s verify math questions step by step. arXiv preprint arXiv:2505.13903. [find](https://scholar.google.com/scholar?q=Chengyu+Shen%2C+Zhen+Hao+Wong%2C+Runming+He%2C+Hao+Liang%2C+Meiyi+Qiang%2C+Zimo+Meng%2C+Zhengyang+Zhao%2C+Bohan+Zeng%2C+Zhengzhou+Zhu%2C+Bin+Cui%2C+et+al%2E+2025%2E+Let%E2%80%99s+verify+math+questions+step+by+step%2E+arXiv+preprint+ar)
41. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. [Large language models encode clinical knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/). Nature, 620(7972):172–180.
42. Mengxuan Sun, Ehud Reiter, Anne E Kiltie, George Ramsay, Lisa Duncan, Peter Murchie, and Rosalind Adam. 2024. Effectiveness of chatgpt in explaining complex medical reports to patients. arXiv preprint arXiv:2406.15963. — [arXiv:2406.15963](https://arxiv.org/abs/2406.15963) [find](https://scholar.google.com/scholar?q=Mengxuan+Sun%2C+Ehud+Reiter%2C+Anne+E+Kiltie%2C+George+Ramsay%2C+Lisa+Duncan%2C+Peter+Murchie%2C+and+Rosalind+Adam%2E+2024%2E+Effectiveness+of+chatgpt+in+explaining+complex+medical+reports+to+patients%2E+arXiv+preprint)
43. Arun James Thirunavukarasu, Refaat Hassan, Shathar Mahmood, Rohan Sanghera, Kara Barzangi, Mo-hanned El Mukashfi, and Sachin Shah. 2023a. Trialling a large language model (chatgpt) in general practice with the applied knowledge test: observational study demonstrating opportunities and limitations in… [find](https://scholar.google.com/scholar?q=Arun+James+Thirunavukarasu%2C+Refaat+Hassan%2C+Shathar+Mahmood%2C+Rohan+Sanghera%2C+Kara+Barzangi%2C+Mo%2Dhanned+El+Mukashfi%2C+and+Sachin+Shah%2E+2023a%2E+Trialling+a+large+language+model+%28chatgpt%29+in+general+practice)
44. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023b. Large language models in medicine. Nature medicine, 29(8):1930– 1940. [find](https://scholar.google.com/scholar?q=Arun+James+Thirunavukarasu%2C+Darren+Shu+Jeng+Ting%2C+Kabilan+Elangovan%2C+Laura+Gutierrez%2C+Ting+Fang+Tan%2C+and+Daniel+Shu+Wei+Ting%2E+2023b%2E+Large+language+models+in+medicine%2E+Nature+medicine%2C+29%288%29%3A1930%E2%80%93+194)
45. Hieu Tran, Zonghai Yao, Junda Wang, Yifan Zhang, Zhichao Yang, and Hong Yu. 2024. [Rare: Retrieval-augmented reasoning enhancement for large language models](https://md.research.elamur.ai/free/en/papers/tran-2025-rare-retrieval-augmented-reasoning-enhancement/). arXiv preprint arXiv:2412.02830.
46. Tao Tu, Mike Schaekermann, Anil Palepu, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Yong Cheng, et al. 2025. Towards conversational diagnostic artificial intelligence. Nature, pages 1–9. [find](https://scholar.google.com/scholar?q=Tao+Tu%2C+Mike+Schaekermann%2C+Anil+Palepu%2C+Khaled+Saab%2C+Jan+Freyberg%2C+Ryutaro+Tanno%2C+Amy+Wang%2C+Brenna+Li%2C+Mohamed+Amin%2C+Yong+Cheng%2C+et+al%2E+2025%2E+Towards+conversational+diagnostic+artificial+intelligence%2E)
47. Junda Wang, Zhichao Yang, Zonghai Yao, and Hong Yu. 2024. Jmlr: Joint medical llm and retrieval training for enhancing reasoning and professional question answering capability. arXiv preprint arXiv:2402.17887. [find](https://scholar.google.com/scholar?q=Junda+Wang%2C+Zhichao+Yang%2C+Zonghai+Yao%2C+and+Hong+Yu%2E+2024%2E+Jmlr%3A+Joint+medical+llm+and+retrieval+training+for+enhancing+reasoning+and+professional+question+answering+capability%2E+arXiv+preprint+arXiv%3A24)
48. Junda Wang, Zonghai Yao, Zhichao Yang, Huixue Zhou, Rumeng Li, Xun Wang, Yucheng Xu, and Hong Yu. 2023. Notechat: a dataset of synthetic doctor-patient conversations conditioned on clinical notes. arXiv preprint arXiv:2310.15959. — [arXiv:2310.15959](https://arxiv.org/abs/2310.15959) [find](https://scholar.google.com/scholar?q=Junda+Wang%2C+Zonghai+Yao%2C+Zhichao+Yang%2C+Huixue+Zhou%2C+Rumeng+Li%2C+Xun+Wang%2C+Yucheng+Xu%2C+and+Hong+Yu%2E+2023%2E+Notechat%3A+a+dataset+of+synthetic+doctor%2Dpatient+conversations+conditioned+on+clinical+notes%2E+arX)
49. Zonghai Yao, Aditya Parashar, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, Zhichao Yang, and Hong Yu. 2024a. Mcqg-srefine: Multiple choice question generation and evaluation with iterative self-critique, correction, and comparison feedback. arXiv preprint arXiv:2410.13191. [find](https://scholar.google.com/scholar?q=Zonghai+Yao%2C+Aditya+Parashar%2C+Huixue+Zhou%2C+Won+Seok+Jang%2C+Feiyun+Ouyang%2C+Zhichao+Yang%2C+and+Hong+Yu%2E+2024a%2E+Mcqg%2Dsrefine%3A+Multiple+choice+question+generation+and+evaluation+with+iterative+self%2Dcritique)
50. Zonghai Yao and Hong Yu. 2025. A survey on llm-based multi-agent ai hospital. [find](https://scholar.google.com/scholar?q=Zonghai+Yao+and+Hong+Yu%2E+2025%2E+A+survey+on+llm%2Dbased+multi%2Dagent+ai+hospital%2E)
51. Zonghai Yao, Zihao Zhang, Chaolong Tang, Xingyu Bian, Youxia Zhao, Zhichao Yang, Junda Wang, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, et al. 2024b. Medqa-cs: Benchmarking large language models clinical skills using an ai-sce framework. arXiv preprint arXiv:2410.01553. — [arXiv:2410.01553](https://arxiv.org/abs/2410.01553) [find](https://scholar.google.com/scholar?q=Zonghai+Yao%2C+Zihao+Zhang%2C+Chaolong+Tang%2C+Xingyu+Bian%2C+Youxia+Zhao%2C+Zhichao+Yang%2C+Junda+Wang%2C+Huixue+Zhou%2C+Won+Seok+Jang%2C+Feiyun+Ouyang%2C+et+al%2E+2024b%2E+Medqa%2Dcs%3A+Benchmarking+large+language+models+clini)
52. Chen Zhang, Luis Fernando D’Haro, Yiming Chen, Malu Zhang, and Haizhou Li. 2024. A comprehensive analysis of the effectiveness of large language models as automatic dialogue evaluators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19515–19524. [find](https://scholar.google.com/scholar?q=Chen+Zhang%2C+Luis+Fernando+D%E2%80%99Haro%2C+Yiming+Chen%2C+Malu+Zhang%2C+and+Haizhou+Li%2E+2024%2E+A+comprehensive+analysis+of+the+effectiveness+of+large+language+models+as+automatic+dialogue+evaluators%2E+In+Proceedings)
53. Wan Zhang and Jing Zhang. 2025. Hallucination mitigation for retrieval-augmented large language models: A review. Mathematics, 13(5):856. [find](https://scholar.google.com/scholar?q=Wan+Zhang+and+Jing+Zhang%2E+2025%2E+Hallucination+mitigation+for+retrieval%2Daugmented+large+language+models%3A+A+review%2E+Mathematics%2C+13%285%29%3A856%2E)
54. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in neural information processing systems, 36:46595–46623.
55. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
56. Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. [Benchmarking retrieval-augmented generation for medicine](https://md.research.elamur.ai/free/en/papers/xiong-2024-benchmarking-retrieval-augmented-generation-medicine/). In Findings of the Association for Computational Linguistics ACL 2024, pages 6233–6251.
57. Guangzhi Xiong, Qiao Jin, Xiao Wang, Minjia Zhang, Zhiyong Lu, and Aidong Zhang. 2025. Improving retrieval-augmented generation in medicine with iterative follow-up questions. In Pacific Symposium on Biocomputing (PSB) 2025, pages 199–214. World Scientific. [find](https://scholar.google.com/scholar?q=Guangzhi+Xiong%2C+Qiao+Jin%2C+Xiao+Wang%2C+Minjia+Zhang%2C+Zhiyong+Lu%2C+and+Aidong+Zhang%2E+2025%2E+Improving+retrieval%2Daugmented+generation+in+medicine+with+iterative+follow%2Dup+questions%2E+In+Pacific+Symposium+on+)
58. Zhichao Yang, Zonghai Yao, Mahbuba Tasmin, Parth Vashisht, Won Seok Jang, Feiyun Ouyang, Beining Wang, David McManus, Dan Berlowitz, and Hong Yu. 2025. Unveiling gpt-4v’s hidden challenges behind high accuracy on usmle questions: Observational study. Journal of Medical Internet Research, 27:e65146. [find](https://scholar.google.com/scholar?q=Zhichao+Yang%2C+Zonghai+Yao%2C+Mahbuba+Tasmin%2C+Parth+Vashisht%2C+Won+Seok+Jang%2C+Feiyun+Ouyang%2C+Beining+Wang%2C+David+McManus%2C+Dan+Berlowitz%2C+and+Hong+Yu%2E+2025%2E+Unveiling+gpt%2D4v%E2%80%99s+hidden+challenges+behind+high)

*Updated 31 October 2026*