---
title: "Improving Medical Reasoning through Retrieval and"
description: "The release of the data, code and 7B/13B weights makes this a reusable recipe for building… 42 references, cited by 33 papers here."
authors: ["Minbyul Jeong", "Jiwoong Sohn", "Mujeen Sung", "Jaewoo Kang"]
year: 2024
venue: "arXiv:2401.15269"
source: "https://arxiv.org/abs/2401.15269"
doi: "10.48550/arxiv.2401.15269"
arxiv: "2401.15269"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jeong-2024-improving-medical-reasoning-through-retrieval/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jeong-2024-improving-medical-reasoning-through-retrieval/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models

# Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models

**Authors:** [Minbyul Jeong](https://md.research.elamur.ai/free/en/authors/minbyul-jeong/), [Jiwoong Sohn](https://md.research.elamur.ai/free/en/authors/jiwoong-sohn/), [Mujeen Sung](https://md.research.elamur.ai/free/en/authors/mujeen-sung/) and [Jaewoo Kang](https://md.research.elamur.ai/free/en/authors/jaewoo-kang/) · **Year:** 2024 · **Venue:** [arXiv:2401.15269](https://md.research.elamur.ai/free/en/venues/bioinformatics/) ([Bioinformatics 2024](https://md.research.elamur.ai/free/en/venues/bioinformatics/2024/)) · **Publisher:** [Oxford University Press](https://md.research.elamur.ai/free/en/publishers/oxford-university-press/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.15269) · [PDF](https://arxiv.org/pdf/2401.15269) · [DOI 10.48550/arxiv.2401.15269](https://doi.org/10.48550/arxiv.2401.15269) · [arXiv 2401.15269](https://arxiv.org/abs/2401.15269) · [OpenAlex](https://openalex.org/W4400128031)

**Citations:** 131 citations (OpenAlex, 2026-10-14) · [33](https://md.research.elamur.ai/free/en/papers/jeong-2024-improving-medical-reasoning-through-retrieval/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
General-purpose retrieval-augmented LLMs often fail in medicine. Methods tuned on open-domain data, such as Self-RAG, tend to pull irrelevant documents and misjudge their usefulness on clinical questions. The authors address this with Self-BioRAG, a biomedical adaptation of Self-RAG. The model decides for itself whether a query needs external evidence, retrieves passages with the MedCPT retriever from biomedical corpora when needed, picks the best passage using reflective-token scores, and grades its own explanations and answers. To train it, they assembled biomedical and clinical instructions and labeled a sample with GPT-4 to train a critic model. The critic then annotated and filtered the full instruction set, and the filtered data was used to fine-tune a generator initialized from Self-RAG weights. Across multiple-choice and long-form medical QA benchmarks, Self-BioRAG beats open models of 7B parameters or fewer and standard RAG. Ablations show that every domain-specific component contributes, with domain instruction tuning contributing most. The release of the data, code and 7B/13B weights makes this a reusable recipe for building self-checking, domain-specialised RAG systems.

## Key points
- **Instruction data:** The pool of about 120k instructions combines Mol-Instructions, MedInstruct and roughly 18k new examples generated in the Self-Instruct style. After the critic model's filtering, 84k instructions with reflective tokens were kept for generator training. The critic itself was trained on 5k examples labeled via the GPT-4 API.
- **Retrieval setup:** MedCPT serves as the retriever and draws on four corpora: medical textbooks, abstracts from PubMed, full-text articles from PMC, and clinical guidelines. Medical Textbook was retrieved from most often for USMLE-style questions.
- **Multiple-choice QA:** Self-BioRAG averages a 7.2% absolute gain over the best open-foundation model of 7B or smaller on MedQA, MedMCQA and MMLU.
- **Long-form QA:** On LiveQA and MedicationQA, it averages about 8 points higher Rouge-1 than plain RAG.
- **Release and venue:** The 7B and 13B models, data and training code are available at github.com/dmis-lab/self-biorag. The paper was accepted at ISMB 2024 (Bioinformatics).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Korea University](https://md.research.elamur.ai/free/en/universities/korea-university/) — University, Seoul, South Korea · 26 works · 9,490 citations
- [Kyung Hee University](https://md.research.elamur.ai/free/en/universities/kyung-hee-university/) — University, Seoul, South Korea · 5 works · 176 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 241 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 184 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

💡 Concepts (1): [💡 Retrieval-augmented generation](https://md.research.elamur.ai/free/en/concepts/retrieval-augmented-generation/) · 50 works

## Cite

Cite as: Minbyul Jeong, Jiwoong Sohn, Mujeen Sung and Jaewoo Kang. 2024. Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models. arXiv:2401.15269.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jeong2024improving,
  title = {{Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models}},
  author = {Minbyul Jeong, Jiwoong Sohn, Mujeen Sung and Jaewoo Kang},
  year = {2024},
  doi = {10.48550/arxiv.2401.15269},
  eprint = {2401.15269},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.15269}
}
```

- [← previous WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://md.research.elamur.ai/free/en/papers/han-2024-wildguard-open-one-stop-moderation/)
- [next → Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://md.research.elamur.ai/free/en/papers/gao-2024-omni-math-universal-olympiad-level/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

42 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abacha, A. B. et al. (2017). Overview of the medical question answering task at trec 2017 liveqa. In TREC. [find](https://scholar.google.com/scholar?q=Abacha%2C+A%2E+B%2E+et+al%2E+%282017%29%2E+Overview+of+the+medical+question+answering+task+at+trec+2017+liveqa%2E+In+TREC%2E)
2. Abacha, A. B. et al. (2019). Bridging the gap between consumers’ medication questions and trusted answers. In MedInfo. [find](https://scholar.google.com/scholar?q=Abacha%2C+A%2E+B%2E+et+al%2E+%282019%29%2E+Bridging+the+gap+between+consumers%E2%80%99+medication+questions+and+trusted+answers%2E+In+MedInfo%2E)
3. Asai, A. et al. (2023). [Self-rag: Learning to retrieve, generate, and critique through self-reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/). arXiv preprint arXiv:2310.11511.
4. Bajaj, P. et al. (2016). [Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). arXiv preprint arXiv:1611.09268.
5. Cao, M. et al. (2022). [Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization](https://md.research.elamur.ai/free/en/papers/cao-2022-hallucinated-but-factual-inspecting-factuality/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
6. Chen, Z. et al. (2023). [Meditron-70b: Scaling medical pretraining for large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/). arXiv preprint arXiv:2311.16079.
7. Christiano, P. F. et al. (2017). Deep reinforcement learning from human preferences. Advances in neural information processing systems. [find](https://scholar.google.com/scholar?q=Christiano%2C+P%2E+F%2E+et+al%2E+%282017%29%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2E)
8. Chung, H. W. et al. (2022). Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E+et+al%2E+%282022%29%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+preprint+arXiv%3A2210%2E11416%2E)
9. Dao, T. et al. (2022). Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E+et+al%2E+%282022%29%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+Neural+Information+Processing+Systems%2E)
10. Fang, Y. et al. (2023). Mol-instructions: A large-scale biomolecular instruction dataset for large language models. arXiv preprint arXiv:2306.08018. [find](https://scholar.google.com/scholar?q=Fang%2C+Y%2E+et+al%2E+%282023%29%2E+Mol%2Dinstructions%3A+A+large%2Dscale+biomolecular+instruction+dataset+for+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E08018%2E)
11. Guo, G. et al. (2003). Knn model-based approach in classification. In On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE: OTM Confederated International Conferences, CoopIS, DOA, and ODBASE 2003, Catania, Sicily, Italy, November 3-7, 2003. Proceedings. [find](https://scholar.google.com/scholar?q=Guo%2C+G%2E+et+al%2E+%282003%29%2E+Knn+model%2Dbased+approach+in+classification%2E+In+On+The+Move+to+Meaningful+Internet+Systems+2003%3A+CoopIS%2C+DOA%2C+and+ODBASE%3A+OTM+Confederated+International+Conferences%2C+CoopIS%2C+DOA%2C)
12. Guu, K. et al. (2020). [Retrieval augmented language model pre-training](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/). In International conference on machine learning.
13. Han, T. et al. (2023). [Medalpaca–an open-source collection of medical conversational ai models and training data](https://md.research.elamur.ai/free/en/papers/han-2023-medalpaca-open-source-collection-medical/). arXiv preprint arXiv:2304.08247.
14. Hendrycks, D. et al. (2020). [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
15. Izacard, G. et al. (2022a). [Few-shot learning with retrieval augmented language models](https://md.research.elamur.ai/free/en/papers/izacard-2022-atlas-few-shot-learning-retrieval/). arXiv preprint arXiv:2208.03299.
16. Izacard, G. et al. (2022b). [Unsupervised dense information retrieval with contrastive learning](https://md.research.elamur.ai/free/en/papers/izacard-2021-unsupervised-dense-information-retrieval-contrastive/). Transactions on Machine Learning Research.
17. Ji, Z. et al. (2023). [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/).
18. ACM Computing Surveys. Jiang, Z. et al. (2023). [Active retrieval augmented generation](https://md.research.elamur.ai/free/en/papers/jiang-2023-active-retrieval-augmented-generation/). arXiv preprint arXiv:2305.06983.
19. Jin, D. et al. (2021). [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences.
20. Jin, Q. et al. (2019). [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
21. Jin, Q. et al. (2023). Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics (Oxford, England). [find](https://scholar.google.com/scholar?q=Jin%2C+Q%2E+et+al%2E+%282023%29%2E+Medcpt%3A+Contrastive+pre%2Dtrained+transformers+with+large%2Dscale+pubmed+search+logs+for+zero%2Dshot+biomedical+information+retrieval%2E+Bioinformatics+%28Oxford%2C+England%29%2E)
22. Kang, M. et al. (2023). [Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks](https://md.research.elamur.ai/free/en/papers/kang-2023-knowledge-augmented-reasoning-distillation-small/). arXiv preprint arXiv:2305.18395.
23. Karpukhin, V. et al. (2020). [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
24. Kitaev, N. and Klein, D. (2018). [Constituency parsing with a self-attentive encoder](https://md.research.elamur.ai/free/en/papers/kitaev-2018-constituency-parsing-self-attentive-encoder/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
25. Kitaev, N. et al. (2019). [Multilingual constituency parsing with self-attention and pre-training](https://md.research.elamur.ai/free/en/papers/kitaev-2019-multilingual-constituency-parsing-self-attention/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
26. Kwon, W. et al. (2023). [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles.
27. Lewis, P. et al. (2020). [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in Neural Information Processing Systems.
28. Li, B. et al. (2023). Meddm: Llm-executable clinical guidance tree for clinical decision-making. arXiv preprint arXiv:2312.02441. — [arXiv:2312.02441](https://arxiv.org/abs/2312.02441) [find](https://scholar.google.com/scholar?q=Li%2C+B%2E+et+al%2E+%282023%29%2E+Meddm%3A+Llm%2Dexecutable+clinical+guidance+tree+for+clinical+decision%2Dmaking%2E+arXiv+preprint+arXiv%3A2312%2E02441%2E+%E2%80%94+%5BarXiv%3A2312%2E02441%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2312%2E02441%29)
29. Lin, C.-Y. (2004). [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out.
30. Mao, Y. et al. (2021). [Generation-augmented retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/mao-2020-generation-augmented-retrieval-open-domain/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
31. Nori, H. et al. (2023). [Capabilities of gpt-4 on medical challenge problems](https://md.research.elamur.ai/free/en/papers/nori-2023-capabilities-gpt-4-medical-challenge/). arXiv preprint arXiv:2303.13375.
32. OpenAI (2023b). Openai [gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
33. Ouyang, L. et al. (2022). [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems.
34. Pal, A. et al. (2022). [Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/). In Conference on Health, Inference, and Learning.
35. Paszke, A. et al. (2019). [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in neural information processing systems.
36. Rajbhandari, S. et al. (2020). Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. [find](https://scholar.google.com/scholar?q=Rajbhandari%2C+S%2E+et+al%2E+%282020%29%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis%2E)
37. IEEE. Schulman, J. et al. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=IEEE%2E+Schulman%2C+J%2E+et+al%2E+%282017%29%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
38. Shao, Z. et al. (2023). [Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy](https://md.research.elamur.ai/free/en/papers/shao-2023-enhancing-retrieval-augmented-large-language/). arXiv preprint arXiv:2305.15294.
39. Singhal, K. et al. (2022). [Large language models encode clinical knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/). arXiv preprint arXiv:2212.13138.
40. Taori, R. et al. (2023). Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. [https://crfm.stanford](https://crfm.stanford). edu/2023/03/13/alpaca. html. [↗](https://crfm.stanford)
41. Taylor, R. et al. (2022). [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/). arXiv preprint arXiv:2211.09085. 8 [https://parser.kitaev.io/*Table](https://parser.kitaev.io/*Table) 9. Statistics of our generated Biomedical Instruction Set.* \\\# of total instances 18,854 avg. instruction length (words) 24.2 avg. input length (words) 35.4 avg… [↗](https://parser.kitaev.io/*Table)
42. Statistics *Fig. 5: Nested piechart of our biomedical instructions root verbs (inner circle) and their four noun objects (outer circle). It signifies the diversity of generated instructions and relates to biomedical terms such as hypothesis, proteins, diagnosis, and symptoms.* 1 Statistics of our ge… [find](https://scholar.google.com/scholar?q=Statistics+%2AFig%2E+5%3A+Nested+piechart+of+our+biomedical+instructions+root+verbs+%28inner+circle%29+and+their+four+noun+objects+%28outer+circle%29%2E+It+signifies+the+diversity+of+generated+instructions+and+relate)

*Updated 31 October 2026*