---
title: "PMC-LLaMA: Towards Building Open-source Language — Wu 2023"
description: "General-purpose LLMs handle everyday dialogue well but often give plausible yet wrong answers in medicine because they lack domain knowledge."
authors: ["Chaoyi Wu", "Weixiong Lin", "Xiaoman Zhang", "Ya Zhang", "Yanfeng Wang", "Weidi Xie"]
year: 2023
venue: "arXiv:2304.14454"
source: "https://arxiv.org/abs/2304.14454"
doi: "10.48550/arxiv.2304.14454"
arxiv: "2304.14454"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2023-pmc-llama-towards-building-open/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2023-pmc-llama-towards-building-open/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / PMC-LLaMA: Towards Building Open-source Language Models for Medicine

# PMC-LLaMA: Towards Building Open-source Language Models for Medicine

**Authors:** Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie · **Year:** 2023 · **Venue:** arXiv:2304.14454 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.14454) · [PDF](https://arxiv.org/pdf/2304.14454) · [DOI 10.48550/arxiv.2304.14454](https://doi.org/10.48550/arxiv.2304.14454) · [arXiv 2304.14454](https://arxiv.org/abs/2304.14454) · [OpenAlex](https://openalex.org/W4367623495)

**Citations:** 49 citations (openalex-snapshot · 2026-10-14) · [93](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
General-purpose LLMs handle everyday dialogue well but often give plausible yet wrong answers in medicine because they lack domain knowledge. The strongest medical models, such as MedPALM-2, do not disclose their data or training details. The authors propose an open, reproducible recipe for adapting LLaMA to medicine in two stages.

1. **Knowledge injection.** The model is further pretrained on MedC-K, a corpus of biomedical papers and medical textbooks. The data mix deliberately favours textbook content.
2. **Instruction tuning.** The model is then tuned on MedC-I, a new instruction set that combines medical QA with rationales, patient–doctor conversations and knowledge-graph prompts. During this stage, loss is computed only on responses.

The resulting 13B-parameter PMC-LLaMA was tested on public medical QA benchmarks. It outperformed LLaMA-2 and even ChatGPT, despite being far smaller. Ablations confirm that each stage contributes to this result. The work matters because it releases the models, code and datasets openly, giving the community a lightweight and transparent baseline for medical LLM research.

## Key points
- **MedC-K corpus:** about 4.8M biomedical papers selected from S2ORC (81.1M papers) by their PMC IDs, totalling roughly 75B tokens, plus 30K medical textbooks yielding about 4B tokens after cleaning.
- **Training data mix:** each batch combines books, papers and general RedPajama-Data text in a 15:4:1 ratio, which emphasises fundamental textbook knowledge.
- **MedC-I instruction dataset:** 202M tokens covering medical conversations, rationale-based QA, and knowledge-graph entity descriptions and relations.
- **Evaluation:** benchmarks were MedQA, MedMCQA and PubMedQA, where PMC-LLaMA (13B) beat ChatGPT (175B) and LLaMA-2 (70B).
- **Availability:** all resources are released at github.com/chaoyi-wu/PMC-LLaMA.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

93 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 89 not yet.

- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://md.research.elamur.ai/free/en/papers/labrak-2024-biomistral-collection-open-source-pretrained/)
- [Faithful Medical Dialogue Generation Using Homo-Heterogeneous Exemplar-based In-Context Knowledge Grounding](https://md.research.elamur.ai/free/en/papers/priya-2026-faithful-medical-dialogue-generation-homo/)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://md.research.elamur.ai/free/en/papers/xiong-2024-benchmarking-retrieval-augmented-generation-medicine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wu2023pmc,
  title = {{PMC-LLaMA: Towards Building Open-source Language Models for Medicine}},
  author = {Wu, Chaoyi and Lin, Weixiong and Zhang, Xiaoman and Zhang, Ya and Wang, Yanfeng and Xie, Weidi},
  year = {2023},
  doi = {10.48550/arxiv.2304.14454},
  eprint = {2304.14454},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.14454}
}
```

- [← previous Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://md.research.elamur.ai/free/en/papers/allen-2023-zhu-physics-language-models-part-3/)
- [next → PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations](https://md.research.elamur.ai/free/en/papers/qi-2023-pragmaticqa-dataset-pragmatic-question-answering/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anil, R.; Dai, A. M.; et al. 2023. PaLM 2 Technical Report. ArXiv, abs/2305.10403. [find](https://scholar.google.com/scholar?q=Anil%2C+R%2E%3B+Dai%2C+A%2E+M%2E%3B+et+al%2E+2023%2E+PaLM+2+Technical+Report%2E+ArXiv%2C+abs%2F2305%2E10403%2E) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
2. Chen, T.; et al. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. Chiang, W.-L.; Li, Z.; and others. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E%3B+et+al%2E+2016%2E+Training+deep+nets+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1604%2E06174%2E+Chiang%2C+W%2E%2DL%2E%3B+Li%2C+Z%2E%3B+and+others%2E+2023%2E+Vicuna%3A+An+Open%2DSource+Chatbot+Impressing+GPT%2D4+with+90%25)
3. Chung, H. W.; et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%3B+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+preprint+arXiv%3A2210%2E11416%2E)
4. Computer, T. 2023. RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset. [find](https://scholar.google.com/scholar?q=Computer%2C+T%2E+2023%2E+RedPajama%3A+An+Open+Source+Recipe+to+Reproduce+LLaMA+training+dataset%2E)
5. Diao, S.; Pan, R.; et al. 2023. LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. [https://optimalscale.github.io/LMFlow/](https://optimalscale.github.io/LMFlow/). [↗](https://optimalscale.github.io/LMFlow/)
6. Du, Z.; et al. 2021. Glm: General language model pre-training with autoregressive blank infilling. arXiv preprint arXiv:2103.10360. [find](https://scholar.google.com/scholar?q=Du%2C+Z%2E%3B+et+al%2E+2021%2E+Glm%3A+General+language+model+pre%2Dtraining+with+autoregressive+blank+infilling%2E+arXiv+preprint+arXiv%3A2103%2E10360%2E) [our card](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/)
7. Han, T.; Adams, L. C.; et al. 2023. MedAlpaca–An Open-Source Collection of Medical Conversational AI Models and Training Data. arXiv preprint arXiv:2304.08247. [find](https://scholar.google.com/scholar?q=Han%2C+T%2E%3B+Adams%2C+L%2E+C%2E%3B+et+al%2E+2023%2E+MedAlpaca%E2%80%93An+Open%2DSource+Collection+of+Medical+Conversational+AI+Models+and+Training+Data%2E+arXiv+preprint+arXiv%3A2304%2E08247%2E) [our card](https://md.research.elamur.ai/free/en/papers/han-2023-medalpaca-open-source-collection-medical/)
8. Iyer, S.; et al. 2022. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017. [find](https://scholar.google.com/scholar?q=Iyer%2C+S%2E%3B+et+al%2E+2022%2E+Opt%2Diml%3A+Scaling+language+model+instruction+meta+learning+through+the+lens+of+generalization%2E+arXiv+preprint+arXiv%3A2212%2E12017%2E) [our card](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/)
9. Jin, D.; Pan, E.; et al. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14): 6421. [find](https://scholar.google.com/scholar?q=Jin%2C+D%2E%3B+Pan%2C+E%2E%3B+et+al%2E+2021%2E+What+disease+does+this+patient+have%3F+a+large%2Dscale+open+domain+question+answering+dataset+from+medical+exams%2E+Applied+Sciences%2C+11%2814%29%3A+6421%2E) [our card](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/)
10. Jin, Q.; et al. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146. [find](https://scholar.google.com/scholar?q=Jin%2C+Q%2E%3B+et+al%2E+2019%2E+Pubmedqa%3A+A+dataset+for+biomedical+research+question+answering%2E+arXiv+preprint+arXiv%3A1909%2E06146%2E) [our card](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/)
11. Kaplan, J.; McCandlish, S.; Henighan, T. J.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D. 2020. Scaling Laws for Neural Language Models. ArXiv, abs/2001.08361. [find](https://scholar.google.com/scholar?q=Kaplan%2C+J%2E%3B+McCandlish%2C+S%2E%3B+Henighan%2C+T%2E+J%2E%3B+Brown%2C+T%2E+B%2E%3B+Chess%2C+B%2E%3B+Child%2C+R%2E%3B+Gray%2C+S%2E%3B+Radford%2C+A%2E%3B+Wu%2C+J%2E%3B+and+Amodei%2C+D%2E+2020%2E+Scaling+Laws+for+Neural+Language+Models%2E+ArXiv%2C+abs%2F2001%2E08361%2E)
12. Kung, T. H.; Cheatham, M.; Medenilla, A.; Sillos, C.; Leon, L. D.; Elepaño, C.; Madriaga, M.; Aggabao, R.; Diaz-Candido, G.; Maningo, J.; and Tseng, V. 2022. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2. [find](https://scholar.google.com/scholar?q=Kung%2C+T%2E+H%2E%3B+Cheatham%2C+M%2E%3B+Medenilla%2C+A%2E%3B+Sillos%2C+C%2E%3B+Leon%2C+L%2E+D%2E%3B+Elepa%C3%B1o%2C+C%2E%3B+Madriaga%2C+M%2E%3B+Aggabao%2C+R%2E%3B+Diaz%2DCandido%2C+G%2E%3B+Maningo%2C+J%2E%3B+and+Tseng%2C+V%2E+2022%2E+Performance+of+ChatGPT+on+USMLE%3A+Potential)
13. Lee, J.; et al. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 1234–1240. [find](https://scholar.google.com/scholar?q=Lee%2C+J%2E%3B+et+al%2E+2020%2E+BioBERT%3A+a+pre%2Dtrained+biomedical+language+representation+model+for+biomedical+text+mining%2E+Bioinformatics%2C+36%284%29%3A+1234%E2%80%931240%2E) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
14. Lindberg, D. A.; Humphreys, B. L.; and McCray, A. T. 1993. The unified medical language system. Yearbook of medical informatics, 2(01): 41–51. [find](https://scholar.google.com/scholar?q=Lindberg%2C+D%2E+A%2E%3B+Humphreys%2C+B%2E+L%2E%3B+and+McCray%2C+A%2E+T%2E+1993%2E+The+unified+medical+language+system%2E+Yearbook+of+medical+informatics%2C+2%2801%29%3A+41%E2%80%9351%2E)
15. Lo, K.; et al. 2020. S2ORC: The Semantic Scholar Open Research Corpus. In 58th Annual Meeting of the ACL, 4969– 4983. Online: ACL. [find](https://scholar.google.com/scholar?q=Lo%2C+K%2E%3B+et+al%2E+2020%2E+S2ORC%3A+The+Semantic+Scholar+Open+Research+Corpus%2E+In+58th+Annual+Meeting+of+the+ACL%2C+4969%E2%80%93+4983%2E+Online%3A+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/lo-2019-s2orc-semantic-scholar-open-research/)
16. Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E%3B+and+Hutter%2C+F%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2E)
17. Luo, R.; Sun, L.; Xia, Y.; Qin, T.; Zhang, S.; Poon, H.; and Liu, T.-Y. 2022. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6): bbac409. [find](https://scholar.google.com/scholar?q=Luo%2C+R%2E%3B+Sun%2C+L%2E%3B+Xia%2C+Y%2E%3B+Qin%2C+T%2E%3B+Zhang%2C+S%2E%3B+Poon%2C+H%2E%3B+and+Liu%2C+T%2E%2DY%2E+2022%2E+BioGPT%3A+generative+pre%2Dtrained+transformer+for+biomedical+text+generation+and+mining%2E+Briefings+in+Bioinformatics%2C+23%286%29%3A+)
18. Nori, H.; King, N.; et al. 2023. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375. [find](https://scholar.google.com/scholar?q=Nori%2C+H%2E%3B+King%2C+N%2E%3B+et+al%2E+2023%2E+Capabilities+of+gpt%2D4+on+medical+challenge+problems%2E+arXiv+preprint+arXiv%3A2303%2E13375%2E) [our card](https://md.research.elamur.ai/free/en/papers/nori-2023-capabilities-gpt-4-medical-challenge/)
19. OpenAI. 2023a. GPT-4 Technical Report. arXiv:2303.08774. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023a%2E+GPT%2D4+Technical+Report%2E+arXiv%3A2303%2E08774%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
20. OpenAI. 2023b. OpenAI. Introducing chatgpt. [https://openai.com/blog/chatgpt/](https://openai.com/blog/chatgpt/). [↗](https://openai.com/blog/chatgpt/)
21. Ouyang, L.; et al. 2022. Training language models to follow instructions with human feedback. NIPS, 35: 27730–27744. [find](https://scholar.google.com/scholar?q=Ouyang%2C+L%2E%3B+et+al%2E+2022%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+NIPS%2C+35%3A+27730%E2%80%9327744%2E)
22. Pal, A.; Umapathi, L. K.; et al. 2022. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, 248–260. [find](https://scholar.google.com/scholar?q=Pal%2C+A%2E%3B+Umapathi%2C+L%2E+K%2E%3B+et+al%2E+2022%2E+MedMCQA%3A+A+large%2Dscale+multi%2Dsubject+multi%2Dchoice+dataset+for+medical+domain+question+answering%2E+In+Conference+on+Health%2C+Inference%2C+and+Learning%2C+248%E2%80%93260%2E) [our card](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/)
23. PMLR. Peng, B.; et al. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277. [find](https://scholar.google.com/scholar?q=PMLR%2E+Peng%2C+B%2E%3B+et+al%2E+2023%2E+Instruction+tuning+with+gpt%2D4%2E+arXiv+preprint+arXiv%3A2304%2E03277%2E) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/)
24. Singhal, K.; Azizi, S.; Tu, T.; Mahdavi, S. S.; Wei, J.; et al. 2022. Large Language Models Encode Clinical Knowledge. arXiv preprint arXiv:2212.13138. [find](https://scholar.google.com/scholar?q=Singhal%2C+K%2E%3B+Azizi%2C+S%2E%3B+Tu%2C+T%2E%3B+Mahdavi%2C+S%2E+S%2E%3B+Wei%2C+J%2E%3B+et+al%2E+2022%2E+Large+Language+Models+Encode+Clinical+Knowledge%2E+arXiv+preprint+arXiv%3A2212%2E13138%2E) [our card](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)
25. Taori, R.; et al. 2023. Stanford Alpaca: An Instruction-following LLaMA model. [https://github.com/tatsu-lab/](https://github.com/tatsu-lab/) stanford_alpaca. [↗](https://github.com/tatsu-lab/)
26. Touvron, H.; Lavril, T.; Izacard, G.; Martinet; et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%3B+Lavril%2C+T%2E%3B+Izacard%2C+G%2E%3B+Martinet%3B+et+al%2E+2023a%2E+Llama%3A+Open+and+efficient+foundation+language+models%2E+arXiv+preprint+arXiv%3A2302%2E13971%2E)
27. Touvron, H.; Martin, L.; Stone, K. R.; et al. 2023b. Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv, abs/2307.09288. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%3B+Martin%2C+L%2E%3B+Stone%2C+K%2E+R%2E%3B+et+al%2E+2023b%2E+Llama+2%3A+Open+Foundation+and+Fine%2DTuned+Chat+Models%2E+ArXiv%2C+abs%2F2307%2E09288%2E)
28. Yunxiang, L.; et al. 2023. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. arXiv preprint arXiv:2303.14070. [find](https://scholar.google.com/scholar?q=Yunxiang%2C+L%2E%3B+et+al%2E+2023%2E+Chatdoctor%3A+A+medical+chat+model+fine%2Dtuned+on+llama+model+using+medical+domain+knowledge%2E+arXiv+preprint+arXiv%3A2303%2E14070%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-chatdoctor-medical-chat-model-fine/)
29. Zhang, K.; et al. 2023. BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks. arXiv preprint arXiv:2305.17100. — [arXiv:2305.17100](https://arxiv.org/abs/2305.17100) [find](https://scholar.google.com/scholar?q=Zhang%2C+K%2E%3B+et+al%2E+2023%2E+BiomedGPT%3A+A+Unified+and+Generalist+Biomedical+Generative+Pre%2Dtrained+Transformer+for+Vision%2C+Language%2C+and+Multimodal+Tasks%2E+arXiv+preprint+arXiv%3A2305%2E17100%2E+%E2%80%94+%5BarXiv%3A2305%2E171)