--- title: "LLaVA-Med: Training a Large Language-and-Vision — Li 2023" description: "General-purpose multimodal assistants perform poorly on biomedical images." authors: ["Chunyuan Li", "Cliff Wong", "Sheng Zhang", "Naoto Usuyama", "Haotian Liu", "Jianwei Yang", "Tristan Naumann", "Hoifung Poon", "Jianfeng Gao"] year: 2023 venue: "arXiv:2306.00890" source: "https://arxiv.org/abs/2306.00890" doi: "10.48550/arxiv.2306.00890" arxiv: "2306.00890" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/li-2023-llava-med-training-large-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-llava-med-training-large-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day # LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day **Authors:** [Chunyuan Li](https://md.research.elamur.ai/free/en/authors/chunyuan-li/), Cliff Wong, [Sheng Zhang](https://md.research.elamur.ai/free/en/authors/sheng-zhang/), Naoto Usuyama, [Haotian Liu](https://md.research.elamur.ai/free/en/authors/haotian-liu/), [Jianwei Yang](https://md.research.elamur.ai/free/en/authors/jianwei-yang/), [Tristan Naumann](https://md.research.elamur.ai/free/en/authors/tristan-naumann/), [Hoifung Poon](https://md.research.elamur.ai/free/en/authors/hoifung-poon/), [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) · **Year:** 2023 · **Venue:** arXiv:2306.00890 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2306.00890) · [PDF](https://arxiv.org/pdf/2306.00890) · [DOI 10.48550/arxiv.2306.00890](https://doi.org/10.48550/arxiv.2306.00890) · [arXiv 2306.00890](https://arxiv.org/abs/2306.00890) · [OpenAlex](https://openalex.org/W7133197087) **Citations:** 485 citations (openalex-snapshot · 2026-10-14) · [191](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary General-purpose multimodal assistants perform poorly on biomedical images. They often decline to answer or produce hallucinated replies. Earlier biomedical VQA systems also mostly pick answers from a fixed set, so they cannot hold free-form conversations. The authors present LLaVA-Med, which adapts the general-domain LLaVA model to biomedicine at low cost. They build training data from PMC-15M, a large collection of figure–caption pairs from PubMed Central. Text-only GPT-4 turns captions, plus sentences in the source papers that reference each figure, into multi-turn dialogues about the image, with no human labeling required. Training follows a two-stage curriculum. First, the model learns biomedical concepts by describing images from their raw captions. Second, it is tuned on the synthetic conversations to handle open-ended instructions. The whole process takes under 15 hours on eight A100 GPUs. After further fine-tuning on VQA-RAD, SLAKE and PathVQA, LLaVA-Med beats earlier supervised state-of-the-art systems on some metrics. The work shows a cheap, annotation-free route to domain-specialized visual chat assistants. ## Key points - Stage 1 (concept alignment) uses 600K image–caption pairs sampled from PMC-15M. Stage 2 uses 60K pairs covering five modalities: CXR, CT, MRI, histopathology and gross pathology. - The architecture links a CLIP vision encoder to a 7B LLaMA/Vicuna language model through a linear projection. Variants with a 13B LM and a BiomedCLIP encoder were also tested. - Instruction data enriched with in-paper figure mentions (60K-IM) gave the best results. For example, VQA-RAD closed-question accuracy reached 84.19, compared with 83.50 for M2I2. The BiomedCLIP variant scored 87.11 on SLAKE open questions, compared with 82.05 for BiomedCLIP. - Longer downstream fine-tuning (around 9 epochs) and the larger 13B model both improved performance. - The model could answer Chinese questions on SLAKE without any Chinese training data. Remaining weaknesses include hallucination and shallow reasoning. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 191 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 183 not yet. - [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://md.research.elamur.ai/free/en/papers/arora-2025-healthbench-evaluating-large-language-models/) - [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/) - [Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-multimodal-arxiv-dataset-improving-scientific/) - [Improved Baselines with Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/) - [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/) - [ViX-Ray: A Vietnamese Chest X-Ray Dataset for Vision-Language Models](https://md.research.elamur.ai/free/en/papers/nguyen-2026-vix-ray-vietnamese-chest-x/) - [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://md.research.elamur.ai/free/en/papers/tang-2024-medagents-large-language-models-as/) - [Representation Engineering: A Top-Down Approach to AI Transparency](https://md.research.elamur.ai/free/en/papers/zou-2023-representation-engineering-top-down-approach/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{li2023llava, title = {{LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day}}, author = {Li, Chunyuan and Wong, Cliff and Zhang, Sheng and Usuyama, Naoto and Liu, Haotian and Yang, Jianwei and Naumann, Tristan and Poon, Hoifung and Gao, Jianfeng}, year = {2023}, doi = {10.48550/arxiv.2306.00890}, eprint = {2306.00890}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2306.00890} } ``` - [← previous LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/) - [next → Llemma: An Open Language Model For Mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) ## References 4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Clinical Camel. [https://wanglab.ml/clinical_camel.html](https://wanglab.ml/clinical_camel.html), 2023. 2 [↗](https://wanglab.ml/clinical_camel.html) 2. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. 1 [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+a) 3. Malek Ayoub, Megan Quamme, Abdul-Rahman K Abdel-Reheem, Poe Lwin, and Megan K Quamme. Covid or not covid? a great mimicker behind the smoke screen. Cureus, 13(11), 2021. 5 [find](https://scholar.google.com/scholar?q=Malek+Ayoub%2C+Megan+Quamme%2C+Abdul%2DRahman+K+Abdel%2DReheem%2C+Poe+Lwin%2C+and+Megan+K+Quamme%2E+Covid+or+not+covid%3F+a+great+mimicker+behind+the+smoke+screen%2E+Cureus%2C+13%2811%29%2C+2021%2E+5) 4. Bappy Basak, Alexander Haragan, Michael Shackcloth, and Joyce Thekkudan. Chondromyxoid fibroma of the rib: A rare benign tumor with potential for local recurrence. Cureus, 13(10), 2021. 5 [find](https://scholar.google.com/scholar?q=Bappy+Basak%2C+Alexander+Haragan%2C+Michael+Shackcloth%2C+and+Joyce+Thekkudan%2E+Chondromyxoid+fibroma+of+the+rib%3A+A+rare+benign+tumor+with+potential+for+local+recurrence%2E+Cureus%2C+13%2810%29%2C+2021%2E+5)