--- title: "HuaTuo: Tuning LLaMA Model with Chinese Medical — Wang 2023" description: "General-purpose LLMs such as LLaMA lack reliable medical expertise." authors: ["Haochun Wang", "Chi Liu", "Nuwa Xi", "Zewen Qiang", "Sendong Zhao", "Bing Qin", "Ting Liu"] year: 2023 venue: "arXiv:2304.06975" source: "https://arxiv.org/abs/2304.06975" doi: "10.48550/arxiv.2304.06975" arxiv: "2304.06975" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wang-2023-huatuo-tuning-llama-model-chinese/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2023-huatuo-tuning-llama-model-chinese/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge # HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge **Authors:** [Haochun Wang](https://md.research.elamur.ai/free/en/authors/haochun-wang/), [Chi Liu](https://md.research.elamur.ai/free/en/authors/chi-liu/), [Nuwa Xi](https://md.research.elamur.ai/free/en/authors/nuwa-xi/), Zewen Qiang, [Sendong Zhao](https://md.research.elamur.ai/free/en/authors/sendong-zhao/), [Bing Qin](https://md.research.elamur.ai/free/en/authors/bing-qin/), [Ting Liu](https://md.research.elamur.ai/free/en/authors/ting-liu/) · **Year:** 2023 · **Venue:** arXiv:2304.06975 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2304.06975) · [PDF](https://arxiv.org/pdf/2304.06975) · [DOI 10.48550/arxiv.2304.06975](https://doi.org/10.48550/arxiv.2304.06975) · [arXiv 2304.06975](https://arxiv.org/abs/2304.06975) · [OpenAlex](https://openalex.org/W4366198844) **Citations:** 94 citations (openalex-snapshot · 2026-10-14) · [76](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary General-purpose LLMs such as LLaMA lack reliable medical expertise. Because they are trained mostly on English text, they also handle Chinese poorly, which makes their health advice risky in Chinese settings. The authors, from Harbin Institute of Technology, address this with HuaTuo, a LLaMA-7B model adapted to Chinese medicine through supervised fine-tuning. To build training data, they sampled facts about diseases, drugs and symptoms from the Chinese medical knowledge graph CMeKG. They then had the OpenAI API turn these facts into question–answer pairs, yielding over 8,000 instances. Since instructions in medical dialogue are nearly identical, the instruction field was dropped and only questions and answers were kept. Tying generation to a knowledge graph is meant to keep answers factually grounded. The authors also propose SUS, a human evaluation scheme rating Safety, Usability and Smoothness. On a Chinese medical test set judged by medically trained annotators, HuaTuo gave the most useful and fluent answers. It stayed nearly as safe as LLaMA, which is cautious but uninformative. The work offers an open recipe for adapting LLMs to a specialised domain in a non-English language. ## Key points - Base model: LLaMA-7B. Knowledge source: CMeKG (structured entries for diseases, drugs, symptoms). Training set: 8,000+ QA instances generated with the OpenAI API from sampled knowledge-graph facts. - New metric SUS (Safety, Usability, Smoothness): five annotators with medical backgrounds scored shuffled responses on a 1–3 scale (1 = not acceptable, 2 = acceptable, 3 = good). - Baselines were LLaMA, Alpaca (80k+ general-domain instructions) and ChatGLM-6B. The text announces four baselines but describes only three. - SUS results (Safety / Usability / Smoothness): - HuaTuo: 2.88 / 2.12 / 2.47 - LLaMA: 2.93 / 1.21 / 1.58 - Alpaca: 2.64 / 2.05 / 2.30 - ChatGLM: 2.59 / 1.93 / 2.41 - Code and model are released at github.com/SCIR-HI/Huatuo-Llama-Med-Chinese. The authors stress the model is for research only and is not a source of medical advice. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) ## Cited by Cited by 76 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 71 not yet. - [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/) - [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-rolellm-benchmarking-eliciting-enhancing-role/) - [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/) - [HuatuoGPT, Towards Taming Language Model to Be a Doctor](https://md.research.elamur.ai/free/en/papers/zhang-2023-huatuogpt-taming-language-model-be/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{wang2023huatuo, title = {{HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge}}, author = {Wang, Haochun and Liu, Chi and Xi, Nuwa and Qiang, Zewen and Zhao, Sendong and Qin, Bing and Liu, Ting}, year = {2023}, doi = {10.48550/arxiv.2304.06975}, eprint = {2304.06975}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2304.06975} } ``` - [← previous How Language Model Hallucinations Can Snowball](https://md.research.elamur.ai/free/en/papers/zhang-2023-language-model-hallucinations-snowball/) - [next → HuatuoGPT, Towards Taming Language Model to Be a Doctor](https://md.research.elamur.ai/free/en/papers/zhang-2023-huatuogpt-taming-language-model-be/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) ## References 13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Zhihong Chen, Junying Chen, Hongbo Zhang, Feng Jiang, Guiming Chen, Fei Yu, Tiannan Wang, Juhao Liang, Chen Zhang, Zhiyi Zhang, Jianquan Li, Xiang Wan, Haizhou Li, and Benyou Wang. 2023. Llm zoo: democratizing chatgpt. [https://github.com/FreedomIntelligence/LLMZoo](https://github.com/FreedomIntelligence/LLMZoo). [↗](https://github.com/FreedomIntelligence/LLMZoo) 2. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320– 33… [find](https://scholar.google.com/scholar?q=Zhengxiao+Du%2C+Yujie+Qian%2C+Xiao+Liu%2C+Ming+Ding%2C+Jiezhong+Qiu%2C+Zhilin+Yang%2C+and+Jie+Tang%2E+2022%2E+Glm%3A+General+language+model+pretraining+with+autoregressive+blank+infilling%2E+In+Proceedings+of+the+60th+An) [our card](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/) 3. Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. 2023. How does chatgpt perform on the united states medical licensing examination? the implications of large language models for medical education and knowledge assessment. JMIR Med… [find](https://scholar.google.com/scholar?q=Aidan+Gilson%2C+Conrad+W+Safranek%2C+Thomas+Huang%2C+Vimig+Socrates%2C+Ling+Chi%2C+Richard+Andrew+Taylor%2C+David+Chartash%2C+et+al%2E+2023%2E+How+does+chatgpt+perform+on+the+united+states+medical+licensing+examination) 4. Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, and You Zhang. 2023. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. [find](https://scholar.google.com/scholar?q=Yunxiang+Li%2C+Zihan+Li%2C+Kai+Zhang%2C+Ruilong+Dan%2C+and+You+Zhang%2E+2023%2E+Chatdoctor%3A+A+medical+chat+model+fine%2Dtuned+on+llama+model+using+medical+domain+knowledge%2E) 5. BYAMBASUREN Odmaa, YANG Yunfei, SUI Zhifang, DAI Damai, CHANG Baobao, LI Sujian, and ZAN Hongying. 2019. Preliminary study on the construction of chinese medical knowledge graph. Journal of Chinese Information Processing, 33(10):1–7. [find](https://scholar.google.com/scholar?q=BYAMBASUREN+Odmaa%2C+YANG+Yunfei%2C+SUI+Zhifang%2C+DAI+Damai%2C+CHANG+Baobao%2C+LI+Sujian%2C+and+ZAN+Hongying%2E+2019%2E+Preliminary+study+on+the+construction+of+chinese+medical+knowledge+graph%2E+Journal+of+Chinese+In) 6. OpenAI. 2022. Chatgpt. [https://chat.openai.com](https://chat.openai.com). [↗](https://chat.openai.com) 7. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+2022%2E+Training+language+models+to+follow+instructi) 8. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/](https://github.com/tatsu-lab/) stanford_alpaca. [↗](https://github.com/tatsu-lab/) 9. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+2023%2E+Llama%3A+Open+and+efficient+f) 10. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+pre) 11. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+2022%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+In+International+Conference) 12. Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao, Yuxiao Liu, Qian Wang, and Dinggang Shen. Doctorglm: Fine-tuning your chinese doctor is not a herculean task. [find](https://scholar.google.com/scholar?q=Honglin+Xiong%2C+Sheng+Wang%2C+Yitao+Zhu%2C+Zihao+Zhao%2C+Yuxiao+Liu%2C+Qian+Wang%2C+and+Dinggang+Shen%2E+Doctorglm%3A+Fine%2Dtuning+your+chinese+doctor+is+not+a+herculean+task%2E) 13. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. GLM-130b: An open bilingual pre-trained model. In The Eleventh… [find](https://scholar.google.com/scholar?q=Aohan+Zeng%2C+Xiao+Liu%2C+Zhengxiao+Du%2C+Zihan+Wang%2C+Hanyu+Lai%2C+Ming+Ding%2C+Zhuoyi+Yang%2C+Yifan+Xu%2C+Wendi+Zheng%2C+Xiao+Xia%2C+Weng+Lam+Tam%2C+Zixuan+Ma%2C+Yufei+Xue%2C+Jidong+Zhai%2C+Wenguang+Chen%2C+Zhiyuan+Liu%2C+Peng+Zh) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/)