---
title: "VinaLLaMA: LLaMA-based Vietnamese Foundation Model (2023)"
description: "Open LLMs such as LLaMA-2 handle Vietnamese poorly because their training data contains few… 31 references, cited by 11 papers here."
authors: ["Quan Nguyen", "Huy Pham", "Dung Dao"]
year: 2023
venue: "arXiv:2312.11011"
source: "https://arxiv.org/abs/2312.11011"
doi: "10.48550/arxiv.2312.11011"
arxiv: "2312.11011"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nguyen-2023-vinallama-llama-based-vietnamese-foundation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nguyen-2023-vinallama-llama-based-vietnamese-foundation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/) / VinaLLaMA: LLaMA-based Vietnamese Foundation Model

# VinaLLaMA: LLaMA-based Vietnamese Foundation Model

**Authors:** [Quan Nguyen](https://md.research.elamur.ai/free/en/authors/quan-nguyen/), Huy Pham and [Dung Dao](https://md.research.elamur.ai/free/en/authors/dung-dao/) · **Year:** 2023 · **Venue:** arXiv:2312.11011 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2312.11011) · [PDF](https://arxiv.org/pdf/2312.11011) · [DOI 10.48550/arxiv.2312.11011](https://doi.org/10.48550/arxiv.2312.11011) · [arXiv 2312.11011](https://arxiv.org/abs/2312.11011) · [OpenAlex](https://openalex.org/W4389984105)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/nguyen-2023-vinallama-llama-based-vietnamese-foundation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Open LLMs such as LLaMA-2 handle Vietnamese poorly because their training data contains few Vietnamese tokens and their tokenizers are not designed for the language. The authors address this by continuing to pretrain LLaMA-2 with BKAI's Vietnamese-oriented tokenizer. The training mix combines public text (books, VnExpress and BaoMoi news, the Vietnamese portion of CulturaX, and English data) with a very large synthetic corpus. That synthetic corpus was built in two stages. First, GPT-4 rewrote sampled passages using custom prompt templates. Then a smaller model, Vietcuna-3B-v2, was trained on those outputs so generation could be scaled more cheaply. The base model was fully fine-tuned on one million bilingual instruction and dialogue samples, and a 2.7B variant was derived through structured pruning. The authors report that VinaLLaMA-7B-chat leads open models on VLSP, VMLU and a Vietnamese Vicuna Benchmark while remaining competitive in English. Because the weights are openly available on HuggingFace, the work gives Vietnamese NLP a reusable foundation model and shows how synthetic data can offset scarce native resources.

## Key points
- **Pretraining data:** About 330B tokens of public data, including 250,000 Vietnamese books, VnExpress/BaoMoi news from 2010 to September 2023 (filtered for harmful content), a CulturaX Vietnamese subset, and 100B English tokens. On top of this come over 500B synthetic Vietnamese tokens. GPT-4 produced about 100K seed samples from 80+ templates, and Vietcuna-3B-v2 generated the bulk. The abstract cites roughly 800B additional tokens in total.
- **Fine-tuning data:** The SFT set has 1M samples. Half are Vietnamese synthetic samples created with Nous Research. The other half are English samples from OpenHermes-2.5 and Capybara. The set covers reasoning, role-play, poetry, coding, function calling and agent prompts, and the model was trained with full fine-tuning for 4 epochs.
- **Compute:** Pretraining ran for one week on 8 nodes with 8× Intel Habana Gaudi2 each. SFT took one day on a Google Cloud TPU v5e node. The 2.7B model was trained in five days on 8× A100 80GB.
- **Evaluation:** Benchmarks were VLSP-LLM 2023 (Vietnamese ARC, HellaSwag, MMLU, TruthfulQA), VMLU, and the Vicuna Benchmark translated by VinAI Research. On the Vicuna Benchmark, VinaLLaMA-7B-chat scored 4.0 in all five categories reported in Table 6, matching ChatGPT.
- **English results:** On the HuggingFace OpenLLM Leaderboard, VinaLLaMA-7B-chat averages 0.5227, compared with 0.5074 for LLaMA-2-7B-chat. Its largest gain is on GSM8K, where it scores 0.3002 against 0.0735.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Quan Nguyen, Huy Pham and Dung Dao. 2023. VinaLLaMA: LLaMA-based Vietnamese Foundation Model. arXiv:2312.11011.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nguyen2023vinallama,
  title = {{VinaLLaMA: LLaMA-based Vietnamese Foundation Model}},
  author = {Quan Nguyen, Huy Pham and Dung Dao},
  year = {2023},
  doi = {10.48550/arxiv.2312.11011},
  eprint = {2312.11011},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.11011}
}
```

- [← previous Susceptibility to Influence of Large Language Models](https://md.research.elamur.ai/free/en/papers/griffin-2023-susceptibility-influence-large-language-models/)
- [next → Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/guo-2023-bridging-gap-between-synthetic-authentic/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/)

## References

31 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, et al. [The falcon series of open language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/). arXiv preprint arXiv:2311.16867, 2023.
2. Zalo AI, Japan Advanced Institute of Science, and Technology. Vmlu: A vietnamese multitask language understanding benchmark suite for large language models. 2023. [find](https://scholar.google.com/scholar?q=Zalo+AI%2C+Japan+Advanced+Institute+of+Science%2C+and+Technology%2E+Vmlu%3A+A+vietnamese+multitask+language+understanding+benchmark+suite+for+large+language+models%2E+2023%2E)
3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609, 2023.
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
5. Le Anh Cuong, Nguyen Trong Hieu, Nguyen Viet Cuong, Nguyen Ngoc Quoc, Le-Minh Nguyen, and Cam-Tu Nguyen. Vlsp 2023 challenge on vietnamese large language models. 2023. [find](https://scholar.google.com/scholar?q=Le+Anh+Cuong%2C+Nguyen+Trong+Hieu%2C+Nguyen+Viet+Cuong%2C+Nguyen+Ngoc+Quoc%2C+Le%2DMinh+Nguyen%2C+and+Cam%2DTu+Nguyen%2E+Vlsp+2023+challenge+on+vietnamese+large+language+models%2E+2023%2E)
6. Llinet Benavides Cesar, Miguel-Ángel Manso-Callejo, and Calimanut-Ionut Cira. Bert (bidirectional encoder representations from transformers) for missing data imputation in solar irradiance time series. Engineering Proceedings, 39(1):26, 2023. [find](https://scholar.google.com/scholar?q=Llinet+Benavides+Cesar%2C+Miguel%2DA%CC%81ngel+Manso%2DCallejo%2C+and+Calimanut%2DIonut+Cira%2E+Bert+%28bidirectional+encoder+representations+from+transformers%29+for+missing+data+imputation+in+solar+irradiance+time+serie)
7. Luigi Daniele and Suphavadeeprasit. Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training. arXiv preprint arXiv:(comming soon), 2023. [find](https://scholar.google.com/scholar?q=Luigi+Daniele+and+Suphavadeeprasit%2E+Amplify%2Dinstruct%3A+Synthetically+generated+diverse+multi%2Dturn+conversations+for+effecient+llm+training%2E+arXiv+preprint+arXiv%3A%28comming+soon%29%2C+2023%2E)
8. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanz… [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+Ce%CC%81sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+Adil+Salim%2C+Shital+Shah%2C+Hark)
9. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
10. BKAI-HUST Foundation Models Lab. Llama-2-bk, 2023. [find](https://scholar.google.com/scholar?q=BKAI%2DHUST+Foundation+Models+Lab%2E+Llama%2D2%2Dbk%2C+2023%2E)
11. Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. [Platypus: Quick, cheap, and powerful refinement of llms](https://md.research.elamur.ai/free/en/papers/lee-2023-platypus-quick-cheap-powerful-refinement/). arXiv preprint arXiv:2308.07317, 2023.
12. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the Assoc…
13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). 2019.
14. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. [Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages](https://md.research.elamur.ai/free/en/papers/nguyen-2023-culturax-cleaned-enormous-multilingual-dataset/), 2023.
15. Dat Quoc Nguyen, Linh The Nguyen, Chi Tran, Dung Ngoc Nguyen, Nhung Nguyen, Thien Huu Nguyen, Dinh Phung, and Hung Bui. Phogpt: Generative pre-training for vietnamese. arXiv preprint arXiv:2311.02945, 2023. — [arXiv:2311.02945](https://arxiv.org/abs/2311.02945) [find](https://scholar.google.com/scholar?q=Dat+Quoc+Nguyen%2C+Linh+The+Nguyen%2C+Chi+Tran%2C+Dung+Ngoc+Nguyen%2C+Nhung+Nguyen%2C+Thien+Huu+Nguyen%2C+Dinh+Phung%2C+and+Hung+Bui%2E+Phogpt%3A+Generative+pre%2Dtraining+for+vietnamese%2E+arXiv+preprint+arXiv%3A2311%2E02945%2C)
16. Quan Nguyen, Huy Pham, and Dung Dao. Openorca-viet. 2023. [find](https://scholar.google.com/scholar?q=Quan+Nguyen%2C+Huy+Pham%2C+and+Dung+Dao%2E+Openorca%2Dviet%2E+2023%2E)
17. Quan Nguyen, Huy Pham, and Dung Dao. Vietcuna. 2023. [find](https://scholar.google.com/scholar?q=Quan+Nguyen%2C+Huy+Pham%2C+and+Dung+Dao%2E+Vietcuna%2E+2023%2E)
18. Duc Q. Nguyen, Sang T. Truong, Toan D. V. Nguyen, Dong D. Le, Nhi N. Truong, and Tho Quan. Ura-llama: Universal adapted large language model for vietnamese. 2023. [find](https://scholar.google.com/scholar?q=Duc+Q%2E+Nguyen%2C+Sang+T%2E+Truong%2C+Toan+D%2E+V%2E+Nguyen%2C+Dong+D%2E+Le%2C+Nhi+N%2E+Truong%2C+and+Tho+Quan%2E+Ura%2Dllama%3A+Universal+adapted+large+language+model+for+vietnamese%2E+2023%2E)
19. Minh Thuan Nguyen, Khanh Tung Tran, Nhu Van Nguyen, and Xuan-Son Vu. ViGPTQA - state-of-the-art LLMs for Vietnamese question answering: System overview, core models training, and evaluations. In Mingxuan Wang and Imed Zitouni, editors, Proceedings of the 2023 Conference on Empirical Methods in Natur… [find](https://scholar.google.com/scholar?q=Minh+Thuan+Nguyen%2C+Khanh+Tung+Tran%2C+Nhu+Van+Nguyen%2C+and+Xuan%2DSon+Vu%2E+ViGPTQA+%2D+state%2Dof%2Dthe%2Dart+LLMs+for+Vietnamese+question+answering%3A+System+overview%2C+core+models+training%2C+and+evaluations%2E+In+Mingx)
20. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
21. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
22. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485– 5551, 2020.
23. teknium. teknium/openhermes-2.5-mistral-7b. 2023. + [find](https://scholar.google.com/scholar?q=teknium%2E+teknium%2Fopenhermes%2D2%2E5%2Dmistral%2D7b%2E+2023%2E+%2B)
24. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
25. VietAI. Gpt-j 6b on vietnamese news, 2021. + [find](https://scholar.google.com/scholar?q=VietAI%2E+Gpt%2Dj+6b+on+vietnamese+news%2C+2021%2E+%2B)
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
27. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In International Conference on Learning Representations, 2021.
28. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Openorca. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Openorca%2E+arXiv+preprint+arXiv%3A2212%2E10560%2C+2022%2E)
29. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=BigScience+Workshop%2C+Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagne%CC%81%2C+Alexandra+Sasha+Luccioni%2C+Franc%CC%A7ois+Yvon%2C+et+al%2E+Bloom%3A+A+176b%2Dparamete)
30. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning, 2023. [find](https://scholar.google.com/scholar?q=Mengzhou+Xia%2C+Tianyu+Gao%2C+Zhiyuan+Zeng%2C+and+Danqi+Chen%2E+Sheared+llama%3A+Accelerating+language+model+pre%2Dtraining+via+structured+pruning%2C+2023%2E)
31. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/), 2023.

*Updated 2 November 2026*