---
title: "Efficient and Effective Text Encoding for Chinese — Cui 2023"
description: "Open LLMs like LLaMA were trained mostly on English and European-language data, and their tokenizer covers fewer than a thousand Chinese characters."
authors: ["Yiming Cui", "Ziqing Yang", "Xin Yao"]
year: 2023
venue: "arXiv:2304.08177"
source: "https://arxiv.org/abs/2304.08177"
doi: "10.48550/arxiv.2304.08177"
arxiv: "2304.08177"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cui-2023-efficient-effective-text-encoding-chinese/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cui-2023-efficient-effective-text-encoding-chinese/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca

# Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca

**Authors:** [Yiming Cui](https://md.research.elamur.ai/free/en/authors/yiming-cui/), [Ziqing Yang](https://md.research.elamur.ai/free/en/authors/ziqing-yang/), Xin Yao · **Year:** 2023 · **Venue:** arXiv:2304.08177 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.08177) · [PDF](https://arxiv.org/pdf/2304.08177) · [DOI 10.48550/arxiv.2304.08177](https://doi.org/10.48550/arxiv.2304.08177) · [arXiv 2304.08177](https://arxiv.org/abs/2304.08177) · [OpenAlex](https://openalex.org/W4366327277)

**Citations:** 70 citations (openalex-snapshot · 2026-10-14) · [215](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Open LLMs like LLaMA were trained mostly on English and European-language data, and their tokenizer covers fewer than a thousand Chinese characters. Other characters fall back to UTF-8 bytes, so one character can cost 3–4 tokens. This inflates sequence length and makes it hard to learn Chinese representations. The authors trained a 20,000-token Chinese SentencePiece model, merged it with LLaMA's vocabulary, and enlarged the embedding and output layers while leaving the original token vectors untouched. The adapted models then received continued pre-training on Chinese text and instruction tuning on Chinese instruction data, using LoRA to keep training cheap. This produced two families: Chinese-LLaMA (base models) and Chinese-Alpaca (instruction-following). Both were later extended to Llama-2 and long-context variants. The new tokenizer roughly halves the token count for Chinese, which doubles effective context and generation speed. On C-Eval, the models are competitive with systems several times larger. All weights and scripts are public, giving a template for adapting LLaMA to other languages.

## Key points
- **Vocabulary extension:** The merged Chinese LLaMA tokenizer grows from 32,000 to 49,953 entries. New embedding rows are appended so the original token vectors stay unchanged.
- **Data scale:** Continued pre-training used 20G of Chinese text for Chinese-LLaMA and 120G for the Plus and Llama-2 versions. Supervised fine-tuning used 2M–4.3M instructions for Alpaca/Alpaca-Plus and 5M for Chinese-Alpaca-2.
- **Model sizes:** The models are built on LLaMA (7B, 13B, 33B) and Llama-2 (7B, 13B). Long-context variants reach 16K tokens via Positional Interpolation and 64K via YaRN, and were evaluated on LongBench.
- **Benchmark results:** Chinese-Alpaca-2-13B reaches 44.0 on the C-Eval test set (5-shot) and 45.5 on CMMLU (few-shot).
- **Released resources:** Everything is available on GitHub in the ymcui/Chinese-LLaMA-Alpaca and Chinese-LLaMA-Alpaca-2 repositories.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

215 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 199 not yet.

- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://md.research.elamur.ai/free/en/papers/alves-2024-tower-open-multilingual-large-language/)
- [Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities](https://md.research.elamur.ai/free/en/papers/arxivrecent-2024-shmidman-adapting-llms-hebrew-unveiling-dictalm/)
- [ALLaM: Large Language Models for Arabic and English](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/)
- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://md.research.elamur.ai/free/en/papers/fei-2023-lawbench-benchmarking-legal-knowledge-large/)
- [Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities](https://md.research.elamur.ai/free/en/papers/fujii-2024-continual-pre-training-cross-lingual/)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/)
- [Lawyer LLaMA Technical Report](https://md.research.elamur.ai/free/en/papers/huang-2023-lawyer-llama-technical-report/)
- [Building Patient Journeys in Hebrew: A Language Model for Clinical Timeline Extraction](https://md.research.elamur.ai/free/en/papers/langstream-2025-hashiloni-building-patient-journeys-hebrew-language/)
- [HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model](https://md.research.elamur.ai/free/en/papers/langstream-2026-kayzer-hebatron-hebrew-specialized-open-weight/)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/)
- [Aligning Large Language Models with Human: A Survey](https://md.research.elamur.ai/free/en/papers/wang-2023-aligning-large-language-models-human/)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/)
- [Baichuan 2: Open Large-scale Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{cui2023efficient,
  title = {{Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca}},
  author = {Cui, Yiming and Yang, Ziqing and Yao, Xin},
  year = {2023},
  doi = {10.48550/arxiv.2304.08177},
  eprint = {2304.08177},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.08177}
}
```

- [← previous Effective Long-Context Scaling of Foundation Models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
- [next → Elaborative Simplification as Implicit Questions Under Discussion](https://md.research.elamur.ai/free/en/papers/wu-2023-elaborative-simplification-as-implicit-questions/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

35 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023. [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Xin+Lv%2C+Jiajie+Zhang%2C+Hongchang+Lyu%2C+Jiankai+Tang%2C+Zhidian+Huang%2C+Zhengxiao+Du%2C+Xiao+Liu%2C+Aohan+Zeng%2C+Lei+Hou%2C+Yuxiao+Dong%2C+Jie+Tang%2C+and+Juanzi+Li%2E+Longbench%3A+A+bilingual%2C+multitask+benchm) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/)
2. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2C+2023%2E)
3. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre-trained models for Chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 657–668, Online, November 2020. Association for C… [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Wanxiang+Che%2C+Ting+Liu%2C+Bing+Qin%2C+Shijin+Wang%2C+and+Guoping+Hu%2E+Revisiting+pre%2Dtrained+models+for+Chinese+natural+language+processing%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Met) [our card](https://md.research.elamur.ai/free/en/papers/cui-2020-revisiting-pre-trained-models-chinese/)
4. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. Pre-training with whole word masking for chinese bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3504–3514, 2021. doi: 10.1109/TASLP.2021.3124365. [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Wanxiang+Che%2C+Ting+Liu%2C+Bing+Qin%2C+and+Ziqing+Yang%2E+Pre%2Dtraining+with+whole+word+masking+for+chinese+bert%2E+IEEE%2FACM+Transactions+on+Audio%2C+Speech%2C+and+Language+Processing%2C+29%3A3504%E2%80%933514%2C+202) [our card](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/)
5. Yiming Cui, Wanxiang Che, Shijin Wang, and Ting Liu. Lert: A linguistically-motivated pre-trained language model. arXiv preprint arXiv:2211.05344, 2022. — [arXiv:2211.05344](https://arxiv.org/abs/2211.05344) [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Wanxiang+Che%2C+Shijin+Wang%2C+and+Ting+Liu%2E+Lert%3A+A+linguistically%2Dmotivated+pre%2Dtrained+language+model%2E+arXiv+preprint+arXiv%3A2211%2E05344%2C+2022%2E+%E2%80%94+%5BarXiv%3A2211%2E05344%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2211)
6. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+Qlora%3A+Efficient+finetuning+of+quantized+llms%2E+arXiv+preprint+arXiv%3A2305%2E14314%2C+2023%2E)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+North+A)
8. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323, 2022. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+GPTQ%3A+Accurate+post%2Dtraining+compression+for+generative+pretrained+transformers%2E+arXiv+preprint+arXiv%3A2210%2E17323%2C+2022%2E)
9. Georgi Gerganov. llama.cpp. [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp), 2023. [↗](https://github.com/ggerganov/llama.cpp)
10. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. arXiv e-prints, art. arXiv:2106.09685, June 2021. doi: 10.48550/arXiv.2106.09685. [find](https://scholar.google.com/scholar?q=Edward+J%2E+Hu%2C+Yelong+Shen%2C+Phillip+Wallis%2C+Zeyuan+Allen%2DZhu%2C+Yuanzhi+Li%2C+Shean+Wang%2C+Lu+Wang%2C+and+Weizhu+Chen%2E+LoRA%3A+Low%2DRank+Adaptation+of+Large+Language+Models%2E+arXiv+e%2Dprints%2C+art%2E+arXiv%3A2106%2E09685)
11. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023. [find](https://scholar.google.com/scholar?q=Yuzhen+Huang%2C+Yuzhuo+Bai%2C+Zhihao+Zhu%2C+Junlei+Zhang%2C+Jinghan+Zhang%2C+Tangjun+Su%2C+Junteng+Liu%2C+Chuancheng+Lv%2C+Yikai+Zhang%2C+Jiayi+Lei%2C+Yao+Fu%2C+Maosong+Sun%2C+and+Junxian+He%2E+C%2Deval%3A+A+multi%2Dlevel+multi%2Ddisc) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/)
12. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Op… [find](https://scholar.google.com/scholar?q=Andreas+Ko%CC%88pf%2C+Yannic+Kilcher%2C+Dimitri+von+Ru%CC%88tte%2C+Sotiris+Anagnostidis%2C+Zhi%2DRui+Tam%2C+Keith+Stevens%2C+Abdullah+Barhoum%2C+Nguyen+Minh+Duc%2C+Oliver+Stanley%2C+Richa%CC%81rd+Nagyfi%2C+Shahul+ES%2C+Sameer+Suri%2C+David+G) [our card](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
13. Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66–71, Brussels, Belgium, November 2018.… [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+SentencePiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+neural+text+processing%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods) [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
14. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785–794, Copenhagen, Denmark, September 2017. Association for Computatio… [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+RACE%3A+Large%2Dscale+ReAding+comprehension+dataset+from+examinations%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natur) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
15. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2023. [find](https://scholar.google.com/scholar?q=Haonan+Li%2C+Yixuan+Zhang%2C+Fajri+Koto%2C+Yifei+Yang%2C+Hai+Zhao%2C+Yeyun+Gong%2C+Nan+Duan%2C+and+Timothy+Baldwin%2E+Cmmlu%3A+Measuring+massive+multitask+language+understanding+in+chinese%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/)
16. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL [https://openreview.net/forum?id=Bkg6RiCqY7](https://openreview.net/forum?id=Bkg6RiCqY7). [↗](https://openreview.net/forum?id=Bkg6RiCqY7)
17. OpenAI. Introducing chatgpt. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt), 2022. [↗](https://openai.com/blog/chatgpt)
18. OpenAI. GPT-4 Technical Report. arXiv e-prints, art. arXiv:2303.08774, March 2023. doi: 10.48550/arXiv.2303.08774. [find](https://scholar.google.com/scholar?q=OpenAI%2E+GPT%2D4+Technical+Report%2E+arXiv+e%2Dprints%2C+art%2E+arXiv%3A2303%2E08774%2C+March+2023%2E+doi%3A+10%2E48550%2FarXiv%2E2303%2E08774%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
19. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeff+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+L%2E+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+John+Schulman%2C+Jacob+Hilton%2C+Fraser+Kelton%2C+Luke+Miller%2C+)
20. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023%2E)
21. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
22. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505–3506, 2020. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26th+ACM+)
23. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagne%CC%81%2C+Alexandra+Sasha+Luccioni%2C+Franc%CC%A7ois+Yvon%2C+Matthias+Galle%CC%81%2C+et+al%2E+Bloom%3A+A+176b%2Dparameter+op)
24. Noam Shazeer. Glu variants improve transformer, 2020. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+Glu+variants+improve+transformer%2C+2020%2E)
25. Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021. [find](https://scholar.google.com/scholar?q=Jianlin+Su%2C+Yu+Lu%2C+Shengfeng+Pan%2C+Bo+Wen%2C+and+Yunfeng+Liu%2E+Roformer%3A+Enhanced+transformer+with+rotary+position+embedding%2C+2021%2E)
26. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023a. [↗](https://github.com/tatsu-lab/stanford_alpaca)
27. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023b. [↗](https://github.com/tatsu-lab/stanford_alpaca)
28. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv prepri… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timothe%CC%81e+Lacroix%2C+Baptiste+Rozie%CC%80re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+Aurelien+Rodriguez%2C+Armand+Joulin%2C+Edo)
29. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Sy… [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+I%2E+Guyon%2C+U%2E+Von+Luxburg%2C+S%2E+Bengio%2C+H%2E+Wall)
31. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Model with Self Generated Instructions. arXiv e-prints, art. arXiv:2212.10560, December 2022. doi: 10.48550/arXiv.2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2DInstruct%3A+Aligning+Language+Model+with+Self+Generated+Instructions%2E+arXiv+e%2Dprints)
32. Bright Xu. Nlp chinese corpus: Large scale chinese corpus for nlp, September 2019. URL https://doi.org/10.5281/zenodo.3402023. — [doi:10.5281/zenodo.3402023](https://doi.org/10.5281/zenodo.3402023) [↗](https://doi.org/10.5281/zenodo.3402023)
33. Ziqing Yang, Zihang Xu, Yiming Cui, Baoxin Wang, Min Lin, Dayong Wu, and Zhigang Chen. CINO: A Chinese minority pre-trained language model. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 3937–3949, Gyeongju, Republic of Korea, October 2022. International Commit… [find](https://scholar.google.com/scholar?q=Ziqing+Yang%2C+Zihang+Xu%2C+Yiming+Cui%2C+Baoxin+Wang%2C+Min+Lin%2C+Dayong+Wu%2C+and+Zhigang+Chen%2E+CINO%3A+A+Chinese+minority+pre%2Dtrained+language+model%2E+In+Proceedings+of+the+29th+International+Conference+on+Compu)
34. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh Inter… [find](https://scholar.google.com/scholar?q=Aohan+Zeng%2C+Xiao+Liu%2C+Zhengxiao+Du%2C+Zihan+Wang%2C+Hanyu+Lai%2C+Ming+Ding%2C+Zhuoyi+Yang%2C+Yifan+Xu%2C+Wendi+Zheng%2C+Xiao+Xia%2C+Weng+Lam+Tam%2C+Zixuan+Ma%2C+Yufei+Xue%2C+Jidong+Zhai%2C+Wenguang+Chen%2C+Zhiyuan+Liu%2C+Peng+Zh) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/)
35. Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, Vancouver, Canada, 2019. URL [https://openreview.net/references/pdf?id=S1qBAf6rr](https://openreview.net/references/pdf?id=S1qBAf6rr). [↗](https://openreview.net/references/pdf?id=S1qBAf6rr)