---
title: "XuanYuan 2.0 — paper by Zhang et al. (2023)"
description: "The only evaluation is a manual comparison against other open Chinese chat models, which the… 22 references, cited by 46 papers here."
authors: ["Xuanyu Zhang", "Qing Yang", "Dongliang Xu"]
year: 2023
venue: "arXiv:2305.12002"
source: "https://arxiv.org/abs/2305.12002"
doi: "10.48550/arxiv.2305.12002"
arxiv: "2305.12002"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2023-xuanyuan-2-0-large-chinese/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2023-xuanyuan-2-0-large-chinese/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters

# XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters

**Authors:** Xuanyu Zhang, Qing Yang and Dongliang Xu · **Year:** 2023 · **Venue:** [arXiv:2305.12002](https://md.research.elamur.ai/free/en/venues/acm-international-conference-on-information-and-knowledge-management-cikm/) ([CIKM 2023](https://md.research.elamur.ai/free/en/venues/acm-international-conference-on-information-and-knowledge-management-cikm/2023/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.12002) · [PDF](https://arxiv.org/pdf/2305.12002) · [DOI 10.48550/arxiv.2305.12002](https://doi.org/10.48550/arxiv.2305.12002) · [arXiv 2305.12002](https://arxiv.org/abs/2305.12002) · [OpenAlex](https://openalex.org/W4387847108)

**Citations:** 79 citations (OpenAlex, 2026-10-14) · [46](https://md.research.elamur.ai/free/en/papers/zhang-2023-xuanyuan-2-0-large-chinese/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The authors address the absence of open-source, very large chat models built for Chinese, and specifically for Chinese finance. Earlier Chinese financial models such as FinBERT, Mengzi and FinT5 stay under one billion parameters. XuanYuan 2.0, from Du Xiaoman Financial, uses the decoder-only BLOOM-176B architecture with ALiBi positions and embedding LayerNorm. Its main methodological contribution is "hybrid-tuning." Instead of running domain pre-training and then instruction tuning as separate steps, the authors shuffle four data types into a single training stream and train in one stage. The four types are general pre-training text, financial pre-training text, general instructions and financial instructions. The goal is to limit catastrophic forgetting of general skills while still adding financial expertise. General instructions were produced with Self-Instruct, and financial ones with Self-QA from structured and unstructured financial sources. The only evaluation is a manual comparison against other open Chinese chat models, which the authors describe as favourable but support with no numbers. The paper matters mainly as an example of mixing domain and general data at very large scale.

## Key points
- The main model has about 176B parameters (70 layers, hidden size 14,336, 112 heads). A smaller XuanYuan2-7B variant has about 7B (30 layers, hidden size 4096). Both use a 250,680-token vocabulary and 2048-token context.
- The hybrid-tuning data mix is:
  - general pre-training: 20%
  - financial pre-training: 13%
  - general instructions: 46%
  - financial instructions: 21%
- Financial instruction data came from Self-QA applied to news, market and analyst reports, social media and company information. General instructions came from Self-Instruct seeded with human-written prompts.
- Training ran on NVIDIA A100 80GB GPUs with DeepSpeed, pipeline parallelism and ZeRO stage 1. Pre-training covered about 366B tokens for the 176B model, followed by about 13B tokens of multitask fine-tuning.
- The predecessor XuanYuan 1.0 ranked first on the CLUE classification leaderboard in 2021. For 2.0, the paper reports only human evaluation, without quantitative scores.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Xuanyu Zhang, Qing Yang and Dongliang Xu. 2023. XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters. arXiv:2305.12002.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2023xuanyuan,
  title = {{XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters}},
  author = {Xuanyu Zhang, Qing Yang and Dongliang Xu},
  year = {2023},
  doi = {10.48550/arxiv.2305.12002},
  eprint = {2305.12002},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.12002}
}
```

- [← previous OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text](https://md.research.elamur.ai/free/en/papers/paster-2023-openwebmath-open-dataset-high-quality/)
- [next → LM vs LM: Detecting Factual Errors via Cross Examination](https://md.research.elamur.ai/free/en/papers/cohen-2023-lm-vs-lm-detecting-factual/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063. [find](https://scholar.google.com/scholar?q=Dogu+Araci%2E+2019%2E+Finbert%3A+Financial+sentiment+analysis+with+pre%2Dtrained+language+models%2E+arXiv+preprint+arXiv%3A1908%2E10063%2E)
2. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+2022%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+arXiv+preprint+arXiv%3A2208%2E07339%2E)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
4. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. [Domain-specific language model pretraining for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/). ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23.
5. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. [Biobert: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/). Bioinformatics, 36(4):1234–1240.
6. Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2021. Finbert: A pre-trained financial language representation model for financial text mining. In Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pages 4513–4519. [find](https://scholar.google.com/scholar?q=Zhuang+Liu%2C+Degen+Huang%2C+Kaiyu+Huang%2C+Zhuang+Li%2C+and+Jun+Zhao%2E+2021%2E+Finbert%3A+A+pre%2Dtrained+financial+language+representation+model+for+financial+text+mining%2E+In+Proceedings+of+the+twenty%2Dninth+intern)
7. Dakuan Lu, Jiaqing Liang, Yipei Xu, Qianyu He, Yipeng Geng, Mengkun Han, Yingsi Xin, Hengkui Wu, and Yanghua Xiao. 2023. Bbt-fin: Comprehensive construction of chinese financial domain pre-trained language model, corpus and benchmark. arXiv preprint arXiv:2302.09432. — [arXiv:2302.09432](https://arxiv.org/abs/2302.09432) [find](https://scholar.google.com/scholar?q=Dakuan+Lu%2C+Jiaqing+Liang%2C+Yipei+Xu%2C+Qianyu+He%2C+Yipeng+Geng%2C+Mengkun+Han%2C+Yingsi+Xin%2C+Hengkui+Wu%2C+and+Yanghua+Xiao%2E+2023%2E+Bbt%2Dfin%3A+Comprehensive+construction+of+chinese+financial+domain+pre%2Dtrained+lan)
8. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
9. Ofir Press, Noah A Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A+Smith%2C+and+Mike+Lewis%2E+2021%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2E)
10. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
11. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551.
12. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Com)
13. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505– 3506. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26t)
14. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
15. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
17. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+pre)
18. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. [Bloomberggpt: A large language model for finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/). arXiv preprint arXiv:2303.17564.
19. Yi Yang, Mark Christopher Siy Uy, and Allen Huang. 2020. Finbert: A pretrained language model for financial communications. arXiv preprint arXiv:2006.08097. [find](https://scholar.google.com/scholar?q=Yi+Yang%2C+Mark+Christopher+Siy+Uy%2C+and+Allen+Huang%2E+2020%2E+Finbert%3A+A+pretrained+language+model+for+financial+communications%2E+arXiv+preprint+arXiv%3A2006%2E08097%2E)
20. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
21. Xuanyu Zhang and Qing Yang. 2023. Self-qa: Unsupervised knowledge guided language model alignment. arXiv preprint. [find](https://scholar.google.com/scholar?q=Xuanyu+Zhang+and+Qing+Yang%2E+2023%2E+Self%2Dqa%3A+Unsupervised+knowledge+guided+language+model+alignment%2E+arXiv+preprint%2E)
22. Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong Wang, and Ming Zhou. 2021. Mengzi: Towards lightweight yet ingenious pre-trained models for chinese. arXiv preprint arXiv:2110.06696. — [arXiv:2110.06696](https://arxiv.org/abs/2110.06696) [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Hanqing+Zhang%2C+Keming+Chen%2C+Yuhang+Guo%2C+Jingyun+Hua%2C+Yulong+Wang%2C+and+Ming+Zhou%2E+2021%2E+Mengzi%3A+Towards+lightweight+yet+ingenious+pre%2Dtrained+models+for+chinese%2E+arXiv+preprint+arXiv%3A2)

*Updated 24 October 2026*