---
title: "Xiezhi — paper by Gu et al. (2023) — summary and ideas"
description: "Benchmarks such as MMLU, BIG-bench and HELM quickly become too easy for modern LLMs and… 47 references, cited by 25 papers here."
authors: ["Zhouhong Gu", "Xiaoxuan Zhu", "Haoning Ye", "Lin Zhang", "Jianchen Wang", "Yixin Zhu", "Sihang Jiang", "Zhuozhi Xiong", "Zihan Li", "Weijie Wu", "Qianyu He", "Rui Xu", "Wenhao Huang", "Jingping Liu", "Zili Wang", "Shusen Wang", "Weiguo Zheng", "Hongwei Feng", "Yanghua Xiao"]
year: 2023
venue: "arXiv:2306.05783"
source: "https://arxiv.org/abs/2306.05783"
doi: "10.48550/arxiv.2306.05783"
arxiv: "2306.05783"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gu-2023-xiezhi-ever-updating-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gu-2023-xiezhi-ever-updating-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/) / Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

# Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

**Authors:** [Zhouhong Gu](https://md.research.elamur.ai/free/en/authors/zhouhong-gu/), Xiaoxuan Zhu, Haoning Ye, [Lin Zhang](https://md.research.elamur.ai/free/en/authors/lin-zhang/), Jianchen Wang, [Yixin Zhu](https://md.research.elamur.ai/free/en/authors/yixin-zhu/), [Sihang Jiang](https://md.research.elamur.ai/free/en/authors/sihang-jiang/), Zhuozhi Xiong, [Zihan Li](https://md.research.elamur.ai/free/en/authors/zihan-li/), Weijie Wu, [Qianyu He](https://md.research.elamur.ai/free/en/authors/qianyu-he/), [Rui Xu](https://md.research.elamur.ai/free/en/authors/rui-xu/), [Wenhao Huang](https://md.research.elamur.ai/free/en/authors/wenhao-huang/), [Jingping Liu](https://md.research.elamur.ai/free/en/authors/jingping-liu/), Zili Wang, Shusen Wang, Weiguo Zheng, Hongwei Feng and [Yanghua Xiao](https://md.research.elamur.ai/free/en/authors/yanghua-xiao/) · **Year:** 2023 · **Venue:** arXiv:2306.05783 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.05783) · [PDF](https://arxiv.org/pdf/2306.05783) · [DOI 10.48550/arxiv.2306.05783](https://doi.org/10.48550/arxiv.2306.05783) · [arXiv 2306.05783](https://arxiv.org/abs/2306.05783) · [OpenAlex](https://openalex.org/W4380355783)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [25](https://md.research.elamur.ai/free/en/papers/gu-2023-xiezhi-ever-updating-benchmark/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Benchmarks such as MMLU, BIG-bench and HELM quickly become too easy for modern LLMs and may leak into their training data. Xiezhi addresses this with a very broad, regularly refreshed test of domain knowledge. Its subject structure follows the Chinese Disciplinary Taxonomy of the Ministry of Education. The authors hand-labelled a seed set of graduate entrance exam questions and used it to train an annotation model. That model then assigned discipline tags to a much larger pool of exam questions and to questions generated automatically from academic surveys, so the benchmark can keep growing with fresh material.

Evaluation also changes. Each question gets 50 candidate answers instead of the usual 4, and the model's generation probabilities rank those candidates. Scores are reported with MRR and Hit@k. Across 47 LLMs, the strongest models beat average human practitioners in science, engineering, agronomy and medicine. Humans still lead clearly in economics, law, education, literature, history and management. Xiezhi separates models of different strength better than earlier benchmarks.

## Key points
- **Scale and structure:** 249,587 multiple-choice questions over 516 disciplines, organised as 13 top-level categories, 118 sub-disciplines and 385 fine-grained fields.
- **Data pipeline:**
  - Xiezhi-Meta has 20,124 manually annotated questions from the Chinese Graduate Entrance Examination.
  - It trains the labelling model, which tags about 170k exam questions and about 80k survey-generated questions.
- **Balanced bilingual subsets:** Xiezhi-Specialty (14,041 single-domain questions) and Xiezhi-Interdiscipline (10,746 multi-domain questions) come in Chinese and English and are less China-centric.
- **Experimental setup:** 47 LLMs were tested, from Bloomz-560m to GPT-4, in 0-, 1- and 3-shot settings on four benchmarks. Xiezhi produced the largest score variance between models.
- **Findings:**
  - More parameters do not always help; Pythia-1.4b beat larger Pythia models.
  - Domain-tuned small models such as DoctorGLM and Baize-Healthcare improved in medicine but lost general ability.
- **Availability:** code and data are released at github.com/MikeGu721/XiezhiBenchmark.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations
- [East China University of Science and Technology](https://md.research.elamur.ai/free/en/universities/east-china-university-of-science-and-technology/) — University, Shanghai, China · 14 works · 87 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, Qianyu He, Rui Xu, Wenhao Huang, Jingping Liu, Zili Wang, Shusen Wang, Weiguo Zheng, Hongwei Feng and Yanghua Xiao. 2023. Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation. arXiv:2306.05783.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gu2023xiezhi,
  title = {{Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation}},
  author = {Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, Qianyu He, Rui Xu, Wenhao Huang, Jingping Liu, Zili Wang, Shusen Wang, Weiguo Zheng, Hongwei Feng and Yanghua Xiao},
  year = {2023},
  doi = {10.48550/arxiv.2306.05783},
  eprint = {2306.05783},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.05783}
}
```

- [← previous M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models](https://md.research.elamur.ai/free/en/papers/liu-2023-m3ke-massive-multi-level/)
- [next → BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models](https://md.research.elamur.ai/free/en/papers/leong-2023-bhasa-holistic-southeast-asian-linguistic/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/)

## References

47 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aribandi, V.; Tay, Y.; Schuster, T.; Rao, J.; Zheng, H. S.; Mehta, S. V.; Zhuang, H.; Tran, V. Q.; Bahri, D.; Ni, J.; et al. 2021. [Ext5: Towards extreme multi-task scaling for transfer learning](https://md.research.elamur.ai/free/en/papers/aribandi-2021-ext5-towards-extreme-multi-task/). arXiv preprint arXiv:2111.10952.
2. Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; et al. 2022. [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073.
3. Biderman, S.; Schoelkopf, H.; Anthony, Q.; Bradley, H.; O’Brien, K.; Hallahan, E.; Khan, M. A.; Purohit, S.; Prashanth, U. S.; Raff, E.; Skowron, A.; Sutawika, L.; and van der Wal, O. 2023. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. arXiv:2304.01373. [find](https://scholar.google.com/scholar?q=Biderman%2C+S%2E%3B+Schoelkopf%2C+H%2E%3B+Anthony%2C+Q%2E%3B+Bradley%2C+H%2E%3B+O%E2%80%99Brien%2C+K%2E%3B+Hallahan%2C+E%2E%3B+Khan%2C+M%2E+A%2E%3B+Purohit%2C+S%2E%3B+Prashanth%2C+U%2E+S%2E%3B+Raff%2C+E%2E%3B+Skowron%2C+A%2E%3B+Sutawika%2C+L%2E%3B+and+van+der+Wal%2C+O%2E+2023%2E+Pythia%3A+A+)
4. Bisk, Y.; Zellers, R.; Gao, J.; Choi, Y.; et al. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, 7432–7439.
5. Black, S.; Biderman, S.; Hallahan, E.; Anthony, Q.; Gao, L.; Golding, L.; He, H.; Leahy, C.; McDonell, K.; Phang, J.; Pieler, M.; Prashanth, U. S.; Purohit, S.; Reynolds, L.; Tow, J.; Wang, B.; and Weinbach, S. 2022. [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/).
6. Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; et al. 2023. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712.
7. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. [find](https://scholar.google.com/scholar?q=Chiang%2C+W%2E%2DL%2E%3B+Li%2C+Z%2E%3B+Lin%2C+Z%2E%3B+Sheng%2C+Y%2E%3B+Wu%2C+Z%2E%3B+Zhang%2C+H%2E%3B+Zheng%2C+L%2E%3B+Zhuang%2C+S%2E%3B+Zhuang%2C+Y%2E%3B+Gonzalez%2C+J%2E+E%2E%3B+Stoica%2C+I%2E%3B+and+Xing%2C+E%2E+P%2E+2023%2E+Vicuna%3A+An+Open%2DSource+Chatbot+Impressing+GPT%2D4+with)
8. Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, E.; Wang, X.; Dehghani, M.; Brahma, S.; et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%3B+Hou%2C+L%2E%3B+Longpre%2C+S%2E%3B+Zoph%2C+B%2E%3B+Tay%2C+Y%2E%3B+Fedus%2C+W%2E%3B+Li%2C+E%2E%3B+Wang%2C+X%2E%3B+Dehghani%2C+M%2E%3B+Brahma%2C+S%2E%3B+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+preprint+arXiv%3A2210%2E1141)
9. Conover, M.; Hayes, M.; Mathur, A.; Meng, X.; Xie, J.; Wan, J.; Shah, S.; Ghodsi, A.; Wendell, P.; Zaharia, M.; and Xin, R. 2023. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM. [https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tu…](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tu…) [↗](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tu…)
10. Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; and Tang, J. 2022. [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 320–335.
11. Fu, Y.; Peng, H.; and Khot, T. 2022. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. [find](https://scholar.google.com/scholar?q=Fu%2C+Y%2E%3B+Peng%2C+H%2E%3B+and+Khot%2C+T%2E+2022%2E+How+does+gpt+obtain+its+ability%3F+tracing+emergent+abilities+of+language+models+to+their+sources%2E)
12. Yao Fu’s Notion. Fu, Y.; Peng, H.; Ou, L.; Sabharwal, A.; and Khot, T. 2023. [Specializing Smaller Language Models towards Multi-Step Reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/). arXiv preprint arXiv:2301.12726. H2O.ai. 2023. h2oGPT - The world’s best open source GPT. [https://github.com/h2oai/h2ogpt](https://github.com/h2oai/h2ogpt). [↗](https://github.com/h2oai/h2ogpt)
13. Hendrycks, D.; Basart, S.; Kadavath, S.; Mazeika, M.; Arora, A.; Guo, E.; Burns, C.; Puranik, S.; He, H.; Song, D.; et al. 2021. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). arXiv preprint arXiv:2105.09938.
14. Hoffmann, J.; Borgeaud, S.; Mensch, A.; Buchatskaya, E.; Cai, T.; Rutherford, E.; de Las Casas, D.; Hendricks, L. A.; Welbl, J.; Clark, A.; et al. 2022. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35: 30016–30030. [find](https://scholar.google.com/scholar?q=Hoffmann%2C+J%2E%3B+Borgeaud%2C+S%2E%3B+Mensch%2C+A%2E%3B+Buchatskaya%2C+E%2E%3B+Cai%2C+T%2E%3B+Rutherford%2C+E%2E%3B+de+Las+Casas%2C+D%2E%3B+Hendricks%2C+L%2E+A%2E%3B+Welbl%2C+J%2E%3B+Clark%2C+A%2E%3B+et+al%2E+2022%2E+An+empirical+analysis+of+compute%2Doptimal+large+)
15. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
16. Hu, Z.; Lan, Y.; Wang, L.; Xu, W.; Lim, E.-P.; Lee, R. K.-W.; Bing, L.; and Poria, S. 2023. [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/). arXiv preprint arXiv:2304.01933.
17. Huang, L.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019. [Cosmos QA: Machine reading comprehension with contextual commonsense reasoning](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/). arXiv preprint arXiv:1909.00277.
18. Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; Lei, J.; et al. 2023. [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). arXiv preprint arXiv:2305.08322.
19. Ji, Y.; Deng, Y.; Gong, Y.; Peng, Y.; Niu, Q.; Zhang, L.; Ma, B.; and Li, X. 2023a. [Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases](https://md.research.elamur.ai/free/en/papers/ji-2023-exploring-impact-instruction-data/). arXiv preprint arXiv:2303.14742.
20. Ji, Y.; Gong, Y.; Deng, Y.; Peng, Y.; Niu, Q.; Ma, B.; and Li, X. 2023b. Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation. arXiv preprint arXiv:2304.07854. — [arXiv:2304.07854](https://arxiv.org/abs/2304.07854) [find](https://scholar.google.com/scholar?q=Ji%2C+Y%2E%3B+Gong%2C+Y%2E%3B+Deng%2C+Y%2E%3B+Peng%2C+Y%2E%3B+Niu%2C+Q%2E%3B+Ma%2C+B%2E%3B+and+Li%2C+X%2E+2023b%2E+Towards+Better+Instruction+Following+Language+Models+for+Chinese%3A+Investigating+the+Impact+of+Training+Data+and+Evaluation%2E+arX)
21. Kpf, A.; Kilcher, Y.; von Rütte, D.; Anagnostidis, S.; Tam, Z.-R.; Stevens, K.; Barhoum, A.; Duc, N. M.; Stanley, O.; Nagyfi, R.; ES, S.; Suri, S.; Glushkov, D.; Dantuluri, A.; Maguire, A.; Schuhmann, C.; Nguyen, H.; and Mattick, A. 2023. [OpenAssistant Conversations – Democratizing Large Language Mo…](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
22. Li, H.; Zhang, Y.; Koto, F.; Yang, Y.; Zhao, H.; Gong, Y.; Duan, N.; and Baldwin, T. 2023. [CMMLU: Measuring massive multitask language understanding in Chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/). arXiv preprint arXiv:2306.09212.
23. Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Liang%2C+P%2E%3B+Bommasani%2C+R%2E%3B+Lee%2C+T%2E%3B+Tsipras%2C+D%2E%3B+Soylu%2C+D%2E%3B+Yasunaga%2C+M%2E%3B+Zhang%2C+Y%2E%3B+Narayanan%2C+D%2E%3B+Wu%2C+Y%2E%3B+Kumar%2C+A%2E%3B+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+arXiv+preprint+arXiv%3A2211%2E091)
24. Liu, C.; Jin, R.; Ren, Y.; Yu, L.; Dong, T.; Peng, X.; Zhang, S.; Peng, J.; Zhang, P.; Lyu, Q.; et al. 2023. [M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models](https://md.research.elamur.ai/free/en/papers/liu-2023-m3ke-massive-multi-level/). arXiv preprint arXiv:2305.10263.
25. Muennighoff, N.; Wang, T.; Sutawika, L.; Roberts, A.; Biderman, S.; Scao, T. L.; Bari, M. S.; Shen, S.; Yong, Z.-X.; Schoelkopf, H.; Tang, X.; Radev, D.; Aji, A. F.; Almubarak, K.; Albanie, S.; Alyafeai, Z.; Webson, A.; Raff, E.; and Raffel, C. 2022. [Crosslingual Generalization through Multitask Fin…](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
26. OpenAI. 2023a. ChatGPT: Optimizing Language Models for Dialogue. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
27. OpenAI. 2023b. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774.
28. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/). Advances in Neural Information Processing Systems, 35: 27730–27744. Perez, E.; Ringer, S.; Lukošiūt…
29. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1): 5485–5551.
30. Sai, A. B.; Mohankumar, A. K.; and Khapra, M. M. 2022. A survey of evaluation metrics used for NLG systems. ACM Computing Surveys (CSUR), 55(2): 1–39. Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; Ilić, S.; Hesslow, D.; Castagné, R.; Luccioni, A. S.; Yvon, F.; Gallé, M.; et al. 2022. Bloom: A 176… [find](https://scholar.google.com/scholar?q=Sai%2C+A%2E+B%2E%3B+Mohankumar%2C+A%2E+K%2E%3B+and+Khapra%2C+M%2E+M%2E+2022%2E+A+survey+of+evaluation+metrics+used+for+NLG+systems%2E+ACM+Computing+Surveys+%28CSUR%29%2C+55%282%29%3A+1%E2%80%9339%2E+Scao%2C+T%2E+L%2E%3B+Fan%2C+A%2E%3B+Akiki%2C+C%2E%3B+Pavlick%2C+E%2E%3B+Ili)
31. Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; et al. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615.
32. StabilityAI. 2023. StableLM: Stability AI Language Models. [https://github.com/Stability-AI/StableLM](https://github.com/Stability-AI/StableLM). [↗](https://github.com/Stability-AI/StableLM)
33. Sun, T.; Xiaotian, Z.; Zhengfu, H.; Peng, L.; Qinyuan, C.; Hang, Y.; Xiangyang, L.; Yunfan, S.; Qiong, T.; Xingjian, Z.; Ke, C.; Yining, Z.; Zhejian, Z.; Ruixiao, L.; Jun, Z.; Yunhua, Z.; Linyang, L.; Xiaogui, Y.; Lingling, W.; Zhangyue, Y.; Xuanjing, H.; and Xipeng, Q. 2023a. FudanNLP Moss. [find](https://scholar.google.com/scholar?q=Sun%2C+T%2E%3B+Xiaotian%2C+Z%2E%3B+Zhengfu%2C+H%2E%3B+Peng%2C+L%2E%3B+Qinyuan%2C+C%2E%3B+Hang%2C+Y%2E%3B+Xiangyang%2C+L%2E%3B+Yunfan%2C+S%2E%3B+Qiong%2C+T%2E%3B+Xingjian%2C+Z%2E%3B+Ke%2C+C%2E%3B+Yining%2C+Z%2E%3B+Zhejian%2C+Z%2E%3B+Ruixiao%2C+L%2E%3B+Jun%2C+Z%2E%3B+Yunhua%2C+Z%2E%3B+Linyang%2C+L%2E%3B)
34. Sun, T.; Zhang, X.; He, Z.; Li, P.; Cheng, Q.; Yan, H.; Liu, X.; Shao, Y.; Tang, Q.; Zhao, X.; Chen, K.; Zheng, Y.; Zhou, Z.; Li, R.; Zhan, J.; Zhou, Y.; Li, L.; Yang, X.; Wu, L.; Yin, Z.; Huang, X.; and Qiu, X. 2023b. [MOSS: An open-source tool-augmented conversational language model from Fudan Univ…](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
35. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. [Stanford Alpaca: An Instruction-following LLaMA model](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). [https://github.com/tatsu-lab/stanford](https://github.com/tatsu-lab/stanford) alpaca. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière… [↗](https://github.com/tatsu-lab/stanford)
36. Wang, E. J.; Alexiuk, C.; Bo, Y.; Yang, Z.; Kwok, K.; Gusev, I.; Echavez, A.; et al. 2023a. Alpaca-LoRA. [https://github.com/tloen/alpaca-lora](https://github.com/tloen/alpaca-lora). [↗](https://github.com/tloen/alpaca-lora)
37. Wang, J.; Liang, Y.; Meng, F.; Shi, H.; Li, Z.; Xu, J.; Qu, J.; and Zhou, J. 2023b. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). arXiv preprint arXiv:2303.04048.
38. Wei, J.; Tay, Y.; Bommasani, R.; Raffel, C.; Zoph, B.; Borgeaud, S.; Yogatama, D.; Bosma, M.; Zhou, D.; Metzler, D.; et al. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682.
39. Xiong, H.; Wang, S.; Zhu, Y.; Zhao, Z.; Liu, Y.; Huang, L.; Wang, Q.; and Shen, D. 2023. DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task. arXiv:2304.01097. [find](https://scholar.google.com/scholar?q=Xiong%2C+H%2E%3B+Wang%2C+S%2E%3B+Zhu%2C+Y%2E%3B+Zhao%2C+Z%2E%3B+Liu%2C+Y%2E%3B+Huang%2C+L%2E%3B+Wang%2C+Q%2E%3B+and+Shen%2C+D%2E+2023%2E+DoctorGLM%3A+Fine%2Dtuning+your+Chinese+Doctor+is+not+a+Herculean+Task%2E+arXiv%3A2304%2E01097%2E)
40. Xu, C.; Guo, D.; Duan, N.; and McAuley, J. 2023. [Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data](https://md.research.elamur.ai/free/en/papers/xu-2023-baize-open-source-chat-model/). arXiv preprint arXiv:2304.01196.
41. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%3B+Holtzman%2C+A%2E%3B+Bisk%2C+Y%2E%3B+Farhadi%2C+A%2E%3B+and+Choi%2C+Y%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
42. Zeng, A.; Liu, X.; Du, Z.; Wang, Z.; Lai, H.; Ding, M.; Yang, Z.; Xu, Y.; Zheng, W.; Xia, X.; Tam, W. L.; Ma, Z.; Xue, Y.; Zhai, J.; Chen, W.; Liu, Z.; Zhang, P.; Dong, Y.; and Tang, J. 2023. [GLM-130B: An Open Bilingual Pre-trained Model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). In The Eleventh International Conference on Learning Represen…
43. Zeng, H. 2023. Measuring Massive Multitask Chinese Understanding. arXiv preprint arXiv:2304.12986. [find](https://scholar.google.com/scholar?q=Zeng%2C+H%2E+2023%2E+Measuring+Massive+Multitask+Chinese+Understanding%2E+arXiv+preprint+arXiv%3A2304%2E12986%2E)
44. Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. 2023. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223.
45. Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2023. [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). arXiv preprint arXiv:2304.06364.
46. Zhou, C.; Li, Q.; Li, C.; Yu, J.; Liu, Y.; Wang, G.; Zhang, K.; Ji, C.; Yan, Q.; He, L.; et al. 2023. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419. [find](https://scholar.google.com/scholar?q=Zhou%2C+C%2E%3B+Li%2C+Q%2E%3B+Li%2C+C%2E%3B+Yu%2C+J%2E%3B+Liu%2C+Y%2E%3B+Wang%2C+G%2E%3B+Zhang%2C+K%2E%3B+Ji%2C+C%2E%3B+Yan%2C+Q%2E%3B+He%2C+L%2E%3B+et+al%2E+2023%2E+A+comprehensive+survey+on+pretrained+foundation+models%3A+A+history+from+bert+to+chatgpt%2E+arXiv+prep)
47. Zhu, M.; Deng, Z.; Xiong, W.; Yu, M.; Zhang, M.; and Wang, W. Y. 2019. Neural Correction Model for Open-Domain Named Entity Recognition. arXiv preprint arXiv:1909.06058. — [arXiv:1909.06058](https://arxiv.org/abs/1909.06058) [find](https://scholar.google.com/scholar?q=Zhu%2C+M%2E%3B+Deng%2C+Z%2E%3B+Xiong%2C+W%2E%3B+Yu%2C+M%2E%3B+Zhang%2C+M%2E%3B+and+Wang%2C+W%2E+Y%2E+2019%2E+Neural+Correction+Model+for+Open%2DDomain+Named+Entity+Recognition%2E+arXiv+preprint+arXiv%3A1909%2E06058%2E+%E2%80%94+%5BarXiv%3A1909%2E06058%5D%28https%3A%2F%2Fa)

*Updated 2 November 2026*