---
title: "SuperCLUE — paper by Xu et al. (2023) — summary and ideas"
description: "Most LLM benchmarks, including Chinese ones such as C-Eval and MMCU, rely on multiple-choice… 32 references, cited by 32 papers here."
authors: ["Liang Xu", "Anqi Li", "Lei Zhu", "Hang Xue", "Changtai Zhu", "Kangkang Zhao", "Haonan He", "Xuanwei Zhang", "Qiyue Kang", "Zhenzhong Lan"]
year: 2023
venue: "arXiv:2307.15020"
source: "https://arxiv.org/abs/2307.15020"
doi: "10.48550/arxiv.2307.15020"
arxiv: "2307.15020"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2023-superclue-comprehensive-chinese-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2023-superclue-comprehensive-chinese-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark

# SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark

**Authors:** [Liang Xu](https://md.research.elamur.ai/free/en/authors/liang-xu/), [Anqi Li](https://md.research.elamur.ai/free/en/authors/anqi-li/), [Lei Zhu](https://md.research.elamur.ai/free/en/authors/lei-zhu/), [Hang Xue](https://md.research.elamur.ai/free/en/authors/hang-xue/), [Changtai Zhu](https://md.research.elamur.ai/free/en/authors/changtai-zhu/), [Kangkang Zhao](https://md.research.elamur.ai/free/en/authors/kangkang-zhao/), Haonan He, [Xuanwei Zhang](https://md.research.elamur.ai/free/en/authors/xuanwei-zhang/), Qiyue Kang and [Zhenzhong Lan](https://md.research.elamur.ai/free/en/authors/zhenzhong-lan/) · **Year:** 2023 · **Venue:** arXiv:2307.15020 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.15020) · [PDF](https://arxiv.org/pdf/2307.15020) · [DOI 10.48550/arxiv.2307.15020](https://doi.org/10.48550/arxiv.2307.15020) · [arXiv 2307.15020](https://arxiv.org/abs/2307.15020) · [OpenAlex](https://openalex.org/W4385373704)

**Citations:** 25 citations (OpenAlex, 2026-10-14) · [32](https://md.research.elamur.ai/free/en/papers/xu-2023-superclue-comprehensive-chinese-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most LLM benchmarks, including Chinese ones such as C-Eval and MMCU, rely on multiple-choice accuracy. That metric says little about how users experience a model in free-form conversation. SuperCLUE addresses this gap for Chinese with three complementary parts:

- **CArena** collects real user queries and pairwise votes from LangYa Leaderboard, an anonymous Elo-based battle platform. Its win-and-tie rates serve as the gold standard.
- **OPEN** holds open-ended single- and multi-turn questions across ten capability categories drawn from annotated user queries. GPT-4 scores the answers automatically.
- **CLOSE** converts each single-turn OPEN item into a multiple-choice question. GPT-3.5 generates the options, which are then manually verified.

Across 11 LLMs, GPT-4 leads by a wide margin, and MiniMax is the strongest Chinese model. GPT-4's judgments agree well with human ratings. Closed-ended accuracy alone does not significantly track user preference, but combining CLOSE with multi-turn OPEN scores predicts CArena results best. The work suggests that Chinese LLM evaluation needs open-ended, conversational testing alongside exam-style questions.

## Key points
- CArena contains about 9.9k user votes collected since 15 May 2023 on LangYa Leaderboard. The compared models include ChatGLM-130B, ChatGLM2-6B, MOSS, Ziya and MiniMax.
- The ten capability categories range from semantic understanding and small talk to code generation, calculation, role-playing and safety. OPEN has 30 single-turn and 30 multi-turn questions per category, for 600 items in total.
- Pearson correlations with CArena are 0.7950 for CLOSE (not significant), 0.8747 for OPEN SINGLE (near significant) and 0.8985 for OPEN MULTIPLE (significant). The best mix, 0.49·CLOSE + 0.51·OPEN MULTIPLE, reaches 0.9397.
- Scores on CLOSE vary much less across models than scores on OPEN SINGLE. The reported coefficients of variation are roughly 0.11 and 0.34, and the paper's ordering of these values is somewhat ambiguous. This means CLOSE separates models poorly.
- Wenxin Yiyan, 360 and SparkDesk were evaluated on OPEN and CLOSE but are missing from CArena because platform access was limited.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Westlake University](https://md.research.elamur.ai/free/en/universities/westlake-university/) — University, Hangzhou, China · 45 works · 8,028 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang and Zhenzhong Lan. 2023. SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark. arXiv:2307.15020.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xu2023superclue,
  title = {{SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark}},
  author = {Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang and Zhenzhong Lan},
  year = {2023},
  doi = {10.48550/arxiv.2307.15020},
  eprint = {2307.15020},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.15020}
}
```

- [← previous RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text](https://md.research.elamur.ai/free/en/papers/zhou-2023-recurrentgpt-interactive-generation-arbitrarily-long/)
- [next → Abstractive Meeting Summarization: A Survey](https://md.research.elamur.ai/free/en/papers/rennard-2023-abstractive-meeting-summarization-survey/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

32 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Ch… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
2. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. [find](https://scholar.google.com/scholar?q=Chiang%2C+W%2E%2DL%2E%3B+Li%2C+Z%2E%3B+Lin%2C+Z%2E%3B+Sheng%2C+Y%2E%3B+Wu%2C+Z%2E%3B+Zhang%2C+H%2E%3B+Zheng%2C+L%2E%3B+Zhuang%2C+S%2E%3B+Zhuang%2C+Y%2E%3B+Gonzalez%2C+J%2E+E%2E%3B+Stoica%2C+I%2E%3B+and+Xing%2C+E%2E+P%2E+2023%2E+Vicuna%3A+An+Open%2DSource+Chatbot+Impressing+GPT%2D4+with)
3. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lo…
4. Dolan, W. B.; and Brockett, C. 2005. [Automatically Constructing a Corpus of Sentential Paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
5. Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; and Tang, J. 2022. [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 320–335.
6. Goyal, T.; Li, J. J.; and Durrett, G. 2023. News Summarization and Evaluation in the Era of GPT-3. arXiv:2209.12356. [find](https://scholar.google.com/scholar?q=Goyal%2C+T%2E%3B+Li%2C+J%2E+J%2E%3B+and+Durrett%2C+G%2E+2023%2E+News+Summarization+and+Evaluation+in+the+Era+of+GPT%2D3%2E+arXiv%3A2209%2E12356%2E)
7. Gu, Z.; Zhu, X.; Ye, H.; Zhang, L.; Wang, J.; Jiang, S.; Xiong, Z.; Li, Z.; He, Q.; Xu, R.; Huang, W.; Wang, Z.; Wang, S.; Zheng, W.; Feng, H.; and Xiao, Y. 2023. [Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation](https://md.research.elamur.ai/free/en/papers/gu-2023-xiezhi-ever-updating-benchmark/). arXiv:2306.05783.
8. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). CoRR, abs/2009.03300.
9. Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; Lei, J.; Fu, Y.; Sun, M.; and He, J. 2023. [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). arXiv preprint arXiv:2305.08322. Kasneci, E.; Seßler, K.; Küchemann, S.; Bannert,…
10. Lee Rodgers, J.; and Nicewander, W. A. 1988. Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1): 59–66. [find](https://scholar.google.com/scholar?q=Lee+Rodgers%2C+J%2E%3B+and+Nicewander%2C+W%2E+A%2E+1988%2E+Thirteen+ways+to+look+at+the+correlation+coefficient%2E+The+American+Statistician%2C+42%281%29%3A+59%E2%80%9366%2E)
11. Lewkowycz, A.; Slone, A.; Andreassen, A.; Freeman, D.; Dyer, E. S.; Mishra, G.; Gur-Ari, G.; Lee, J.; Sohl-dickstein, J.; Chiafullo, K.; Fedus, L. B.; Fiedel, N.; Liu, R.; Misra, V.; and Ramasesh, V. V. 2022. [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language model…](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
12. Li, X.; Zhang, T.; Dubois, Y.; Taori, R.; Gulrajani, I.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. [https://github.com/tatsu-lab/alpaca](https://github.com/tatsu-lab/alpaca) eval. [↗](https://github.com/tatsu-lab/alpaca)
13. Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; Newman, B.; Yuan, B.; Yan, B.; Zhang, C.; Cosgrove, C.; Manning, C. D.; Ré, C.; Acosta-Navas, D.; Hudson, D. A.; Zelikman, E.; Durmus, E.; Ladhak, F.; Rong, F.; Ren, H.; Yao, H.; Wa… [find](https://scholar.google.com/scholar?q=Liang%2C+P%2E%3B+Bommasani%2C+R%2E%3B+Lee%2C+T%2E%3B+Tsipras%2C+D%2E%3B+Soylu%2C+D%2E%3B+Yasunaga%2C+M%2E%3B+Zhang%2C+Y%2E%3B+Narayanan%2C+D%2E%3B+Wu%2C+Y%2E%3B+Kumar%2C+A%2E%3B+Newman%2C+B%2E%3B+Yuan%2C+B%2E%3B+Yan%2C+B%2E%3B+Zhang%2C+C%2E%3B+Cosgrove%2C+C%2E%3B+Manning%2C+C%2E+D%2E%3B+Re%CC%81%2C+C%2E%3B+A)
14. Nov, O.; Singh, N.; and Mann, D. 2023. Putting ChatGPT’s Medical Advice to the (Turing) Test. arXiv:2301.10035. — [arXiv:2301.10035](https://arxiv.org/abs/2301.10035) [find](https://scholar.google.com/scholar?q=Nov%2C+O%2E%3B+Singh%2C+N%2E%3B+and+Mann%2C+D%2E+2023%2E+Putting+ChatGPT%E2%80%99s+Medical+Advice+to+the+%28Turing%29+Test%2E+arXiv%3A2301%2E10035%2E+%E2%80%94+%5BarXiv%3A2301%2E10035%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2301%2E10035%29)
15. OpenAI. 2023. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774.
16. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. [Training language models to follow instruction…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
17. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). arXiv preprint arXiv:1606.05250.
18. Sallam, M. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare, volume 11, 887. [find](https://scholar.google.com/scholar?q=Sallam%2C+M%2E+2023%2E+ChatGPT+utility+in+healthcare+education%2C+research%2C+and+practice%3A+systematic+review+on+the+promising+perspectives+and+valid+concerns%2E+In+Healthcare%2C+volume+11%2C+887%2E)
19. MDPI. Sarlin, P.; DeTone, D.; Malisiewicz, T.; and Rabinovich, A. 2019. SuperGlue: Learning Feature Matching with Graph Neural Networks. CoRR, abs/1911.11763. [find](https://scholar.google.com/scholar?q=MDPI%2E+Sarlin%2C+P%2E%3B+DeTone%2C+D%2E%3B+Malisiewicz%2C+T%2E%3B+and+Rabinovich%2C+A%2E+2019%2E+SuperGlue%3A+Learning+Feature+Matching+with+Graph+Neural+Networks%2E+CoRR%2C+abs%2F1911%2E11763%2E)
20. Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C. D.; Ng, A. Y.; and Potts, C. 2013. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 conference on empirical methods in natural language processing, 1631–1642.
21. Spearman, C. 1987. The Proof and Measurement of Association between Two Things. The American Journal of Psychology, 100(3/4): 441–471. [find](https://scholar.google.com/scholar?q=Spearman%2C+C%2E+1987%2E+The+Proof+and+Measurement+of+Association+between+Two+Things%2E+The+American+Journal+of+Psychology%2C+100%283%2F4%29%3A+441%E2%80%93471%2E)
22. Sun, T.; Zhang, X.; He, Z.; Li, P.; Cheng, Q.; Yan, H.; Liu, X.; Shao, Y.; Tang, Q.; Zhao, X.; Chen, K.; Zheng, Y.; Zhou, Z.; Li, R.; Zhan, J.; Zhou, Y.; Li, L.; Yang, X.; Wu, L.; Yin, Z.; Huang, X.; and Qiu, X. 2023. [MOSS: Training Conversational Language Models from Synthetic Data](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). Touvron, H.; La…
23. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2018. [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 353– 355. Brussels, Belgium: As…
24. Wang, J.; Zhang, Y.; Zhang, L.; Yang, P.; Gao, X.; Wu, Z.; Dong, X.; He, J.; Zhuo, J.; Yang, Q.; Huang, Y.; Li, X.; Wu, Y.; Lu, J.; Zhu, X.; Chen, W.; Han, T.; Pan, K.; Wang, R.; Wang, H.; Wu, X.; Zeng, Z.; Chen, C.; Gan, R.; and Zhang, J. 2022. Fengshenbang 1.0: Being the Foundation of Chinese Cogn… — [arXiv:2209.02970](https://arxiv.org/abs/2209.02970) [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%3B+Zhang%2C+Y%2E%3B+Zhang%2C+L%2E%3B+Yang%2C+P%2E%3B+Gao%2C+X%2E%3B+Wu%2C+Z%2E%3B+Dong%2C+X%2E%3B+He%2C+J%2E%3B+Zhuo%2C+J%2E%3B+Yang%2C+Q%2E%3B+Huang%2C+Y%2E%3B+Li%2C+X%2E%3B+Wu%2C+Y%2E%3B+Lu%2C+J%2E%3B+Zhu%2C+X%2E%3B+Chen%2C+W%2E%3B+Han%2C+T%2E%3B+Pan%2C+K%2E%3B+Wang%2C+R%2E%3B+Wang%2C+H%2E%3B+Wu%2C+X%2E%3B+Zen)
25. Williams, A.; Nangia, N.; and Bowman, S. 2018. [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 11…
26. Xu, L.; Hu, H.; Zhang, X.; Li, L.; Cao, C.; Li, Y.; Xu, Y.; Sun, K.; Yu, D.; Yu, C.; Tian, Y.; Dong, Q.; Liu, W.; Shi, B.; Cui, Y.; Li, J.; Zeng, J.; Wang, R.; Xie, W.; Li, Y.; Patterson, Y.; Tian, Z.; Zhang, Y.; Zhou, H.; Liu, S.; Zhao, Z.; Zhao, Q.; Yue, C.; Zhang, X.; Yang, Z.; Richardson, K.; an… — [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/)
27. Zeng, A.; Liu, X.; Du, Z.; Wang, Z.; Lai, H.; Ding, M.; Yang, Z.; Xu, Y.; Zheng, W.; Xia, X.; et al. 2022. [Glm-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). arXiv preprint arXiv:2210.02414.
28. Zeng, H. 2023. Measuring Massive Multitask Chinese Understanding. arXiv:2304.12986. [find](https://scholar.google.com/scholar?q=Zeng%2C+H%2E+2023%2E+Measuring+Massive+Multitask+Chinese+Understanding%2E+arXiv%3A2304%2E12986%2E)
29. Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; Mihaylov, T.; Ott, M.; Shleifer, S.; Shuster, K.; Simig, D.; Koura, P. S.; Sridhar, A.; Wang, T.; and Zettlemoyer, L. 2022. [OPT: Open Pre-trained Transformer Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv:2205.01068.
30. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. [Judging LLM-as-a-judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv:2306.05685.
31. Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A. S. S.; Chen, W.; and Duan, N. 2023. [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). ArXiv, abs/2304.06364.
32. Zhuang, L.; Wayne, L.; Ya, S.; and Jun, Z. 2021. [A Robustly Optimized BERT Pre-training Approach with Post-training](https://md.research.elamur.ai/free/en/papers/zhuang-2021-robustly-optimized-bert-pre-training/). In Proceedings of the 20th Chinese National Conference on Computational Linguistics, 1218–1227. Huhhot, China: Chinese Information Processing Society of China.

*Updated 31 October 2026*