---
title: "大语言模型评测综述 — 罗文 2024"
description: "This survey from Peking University examines how large language models (LLMs) should be evaluated."
authors: ["罗文", "王厚峰"]
year: 2024
venue: "中文信息学报 (Journal of Chinese Information Processing)"
source: "http://jcip.cipsc.org.cn/article/id/zwxxxb_3659"
licence: "Free access (jcip.cipsc.org.cn), no open licence stated — readonly; © 中文信息学报"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/luo-2024-evaluating-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/luo-2024-evaluating-large-language/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2024](https://md.research.elamur.ai/free/papers/2024/page/4/) / 大语言模型评测综述

# 大语言模型评测综述

**Authors:** 罗文, 王厚峰 · **Year:** 2024 · **Venue:** 中文信息学报 (Journal of Chinese Information Processing) · **Licence:** Free access (jcip.cipsc.org.cn), no open licence stated — readonly; © 中文信息学报

**Links:** [Publisher's page](http://jcip.cipsc.org.cn/article/id/zwxxxb_3659) · [PDF](http://jcip.cipsc.org.cn/cn/article/pdf/preview/zwxxxb_3659.pdf)

## Summary
This survey from Peking University examines how large language models (LLMs) should be evaluated. As LLMs become more general-purpose, the older approach of scoring one model on one task with one metric no longer works well. The authors identify three problems with it. Benchmarks are too narrow to separate strong models. Test data often comes from limited domains or populations. Properties beyond raw accuracy, such as reasoning, robustness, fairness and harmful output, are largely ignored.

The paper is a literature review and introduces no new dataset. It sorts benchmarks into two groups. The "classical" paradigm compares model outputs against reference answers for NLU and NLG tasks. The "new" paradigm combines many datasets and also probes alignment with human preferences, knowledge, reasoning and safety. The authors then cover evaluation metrics and methods, point out weaknesses such as unreasonable task design and results that are hard to interpret, and suggest future research directions. Chinese-language benchmarks get particular attention, which makes the survey a useful map for anyone building evaluation suites for languages other than English.

## Key points
- **Classical benchmarks surveyed:**
  - GLUE has 9 NLU tasks, and SuperGLUE adds five harder ones.
  - XTREME spans 40 languages and 9 tasks; XGLUE has 11 tasks across 19 languages.
  - CLUE (2020) was the first large-scale Chinese NLU benchmark.
  - BLEU, METEOR and ROUGE are discussed as generation metrics.
- **Integrated and LLM-scale benchmarks:**
  - CUGE organizes 21 datasets into 7 language functions and 18 NLP tasks.
  - EleutherAI LM Harness covers 200+ datasets.
  - BIG-bench (Google, 2022) took 442 researchers two years and covers 200+ datasets.
- **Preference-alignment benchmarks:**
  - MT-Bench has 80 multi-turn questions across 8 scenarios.
  - Chatbot Arena collected about 30,000 crowdsourced pairwise judgments in its first month.
  - SuperCLUE-Open has 1,200 Chinese multi-turn questions.
- **C-EVAL** contains 13,948 multiple-choice questions spanning 52 subjects and four difficulty levels. To reduce contamination, its questions come mainly from mock and local exams. A harder subset, C-EVAL HARD, is also provided.
- **SAFETYPROMPTS** tests the safety of Chinese LLMs using 8 safety scenarios (e.g., insult, privacy, mental health) and 6 instruction-attack types (e.g., goal hijacking, prompt leaking).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{2024,
  title = {{大语言模型评测综述}},
  author = {罗文 and 王厚峰},
  journal = {中文信息学报 (Journal of Chinese Information Processing)},
  year = {2024},
  url = {http://jcip.cipsc.org.cn/article/id/zwxxxb_3659}
}
```

- [← previous 大语言模型的幻觉问题研究综述](https://md.research.elamur.ai/free/papers/liu-2024-hallucinations-large-language/)
- [next → 探索中文预训练模型的混合粒度编码和IDF遮蔽](https://md.research.elamur.ai/free/papers/shao-2024-exploring-chinese-pre/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/papers/2024/page/4/)