---
title: "CLiMP: A Benchmark for Chinese Language Model Evaluation"
description: "Most suites for testing the grammatical knowledge of language models are built for… 81.8% accuracy. 25 references, cited by 48 papers here."
authors: ["Beilei Xiang", "Changbing Yang", "Yu Li", "Alex Warstadt", "Katharina Kann"]
year: 2021
venue: "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume"
source: "https://aclanthology.org/2021.eacl-main.242/"
doi: "10.18653/v1/2021.eacl-main.242"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiang-2021-climp-benchmark-chinese-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiang-2021-climp-benchmark-chinese-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / CLiMP: A Benchmark for Chinese Language Model Evaluation

# CLiMP: A Benchmark for Chinese Language Model Evaluation

**Authors:** Beilei Xiang, [Changbing Yang](https://md.research.elamur.ai/free/en/authors/changbing-yang/), [Yu Li](https://md.research.elamur.ai/free/en/authors/yu-li/), [Alex Warstadt](https://md.research.elamur.ai/free/en/authors/alex-warstadt/) and [Katharina Kann](https://md.research.elamur.ai/free/en/authors/katharina-kann/) · **Year:** 2021 · **Venue:** [Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2021](https://md.research.elamur.ai/free/en/venues/eacl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.eacl-main.242/) · [PDF](https://aclanthology.org/2021.eacl-main.242.pdf) · [DOI 10.18653/v1/2021.eacl-main.242](https://doi.org/10.18653/v1/2021.eacl-main.242) · [OpenAlex](https://openalex.org/W3155192455)

**Citations:** 25 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/xiang-2021-climp-benchmark-chinese-language-model/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most suites for testing the grammatical knowledge of language models are built for English. Earlier non-English work mostly checks subject–verb agreement, which Mandarin does not have. The authors address this gap with CLiMP, a collection of Mandarin minimal pairs. Each pair contains a grammatical and an ungrammatical sentence that differ by one edit.

The sentences were generated semi-automatically from grammar templates. The templates draw on an annotated lexicon translated from BLiMP's vocabulary and extended with Chinese-specific items such as classifiers and coverbs. Native speakers validated the pairs.

Eleven models were tested: Chinese BERT, six LSTMs and four 5-gram models, trained on Chinese Wikipedia with word or character tokenization. Each model passes a pair if it gives the grammatical sentence the higher probability; BERT is scored with pseudo-log-likelihood. Chinese BERT averaged 81.8% accuracy, while the LSTMs and 5-grams stayed only modestly above the 50% chance level.

The models handle local selectional restrictions fairly well. They break down on binding, filler-gap dependencies and the bǎ construction. This suggests weak hierarchical and long-distance knowledge in Chinese LMs, and CLiMP gives a benchmark for tracking future progress.

## Key points
- CLiMP has 16 paradigms of 1,000 minimal pairs each, covering 9 Mandarin phenomena. The vocabulary contains 3,456 words annotated with 84 features.
- Validation used 22 native-speaker annotators. A coverb-direction paradigm was dropped for falling below 85% agreement. Human agreement is 95.8% overall and 97.1% on a 240-pair resample, compared with 88.6% for BLiMP.
- Chinese BERT scored 81.8% overall, about 14 points below humans. LSTMs ranged from 60.4% to 66.0%, and 5-grams (KenLM) from 55.9% to 65.7%.
- Classifier–noun agreement and verb complements were the easiest phenomena; BERT scored about 93% on both. Binding sat near chance (about 50%) for every model.
- BERT did poorly on NP head finality (53.1%) but well on coverbs (87.9%). Larger corpora helped the 5-grams but barely changed LSTM results. Character tokenization helped the 5-grams.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 136 works · 86,390 citations
- [University of Colorado Boulder](https://md.research.elamur.ai/free/en/universities/university-of-colorado-boulder/) — University, Boulder, United States · 52 works · 49,647 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt and Katharina Kann. 2021. CLiMP: A Benchmark for Chinese Language Model Evaluation. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xiang2021climp,
  title = {{CLiMP: A Benchmark for Chinese Language Model Evaluation}},
  author = {Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt and Katharina Kann},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.242},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.242}
}
```

- [← previous SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images](https://md.research.elamur.ai/free/en/papers/dimitrov-2021-semeval-2021-task-6-detection/)
- [next → Hate Towards the Political Opponent: A Twitter Corpus Study of the 2020 US Elections on the Basis of Offensive Speech and Stance Detection](https://md.research.elamur.ai/free/en/papers/grimminger-2021-hate-political-opponent-twitter-corpus/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

25 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D%2E+Manning%2E+2019%2E+What+does+BERT+look+at%3F+an+analysis+of+BERT%E2%80%99s+attention%2E+In+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+an)
2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
4. Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sentence vector representations. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1790–1801, Santa Fe, New Mexico, USA. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2C+Ahmed+Elgohary%2C+Colin+Phillips%2C+and+Philip+Resnik%2E+2018%2E+Assessing+composition+in+sentence+vector+representations%2E+In+Proceedings+of+the+27th+International+Conference+on+Computationa)
5. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. [Colorless green recurrent networks dream hierarchically](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
6. John Hewitt and Christopher D. Manning. 2019. [A structural probe for finding syntax in word representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129…
7. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
8. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2020%2E+A+systematic+assessment+of+syntactic+generalization+in+neural+language+models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+th)
9. Dan Jurafsky and James H. Martin. 2009. Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. Pearson Prentice Hall, Upper Saddle River, N.J. [find](https://scholar.google.com/scholar?q=Dan+Jurafsky+and+James+H%2E+Martin%2E+2009%2E+Speech+and+language+processing%3A+An+introduction+to+natural+language+processing%2C+computational+linguistics%2C+and+speech+recognition%2E+Pearson+Prentice+Hall%2C+Upper+)
10. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. [Assessing the ability of LSTMs to learn syntax-sensitive dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). Transactions of the Association for Computational Linguistics, 4:521– 535.
11. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1192%E2%80%931202%2C+Bruss)
12. Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020. [Cross-linguistic syntactic evaluation of word prediction models](https://md.research.elamur.ai/free/en/papers/mueller-2020-cross-linguistic-syntactic-evaluation-word/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5523–5539, Online. Association for Comput…
13. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
14. Yip Po-Ching and Don Rimmington. 2015. Chinese: A comprehensive grammar. Routledge. [find](https://scholar.google.com/scholar?q=Yip+Po%2DChing+and+Don+Rimmington%2E+2015%2E+Chinese%3A+A+comprehensive+grammar%2E+Routledge%2E)
15. Shauli Ravfogel, Yoav Goldberg, and Francis Tyers. 2018. Can LSTM learn to capture agreement? the case of basque. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 98–107, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shauli+Ravfogel%2C+Yoav+Goldberg%2C+and+Francis+Tyers%2E+2018%2E+Can+LSTM+learn+to+capture+agreement%3F+the+case+of+basque%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+and+Interpreting+Neur)
16. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2021. [A Primer in BERTology: What We Know About How BERT Works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). Transactions of the Association for Computational Linguistics, 8(0):842– 866.
17. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. [Masked language model scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics.
18. Joonbo Shin, Yoonhyung Lee, and Kyomin Jung. 2019. Effective sentence scoring method using BERT for speech recognition. In Asian Conference on Machine Learning, pages 1081–1093. [find](https://scholar.google.com/scholar?q=Joonbo+Shin%2C+Yoonhyung+Lee%2C+and+Kyomin+Jung%2E+2019%2E+Effective+sentence+scoring+method+using+BERT+for+speech+recognition%2E+In+Asian+Conference+on+Machine+Learning%2C+pages+1081%E2%80%931093%2E)
19. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. [What do you learn from context? probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/). In International Conference on Lear…
20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc.
21. Jesse Vig. 2019. [A multiscale visualization of attention in the transformer model](https://md.research.elamur.ai/free/en/papers/vig-2019-multiscale-visualization-attention-transformer-model/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37–42, Florence, Italy. Association for Computational Linguistics.
22. Alex Wang and Kyunghyun Cho. 2019. [BERT has a mouth, and it must speak: BERT as a Markov random field language model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/). In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30–36, Minneapolis, Minnesota. Association for Computational Linguistics.
23. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo-nananey, Wei Peng, Sheng-Fu Wang, and Samuel Bowman. 2020. [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8(0):377–392.
24. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
25. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. [What do rnn language models learn about filler–gap dependencies?](https://md.research.elamur.ai/free/en/papers/wilcox-2018-rnn-language-models-learn-about/) In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221.

*Updated 24 October 2026*