---
title: "SLING: Sino Linguistic Evaluation of Large Language Models"
description: "The authors build SLING, a set of about 38K Mandarin minimal pairs covering nine… 97.1% accuracy. 60 references, cited by 33 papers here."
authors: ["Yixiao Song", "Kalpesh Krishna", "Rajesh Bhatt", "Mohit Iyyer"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.305/"
doi: "10.18653/v1/2022.emnlp-main.305"
arxiv: "2210.11689"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/song-2022-sling-sino-linguistic-evaluation-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/song-2022-sling-sino-linguistic-evaluation-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / SLING: Sino Linguistic Evaluation of Large Language Models

# SLING: Sino Linguistic Evaluation of Large Language Models

**Authors:** [Yixiao Song](https://md.research.elamur.ai/free/en/authors/yixiao-song/), [Kalpesh Krishna](https://md.research.elamur.ai/free/en/authors/kalpesh-krishna/), [Rajesh Bhatt](https://md.research.elamur.ai/free/en/authors/rajesh-bhatt/) and [Mohit Iyyer](https://md.research.elamur.ai/free/en/authors/mohit-iyyer/) · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.305/) · [PDF](https://aclanthology.org/2022.emnlp-main.305.pdf) · [DOI 10.18653/v1/2022.emnlp-main.305](https://doi.org/10.18653/v1/2022.emnlp-main.305) · [arXiv 2210.11689](https://arxiv.org/abs/2210.11689) · [OpenAlex](https://openalex.org/W4385573092)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [33](https://md.research.elamur.ai/free/en/papers/song-2022-sling-sino-linguistic-evaluation-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Measuring how much grammar pretrained Chinese language models actually know requires clean test data. The earlier CLiMP benchmark falls short because its vocabulary was translated from the English BLiMP and several of its templates produce pairs with no real acceptability contrast.

The authors build SLING, a set of about 38K Mandarin minimal pairs covering nine phenomena (six syntactic, three semantic) across 38 paradigms. Rather than translating, they extract subtrees from linguist-annotated parses in the Chinese Treebank 9.0 and edit them with hand-designed templates, so that one sentence in each pair becomes unacceptable. A model gets an item right when it assigns the acceptable sentence a lower (pseudo-)perplexity.

Across 18 public monolingual and multilingual LMs, mean accuracy is 69.7%, compared with 97.1% for native speakers. The relatively small bert-base-chinese scores highest at 84.8%, outperforming much larger models. Models also show strong gender and number biases and cope better with local than with hierarchical structure.

SLING therefore provides a more trustworthy and more discriminative diagnostic for Chinese LMs.

## Key points
- CLiMP's real vocabulary is only 1,272 types (230 of them proper names), not the reported 3,456. SLING draws on 11,988 types from the Chinese Treebank 9.0. A few classifier-noun pairs (196) had to be written by hand.
- In human validation, 16 native speakers reached 97.1% accuracy with Fleiss' κ = 0.88.
- Of the 18 evaluated models, 10 are monolingual and 8 multilingual (e.g., bert-base-chinese, PanGu-α, CPM, mT5, XLM-R). Monolingual models generally do better, and larger size does not guarantee better scores.
- Model rankings differ between benchmarks: chinese-pert-base is best on CLiMP (81.2%), while bert-base-chinese is best on SLING. Accuracy varies far more across phenomena on SLING (SD 24.1%) than on CLiMP (13.2%).
- Eight baseline paradigms reveal gender and number bias. Models also struggle with ambiguous items such as le, haishi, and wh-phrases.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) — University, Amherst Center, United States · 106 works · 66,692 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Yixiao Song, Kalpesh Krishna, Rajesh Bhatt and Mohit Iyyer. 2022. SLING: Sino Linguistic Evaluation of Large Language Models. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{song2022sling,
  title = {{SLING: Sino Linguistic Evaluation of Large Language Models}},
  author = {Yixiao Song, Kalpesh Krishna, Rajesh Bhatt and Mohit Iyyer},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.305},
  eprint = {2210.11689},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.305}
}
```

- [← previous PropBank Comes of Age—Larger, Smarter, and more Diverse](https://md.research.elamur.ai/free/en/papers/pradhan-2022-propbank-comes-age-larger-smarter/)
- [next → Czech Grammar Error Correction with a Large and Diverse Corpus](https://md.research.elamur.ai/free/en/papers/naplava-2022-czech-grammar-error-correction-large/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

60 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Barbara Abbott. 1993. A pragmatic account of the definiteness effect in existencial sentences. Journal of Pragmatics, 19(1):39–55. [find](https://scholar.google.com/scholar?q=Barbara+Abbott%2E+1993%2E+A+pragmatic+account+of+the+definiteness+effect+in+existencial+sentences%2E+Journal+of+Pragmatics%2C+19%281%29%3A39%E2%80%9355%2E)
2. Márta Abrusán. 2019. Semantic anomaly, pragmatic infelicity, and ungrammaticality. Annual Review of Linguistics, 5:329–351. [find](https://scholar.google.com/scholar?q=M%C3%A1rta+Abrus%C3%A1n%2E+2019%2E+Semantic+anomaly%2C+pragmatic+infelicity%2C+and+ungrammaticality%2E+Annual+Review+of+Linguistics%2C+5%3A329%E2%80%93351%2E)
3. Sigrid Beck. 2006. Intervention effects follow from focus interpretation. Natural Language Semantics, 14(1):1–56. [find](https://scholar.google.com/scholar?q=Sigrid+Beck%2E+2006%2E+Intervention+effects+follow+from+focus+interpretation%2E+Natural+Language+Semantics%2C+14%281%29%3A1%E2%80%9356%2E)
4. Shikha Bordia and Samuel R. Bowman. 2019. [Identifying and reducing gender bias in word-level language models](https://md.research.elamur.ai/free/en/papers/bordia-2019-identifying-reducing-gender-bias-word/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 7–15, Minneapolis, Minnesota. Associati…
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Li Chen. 2012. Chinese polarity items. Ph.D. thesis, City University of Hong Kong. [find](https://scholar.google.com/scholar?q=Li+Chen%2E+2012%2E+Chinese+polarity+items%2E+Ph%2ED%2E+thesis%2C+City+University+of+Hong+Kong%2E)
7. Lisa Lai-Shen Cheng. 1994. Wh-words as polarity items. Chinese Languages and Linguistics, 2:615– 640. [find](https://scholar.google.com/scholar?q=Lisa+Lai%2DShen+Cheng%2E+1994%2E+Wh%2Dwords+as+polarity+items%2E+Chinese+Languages+and+Linguistics%2C+2%3A615%E2%80%93+640%2E)
8. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D%2E+Manning%2E+2019%2E+What+does+BERT+look+at%3F+an+analysis+of+BERT%E2%80%99s+attention%2E+In+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+an)
9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
10. Yiming Cui, Ziqing Yang, and Ting Liu. 2022. Pert: Pre-training bert with permuted language model. arXiv preprint arXiv:2203.06906. — [arXiv:2203.06906](https://arxiv.org/abs/2203.06906) [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Ziqing+Yang%2C+and+Ting+Liu%2E+2022%2E+Pert%3A+Pre%2Dtraining+bert+with+permuted+language+model%2E+arXiv+preprint+arXiv%3A2203%2E06906%2E+%E2%80%94+%5BarXiv%3A2203%2E06906%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2203%2E06906%29)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL-HLT, pages 4171– 4186.
12. Richard Futrell, Ethan Wilcox, Takashi Morita, and Roger Levy. 2018. [RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency](https://md.research.elamur.ai/free/en/papers/futrell-2018-rnns-psycholinguistic-subjects-syntactic-state/). arXiv preprint arXiv:1809.01329.
13. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori… [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Tosin+Adewumi%2C+Karmanya+Aggarwal%2C+Pawan+Sasanka+Ammanamanchi%2C+Anuoluwapo+Aremu%2C+Antoine+Bosselut%2C+Khyathi+Raghavi+Chandu%2C+Miruna%2DAdriana+Clinciu%2C+Dipanjan+Das%2C+Kaustubh+Dhole%2C+Wany)
14. Anastasia Giannakidou, Claudia Maienborn, Klaus von Heusinger, and Paul Portner. 2019. Negative and positive polarity items. Semantics—Sentence and information structure, pages 69–134. [find](https://scholar.google.com/scholar?q=Anastasia+Giannakidou%2C+Claudia+Maienborn%2C+Klaus+von+Heusinger%2C+and+Paul+Portner%2E+2019%2E+Negative+and+positive+polarity+items%2E+Semantics%E2%80%94Sentence+and+information+structure%2C+pages+69%E2%80%93134%2E)
15. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2019. [Colorless green recurrent networks dream hierarchically](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/). Proceedings of the Society for Computation in Linguistics, 2(1):363–364.
16. John Hale. 2001. [A probabilistic earley parser as a psycholinguistic model](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/). In Second meeting of the north american chapter of the association for computational linguistics.
17. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, Chi… [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Percy+Liang%2E+2019%2E+Designing+and+interpreting+probes+with+control+tasks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Internati)
18. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. [Surface form competition: Why the highest probability answer isn’t always right](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, Online and Punta Cana, Dominic…
19. Jianhua Hu and Haihua Pan. 2008. Focus and the basic function of Chinese existential you-sentences. In Existence: Semantics and syntax, pages 133–145. Springer. [find](https://scholar.google.com/scholar?q=Jianhua+Hu+and+Haihua+Pan%2E+2008%2E+Focus+and+the+basic+function+of+Chinese+existential+you%2Dsentences%2E+In+Existence%3A+Semantics+and+syntax%2C+pages+133%E2%80%93145%2E+Springer%2E)
20. Cheng-Teh James Huang, Yen-hui Audrey Li, and Yafei Li. 2009. The syntax of Chinese, volume 10. Cambridge University Press Cambridge. [find](https://scholar.google.com/scholar?q=Cheng%2DTeh+James+Huang%2C+Yen%2Dhui+Audrey+Li%2C+and+Yafei+Li%2E+2009%2E+The+syntax+of+Chinese%2C+volume+10%2E+Cambridge+University+Press+Cambridge%2E)
21. Edward L Keenan. 1987. A semantic definition of “indefinite NP". In Eric J. Reuland and Alice G. B. Ter Meulen, editors, The Representation of (In)Definiteness, pages 286–317. Mit Press. [find](https://scholar.google.com/scholar?q=Edward+L+Keenan%2E+1987%2E+A+semantic+definition+of+%E2%80%9Cindefinite+NP%22%2E+In+Eric+J%2E+Reuland+and+Alice+G%2E+B%2E+Ter+Meulen%2C+editors%2C+The+Representation+of+%28In%29Definiteness%2C+pages+286%E2%80%93317%2E+Mit+Press%2E)
22. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Rebecca+Knowles%2E+2017%2E+Six+challenges+for+neural+machine+translation%2E+In+Proceedings+of+the+First+Workshop+on+Neural+Machine+Translation%2C+pages+28%E2%80%9339%2C+Vancouver%2E+Association+for+Comp)
23. Rajesh Kumar. 2013. The syntax of negation and the licensing of negative polarity items in Hindi. Routledge. [find](https://scholar.google.com/scholar?q=Rajesh+Kumar%2E+2013%2E+The+syntax+of+negation+and+the+licensing+of+negative+polarity+items+in+Hindi%2E+Routledge%2E)
24. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, pages 159–174. [find](https://scholar.google.com/scholar?q=J+Richard+Landis+and+Gary+G+Koch%2E+1977%2E+The+measurement+of+observer+agreement+for+categorical+data%2E+Biometrics%2C+pages+159%E2%80%93174%2E)
25. Jo-Wang Lin. 1998. On existential polarity-wh-0phrases in Chinese. Journal of East Asian Linguistics, 7(3):219–255. [find](https://scholar.google.com/scholar?q=Jo%2DWang+Lin%2E+1998%2E+On+existential+polarity%2Dwh%2D0phrases+in+Chinese%2E+Journal+of+East+Asian+Linguistics%2C+7%283%29%3A219%E2%80%93255%2E)
26. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. [Assessing the ability of lstms to learn syntax-sensitive dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). Transactions of the Association for Computational Linguistics, 4:521–535.
27. Angelina McMillan-Major, Salomey Osei, Juan Diego Rodriguez, Pawan Sasanka Ammanamanchi, Sebastian Gehrmann, and Yacine Jernite. 2021. Reusable templates and guides for documenting datasets and models for natural language processing and generation: A case study of the HuggingFace and GEM data and mo… [find](https://scholar.google.com/scholar?q=Angelina+McMillan%2DMajor%2C+Salomey+Osei%2C+Juan+Diego+Rodriguez%2C+Pawan+Sasanka+Ammanamanchi%2C+Sebastian+Gehrmann%2C+and+Yacine+Jernite%2E+2021%2E+Reusable+templates+and+guides+for+documenting+datasets+and+models)
28. Haihua Pan and Peppina Lee. 2004. The role of pragmatics in interpreting the Chinese perfective markers-guo and-le. Journal of Pragmatics, 36(3):441–466. [find](https://scholar.google.com/scholar?q=Haihua+Pan+and+Peppina+Lee%2E+2004%2E+The+role+of+pragmatics+in+interpreting+the+Chinese+perfective+markers%2Dguo+and%2Dle%2E+Journal+of+Pragmatics%2C+36%283%29%3A441%E2%80%93466%2E)
29. Yip Po-Ching and Don Rimmington. 2015. Chinese: A comprehensive grammar. Routledge. [find](https://scholar.google.com/scholar?q=Yip+Po%2DChing+and+Don+Rimmington%2E+2015%2E+Chinese%3A+A+comprehensive+grammar%2E+Routledge%2E)
30. Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2020. [Masked language model scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712.
31. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. [Ernie: Enhanced representation through knowledge integration](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/). arXiv preprint arXiv:1904.09223.
32. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601.
33. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry,… — [Datasets: A Community Library for Natural Language Processing](https://md.research.elamur.ai/free/en/papers/lhoest-2021-datasets-community-library-natural-language/)
34. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019b. [What do you learn from context? Probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/). In 7th International Conference on Learni…
35. Jing Lin and Anastasia Giannakidou. 2015. No exhaustivity for the mandarin NPI shenme. Unpublished Manuscript. [find](https://scholar.google.com/scholar?q=Jing+Lin+and+Anastasia+Giannakidou%2E+2015%2E+No+exhaustivity+for+the+mandarin+NPI+shenme%2E+Unpublished+Manuscript%2E)
36. Leslie Fu-mei Wang. 2002. From a motion verb to an aspect marker: A study of guo in Mandarin Chinese. Concentric: Studies in Linguistics, 28(2):57–84. [find](https://scholar.google.com/scholar?q=Leslie+Fu%2Dmei+Wang%2E+2002%2E+From+a+motion+verb+to+an+aspect+marker%3A+A+study+of+guo+in+Mandarin+Chinese%2E+Concentric%3A+Studies+in+Linguistics%2C+28%282%29%3A57%E2%80%9384%2E)
37. Ildikó Tóth. 1999. Negative polarity item licensing in Hungarian. Acta Linguistica Hungarica, 46(1):119– 142. [find](https://scholar.google.com/scholar?q=Ildik%C3%B3+T%C3%B3th%2E+1999%2E+Negative+polarity+item+licensing+in+Hungarian%2E+Acta+Linguistica+Hungarica%2C+46%281%29%3A119%E2%80%93+142%2E)
38. Elena Voita and Ivan Titov. 2020. [Information-theoretic probing with minimum description length](https://md.research.elamur.ai/free/en/papers/voita-2020-information-theoretic-probing-minimum-description/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183–196, Online. Association for Computational Linguistics.
39. Lianqing Wang. 1994. Origin and development of classifiers in Chinese. Ph.D. thesis, The Ohio State University. future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. [find](https://scholar.google.com/scholar?q=Lianqing+Wang%2E+1994%2E+Origin+and+development+of+classifiers+in+Chinese%2E+Ph%2ED%2E+thesis%2C+The+Ohio+State+University%2E+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E+Transactions+of+the+Association+for+Computa)
40. Yu-Fang Flora Wang and Miao-Ling Hsieh. 1996. A syntactic study of the Chinese negative polarity item renhe. Cahiers de linguistique-Asie orientale, 25(1):35–62. [find](https://scholar.google.com/scholar?q=Yu%2DFang+Flora+Wang+and+Miao%2DLing+Hsieh%2E+1996%2E+A+syntactic+study+of+the+Chinese+negative+polarity+item+renhe%2E+Cahiers+de+linguistique%2DAsie+orientale%2C+25%281%29%3A35%E2%80%9362%2E)
41. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In NAACL-HLT.
42. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. [Blimp: The benchmark of linguistic minimal pairs for English](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8:377–392.
43. Nianwen Xue, Xiuhong Zhang, Zixin Jiang, Martha Palmer, Fei Xia, Fu-Dong Chiou, and Meiyu Chang. 2016. Chinese Treebank 9.0 LDC2016T13. [find](https://scholar.google.com/scholar?q=Nianwen+Xue%2C+Xiuhong+Zhang%2C+Zixin+Jiang%2C+Martha+Palmer%2C+Fei+Xia%2C+Fu%2DDong+Chiou%2C+and+Meiyu+Chang%2E+2016%2E+Chinese+Treebank+9%2E0+LDC2016T13%2E)
44. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
45. Huiying Wen. 2020. Relative clauses in Mandarin Chinese. Queen Mary’s Occasional Papers Advancing Linguistics (OPAL, no. 46). [find](https://scholar.google.com/scholar?q=Huiying+Wen%2E+2020%2E+Relative+clauses+in+Mandarin+Chinese%2E+Queen+Mary%E2%80%99s+Occasional+Papers+Advancing+Linguistics+%28OPAL%2C+no%2E+46%29%2E)
46. Ethan Wilcox, Peng Qian, Richard Futrell, Miguel Ballesteros, and Roger Levy. 2019. Structural supervision improves learning of non-local grammatical dependencies. In Proceedings of NAACL-HLT, pages 3302–3312. [find](https://scholar.google.com/scholar?q=Ethan+Wilcox%2C+Peng+Qian%2C+Richard+Futrell%2C+Miguel+Ballesteros%2C+and+Roger+Levy%2E+2019%2E+Structural+supervision+improves+learning+of+non%2Dlocal+grammatical+dependencies%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+3)
47. Ying Wu. 2010. “haishi” de duoyixing yu xide nandu [the polysemy and the acquisition difficulty of haishi]. TCSOL Studies, pages 41–48. [find](https://scholar.google.com/scholar?q=Ying+Wu%2E+2010%2E+%E2%80%9Chaishi%E2%80%9D+de+duoyixing+yu+xide+nandu+%5Bthe+polysemy+and+the+acquisition+difficulty+of+haishi%5D%2E+TCSOL+Studies%2C+pages+41%E2%80%9348%2E)
48. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+%C5%81uk)
49. Fei Xia. 2000. The segmentation guidelines for the penn chinese treebank 3.0. IRCS Technical Reports Series. 37. [find](https://scholar.google.com/scholar?q=Fei+Xia%2E+2000%2E+The+segmentation+guidelines+for+the+penn+chinese+treebank+3%2E0%2E+IRCS+Technical+Reports+Series%2E+37%2E)
50. Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. [CLiMP: A benchmark for Chinese language model evaluation](https://md.research.elamur.ai/free/en/papers/xiang-2021-climp-benchmark-chinese-language-model/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2784–2790, Online. Association fo…
51. Keiko Yoshimura. 2007. Focus and polarity: even and only in Japanese. The University of Chicago. [find](https://scholar.google.com/scholar?q=Keiko+Yoshimura%2E+2007%2E+Focus+and+polarity%3A+even+and+only+in+Japanese%2E+The+University+of+Chicago%2E)
52. Jeremy Zehr and Florian Schwarz. 2018. Penncontroller for internet based experiments. [find](https://scholar.google.com/scholar?q=Jeremy+Zehr+and+Florian+Schwarz%2E+2018%2E+Penncontroller+for+internet+based+experiments%2E)
53. Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, Chen Li, Ziyan Gong, Yifan Yao, Xinjing Huang, Jun Wang, Jianfeng Yu, Qi Guo, Yue Yu, Yan Zhang, Jin Wang, Hengtao Tao, Dasen Yan, Zexuan Yi, Fang Peng, Fangqing Jiang, Han Zhang,… — [PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation](https://md.research.elamur.ai/free/en/papers/zeng-2021-pangu-alpha-large-scale/)
54. Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, et al. 2021a. [CPM: A large-scale generative Chinese pre-trained language model](https://md.research.elamur.ai/free/en/papers/zhang-2020-cpm-large-scale-generative/). AI Open, 2:93–99.
55. Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong Wang, and Ming Zhou. 2021b. Mengzi: Towards lightweight yet ingenious pre-trained models for chinese. arXiv preprint arXiv:2110.06696. — [arXiv:2110.06696](https://arxiv.org/abs/2110.06696) [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Hanqing+Zhang%2C+Keming+Chen%2C+Yuhang+Guo%2C+Jingyun+Hua%2C+Yulong+Wang%2C+and+Ming+Zhou%2E+2021b%2E+Mengzi%3A+Towards+lightweight+yet+ingenious+pre%2Dtrained+models+for+chinese%2E+arXiv+preprint+arXiv%3A)
56. Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, and Xiaoyong Du. 2019. [UER: An open-source toolkit for pre-training models](https://md.research.elamur.ai/free/en/papers/zhao-2019-uer-open-source-toolkit-pre/). EMNLP-IJCNLP.
57. Min Zhou and Jingquan Han. 2012. A phase-based approach to the derivation of relative constructions in Mandarin Chinese. Journal of Foreign Languages, 3(002). [find](https://scholar.google.com/scholar?q=Min+Zhou+and+Jingquan+Han%2E+2012%2E+A+phase%2Dbased+approach+to+the+derivation+of+relative+constructions+in+Mandarin+Chinese%2E+Journal+of+Foreign+Languages%2C+3%28002%29%2E)
58. Alessandro Zucchi. 1995. The ingredients of definiteness and the definiteness effect. Natural Language Semantics, 3(1):33–78. [find](https://scholar.google.com/scholar?q=Alessandro+Zucchi%2E+1995%2E+The+ingredients+of+definiteness+and+the+definiteness+effect%2E+Natural+Language+Semantics%2C+3%281%29%3A33%E2%80%9378%2E)
59. Liejiong Xu. 1995. Definiteness effects on Chinese word order. Cahiers de linguistique-Asie orientale, 24(1):29–48. [find](https://scholar.google.com/scholar?q=Liejiong+Xu%2E+1995%2E+Definiteness+effects+on+Chinese+word+order%2E+Cahiers+de+linguistique%2DAsie+orientale%2C+24%281%29%3A29%E2%80%9348%2E)
60. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Aditya+Barua%2C+Noah+Constant%2C+Rami+Al%2DRfou%2C+Sharan+Narang%2C+Mihir+Kale%2C+Adam+Roberts%2C+and+Colin+Raffel%2E+2022%2E+ByT5%3A+Towards+a+token%2Dfree)

*Updated 29 October 2026*