---
title: "ChID: A Large-scale Chinese IDiom Dataset for Cloze Test (2019)"
description: "To fill this gap, the authors build ChID, a large corpus drawn from news, novels and essays. 23.3 points. 39 references, cited by 62 papers here."
authors: ["Chujie Zheng", "Minlie Huang", "Aixin Sun"]
year: 2019
venue: "arXiv:1906.01265"
source: "https://arxiv.org/abs/1906.01265"
doi: "10.48550/arxiv.1906.01265"
arxiv: "1906.01265"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zheng-2019-chid-large-scale-chinese/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zheng-2019-chid-large-scale-chinese/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / ChID: A Large-scale Chinese IDiom Dataset for Cloze Test

# ChID: A Large-scale Chinese IDiom Dataset for Cloze Test

**Authors:** [Chujie Zheng](https://md.research.elamur.ai/free/en/authors/chujie-zheng/), [Minlie Huang](https://md.research.elamur.ai/free/en/authors/minlie-huang/) and [Aixin Sun](https://md.research.elamur.ai/free/en/authors/aixin-sun/) · **Year:** 2019 · **Venue:** [arXiv:1906.01265](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2019](https://md.research.elamur.ai/free/en/venues/acl/2019/)) · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1906.01265) · [PDF](https://arxiv.org/pdf/1906.01265) · [DOI 10.48550/arxiv.1906.01265](https://doi.org/10.48550/arxiv.1906.01265) · [arXiv 1906.01265](https://arxiv.org/abs/1906.01265) · [OpenAlex](https://openalex.org/W2949884065)

**Citations:** 57 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/zheng-2019-chid-large-scale-chinese/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Chinese cloze-style reading comprehension has had few benchmarks, and none of them target chengyu (成语). These four-character idioms are often non-compositional and metaphorical, and many have near-synonyms that differ only subtly in usage. To fill this gap, the authors build ChID, a large corpus drawn from news, novels and essays. In each passage, idioms are masked and must be recovered from a list of seven candidates. The authors design candidate lists with embedding-based similarity, which lets them control difficulty by mixing near-synonymous distractors with unrelated ones. They benchmark three cloze models (LM, AR, SAR) and compare ways of representing idioms. The best system, AR, still trails human readers by roughly 15–23 accuracy points. The gap is largest on out-of-domain passages, which contain more rare idioms. Treating each idiom as a single pretrained token works clearly better than building its meaning from its characters. ChID therefore offers a demanding benchmark for figurative-language understanding in Chinese machine reading.

## Key points
- **Dataset size and coverage:** ChID contains 581K passages and 729K blanks, covering 3,848 idioms from three domains (news, novels, essays).
- **Candidate design:** In the main setting, each blank offers the gold idiom, three similar idioms and three random ones. The extra test sets Ran (only dissimilar distractors) and Sim (distractors from the top-10 similar idioms) show that similar distractors make the task harder for both humans and models.
- **Near-synonym study:** Annotators examined 200 sampled idioms (Fleiss' kappa 0.479). About 89.5% had at least one near-synonym, and 23% had four or more.
- **Human vs. model accuracy:** AR, the best model, scores 72.4 on Test and 62.9 on Out. Humans score 87.1 and 86.2 on the same sets, a gap of 14.6 to 23.3 points.
- **Idiom representation:** Using a whole-idiom embedding beats both averaging character embeddings and feeding the characters through an MLP, across all models. For example, AR on Test scores 72.4 with idiom embeddings, 66.5 with the MLP and 63.8 with averaged characters (p \< 0.01).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 137 works · 83,262 citations
- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 73 works · 24,390 citations
- [Beijing National Research Center for Information Science and Technology](https://md.research.elamur.ai/free/en/labs/beijing-national-research-center-for-information-science-and-technology/) — Research institute, Beijing, China · 21 works · 5,843 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 260 works

🧭 Topics (1): [🧷 Idioms & multiword expressions](https://md.research.elamur.ai/free/en/topics/multiword-expressions/) · 155 works

## Cite

Cite as: Chujie Zheng, Minlie Huang and Aixin Sun. 2019. ChID: A Large-scale Chinese IDiom Dataset for Cloze Test. arXiv:1906.01265.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zheng2019chid,
  title = {{ChID: A Large-scale Chinese IDiom Dataset for Cloze Test}},
  author = {Chujie Zheng, Minlie Huang and Aixin Sun},
  year = {2019},
  doi = {10.48550/arxiv.1906.01265},
  eprint = {1906.01265},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1906.01265}
}
```

- [← previous On Measuring Gender Bias in Translation of Gender-neutral Pronouns](https://md.research.elamur.ai/free/en/papers/cho-2019-measuring-gender-bias-translation-gender/)
- [next → Encode, Tag, Realize: High-Precision Text Editing](https://md.research.elamur.ai/free/en/papers/malmi-2019-encode-tag-realize-high-precision/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

39 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16)… [find](https://scholar.google.com/scholar?q=Mart%C4%B1%CC%81n+Abadi%2C+Paul+Barham%2C+Jianmin+Chen%2C+Zhifeng+Chen%2C+Andy+Davis%2C+Jeffrey+Dean%2C+Matthieu+Devin%2C+Sanjay+Ghemawat%2C+Geoffrey+Irving%2C+Michael+Isard%2C+et+al%2E+2016%2E+Tensorflow%3A+A+system+for+large%2Dscale+mac)
2. Dimitra Anastasiou. 2010. Idiom treatment experiments in machine translation. Cambridge Scholars Publishing. [find](https://scholar.google.com/scholar?q=Dimitra+Anastasiou%2E+2010%2E+Idiom+treatment+experiments+in+machine+translation%2E+Cambridge+Scholars+Publishing%2E)
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In International Conference on Learning Representations.
4. Christopher M Bishop et al. 1995. Neural networks for pattern recognition. Oxford university press. [find](https://scholar.google.com/scholar?q=Christopher+M+Bishop+et+al%2E+1995%2E+Neural+networks+for+pattern+recognition%2E+Oxford+university+press%2E)
5. Cristina Cacciari and Patrizia Tabossi. 2014. Idioms: Processing, structure, and interpretation. Psychology Press. [find](https://scholar.google.com/scholar?q=Cristina+Cacciari+and+Patrizia+Tabossi%2E+2014%2E+Idioms%3A+Processing%2C+structure%2C+and+interpretation%2E+Psychology+Press%2E)
6. Fabienne Cap, Manju Nirmal, Marion Weller, and Sabine Schulte Im Walde. 2015. How to account for idiomatic german support verb constructions in statistical machine translation. In Proceedings of the 11th Workshop on Multiword Expressions, pages 19–28. [find](https://scholar.google.com/scholar?q=Fabienne+Cap%2C+Manju+Nirmal%2C+Marion+Weller%2C+and+Sabine+Schulte+Im+Walde%2E+2015%2E+How+to+account+for+idiomatic+german+support+verb+constructions+in+statistical+machine+translation%2E+In+Proceedings+of+the+1)
7. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. [A thorough examination of the cnn/daily mail reading comprehension task](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 2358–2367.
8. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-overattention neural networks for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 593–602. [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Zhipeng+Chen%2C+Si+Wei%2C+Shijin+Wang%2C+Ting+Liu%2C+and+Guoping+Hu%2E+2017%2E+Attention%2Doverattention+neural+networks+for+reading+comprehension%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Assoc)
9. Yiming Cui, Ting Liu, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2018. Dataset for the 7 Jon Jonz. 1991. Cloze item types and second language Our dataset is available at [https://github.com/chujiezheng/ChID-Dataset](https://github.com/chujiezheng/ChID-Dataset). comprehension. Language testing, 8(1):1–22. first evaluation on chinese ma… [↗](https://github.com/chujiezheng/ChID-Dataset)
10. Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. 2016. Consensus attention-based neural networks for chinese reading comprehension. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1777–1786. [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Ting+Liu%2C+Zhipeng+Chen%2C+Shijin+Wang%2C+and+Guoping+Hu%2E+2016%2E+Consensus+attention%2Dbased+neural+networks+for+chinese+reading+comprehension%2E+In+Proceedings+of+COLING+2016%2C+the+26th+Internationa)
11. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1832–1846. [find](https://scholar.google.com/scholar?q=Bhuwan+Dhingra%2C+Hanxiao+Liu%2C+Zhilin+Yang%2C+William+Cohen%2C+and+Ruslan+Salakhutdinov%2E+2017%2E+Gated%2Dattention+readers+for+text+comprehension%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+fo)
12. Terrence L Fine. 1999. Feedforward Neural Network Methodology. Springer Science & Business Media. [find](https://scholar.google.com/scholar?q=Terrence+L+Fine%2E+1999%2E+Feedforward+Neural+Network+Methodology%2E+Springer+Science+%26+Business+Media%2E)
13. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [find](https://scholar.google.com/scholar?q=Joseph+L+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+bulletin%2C+76%285%29%3A378%2E)
14. Sandra S Fotos. 1991. The cloze test as an integrative measure of efl proficiency: A substitute for essays on college entrance examinations? Language learning, 41(3):313–336. [find](https://scholar.google.com/scholar?q=Sandra+S+Fotos%2E+1991%2E+The+cloze+test+as+an+integrative+measure+of+efl+proficiency%3A+A+substitute+for+essays+on+college+entrance+examinations%3F+Language+learning%2C+41%283%29%3A313%E2%80%93336%2E)
15. Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et al. 2018. [Dureader: a chinese machine reading comprehension dataset from real-world applications](https://md.research.elamur.ai/free/en/papers/he-2017-dureader-chinese-machine-reading/). In Proceedings of the Workshop on Machine Reading for Question Answering, pages 37–46.
16. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In Advances in Neural Information Processing Systems, pages 1693– 1701.
17. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. [The goldilocks principle: Reading children’s books with explicit memory representations](https://md.research.elamur.ai/free/en/papers/hill-2015-goldilocks-principle-reading-children-s/). In International Conference on Learning Representations.
18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
19. Ray Jackendoff and Ray S Jackendoff. 2002. Foundations of language: Brain, meaning, grammar, evolution. Oxford University Press, USA. [find](https://scholar.google.com/scholar?q=Ray+Jackendoff+and+Ray+S+Jackendoff%2E+2002%2E+Foundations+of+language%3A+Brain%2C+meaning%2C+grammar%2C+evolution%2E+Oxford+University+Press%2C+USA%2E)
20. Zhiying Jiang, Boliang Zhang, Lifu Huang, and Heng Ji. 2018. Chengyu cloze test. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 154–158. [find](https://scholar.google.com/scholar?q=Zhiying+Jiang%2C+Boliang+Zhang%2C+Lifu+Huang%2C+and+Heng+Ji%2E+2018%2E+Chengyu+cloze+test%2E+In+Proceedings+of+the+Thirteenth+Workshop+on+Innovative+Use+of+NLP+for+Building+Educational+Applications%2C+pages+154%E2%80%93158)
21. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+International+Conference+on+Learning+Representations%2E)
22. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [Race: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794.
23. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). In International Conference on Learning Representations Workshop.
24. Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Michael+Roth%2C+Annie+Louis%2C+Nathanael+Chambers%2C+and+James+Allen%2E+2017%2E)
25. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46–51. [find](https://scholar.google.com/scholar?q=Lsdsem+2017+shared+task%3A+The+story+cloze+test%2E+In+Proceedings+of+the+2nd+Workshop+on+Linking+Models+of+Lexical%2C+Sentential+and+Discourse%2Dlevel+Semantics%2C+pages+46%E2%80%9351%2E)
26. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. [Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). arXiv preprint arXiv:1611.09268.
27. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. [Who did what: A large-scale person-centered cloze dataset](https://md.research.elamur.ai/free/en/papers/onishi-2016-who-did-what-large-scale/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2230– 2235.
28. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. [The lambada dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). In Proceedings of the 54th Annual Meeting of the Association f…
29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
30. Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. [Coqa: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). arXiv preprint arXiv:1808.07042.
31. Giancarlo D. Salton, Robert J. Ross, and John D. Kelleher. 2014. Evaluation of a substitution method for idiom transformation in statistical machine translation. In MWE@EACL. [find](https://scholar.google.com/scholar?q=Giancarlo+D%2E+Salton%2C+Robert+J%2E+Ross%2C+and+John+D%2E+Kelleher%2E+2014%2E+Evaluation+of+a+substitution+method+for+idiom+transformation+in+statistical+machine+translation%2E+In+MWE%40EACL%2E)
32. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. [Bidirectional attention flow for machine comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/). In International Conference on Learning Representations.
33. Yutong Shao, Rico Sennrich, Bonnie L. Webber, and Federico Fancellu. 2017. Evaluating machine translation performance on chinese idioms with a blacklist method. In Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Yutong+Shao%2C+Rico+Sennrich%2C+Bonnie+L%2E+Webber%2C+and+Federico+Fancellu%2E+2017%2E+Evaluating+machine+translation+performance+on+chinese+idioms+with+a+blacklist+method%2E+In+Language+Resources+and+Evaluation+Co)
34. Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. [Directional skip-gram: Explicitly distinguishing left and right context for word embeddings](https://md.research.elamur.ai/free/en/papers/song-2018-directional-skip-gram-explicitly-distinguishing/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol…
35. M Sun, J Li, Z Guo, Z Yu, Y Zheng, X Si, and Z Liu. 2016. Thuctc: an efficient chinese text classifier. GitHub Repository. [find](https://scholar.google.com/scholar?q=M+Sun%2C+J+Li%2C+Z+Guo%2C+Z+Yu%2C+Y+Zheng%2C+X+Si%2C+and+Z+Liu%2E+2016%2E+Thuctc%3A+an+efficient+chinese+text+classifier%2E+GitHub+Repository%2E)
36. Wilson L Taylor. 1953. “cloze procedure”: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433. [find](https://scholar.google.com/scholar?q=Wilson+L+Taylor%2E+1953%2E+%E2%80%9Ccloze+procedure%E2%80%9D%3A+A+new+tool+for+measuring+readability%2E+Journalism+Bulletin%2C+30%284%29%3A415%E2%80%93433%2E)
37. Annie Tremblay. 2011. Proficiency assessment standards in second language acquisition research:“clozing” the gap. Studies in Second Language Acquisition, 33(3):339–372. [find](https://scholar.google.com/scholar?q=Annie+Tremblay%2E+2011%2E+Proficiency+assessment+standards+in+second+language+acquisition+research%3A%E2%80%9Cclozing%E2%80%9D+the+gap%2E+Studies+in+Second+Language+Acquisition%2C+33%283%29%3A339%E2%80%93372%2E)
38. Alison Wray. 2002. Formulaic language and the lexicon. [find](https://scholar.google.com/scholar?q=Alison+Wray%2E+2002%2E+Formulaic+language+and+the+lexicon%2E)
39. Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy. 2018. Large-scale cloze test dataset created by teachers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2344–2356. [find](https://scholar.google.com/scholar?q=Qizhe+Xie%2C+Guokun+Lai%2C+Zihang+Dai%2C+and+Eduard+Hovy%2E+2018%2E+Large%2Dscale+cloze+test+dataset+created+by+teachers%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C)

*Updated 24 October 2026*