---
title: "CMATH — paper by Wei et al. (2023) — summary and ideas"
description: "The authors, from Xiaomi AI Lab, introduce CMATH, a test-only collection of about… 60% accuracy. 24 references, cited by 57 papers here."
authors: ["Tianwen Wei", "Jian Luan", "Wei Liu", "Shuang Dong", "Bin Wang"]
year: 2023
venue: "arXiv:2306.16636"
source: "https://arxiv.org/abs/2306.16636"
doi: "10.48550/arxiv.2306.16636"
arxiv: "2306.16636"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wei-2023-cmath-language-model-pass/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wei-2023-cmath-language-model-pass/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?

# CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?

**Authors:** [Tianwen Wei](https://md.research.elamur.ai/free/en/authors/tianwen-wei/), Jian Luan, [Wei Liu](https://md.research.elamur.ai/free/en/authors/wei-liu/), Shuang Dong and [Bin Wang](https://md.research.elamur.ai/free/en/authors/bin-wang/) · **Year:** 2023 · **Venue:** arXiv:2306.16636 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.16636) · [PDF](https://arxiv.org/pdf/2306.16636) · [DOI 10.48550/arxiv.2306.16636](https://doi.org/10.48550/arxiv.2306.16636) · [arXiv 2306.16636](https://arxiv.org/abs/2306.16636) · [OpenAlex](https://openalex.org/W4382765605)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/wei-2023-cmath-language-model-pass/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

Existing math benchmarks for large language models are mostly English, often pitched at college level, and produce scores that are hard for non-specialists to interpret. The authors, from Xiaomi AI Lab, introduce CMATH, a test-only collection of about 1.7k Chinese elementary school word problems taken from freely available workbooks and exams. Problems with images were removed, and the remaining text was deduplicated, cleaned and checked by hand. Each item carries a school grade (1–6), a numeric answer, a manually counted number of reasoning steps and an automatically computed digit count. With these labels, results can be reported in school terms, for example the highest grade a model reliably passes. The authors ran ten Chinese-capable chat and foundation models zero-shot, and only GPT-4 reached the 60% pass mark in every grade. A second test added irrelevant numeric details to problems. Every model except GPT-4 lost much of its accuracy. CMATH gives an easy-to-read measure of arithmetic and reasoning ability, and of robustness, for Chinese LLMs.

## Key points

- **Dataset size by grade:** grades 1–6 contain 254, 353, 337, 220, 227 and 298 problems. Average length, step count and digit count generally rise with grade (steps go from 1.3 to 3.0).
- **Models tested:** GPT-4, ChatGPT (gpt-3.5-turbo), Chinese-Alpaca-13B/33B, MOSS-16B, Ziya-LLaMA-13B, RWKV-7B, Baichuan-7B, ChatGLM-6B and ChatGLM2-6B. All were evaluated zero-shot, with no extra prompting.
- **Main result:** GPT-4 was the only model to reach at least 60% accuracy across all six grades. The other models failed at various grade levels.
- **Distractor set:** a separate set of 60 problems (10 per grade), each extended with 1–5 inserted irrelevant details containing one number apiece. ChatGPT lost roughly 30% accuracy with only two distractors, while GPT-4 declined only slightly.
- **Advantage over earlier Chinese datasets:** Math23K and APE210K lack fine-grained labels such as grade or step count. AGI-Eval and C-Eval contain much harder material, ranging from middle school to college level.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 140 works · [➗ Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) · 21 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong and Bin Wang. 2023. CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?. arXiv:2306.16636.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wei2023cmath,
  title = {{CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?}},
  author = {Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong and Bin Wang},
  year = {2023},
  doi = {10.48550/arxiv.2306.16636},
  eprint = {2306.16636},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.16636}
}
```

- [← previous FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models](https://md.research.elamur.ai/free/en/papers/palta-2023-fork-bite-sized-test-set/)
- [next → Dynamic Routing Transformer Network for Multimodal Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/tian-2023-dynamic-routing-transformer-network-multimodal/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

24 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. [Mathqa: Towards interpretable math word problem solving with operation-based formalisms](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/).
2. Baichuan-7B. — [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/) [https://github.com/baichuan-inc/baichuan-7B/blob/](https://github.com/baichuan-inc/baichuan-7B/blob/) main/README_EN.md . Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.… [↗](https://github.com/baichuan-inc/baichuan-7B/blob/)
3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/).
4. Yiming Cui, Ziqing Yang, and Xin Yao. 2023. [Efficient and effective text encoding for chinese llama and alpaca](https://md.research.elamur.ai/free/en/papers/cui-2023-efficient-effective-text-encoding-chinese/). arXiv preprint arXiv:2304.08177.
5. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. [GLM: general language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). pages 320–335.
6. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/).
7. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/).
8. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/).
9. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. [Ceval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). arXiv preprint arXiv:2305.08322.
10. IDEA-CCNL. 2023. Idea-ccnl/ziya-llama-13b-v1. [https://huggingface.co/IDEA-CCNL/](https://huggingface.co/IDEA-CCNL/) Ziya-LLaMA-13B-v1/blob/main/README.md . [↗](https://huggingface.co/IDEA-CCNL/)
11. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2023%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E)
12. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
13. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
14. RWKV-4-raven. [https://huggingface.co/BlinkDL/rwkv-4-raven](https://huggingface.co/BlinkDL/rwkv-4-raven) . [↗](https://huggingface.co/BlinkDL/rwkv-4-raven)
15. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito,… [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Eric+Alcaide%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Samuel+Arcadinho%2C+Huanqi+Cao%2C+Xin+Cheng%2C+Michael+Chung%2C+Matteo+Grella%2C+Kranthi+Kiran+GV%2C+Xuzheng+He%2C+Haowen+Hou%2C+Przemyslaw+Kazienko%2C+Jan+Kocon%2C+J)
16. Tianxiang Sun and Xipeng Qiu. 2023. MOSS. Github. [https://github.com/OpenLMLab/MOSS/](https://github.com/OpenLMLab/MOSS/) blob/main/README_en.md . [↗](https://github.com/OpenLMLab/MOSS/)
17. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/).
18. THUDM. 2023a. ChatGLM-6B. [https://github.com/THUDM/ChatGLM-6B/blob/main/README_en.md](https://github.com/THUDM/ChatGLM-6B/blob/main/README_en.md) . [↗](https://github.com/THUDM/ChatGLM-6B/blob/main/README_en.md)
19. ChatGLM2-6B. [https://github.com/THUDM/ChatGLM2-6B/blob/main/](https://github.com/THUDM/ChatGLM2-6B/blob/main/) README_EN.md . [↗](https://github.com/THUDM/ChatGLM2-6B/blob/main/)
20. Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 845–854, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yan+Wang%2C+Xiaojiang+Liu%2C+and+Shuming+Shi%2E+2017%2E+Deep+neural+solver+for+math+word+problems%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+845%E2%80%93854%2C+Cop)
21. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/).
22. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. [GLM-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). arXiv preprint arXiv:2210.02414.
23. Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020. Ape210k: A large-scale and template-rich dataset of math word problems. [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Mingyue+Shang%2C+Yang+Liu%2C+Liang+Wang%2C+and+Jingming+Liu%2E+2020%2E+Ape210k%3A+A+large%2Dscale+and+template%2Drich+dataset+of+math+word+problems%2E)
24. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/).

*Updated 24 October 2026*