--- title: "SuperCLUE-Math6 — paper by Xu et al. (2024)" description: "The authors also propose a scoring scheme that gives more weight to problems needing… 98% accuracy. 7 references, cited by 4 papers here." authors: ["Liang Xu", "Hang Xue", "Lei Zhu", "Kangkang Zhao"] year: 2024 venue: "arXiv:2401.11819" source: "https://arxiv.org/abs/2401.11819" doi: "10.48550/arxiv.2401.11819" arxiv: "2401.11819" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/xu-2024-superclue-math6-graded-multi/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2024-superclue-math6-graded-multi/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) / SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese # SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese **Authors:** [Liang Xu](https://md.research.elamur.ai/free/en/authors/liang-xu/), [Hang Xue](https://md.research.elamur.ai/free/en/authors/hang-xue/), [Lei Zhu](https://md.research.elamur.ai/free/en/authors/lei-zhu/) and [Kangkang Zhao](https://md.research.elamur.ai/free/en/authors/kangkang-zhao/) · **Year:** 2024 · **Venue:** arXiv:2401.11819 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2401.11819) · [PDF](https://arxiv.org/pdf/2401.11819) · [DOI 10.48550/arxiv.2401.11819](https://doi.org/10.48550/arxiv.2401.11819) · [arXiv 2401.11819](https://arxiv.org/abs/2401.11819) · [OpenAlex](https://openalex.org/W4391158660) **Citations:** 2 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/xu-2024-superclue-math6-graded-multi/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Popular math benchmarks such as GSM8K are English-only and do not probe multi-step or sustained inference well. To address this for Chinese LLMs, the SuperCLUE team built SuperCLUE-Math6 (SC-Math6). It contains 1072 grade-school word problems, drawn from Chinese exams and textbooks and manually rewritten to be unique. Each problem has a human-written natural-language solution, an annotated count of reasoning steps, and a follow-up question that builds on it, for 2144 items in total. Annotators solved every item themselves to check it. The authors also propose a scoring scheme that gives more weight to problems needing more steps. This step-weighted score is averaged equally with plain accuracy, and the result is mapped to five reasoning levels. Testing 13 models showed a clear tiering, with GPT-4 variants at the top. Every model scored lower on follow-up questions than on the initial ones. SC-Math6 matters because it gives Chinese LLMs a harder, interaction-aware math testbed whose results are easier to interpret than a single accuracy number. ## Key points - **Dataset design:** - The dataset has 1072 problem/follow-up pairs (2144 questions). - Steps are distributed roughly as follows: 15–20% need 1 step, 15–20% need 2, 45–50% need 3, and 5–10% need 4–5. - A spot check found 1 questionable pair out of 50, giving about 98% accuracy. - **Scoring method:** - The Comprehensive Score is an equal mix of two parts. The first is a Reasoning Steps Score, weighted by step count. The second is an Overall Accuracy Score, which averages per-question accuracy with strict pair accuracy (both turns must be correct). - Models whose scores fall within 5 points of each other share a level. - **Results:** - GPT-4-1106-Preview ranked first with 90.71. GPT-4, Ernie-bot 4.0, GLM-4 and Xinghuo 3.5 also reached Level 5. - Chinese-Alpaca2-13B scored lowest at 20.55. - **Follow-up drop:** - GPT-4-1106-Preview fell from 95.43% to 89.37% on follow-up questions. - Xinghuo 3.0 showed the largest drop, losing about 45 points. - **Other findings:** - The average model score was 55.34 on SC-Math6, compared with 59.21 on GSM8K. - Higher scores were associated with better compliance with output-format instructions and, in some cases, with longer answers. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 189 works · [➗ Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) · 52 works 🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works ## Cite Cite as: Liang Xu, Hang Xue, Lei Zhu and Kangkang Zhao. 2024. SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese. arXiv:2401.11819. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{xu2024superclue, title = {{SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese}}, author = {Liang Xu, Hang Xue, Lei Zhu and Kangkang Zhao}, year = {2024}, doi = {10.48550/arxiv.2401.11819}, eprint = {2401.11819}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2401.11819} } ``` - [← previous SMASH at StanceEval 2024: Prompt Engineering LLMs for Arabic Stance Detection](https://md.research.elamur.ai/free/en/papers/hariri-2024-smash-stanceeval-2024-prompt-engineering/) - [next → The Qiyas Benchmark: Measuring ChatGPT Mathematical and Language Understanding in Arabic](https://md.research.elamur.ai/free/en/papers/khalifa-2024-qiyas-benchmark-measuring-chatgpt-mathematical/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) ## References 7 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. OpenAI. 2023. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774. 2. Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and others, 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv:2110.14168. 3. Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob, 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv:2103.03874. 4. Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and others, 2023. [Judging LLM-as-a-judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv:2306.05685. 5. Li, Xuechen and Zhang, Tianyi and Dubois, Yann and Taori, Rohan and Gulrajani, Ishaan and Guestrin, Carlos and Liang, Percy and Hashimoto, Tatsunori B, 2023. Alpacaeval: An automatic evaluator of instruction-following models. GitHub repository. [find](https://scholar.google.com/scholar?q=Li%2C+Xuechen+and+Zhang%2C+Tianyi+and+Dubois%2C+Yann+and+Taori%2C+Rohan+and+Gulrajani%2C+Ishaan+and+Guestrin%2C+Carlos+and+Liang%2C+Percy+and+Hashimoto%2C+Tatsunori+B%2C+2023%2E+Alpacaeval%3A+An+automatic+evaluator+of+inst) 6. Keisuke Sakaguchi and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi, 2019. An Adversarial Winograd Schema Challenge at Scale. arXiv:1907.10641. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi+and+Ronan+Le+Bras+and+Chandra+Bhagavatula+and+Yejin+Choi%2C+2019%2E+An+Adversarial+Winograd+Schema+Challenge+at+Scale%2E+arXiv%3A1907%2E10641%2E) 7. Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord, 2018. [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv:1803.05457. *Figure 5: Relation between reasoning steps and Mean Accuracy* *Updated 31 October 2026*