---
title: "Evaluating the Capability of Large-scale Language Models on"
description: "Earlier English GEC studies found that LLMs trail specialized models. 18 references, cited by 12 papers here."
authors: ["Fanyi Qu", "Chenming Tang", "Yunfang Wu"]
year: 2023
venue: "arXiv:2307.03972"
source: "https://arxiv.org/abs/2307.03972"
doi: "10.48550/arxiv.2307.03972"
arxiv: "2307.03972"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qu-2023-evaluating-capability-large-scale-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qu-2023-evaluating-capability-large-scale-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Evaluating the Capability of Large-scale Language Models on Chinese Grammatical Error Correction Task

# Evaluating the Capability of Large-scale Language Models on Chinese Grammatical Error Correction Task

**Authors:** Fanyi Qu, [Chenming Tang](https://md.research.elamur.ai/free/en/authors/chenming-tang/) and [Yunfang Wu](https://md.research.elamur.ai/free/en/authors/yunfang-wu/) · **Year:** 2023 · **Venue:** arXiv:2307.03972 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.03972) · [PDF](https://arxiv.org/pdf/2307.03972) · [DOI 10.48550/arxiv.2307.03972](https://doi.org/10.48550/arxiv.2307.03972) · [arXiv 2307.03972](https://arxiv.org/abs/2307.03972) · [OpenAlex](https://openalex.org/W4383987527)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/qu-2023-evaluating-capability-large-scale-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This report asks whether instruction-following large language models can handle Chinese grammatical error correction (CGEC) as well as dedicated systems. Earlier English GEC studies found that LLMs trail specialized models. The authors prompted 12 LLMs on four CGEC benchmarks. The models included the GPT family, Doubao, DeepSeek, ChatGLM/GLM, LLaMA and Qwen, spanning open 7B models and commercial APIs. Two benchmarks contain errors by learners of Chinese (NLPCC-2018, MuCGEC), and two come from native-speaker language exams (FCGEC, NaCGEC). Outputs were scored with precision, recall and F0.5 at both character and word level.

No LLM reached the previous SOTA. The authors attribute this mainly to over-correction, where models rewrite text that was already acceptable. Results also depended on the data source: most models did better on learner errors, which are mostly lexical confusions, than on the more structural errors in exam data. Reasoning-oriented models generally failed to beat cheaper general-purpose models. The findings suggest that practical CGEC deployments should prefer general-purpose LLMs, and that over-correction is the key problem to solve.

## Key points
- **Data:** NLPCC-2018 test set (2000 sentences), MuCGEC validation (1136), FCGEC validation (2000) and NaCGEC validation (500).
- **Metrics:** Word-level scores used the official M2 scorer with PKUNLP segmentation, and char-level scores used ChERRANT. Char-level scores came out notably lower than word-level ones, which the authors could not explain.
- **Strongest models:** Doubao-1.5-pro and DeepSeek-V3 led. For example, the best char-level F0.5 on NaCGEC was 38.25 (Doubao), against a SOTA of 49.78. The best word-level F0.5 on NLPCC was 40.89 (DeepSeek-V3), against a SOTA of 47.80.
- **Reasoning vs. general models:** o3-mini and DeepSeek-R1 largely failed to surpass GPT-4o-mini and DeepSeek-V3. QwQ-32B beat Qwen-2.5-7B-Instruct but needed about 64 GPU-hours versus roughly 1.
- **Other trends:** Newer and larger models scored higher; LLaMA-3.1-8B, for instance, beat LLaMA-7B on every dataset. The strong exam-data results of some models may reflect training-data leakage.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🖍️ Grammatical error correction & learner language](https://md.research.elamur.ai/free/en/topics/grammatical-error-correction/) · 156 works

## Cite

Cite as: Fanyi Qu, Chenming Tang and Yunfang Wu. 2023. Evaluating the Capability of Large-scale Language Models on Chinese Grammatical Error Correction Task. arXiv:2307.03972.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{qu2023evaluating,
  title = {{Evaluating the Capability of Large-scale Language Models on Chinese Grammatical Error Correction Task}},
  author = {Fanyi Qu, Chenming Tang and Yunfang Wu},
  year = {2023},
  doi = {10.48550/arxiv.2307.03972},
  eprint = {2307.03972},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.03972}
}
```

- [← previous DiPlomat: A Dialogue Dataset for Situated Pragmatic Reasoning](https://md.research.elamur.ai/free/en/papers/li-2023-diplomat-dialogue-dataset-situated-pragmatic/)
- [next → Extension Multi30K: Multimodal Dataset for Integrated Vision and Language Research in Ukrainian](https://md.research.elamur.ai/free/en/papers/saichyshyna-2023-extension-multi30k-multimodal-dataset-integrated/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… — [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
2. Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 3-8, 2012, Montréal, Canada, pages 568–572. The Association for Comp… [find](https://scholar.google.com/scholar?q=Daniel+Dahlmeier+and+Hwee+Tou+Ng%2E+2012%2E+Better+evaluation+for+grammatical+error+correction%2E+In+Human+Language+Technologies%3A+Conference+of+the+North+American+Chapter+of+the+Association+of+Computational)
3. DeepSeek-AI. 2024. Deepseek-v3 technical report. ArXiv, abs/2412.19437. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+ArXiv%2C+abs%2F2412%2E19437%2E)
4. DeepSeek-AI. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). ArXiv, abs/2501.12948.
5. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. [Glm: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335…
6. Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023. Is chatgpt a highly fluent grammatical error correction system? A comprehensive evaluation. CoRR, abs/2304.01746. [find](https://scholar.google.com/scholar?q=Tao+Fang%2C+Shu+Yang%2C+Kaixin+Lan%2C+Derek+F%2E+Wong%2C+Jinpeng+Hu%2C+Lidia+S%2E+Chao%2C+and+Yue+Zhang%2E+2023%2E+Is+chatgpt+a+highly+fluent+grammatical+error+correction+system%3F+A+comprehensive+evaluation%2E+CoRR%2C+abs%2F230)
7. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. [Is chatgpt A good translator?](https://md.research.elamur.ai/free/en/papers/jiao-2023-chatgpt-good-translator-yes-gpt/) A preliminary study. CoRR, abs/2301.08745.
8. Xianzhi Li, Xiaodan Zhu, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. 2023. Are chatgpt and GPT-4 general-purpose solvers for financial text analytics? an examination on several typical tasks. CoRR, abs/2305.05862. — [arXiv:2305.05862](https://arxiv.org/abs/2305.05862) [find](https://scholar.google.com/scholar?q=Xianzhi+Li%2C+Xiaodan+Zhu%2C+Zhiqiang+Ma%2C+Xiaomo+Liu%2C+and+Sameena+Shah%2E+2023%2E+Are+chatgpt+and+GPT%2D4+general%2Dpurpose+solvers+for+financial+text+analytics%3F+an+examination+on+several+typical+tasks%2E+CoRR%2C+abs)
9. Shirong Ma, Yinghui Li, Rongyi Sun, Qingyu Zhou, Shulin Huang, Ding Zhang, Li Yangning, Ruiyang Liu, Zhongli Li, Yunbo Cao, et al. 2022. [Linguistic rules-based corpus generation for native chinese grammatical error correction](https://md.research.elamur.ai/free/en/papers/ma-2022-linguistic-rules-based-corpus-generation/). In Findings of the Association for Computational Linguistics: EMNLP 2022.
10. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
11. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, ab…
12. Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael R. Lyu. 2023. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. CoRR, abs/2303.13648. [find](https://scholar.google.com/scholar?q=Haoran+Wu%2C+Wenxuan+Wang%2C+Yuxuan+Wan%2C+Wenxiang+Jiao%2C+and+Michael+R%2E+Lyu%2E+2023%2E+Chatgpt+or+grammarly%3F+evaluating+chatgpt+on+grammatical+error+correction+benchmark%2E+CoRR%2C+abs%2F2303%2E13648%2E)
13. Lvxiaowei Xu, Jianwang Wu, Jiawei Peng, Jiayu Fu, and Ming Cai. 2022. FCGEC: fine-grained corpus for chinese grammatical error correction. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1900–1918. Association for… [find](https://scholar.google.com/scholar?q=Lvxiaowei+Xu%2C+Jianwang+Wu%2C+Jiawei+Peng%2C+Jiayu+Fu%2C+and+Ming+Cai%2E+2022%2E+FCGEC%3A+fine%2Dgrained+corpus+for+chinese+grammatical+error+correction%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A)
14. Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei… [find](https://scholar.google.com/scholar?q=Qwen+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Y)
15. Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023. [Exploring the limits of chatgpt for query or aspect-based text summarization](https://md.research.elamur.ai/free/en/papers/yang-2023-exploring-limits-chatgpt-query-aspect/). CoRR, abs/2302.08081.
16. Team Glm Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Ming yue Liu, Minlie Huang, Peng Zhang, Qinka… — [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/)
17. Yue Zhang, Zhenghua Li, Zuyi Bao, Jiacheng Li, Bo Zhang, Chen Li, Fei Huang, and Min Zhang. 2022. Mucgec: a multi-reference multi-source evaluation dataset for chinese grammatical error correction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computation… [find](https://scholar.google.com/scholar?q=Yue+Zhang%2C+Zhenghua+Li%2C+Zuyi+Bao%2C+Jiacheng+Li%2C+Bo+Zhang%2C+Chen+Li%2C+Fei+Huang%2C+and+Min+Zhang%2E+2022%2E+Mucgec%3A+a+multi%2Dreference+multi%2Dsource+evaluation+dataset+for+chinese+grammatical+error+correction%2E+In)
18. Yuanyuan Zhao, Nan Jiang, Weiwei Sun, and Xiaojun Wan. 2018. Overview of the NLPCC 2018 shared task: Grammatical error correction. In Natural Language Processing and Chinese Computing - 7th CCF International Conference, NLPCC 2018, Hohhot, China, August 26-30, 2018, Proceedings, Part II, volume 1110… [find](https://scholar.google.com/scholar?q=Yuanyuan+Zhao%2C+Nan+Jiang%2C+Weiwei+Sun%2C+and+Xiaojun+Wan%2E+2018%2E+Overview+of+the+NLPCC+2018+shared+task%3A+Grammatical+error+correction%2E+In+Natural+Language+Processing+and+Chinese+Computing+%2D+7th+CCF+Intern)

*Updated 31 October 2026*