---
title: "ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical"
description: "The authors compare it with the commercial assistant Grammarly and with GECToR, a strong… 23 references, cited by 71 papers here."
authors: ["Haoran Wu", "Wenxuan Wang", "Yuxuan Wan", "Wenxiang Jiao", "Michael Lyu"]
year: 2023
venue: "arXiv:2303.13648"
source: "https://arxiv.org/abs/2303.13648"
doi: "10.48550/arxiv.2303.13648"
arxiv: "2303.13648"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2023-chatgpt-grammarly-evaluating-chatgpt-grammatical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2023-chatgpt-grammarly-evaluating-chatgpt-grammatical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark

# ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark

**Authors:** Haoran Wu, [Wenxuan Wang](https://md.research.elamur.ai/free/en/authors/wenxuan-wang/), [Yuxuan Wan](https://md.research.elamur.ai/free/en/authors/yuxuan-wan/), [Wenxiang Jiao](https://md.research.elamur.ai/free/en/authors/wenxiang-jiao/) and [Michael Lyu](https://md.research.elamur.ai/free/en/authors/michael-r-lyu/) · **Year:** 2023 · **Venue:** arXiv:2303.13648 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.13648) · [PDF](https://arxiv.org/pdf/2303.13648) · [DOI 10.48550/arxiv.2303.13648](https://doi.org/10.48550/arxiv.2303.13648) · [arXiv 2303.13648](https://arxiv.org/abs/2303.13648) · [OpenAlex](https://openalex.org/W4361019538)

**Citations:** 47 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/wu-2023-chatgpt-grammarly-evaluating-chatgpt-grammatical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This early technical report asks whether ChatGPT can serve as a grammatical error correction (GEC) system. The authors compare it with the commercial assistant Grammarly and with GECToR, a strong research model built on RoBERTa. They use 100 sentences from the CoNLL-2014 test set, which contains essays by non-native English writers. ChatGPT was queried by hand with a single instruction prompt and scored with the official CoNLL-2014 scorer (Precision, Recall, F0.5). By these automatic measures, ChatGPT ranks last, and the gap widens on longer sentences. Inspecting the outputs explains much of this. ChatGPT tends to rephrase and restructure sentences rather than make minimal fixes, so its fluent and grammatical rewrites still diverge from the reference edits. A small manual annotation shows that ChatGPT misses fewer errors but adds many unrequested changes. The authors argue that reference-based metrics undervalue fluency-oriented correctors and that ChatGPT is a promising GEC tool. Caveats include the tiny sample, a single prompt, and no in-context examples.

## Key points
- **Overall scores (Precision / Recall / F0.5):**
  - GECToR: 71.2 / 38.4 / 60.8
  - Grammarly: 67.3 / 51.1 / 63.3
  - ChatGPT: 51.2 / 62.8 / 53.1

  ChatGPT has the highest recall, GECToR the highest precision, and Grammarly the best F0.5.
- **Sentence length:** The test set was split into short, medium and long thirds. ChatGPT comes close to Grammarly on short sentences but drops sharply on long ones.
- **Grammarly as a second pass:** Running Grammarly over the systems' outputs barely changes ChatGPT's results (+0.5 F0.5), mostly through punctuation fixes. For GECToR, it adds +2.1 F0.5 and +16.5 recall, meaning GECToR leaves many errors in place.
- **Human evaluation:** Two annotators labelled 20 sentences, counting under-, mis- and over-corrections:
  - GECToR: 13 / 4 / 0
  - Grammarly: 14 / 0 / 1
  - ChatGPT: 3 / 3 / 30
- **Setup limitations:** The study uses only the CoNLL-2014 data and one zero-shot prompt. It does not test few-shot or chain-of-thought prompting, and it relies only on M²-style metrics, with no pretraining-based metrics.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 163 works · 21,032 citations
- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 159 works · 16,154 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🖍️ Grammatical error correction & learner language](https://md.research.elamur.ai/free/en/topics/grammatical-error-correction/) · 162 works

## Cite

Cite as: Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao and Michael Lyu. 2023. ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark. arXiv:2303.13648.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wu2023chatgpt,
  title = {{ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark}},
  author = {Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao and Michael Lyu},
  year = {2023},
  doi = {10.48550/arxiv.2303.13648},
  eprint = {2303.13648},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.13648}
}
```

- [← previous Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors](https://md.research.elamur.ai/free/en/papers/zhang-2023-aligning-instruction-tasks-unlocks-large/)
- [next → MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/juraska-2023-metricx-23-google-submission-wmt/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. [A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). ArXiv.
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). NeurIPS.
3. Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. [The bea-2019 shared task on grammatical error correction](https://md.research.elamur.ai/free/en/papers/bryant-2019-bea-2019-shared-task-grammatical/). In BEA@ACL.
4. Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2022. Grammatical error correction: A survey of the state of the art. ArXiv. [find](https://scholar.google.com/scholar?q=Christopher+Bryant%2C+Zheng+Yuan%2C+Muhammad+Reza+Qorib%2C+Hannan+Cao%2C+Hwee+Tou+Ng%2C+and+Ted+Briscoe%2E+2022%2E+Grammatical+error+correction%3A+A+survey+of+the+state+of+the+art%2E+ArXiv%2E)
5. Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. NeruIPS. [find](https://scholar.google.com/scholar?q=Paul+Francis+Christiano%2C+Jan+Leike%2C+Tom+B%2E+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+NeruIPS%2E)
6. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and J J Berner. 2023. [Mathematical capabilities of chatgpt](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/). ArXiv.
7. Peiyuan Gong, Xuebo Liu, Heyan Huang, and Min Zhang. 2022. Revisiting grammatical error correction evaluation and beyond. EMNLP. [find](https://scholar.google.com/scholar?q=Peiyuan+Gong%2C+Xuebo+Liu%2C+Heyan+Huang%2C+and+Min+Zhang%2E+2022%2E+Revisiting+grammatical+error+correction+evaluation+and+beyond%2E+EMNLP%2E)
8. Grammarly. 2023. Grammarly website about us page. [find](https://scholar.google.com/scholar?q=Grammarly%2E+2023%2E+Grammarly+website+about+us+page%2E)
9. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT a good translator? a preliminary study. In ArXiv. [find](https://scholar.google.com/scholar?q=Wenxiang+Jiao%2C+Wenxuan+Wang%2C+Jen+tse+Huang%2C+Xing+Wang%2C+and+Zhaopeng+Tu%2E+2023%2E+Is+ChatGPT+a+good+translator%3F+a+preliminary+study%2E+In+ArXiv%2E)
10. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The conll-2014 shared task on grammatical error correction. In CoNLL. [find](https://scholar.google.com/scholar?q=Hwee+Tou+Ng%2C+Siew+Mei+Wu%2C+Ted+Briscoe%2C+Christian+Hadiwinoto%2C+Raymond+Hendy+Susanto%2C+and+Christopher+Bryant%2E+2014%2E+The+conll%2D2014+shared+task+on+grammatical+error+correction%2E+In+CoNLL%2E)
11. Reham Omar, Omij Mangukiya, Panos Kalnis, and Essam Mansour. 2023. Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots. ArXiv. [find](https://scholar.google.com/scholar?q=Reham+Omar%2C+Omij+Mangukiya%2C+Panos+Kalnis%2C+and+Essam+Mansour%2E+2023%2E+Chatgpt+versus+traditional+question+answering+for+knowledge+graphs%3A+Current+status+and+future+directions+towards+knowledge+graph+chat)
12. Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem N. Chernodub, and Oleksandr Skurzhanskyi. 2020. Gector – grammatical error correction: Tag, not rewrite. In Workshop on Innovative Use of NLP for Building Educational Applications. [find](https://scholar.google.com/scholar?q=Kostiantyn+Omelianchuk%2C+Vitaliy+Atrasevych%2C+Artem+N%2E+Chernodub%2C+and+Oleksandr+Skurzhanskyi%2E+2020%2E+Gector+%E2%80%93+grammatical+error+correction%3A+Tag%2C+not+rewrite%2E+In+Workshop+on+Innovative+Use+of+NLP+for+Buil)
13. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv.
14. Sebastian Ruder. 2022. NLP-progress. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2E+2022%2E+NLP%2Dprogress%2E)
15. Joel R. Tetreault, Keisuke Sakaguchi, and Courtney Napoles. 2017. Jfleg: A fluency corpus and benchmark for grammatical error correction. In EACL. [find](https://scholar.google.com/scholar?q=Joel+R%2E+Tetreault%2C+Keisuke+Sakaguchi%2C+and+Courtney+Napoles%2E+2017%2E+Jfleg%3A+A+fluency+corpus+and+benchmark+for+grammatical+error+correction%2E+In+EACL%2E)
16. Wenxuan Wang, Wenxiang Jiao, Yongchang Hao, Xing Wang, Shuming Shi, Zhaopeng Tu, and Michael Lyu. 2022. Understanding and improving sequence-to-sequence pretraining for neural machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Wenxuan+Wang%2C+Wenxiang+Jiao%2C+Yongchang+Hao%2C+Xing+Wang%2C+Shuming+Shi%2C+Zhaopeng+Tu%2C+and+Michael+Lyu%2E+2022%2E+Understanding+and+improving+sequence%2Dto%2Dsequence+pretraining+for+neural+machine+translation%2E+In+)
17. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). ICLR.
18. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoc Le, and Denny Zhou. 2022b. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). NeurIPS.
19. Wikipedia contributors. 2023a. F-score — Wikipedia, the free encyclopedia. [Online; accessed 5-March-2023]. [find](https://scholar.google.com/scholar?q=Wikipedia+contributors%2E+2023a%2E+F%2Dscore+%E2%80%94+Wikipedia%2C+the+free+encyclopedia%2E+%5BOnline%3B+accessed+5%2DMarch%2D2023%5D%2E)
20. Wikipedia contributors. 2023b. Grammarly — Wikipedia, the free encyclopedia. [Online; accessed 2-March-2023]. [find](https://scholar.google.com/scholar?q=Wikipedia+contributors%2E+2023b%2E+Grammarly+%E2%80%94+Wikipedia%2C+the+free+encyclopedia%2E+%5BOnline%3B+accessed+2%2DMarch%2D2023%5D%2E)
21. Chun Xia and Lingming Zhang. 2023. Conversational automated program repair. ArXiv. [find](https://scholar.google.com/scholar?q=Chun+Xia+and+Lingming+Zhang%2E+2023%2E+Conversational+automated+program+repair%2E+ArXiv%2E)
22. Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023. [Exploring the limits of chatgpt for query or aspect-based text summarization](https://md.research.elamur.ai/free/en/papers/yang-2023-exploring-limits-chatgpt-query-aspect/). ArXiv.
23. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. ArXiv. [find](https://scholar.google.com/scholar?q=Qihuang+Zhong%2C+Liang+Ding%2C+Juhua+Liu%2C+Bo+Du%2C+and+Dacheng+Tao%2E+2023%2E+Can+chatgpt+understand+too%3F+a+comparative+study+on+chatgpt+and+fine%2Dtuned+bert%2E+ArXiv%2E)

*Updated 2 November 2026*