---
title: "Analyzing the Performance of GPT-3.5 and GPT-4 in"
description: "This study fills that gap by testing a GPT-3.5 model (text-davinci-003) and a GPT-4 model… 29 references, cited by 56 papers here."
authors: ["Steven Coyne", "Keisuke Sakaguchi", "Diana Galvan-Sosa", "Michael Zock", "Kentaro Inui"]
year: 2023
venue: "arXiv:2303.14342"
source: "https://arxiv.org/abs/2303.14342"
doi: "10.48550/arxiv.2303.14342"
arxiv: "2303.14342"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/coyne-2023-analyzing-performance-gpt-3-5/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/coyne-2023-analyzing-performance-gpt-3-5/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction

# Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction

**Authors:** [Steven Coyne](https://md.research.elamur.ai/free/en/authors/steven-coyne/), [Keisuke Sakaguchi](https://md.research.elamur.ai/free/en/authors/keisuke-sakaguchi/), [Diana Galvan-Sosa](https://md.research.elamur.ai/free/en/authors/diana-galvan-sosa/), [Michael Zock](https://md.research.elamur.ai/free/en/authors/michael-zock/) and [Kentaro Inui](https://md.research.elamur.ai/free/en/authors/kentaro-inui/) · **Year:** 2023 · **Venue:** arXiv:2303.14342 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.14342) · [PDF](https://arxiv.org/pdf/2303.14342) · [DOI 10.48550/arxiv.2303.14342](https://doi.org/10.48550/arxiv.2303.14342) · [arXiv 2303.14342](https://arxiv.org/abs/2303.14342) · [OpenAlex](https://openalex.org/W4361192999)

**Citations:** 25 citations (OpenAlex, 2026-10-14) · [56](https://md.research.elamur.ai/free/en/papers/coyne-2023-analyzing-performance-gpt-3-5/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

OpenAI's technical reports give no benchmark results for grammatical error correction (GEC). This study fills that gap by testing a GPT-3.5 model (text-davinci-003) and a GPT-4 model (gpt-4-0314) on standard GEC benchmarks. The task is framed as sentence-level revision: the model gets one possibly faulty sentence and returns one corrected version.

The authors compared ten zero-shot prompts at three temperature settings, using GLEU on the JFLEG development set to choose the best one. They then tried few-shot variants. The chosen prompt was evaluated on the BEA-2019 and JFLEG test sets. Human raters scored GPT outputs alongside source sentences, human references and two baseline GEC systems.

Both models handled the task reliably once given a well-designed prompt. GPT-4 set a new best score on JFLEG. However, the models tend toward fluency rewrites and sometimes over-edit, which hurts them on BEA-2019, a benchmark that rewards minimal edits.

The work shows that prompt wording, task definition and benchmark choice strongly affect how LLM-based GEC is judged.

## Key points

- **Best zero-shot prompt (#10):** an explicit instruction to fix grammar and spelling and to copy the sentence unchanged if it is already correct. On the JFLEG dev set it reached GLEU 0.582 (GPT-3.5) and 0.601 (GPT-4). A plain "more fluent" prompt scored only 0.314 and 0.245.
- **Temperature:** the authors tested τ of 0.1, 0.5 and 0.9. Lower temperature consistently gave better results.
- **Prompt sensitivity:** small changes, such as adding quotes or field labels, noticeably changed both correction quality and output format. Few-shot prompts used up to six example pairs.
- **Human evaluation:** about 200 candidate sets were rated. Baselines beat the GPT models in 4 (BEA-2019) and 7 (JFLEG) cases, while human references beat them in 13 and 10 cases.
- **Error patterns:** the models sometimes changed a sentence's meaning or completed fragments with invented content. Occasionally they under-edited instead, leaving ungrammatical spans unchanged.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Aix-Marseille Université](https://md.research.elamur.ai/free/en/universities/aix-marseille-universite/) — University, Marseille, France · 25 works · 39,241 citations
- [Tohoku University](https://md.research.elamur.ai/free/en/universities/tohoku-university/) — University, Sendai, Japan · 23 works · 595 citations
- [RIKEN](https://md.research.elamur.ai/free/en/labs/riken/) — Research institute, Wako, Japan · 30 works · 607 citations
- [Laboratoire d’Informatique et Systèmes](https://md.research.elamur.ai/free/en/labs/laboratoire-d-informatique-et-systemes/) — Research institute, Marseille, France · 10 works · 187 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖍️ Grammatical error correction & learner language](https://md.research.elamur.ai/free/en/topics/grammatical-error-correction/) · 162 works

## Cite

Cite as: Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock and Kentaro Inui. 2023. Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction. arXiv:2303.14342.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{coyne2023analyzing,
  title = {{Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction}},
  author = {Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock and Kentaro Inui},
  year = {2023},
  doi = {10.48550/arxiv.2303.14342},
  eprint = {2303.14342},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.14342}
}
```

- [← previous TableLlama: Towards Open Large Generalist Models for Tables](https://md.research.elamur.ai/free/en/papers/zhang-2023-tablellama-towards-open-large-generalist/)
- [next → Attribution Patching Outperforms Automated Circuit Discovery](https://md.research.elamur.ai/free/en/papers/syed-2023-attribution-patching-outperforms-automated-circuit/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
2. Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. [The BEA-2019 shared task on grammatical error correction](https://md.research.elamur.ai/free/en/papers/bryant-2019-bea-2019-shared-task-grammatical/). In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–75, Florence, Italy. Association for Computatio…
3. Shamil Chollampatt and Hwee Tou Ng. 2018. A multi-layer convolutional encoder-decoder neural network for grammatical error correction. In AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Shamil+Chollampatt+and+Hwee+Tou+Ng%2E+2018%2E+A+multi%2Dlayer+convolutional+encoder%2Ddecoder+neural+network+for+grammatical+error+correction%2E+In+AAAI+Conference+on+Artificial+Intelligence%2E)
4. Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023. [Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation](https://md.research.elamur.ai/free/en/papers/fang-2023-chatgpt-highly-fluent-grammatical-error/).
5. Roman Grundkiewicz and Marcin Junczys-Dowmunt. 2018. Near human-level performance in grammatical error correction with hybrid machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… [find](https://scholar.google.com/scholar?q=Roman+Grundkiewicz+and+Marcin+Junczys%2DDowmunt%2E+2018%2E+Near+human%2Dlevel+performance+in+grammatical+error+correction+with+hybrid+machine+translation%2E+In+Proceedings+of+the+2018+Conference+of+the+North+Am)
6. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In International Conference on Learning Representations.
7. Md Asadul Islam and Enrico Magnani. 2021. Is this the end of the gold standard? a straightforward reference-less grammatical error correction metric. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3009–3015, Online and Punta Cana, Dominican Republic.… [find](https://scholar.google.com/scholar?q=Md+Asadul+Islam+and+Enrico+Magnani%2E+2021%2E+Is+this+the+end+of+the+gold+standard%3F+a+straightforward+reference%2Dless+grammatical+error+correction+metric%2E+In+Proceedings+of+the+2021+Conference+on+Empirical)
8. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. [How can we know what language models know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) Transactions of the Association for Computational Linguistics, 8:423–438.
9. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human… [find](https://scholar.google.com/scholar?q=Marcin+Junczys%2DDowmunt%2C+Roman+Grundkiewicz%2C+Shubha+Guha%2C+and+Kenneth+Heafield%2E+2018%2E+Approaching+neural+grammatical+error+correction+as+a+low%2Dresource+machine+translation+task%2E+In+Proceedings+of+the+2)
10. Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4248–4254, Onli… [find](https://scholar.google.com/scholar?q=Masahiro+Kaneko%2C+Masato+Mita%2C+Shun+Kiyono%2C+Jun+Suzuki%2C+and+Kentaro+Inui%2E+2020%2E+Encoder%2Ddecoder+models+can+benefit+from+pre%2Dtrained+masked+language+models+in+grammatical+error+correction%2E+In+Proceeding)
11. Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019. An empirical study of incorporating pseudo data into grammatical error correction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Na… [find](https://scholar.google.com/scholar?q=Shun+Kiyono%2C+Jun+Suzuki%2C+Masato+Mita%2C+Tomoya+Mizumoto%2C+and+Kentaro+Inui%2E+2019%2E+An+empirical+study+of+incorporating+pseudo+data+into+grammatical+error+correction%2E+In+Proceedings+of+the+2019+Conference+)
12. Zhenghao Liu, Xiaoyuan Yi, Maosong Sun, Liner Yang, and Tat-Seng Chua. 2021. Neural quality estimation with multiple hypotheses for grammatical error correction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… [find](https://scholar.google.com/scholar?q=Zhenghao+Liu%2C+Xiaoyuan+Yi%2C+Maosong+Sun%2C+Liner+Yang%2C+and+Tat%2DSeng+Chua%2E+2021%2E+Neural+quality+estimation+with+multiple+hypotheses+for+grammatical+error+correction%2E+In+Proceedings+of+the+2021+Conference+)
13. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volum… [find](https://scholar.google.com/scholar?q=Courtney+Napoles%2C+Keisuke+Sakaguchi%2C+Matt+Post%2C+and+Joel+Tetreault%2E+2015%2E+Ground+truth+for+grammatical+error+correction+metrics%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Comput)
14. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 229–234, Valencia, Spai… [find](https://scholar.google.com/scholar?q=Courtney+Napoles%2C+Keisuke+Sakaguchi%2C+and+Joel+Tetreault%2E+2017%2E+JFLEG%3A+A+fluency+corpus+and+benchmark+for+grammatical+error+correction%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+)
15. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–14, Baltimore, Ma… [find](https://scholar.google.com/scholar?q=Hwee+Tou+Ng%2C+Siew+Mei+Wu%2C+Ted+Briscoe%2C+Christian+Hadiwinoto%2C+Raymond+Hendy+Susanto%2C+and+Christopher+Bryant%2E+2014%2E+The+CoNLL%2D2014+shared+task+on+grammatical+error+correction%2E+In+Proceedings+of+the+Eigh)
16. Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR – grammatical error correction: Tag, not rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163–170, Seattle, WA, USA → Online. A… [find](https://scholar.google.com/scholar?q=Kostiantyn+Omelianchuk%2C+Vitaliy+Atrasevych%2C+Artem+Chernodub%2C+and+Oleksandr+Skurzhanskyi%2E+2020%2E+GECToR+%E2%80%93+grammatical+error+correction%3A+Tag%2C+not+rewrite%2E+In+Proceedings+of+the+Fifteenth+Workshop+on+Inno)
17. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
18. Robert Ostling and Murathan Kurfalı. 2022. Really good grammatical error correction, and how to evaluate it. Swedish Language Technology Conference (SLTC). [find](https://scholar.google.com/scholar?q=Robert+Ostling+and+Murathan+Kurfal%C4%B1%2E+2022%2E+Really+good+grammatical+error+correction%2C+and+how+to+evaluate+it%2E+Swedish+Language+Technology+Conference+%28SLTC%29%2E)
19. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
20. Muhammad Qorib, Seung-Hoon Na, and Hwee Tou Ng. 2022. Frustratingly easy system combination for grammatical error correction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1964–1974, Seattle, U… [find](https://scholar.google.com/scholar?q=Muhammad+Qorib%2C+Seung%2DHoon+Na%2C+and+Hwee+Tou+Ng%2E+2022%2E+Frustratingly+easy+system+combination+for+grammatical+error+correction%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the)
21. Keisuke Sakaguchi and Benjamin Van Durme. 2018. Efficient online scalar annotation with bounded support. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 208–218, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi+and+Benjamin+Van+Durme%2E+2018%2E+Efficient+online+scalar+annotation+with+bounded+support%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vol)
22. Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022. [PEER: A Collaborative Language Model](https://md.research.elamur.ai/free/en/papers/schick-2022-peer-collaborative-language-model/).
23. Timo Schick and Hinrich Schütze. 2021. [It’s not just size that matters: Small language models are also few-shot learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Associ…
24. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin…
25. Felix Stahlberg and Shankar Kumar. 2021. Synthetic data generation for grammatical error correction with tagged corruption models. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 37–47, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Felix+Stahlberg+and+Shankar+Kumar%2E+2021%2E+Synthetic+data+generation+for+grammatical+error+correction+with+tagged+corruption+models%2E+In+Proceedings+of+the+16th+Workshop+on+Innovative+Use+of+NLP+for+Buil)
26. Maksym Tarnavskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022. Ensembling and knowledge distilling of large sequence taggers for grammatical error correction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3842–3852, D… [find](https://scholar.google.com/scholar?q=Maksym+Tarnavskyi%2C+Artem+Chernodub%2C+and+Kostiantyn+Omelianchuk%2E+2022%2E+Ensembling+and+knowledge+distilling+of+large+sequence+taggers+for+grammatical+error+correction%2E+In+Proceedings+of+the+60th+Annual+)
27. Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023. [Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark](https://md.research.elamur.ai/free/en/papers/wu-2023-chatgpt-grammarly-evaluating-chatgpt-grammatical/).
28. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2021. LM-critic: Language models for unsupervised grammatical error correction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7752–7763, Online and Punta Cana, Dominican Republic. Association for C… [find](https://scholar.google.com/scholar?q=Michihiro+Yasunaga%2C+Jure+Leskovec%2C+and+Percy+Liang%2E+2021%2E+LM%2Dcritic%3A+Language+models+for+unsupervised+grammatical+error+correction%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natura)
29. Michael Zock and Debela Tesfaye Gemechu. 2017. Use your mind and learn to write: The problem of producing coherent text. In Cognitive Approach to Natural Language Processing, pages 129–158. Elsevier. [find](https://scholar.google.com/scholar?q=Michael+Zock+and+Debela+Tesfaye+Gemechu%2E+2017%2E+Use+your+mind+and+learn+to+write%3A+The+problem+of+producing+coherent+text%2E+In+Cognitive+Approach+to+Natural+Language+Processing%2C+pages+129%E2%80%93158%2E+Elsevier%2E)

*Updated 2 November 2026*