--- title: "Reference-based Metrics can be Replaced with — Asano 2017" description: "Grammatical error correction (GEC) systems are usually evaluated by comparing their outputs with reference corrections." authors: ["Hiroki Asano", "Tomoya Mizumoto", "Kentaro Inui"] year: 2017 venue: "Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)" source: "https://aclanthology.org/I17-2058/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/asano-2017-reference-based-metrics-can-be/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/asano-2017-reference-based-metrics-can-be/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2017](https://md.research.elamur.ai/free/papers/2017/page/3/) / Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems # Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems **Authors:** Hiroki Asano, [Tomoya Mizumoto](https://md.research.elamur.ai/free/authors/tomoya-mizumoto/), [Kentaro Inui](https://md.research.elamur.ai/free/authors/kentaro-inui/) · **Year:** 2017 · **Venue:** Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/I17-2058/) · [PDF](https://aclanthology.org/I17-2058.pdf) · [OpenAlex](https://openalex.org/W2772982633) **Citations:** 21 citations (openalex-snapshot · 2026-10-14) · [28](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Grammatical error correction (GEC) systems are usually evaluated by comparing their outputs with reference corrections. However, references have to be created manually, which is expensive, and they rarely cover all acceptable variants. The earlier reference-less metric of Napoles et al. (2016) took only grammaticality into account and performed worse than reference-based metrics. The authors extend it with two more criteria. Grammaticality is scored by a logistic regression over linguistic features, augmented with the number of errors found by LanguageTool; the model is trained on GUG. Fluency is determined by the normalized score of a recurrent language model. Meaning preservation is measured with METEOR between the system output and the source sentence. The weights of the three components are tuned on JFLEG. Testing is carried out on the outputs of the 12 CoNLL-2014 systems for which a human ranking exists. The combined metric agrees with human judgments better than M2 and GLEU+, even though those used 18 references per sentence. Unlike them, it also prefers extensive "fluency" edits over minimal ones. According to the authors, this is the first empirical evidence that in GEC reference-less evaluation can replace reference-based evaluation. The approach is especially useful for languages that lack annotated resources. ## Key points - Correlation with the human ranking of the 12 CoNLL-2014 systems (Spearman's ρ / Pearson's r): - the proposed combination: 0.874 / 0.878; - GLEU+: 0.857 / 0.843; - M2: 0.648 / 0.632; - grammaticality only: 0.835 / 0.759; - METEOR only: ρ = −0.192. - Components of the metric: - the grammaticality model achieves 78.9% accuracy on GUG (the original model of Heilman et al. had 77.2%); - fluency uses an RNN LM (faster-rnnlm) trained on 10 million sentences from BNC and Wikipedia; its scores correlate with human acceptability judgments at r = 0.395; - METEOR is computed over content words only, taking into account inflection, synonyms and spelling errors. - Weights tuned on JFLEG: α = 0.07 (grammaticality), β = 0.83 (fluency), γ = 0.10 (meaning). The authors caution that these weights do not indicate the relative importance of the components, since their values have different ranges. - Experiment with minimal and "fluency" edits from the dataset of Sakaguchi et al. (2016): - the combined metric ranks the fluency edits of both editors highest (Flu-A 0.865, Flu-B 0.854); - GLEU+ always prefers minimal edits; - M2 also ranks a minimal edit (Min-B) first. - Limitations: - METEOR does not recognize paraphrases that are close in meaning; - spelling errors distort the fluency score. An oracle combination of the components yields ρ = 0.951 and r = 0.923, i.e., there is room for improvement. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) ## Cited by Cited by 28 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 26 not yet. - [Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://md.research.elamur.ai/free/papers/alhafni-2023-advancements-arabic-grammatical-error-detection/) - [The BEA-2019 Shared Task on Grammatical Error Correction](https://md.research.elamur.ai/free/papers/bryant-2019-bea-2019-shared-task-grammatical/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{asano2017reference, title = {{Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems}}, author = {Asano, Hiroki and Mizumoto, Tomoya and Inui, Kentaro}, booktitle = {Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)}, year = {2017}, url = {https://aclanthology.org/I17-2058/} } ``` - [← previous Reading Wikipedia to Answer Open-Domain Questions](https://md.research.elamur.ai/free/papers/chen-2017-reading-wikipedia-answer-open-domain/) - [next → Reflexive verbs in Hebrew: Deep unaccusativity meets lexical semantics](https://md.research.elamur.ai/free/papers/lingbulk-2017-kastner-reflexive-verbs-hebrew-deep-unaccusativity/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/papers/2017/page/3/) ## References 14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013. TOEFL11: A Corpus of Non-Native English. Technical report, Educational Testing Service. [find](https://scholar.google.com/scholar?q=Daniel+Blanchard%2C+Joel+Tetreault%2C+Derrick+Higgins%2C+Aoife+Cahill%2C+and+Martin+Chodorow%2E+2013%2E+TOEFL11%3A+A+Corpus+of+Non%2DNative+English%2E+Technical+report%2C+Educational+Testing+Service%2E) 2. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground Truth for Grammatical Error Correction Metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volum… [find](https://scholar.google.com/scholar?q=Courtney+Napoles%2C+Keisuke+Sakaguchi%2C+Matt+Post%2C+and+Joel+Tetreault%2E+2015%2E+Ground+Truth+for+Grammatical+Error+Correction+Metrics%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Comput) 3. BNC Consortium. 2007. The British National Corpus. version 3 (BNC XML Edition). Distributed by Oxford University Computing Services on behalf of the BNC Consortium. [find](https://scholar.google.com/scholar?q=BNC+Consortium%2E+2007%2E+The+British+National+Corpus%2E+version+3+%28BNC+XML+Edition%29%2E+Distributed+by+Oxford+University+Computing+Services+on+behalf+of+the+BNC+Consortium%2E) 4. Christopher Bryant and Hwee Tou Ng. 2015. How Far are We from Fully Automatic High Quality Grammatical Error Correction? In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Christopher+Bryant+and+Hwee+Tou+Ng%2E+2015%2E+How+Far+are+We+from+Fully+Automatic+High+Quality+Grammatical+Error+Correction%3F+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computational+) 5. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2016. There’s No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2109–2115. [find](https://scholar.google.com/scholar?q=Courtney+Napoles%2C+Keisuke+Sakaguchi%2C+and+Joel+Tetreault%2E+2016%2E+There%E2%80%99s+No+Comparison%3A+Reference%2Dless+Evaluation+Metrics+in+Grammatical+Error+Correction%2E+In+Proceedings+of+the+2016+Conference+on+Empiri) 6. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A Fluency Corpus and B enchmark for Grammatical Error Correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 229–234. [find](https://scholar.google.com/scholar?q=Courtney+Napoles%2C+Keisuke+Sakaguchi%2C+and+Joel+Tetreault%2E+2017%2E+JFLEG%3A+A+Fluency+Corpus+and+B+enchmark+for+Grammatical+Error+Correction%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of) 7. Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568–572. [find](https://scholar.google.com/scholar?q=Daniel+Dahlmeier+and+Hwee+Tou+Ng%2E+2012%2E+Better+evaluation+for+grammatical+error+correction%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Lin) 8. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–14. [find](https://scholar.google.com/scholar?q=Hwee+Tou+Ng%2C+Siew+Mei+Wu%2C+Ted+Briscoe%2C+Christian+Hadiwinoto%2C+Raymond+Hendy+Susanto%2C+and+Christopher+Bryant%2E+2014%2E+The+CoNLL%2D2014+Shared+Task+on+Grammatical+Error+Correction%2E+In+Proceedings+of+the+Eigh) 9. Michael Denkowski and Alon Lavie. 2014. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Proceedings of the 7 The weights were set to a value that achieves the highest correlation on the CoNLL dataset. [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Alon+Lavie%2E+2014%2E+Meteor+Universal%3A+Language+Specific+Translation+Evaluation+for+Any+Target+Language%2E+In+Proceedings+of+the+7+The+weights+were+set+to+a+value+that+achieves+the+hi) [our card](https://md.research.elamur.ai/free/papers/denkowski-2014-meteor-universal-language-specific-translation/) 10. Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 Shared Task on Grammatical Error Correction. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 1–12. [find](https://scholar.google.com/scholar?q=Hwee+Tou+Ng%2C+Siew+Mei+Wu%2C+Yuanbin+Wu%2C+Christian+Hadiwinoto%2C+and+Joel+Tetreault%2E+2013%2E+The+CoNLL%2D2013+Shared+Task+on+Grammatical+Error+Correction%2E+In+Proceedings+of+the+Seventeenth+Conference+on+Comput) 11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+Method+for+Automatic+Evaluation+of+Machine+Translation%2E+In+Proceedings+of+40th+Annual+Meeting+of+the+Association+for+Computat) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) 12. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English Gigaword Fifth Edition LDC2011T07. Philadelphia: Linguistic Data Consortium. [find](https://scholar.google.com/scholar?q=Robert+Parker%2C+David+Graff%2C+Junbo+Kong%2C+Ke+Chen%2C+and+Kazuaki+Maeda%2E+2011%2E+English+Gigaword+Fifth+Edition+LDC2011T07%2E+Philadelphia%3A+Linguistic+Data+Consortium%2E) 13. Barbara Rychalska, Katarzyna Pakulska, Krystyna Chodorowska, Wojciech Walczak, and Piotr An-druszkiewicz. 2016. Samsung Poland NLP Team at SemEval-2016 Task 1: Necessity for diversity; combining recursive autoencoders, WordNet and ensemble methods to measure semantic similarity. In Proceedings of th… [find](https://scholar.google.com/scholar?q=Barbara+Rychalska%2C+Katarzyna+Pakulska%2C+Krystyna+Chodorowska%2C+Wojciech+Walczak%2C+and+Piotr+An%2Ddruszkiewicz%2E+2016%2E+Samsung+Poland+NLP+Team+at+SemEval%2D2016+Task+1%3A+Necessity+for+diversity%3B+combining+recur) 14. Keisuke Sakaguchi, Courtney Napoles, Matt Post, and Joel Tetreault. 2016. Reassessing the Goals of Grammatical Error Correction: Fluency Instead of Grammaticality. Transactions of the Association for Computational Linguistics, 4:169–182. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Courtney+Napoles%2C+Matt+Post%2C+and+Joel+Tetreault%2E+2016%2E+Reassessing+the+Goals+of+Grammatical+Error+Correction%3A+Fluency+Instead+of+Grammaticality%2E+Transactions+of+the+Association+for+)