---
title: "Automated Essay Scoring and Language — Wilkens 2026"
description: "Most automated essay scoring (AES) research is benchmarked on the English ASAP dataset with a single metric, typically QWK."
authors: ["Rodrigo Wilkens", "Rémi Cardon", "Vincent Folny", "Thomas François"]
year: 2026
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2026.tacl-1.44/"
doi: "10.1162/tacl.a.693"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wilkens-2026-automated-essay-scoring-language-certification/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wilkens-2026-automated-essay-scoring-language-certification/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / Automated Essay Scoring and Language Certification: Assessing Generalizability, Agreement and Validity for French

# Automated Essay Scoring and Language Certification: Assessing Generalizability, Agreement and Validity for French

**Authors:** [Rodrigo Wilkens](https://md.research.elamur.ai/free/en/authors/rodrigo-wilkens/), Rémi Cardon, Vincent Folny, [Thomas François](https://md.research.elamur.ai/free/en/authors/thomas-francois/) · **Year:** 2026 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.tacl-1.44/) · [PDF](https://aclanthology.org/2026.tacl-1.44.pdf) · [DOI 10.1162/tacl.a.693](https://doi.org/10.1162/tacl.a.693) · [OpenAlex](https://openalex.org/W7165010782)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
Most automated essay scoring (AES) research is benchmarked on the English ASAP dataset with a single metric, typically QWK. That narrow evaluation does not meet the needs of high-stakes language certification. The authors address this for French using essays from the TCF (Test de connaissance du Français), administered by France Éducation international. They turn the argument-based validation (ABV) framework of Williamson et al. into a practical tool by attaching automatic NLP measures to its abstract areas: agreement with human raters, generalization, correlations with linguistic features, fairness checks, and prediction-error and calibration analysis. They use this framework to compare 8 model architectures, including hybrids that combine CamemBERT with linguistic features. The models are trained on a large core exam corpus and tested on a separate, heavily rated generalization corpus. The new models outperform earlier French AES systems. The analysis also shows that standard metrics can mislead and that two raters per essay underestimate model quality. The work gives French AES a stronger baseline and a reusable evaluation method for certification settings.

## Key points
- **Data:** The core corpus has about 27k TCF essays, each scored by 2 raters. The generalization corpus has 961 essays, each scored by at least nine raters. The authors describe the core set as the largest calibrated corpus used for French AES in certification. The previous largest public FFL corpus, TCFLE-8, had 6.5k essays.
- **Models:** Eight architectures were compared, all built on CamemBERT, since very few French encoders are available. Code for the hybrid architectures is hosted at gitlab.com/rswilkens/linguistic-features-in-transformers.
- **Metric pitfalls:** QWK and correlation can look strong even when exact CEFR levels are missed. F1 and exact accuracy, in turn, cannot show whether a model grasps the ordering of the level scale.
- **Rater count:** With only two raters, measured model performance is underestimated, most visibly at level C2. The authors argue the field needs well-calibrated generalization corpora.
- **Fairness and calibration:** No bias by language, gender, or task appeared on the core corpus or on the generalization corpus. Some bias did emerge on essays where raters disagreed. Calibration analysis (ECE) showed the models are well calibrated and make fewer errors than individual human raters. France Éducation international now uses an automated scorer alongside human raters.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{wilkens2026automated,
  title = {{Automated Essay Scoring and Language Certification: Assessing Generalizability, Agreement and Validity for French}},
  author = {Wilkens, Rodrigo and Cardon, Rémi and Folny, Vincent and François, Thomas},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.693},
  url = {https://doi.org/10.1162/tacl.a.693}
}
```

- [← previous AuthorMix: Modular Authorship Style Transfer via Layer-wise Adapter Mixing](https://md.research.elamur.ai/free/en/papers/thillainathan-2026-authormix-modular-authorship-style-transfer/)
- [next → Automated Scoring of Arabic Text Using Large Language Models: A Literature Review](https://md.research.elamur.ai/free/en/papers/dahimi-2026-automated-scoring-arabic-text-using/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

69 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AERA, APA, and NCME associations. 2014. Standards for educational and psychological testing. American Educational Research Association. [find](https://scholar.google.com/scholar?q=AERA%2C+APA%2C+and+NCME+associations%2E+2014%2E+Standards+for+educational+and+psychological+testing%2E+American+Educational+Research+Association%2E)
2. Y Attali. 2009. Evaluating automated scoring for operational use in consequential language assessment—the ets experience. In annual meeting of the National Council on Measurement in Education, San Diego, CA. [find](https://scholar.google.com/scholar?q=Y+Attali%2E+2009%2E+Evaluating+automated+scoring+for+operational+use+in+consequential+language+assessment%E2%80%94the+ets+experience%2E+In+annual+meeting+of+the+National+Council+on+Measurement+in+Education%2C+San+Die)
3. Jill Burstein and Martin Chodorow. 1999. Automated essay scoring for nonnative english speakers. In Computer mediated language assessment and evaluation in natural language processing. [find](https://scholar.google.com/scholar?q=Jill+Burstein+and+Martin+Chodorow%2E+1999%2E+Automated+essay+scoring+for+nonnative+english+speakers%2E+In+Computer+mediated+language+assessment+and+evaluation+in+natural+language+processing%2E)
4. Jill Burstein, Karen Kukich, Susanne Wolff, Chi Lu, Martin Chodorow, Lisa Braden-Harder, and Mary Dee Harris. 1998. Automated scoring using a hybrid feature identification technique. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 206–210, Montreal, Quebec, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jill+Burstein%2C+Karen+Kukich%2C+Susanne+Wolff%2C+Chi+Lu%2C+Martin+Chodorow%2C+Lisa+Braden%2DHarder%2C+and+Mary+Dee+Harris%2E+1998%2E+Automated+scoring+using+a+hybrid+feature+identification+technique%2E+In+36th+Annual+Me)
5. Carol A Chapelle, Mary K Enright, and Joan M Jamieson. 2008. Building a validity argument for the Test of English as a Foreign Language. Routledge. [find](https://scholar.google.com/scholar?q=Carol+A+Chapelle%2C+Mary+K+Enright%2C+and+Joan+M+Jamieson%2E+2008%2E+Building+a+validity+argument+for+the+Test+of+English+as+a+Foreign+Language%2E+Routledge%2E)
6. Beata Beigman Klebanov and Nitin Madnani. 2020. Automated evaluation of writing – 50 years and counting. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7796–7810, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Beata+Beigman+Klebanov+and+Nitin+Madnani%2E+2020%2E+Automated+evaluation+of+writing+%E2%80%93+50+years+and+counting%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pag)
7. Brian E Clauser, Michael T Kane, and David B Swanson. 2002. Validity issues for performance-based tests scored with computer-automated scoring systems. Applied Measurement in Education, 15(4):413–432. [↗](https://doi.org/10.1207/s15324818ame1504_05)
8. Randy Elliot Bennett and Isaac I Bejar. 1998. Validity and automad scoring: It’s not only the scoring. Educational Measurement: Issues and Practice, 17(4):9–17. [↗](https://doi.org/10.1111/j.1745-3992.1998.tb00631.x)
9. Open Cambridge Learner Corpus. 2017. Distributed by lexical computing limited on behalf of cambridge university press and cambridge english language assessment. [find](https://scholar.google.com/scholar?q=Open+Cambridge+Learner+Corpus%2E+2017%2E+Distributed+by+lexical+computing+limited+on+behalf+of+cambridge+university+press+and+cambridge+english+language+assessment%2E)
10. Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013. Toefl11: A corpus of non-native english. ETS Research Report Series, 2013(2):i–15. [find](https://scholar.google.com/scholar?q=Daniel+Blanchard%2C+Joel+Tetreault%2C+Derrick+Higgins%2C+Aoife+Cahill%2C+and+Martin+Chodorow%2E+2013%2E+Toefl11%3A+A+corpus+of+non%2Dnative+english%2E+ETS+Research+Report+Series%2C+2013%282%29%3Ai%E2%80%9315%2E)
11. Council of Europe. 2020. Common European Framework of Reference for Languages: Learning, Teaching, Assessment – Companion Volume. Council of Europe Publishing. [find](https://scholar.google.com/scholar?q=Council+of+Europe%2E+2020%2E+Common+European+Framework+of+Reference+for+Languages%3A+Learning%2C+Teaching%2C+Assessment+%E2%80%93+Companion+Volume%2E+Council+of+Europe+Publishing%2E)
12. Renske Bouwer, Anton Béguin, Ted Sanders, and Huub Van den Bergh. 2015. Effect of genre on the generalizability of writing scores. Language Testing, 32(1):83–100. [↗](https://doi.org/10.1177/0265532214542994)
13. Adriane Boyd, Jirka Hana, Lionel Nicolas, Detmar Meurers, Katrin Wisniewski, Andrea Abel, Karin Schöne, Barbora Stindlová, and Chiara Vettori. 2014. The MERLIN corpus: Learner language and the CEFR. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, pages 1281––1288. [find](https://scholar.google.com/scholar?q=Adriane+Boyd%2C+Jirka+Hana%2C+Lionel+Nicolas%2C+Detmar+Meurers%2C+Katrin+Wisniewski%2C+Andrea+Abel%2C+Karin+Sch%C3%B6ne%2C+Barbora+Stindlov%C3%A1%2C+and+Chiara+Vettori%2E+2014%2E+The+MERLIN+corpus%3A+Learner+language+and+the+CEFR%2E+I)
14. Brent Bridgeman, Catherine Trapani, and Yigal Attali. 2009. Considering fairness and validity in evaluating automated scoring. In annual meeting of the National Council on Measurement in Education, San Diego, CA. [find](https://scholar.google.com/scholar?q=Brent+Bridgeman%2C+Catherine+Trapani%2C+and+Yigal+Attali%2E+2009%2E+Considering+fairness+and+validity+in+evaluating+automated+scoring%2E+In+annual+meeting+of+the+National+Council+on+Measurement+in+Education%2C+Sa)
15. Bastien De Clercq and Alex Housen. 2017. A cross-linguistic perspective on syntactic complexity in l2 development: Syntactic elaboration and diversity. The Modern Language Journal, 101(2):315–334. [find](https://scholar.google.com/scholar?q=Bastien+De+Clercq+and+Alex+Housen%2E+2017%2E+A+cross%2Dlinguistic+perspective+on+syntactic+complexity+in+l2+development%3A+Syntactic+elaboration+and+diversity%2E+The+Modern+Language+Journal%2C+101%282%29%3A315%E2%80%93334%2E)
16. Afrizal Doewes, Nughthoh Kurdhi, and Akrati Saxena. 2023. Evaluating quadratic weighted kappa as the standard performance metric for automated essay scoring. In 16th International Conference on Educational Data Mining, EDM 2023, pages 103–113. International Educational Data Mining Society (IEDMS). [find](https://scholar.google.com/scholar?q=Afrizal+Doewes%2C+Nughthoh+Kurdhi%2C+and+Akrati+Saxena%2E+2023%2E+Evaluating+quadratic+weighted+kappa+as+the+standard+performance+metric+for+automated+essay+scoring%2E+In+16th+International+Conference+on+Educat)
17. T. Eckes. 2009. Quantitative Data Analysis for Language Assessment Volume I: Fundamental Techniques, 1 edition. Routledge. [find](https://scholar.google.com/scholar?q=T%2E+Eckes%2E+2009%2E+Quantitative+Data+Analysis+for+Language+Assessment+Volume+I%3A+Fundamental+Techniques%2C+1+edition%2E+Routledge%2E)
18. Fanny Forsberg Lundell and Christina Lindqvist. 2014. Vocabulary aspects of advanced l2 french: Do lexical formulaic sequences and lexical richness develop at the same rate? In The Acquisition of French as a Second Language: New developmental perspectives, pages 75–94. John Benjamins Publishing Company. [find](https://scholar.google.com/scholar?q=Fanny+Forsberg+Lundell+and+Christina+Lindqvist%2E+2014%2E+Vocabulary+aspects+of+advanced+l2+french%3A+Do+lexical+formulaic+sequences+and+lexical+richness+develop+at+the+same+rate%3F+In+The+Acquisition+of+Fren)
19. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. [↗](https://doi.org/10.48550/arxiv.1706.04599)
20. Jason Sebastian Kusuma, Kevin Halim, Edgard Jonathan Putra Pranoto, Bayu Kanigoro, and Edy Irwansyah. 2022. Automated essay scoring using machine learning. In 2022 4th International Conference on Cybernetics and Intelligent System (ICORIS), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=Jason+Sebastian+Kusuma%2C+Kevin+Halim%2C+Edgard+Jonathan+Putra+Pranoto%2C+Bayu+Kanigoro%2C+and+Edy+Irwansyah%2E+2022%2E+Automated+essay+scoring+using+machine+learning%2E+In+2022+4th+International+Conference+on+Cybe)
21. Benoit Lemaire and Philippe Dessus. 2001. A System to Assess the Semantic Content of Student Essays. Journal of Educational Computing Research, 24(3):305–320. [↗](https://doi.org/10.2190/g649-0r9c-c021-p6x3)
22. Guoxi Liang, Byung-Won On, Dongwon Jeong, Hyun-Chul Kim, and Gyu Sang Choi. 2018. Automated essay scoring: A siamese bidirectional lstm neural network architecture. Symmetry, 10(12):682. [↗](https://doi.org/10.3390/sym10120682)
23. Kilem L Gwet. 2014. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC. [find](https://scholar.google.com/scholar?q=Kilem+L+Gwet%2E+2014%2E+Handbook+of+inter%2Drater+reliability%3A+The+definitive+guide+to+measuring+the+extent+of+agreement+among+raters%2E+Advanced+Analytics%2C+LLC%2E)
24. Jinyan Huang and Patrick B Whipple. 2023. Rater variability and reliability of constructed response questions in new york state high-stakes tests of english language arts and mathematics: implications for educational assessment policy. Humanities and Social Sciences Communications, 10(1):1–10. [↗](https://doi.org/10.1057/s41599-023-02385-4)
25. Shi Huawei and Vahid Aryadoust. 2023. A systematic review of automated writing evaluation systems. Education and Information Technologies, 28(1):771–795. [find](https://scholar.google.com/scholar?q=Shi+Huawei+and+Vahid+Aryadoust%2E+2023%2E+A+systematic+review+of+automated+writing+evaluation+systems%2E+Education+and+Information+Technologies%2C+28%281%29%3A771%E2%80%93795%2E)
26. Zhiwei Jiang, Tianyi Gao, Yafeng Yin, Meng Liu, Hua Yu, Zifeng Cheng, and Qing Gu. 2023. Improving domain generalization for prompt-aware essay scoring via disentangled representation learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12456–12470, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhiwei+Jiang%2C+Tianyi+Gao%2C+Yafeng+Yin%2C+Meng+Liu%2C+Hua+Yu%2C+Zifeng+Cheng%2C+and+Qing+Gu%2E+2023%2E+Improving+domain+generalization+for+prompt%2Daware+essay+scoring+via+disentangled+representation+learning%2E+In+Pro)
27. Cancan Jin, Ben He, Kai Hui, and Le Sun. 2018. Tdnn: a two-stage deep neural network for prompt-independent automated essay scoring. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1088–1097. [find](https://scholar.google.com/scholar?q=Cancan+Jin%2C+Ben+He%2C+Kai+Hui%2C+and+Le+Sun%2E+2018%2E+Tdnn%3A+a+two%2Dstage+deep+neural+network+for+prompt%2Dindependent+automated+essay+scoring%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Co)
28. Michael T Kane. 2013. Validating the interpretations and uses of test scores. Journal of educational measurement, 50(1):1–73. [↗](https://doi.org/10.1111/jedm.12000)
29. Beata Beigman Klebanov and Nitin Madnani. 2021. Automated essay scoring. Synthesis Lectures on Human Language Technologies, 14(5):1–314. [find](https://scholar.google.com/scholar?q=Beata+Beigman+Klebanov+and+Nitin+Madnani%2E+2021%2E+Automated+essay+scoring%2E+Synthesis+Lectures+on+Human+Language+Technologies%2C+14%285%29%3A1%E2%80%93314%2E)
30. Susan Lottridge, Chris Ormerod, and Amir Jafari. 2023. Psychometric considerations when using deep learning for automated scoring. Advancing natural language processing in educational assessment, pages 15–30. [↗](https://doi.org/10.4324/9781003278658-3)
31. Anastassia Loukina, Nitin Madnani, and Klaus Zechner. 2019. The many dimensions of algorithmic fairness in educational applications. In Proceedings of the fourteenth workshop on innovative use of NLP for building educational applications, pages 1–10. [find](https://scholar.google.com/scholar?q=Anastassia+Loukina%2C+Nitin+Madnani%2C+and+Klaus+Zechner%2E+2019%2E+The+many+dimensions+of+algorithmic+fairness+in+educational+applications%2E+In+Proceedings+of+the+fourteenth+workshop+on+innovative+use+of+NLP+)
32. Anastassia Loukina, Klaus Zechner, James Bruno, and Beata Beigman Klebanov. 2018. Using exemplar responses for training and evaluating automated speech scoring systems. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–12, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anastassia+Loukina%2C+Klaus+Zechner%2C+James+Bruno%2C+and+Beata+Beigman+Klebanov%2E+2018%2E+Using+exemplar+responses+for+training+and+evaluating+automated+speech+scoring+systems%2E+In+Proceedings+of+the+Thirteent)
33. Nitin Madnani and Aoife Cahill. 2018. Automated scoring: Beyond natural language processing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1099– 1109, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nitin+Madnani+and+Aoife+Cahill%2E+2018%2E+Automated+scoring%3A+Beyond+natural+language+processing%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+pages+1099%E2%80%93+1109%2C+Santa+F)
34. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Louis+Martin%2C+Benjamin+Muller%2C+Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Yoann+Dupont%2C+Laurent+Romary%2C+%C3%89ric+de+la+Clergerie%2C+Djam%C3%A9+Seddah%2C+and+Beno%C3%AEt+Sagot%2E+2020%2E+CamemBERT%3A+a+tasty+French+language+model%2E+In+Proceed) [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
35. Elijah Mayfield and Alan W Black. 2020. Should you fine-tune bert for automated essay scoring? In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 151–162. 6f410437ceb2c6f2762fbcdfa8a28e8c/ 2021_PTEA_White_Paper_Institutions_ Automated_Scoring_White_ Paper-May-2018.pdf, accessed june 30th, 2024. [find](https://scholar.google.com/scholar?q=Elijah+Mayfield+and+Alan+W+Black%2E+2020%2E+Should+you+fine%2Dtune+bert+for+automated+essay+scoring%3F+In+Proceedings+of+the+Fifteenth+Workshop+on+Innovative+Use+of+NLP+for+Building+Educational+Applications%2C+)
36. Daniel F McCaffrey, Jodi M Casabianca, Kathryn L Ricker-Pedley, René R Lawless, and Cathy Wendler. 2022. Best practices for constructed-response scoring. ETS Research Report Series, 2022(1):1–58. [↗](https://doi.org/10.1002/ets2.12358)
37. Chaitanya Ramineni, Catherine S. Trapani, David M. Williamson, Tim Davey, and Brent Bridgeman. 2012. Evaluation of the e-rater® scoring engine for the toefl® independent and integrated prompts. ETS Research Report Series, 2012(1):i–51. [↗](https://doi.org/10.1002/j.2333-8504.2012.tb02288.x)
38. Amália Mendes, Sandra Antunes, Maarten Janssen, and Anabela Gonçalves. 2016. The COPLE2 corpus: a learner corpus for Portuguese. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3207–3214, Portorož, Slovenia. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Am%C3%A1lia+Mendes%2C+Sandra+Antunes%2C+Maarten+Janssen%2C+and+Anabela+Gon%C3%A7alves%2E+2016%2E+The+COPLE2+corpus%3A+a+learner+corpus+for+Portuguese%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resour)
39. Bojana Ranković, Sarah Smirnow, Martin Jaggi, and Martin J. Tomasik. 2020. Automated Essay Scoring in Foreign Language Students Based on Deep Contextualised Word Representations. In LAK20-10th International Conference on Learning Analytics & Knowledge. Issue: CONF. [find](https://scholar.google.com/scholar?q=Bojana+Rankovic%CC%81%2C+Sarah+Smirnow%2C+Martin+Jaggi%2C+and+Martin+J%2E+Tomasik%2E+2020%2E+Automated+Essay+Scoring+in+Foreign+Language+Students+Based+on+Deep+Contextualised+Word+Representations%2E+In+LAK20%2D10th+Intern)
40. Samuel Messick. 1990. Validity of test interpretation and use. Technical Report ETS-RR-90-11, ETS, Princeton, N.J. [↗](https://doi.org/10.1002/j.2333-8504.1990.tb01343.x)
41. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. [find](https://scholar.google.com/scholar?q=Atsushi+Mizumoto+and+Masaki+Eguchi%2E+2023%2E+Exploring+the+potential+of+using+an+ai+language+model+for+automated+essay+scoring%2E+Research+Methods+in+Applied+Linguistics%2C+2%282%29%3A100050%2E)
42. Ricardo Muñoz Sánchez, David Alfter, Simon Dobnik, Maria Irena Szawerna, and Elena Volodina. 2024. Jingle BERT, jingle BERT, frozen all the way: Freezing layers to identify CEFR levels of second language learners using BERT. In Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning, pages 137–152, Rennes, France. LiU Electronic Press. [find](https://scholar.google.com/scholar?q=Ricardo+Mu%C3%B1oz+S%C3%A1nchez%2C+David+Alfter%2C+Simon+Dobnik%2C+Maria+Irena+Szawerna%2C+and+Elena+Volodina%2E+2024%2E+Jingle+BERT%2C+jingle+BERT%2C+frozen+all+the+way%3A+Freezing+layers+to+identify+CEFR+levels+of+second+langu)
43. Diane Nicholls. 2003. The Cambridge Learner Corpus: Error coding and analysis for lexicography and ELT. In Proceedings of the Corpus Linguistics 2003 conference, volume 16, pages 572–581. [find](https://scholar.google.com/scholar?q=Diane+Nicholls%2E+2003%2E+The+Cambridge+Learner+Corpus%3A+Error+coding+and+analysis+for+lexicography+and+ELT%2E+In+Proceedings+of+the+Corpus+Linguistics+2003+conference%2C+volume+16%2C+pages+572%E2%80%93581%2E)
44. Nicholas Parslow. 2015. Automated Analysis of L2 French Writing: a preliminary study. Master’s thesis. Publisher: Unpublished. [↗](https://doi.org/10.13140/rg.2.1.2833.5204)
45. Pearson. 2019. Pte academic automated scoring. retrieved from: https://assets.ctfassets.net/ yqwtwibiobs4/018RxttvPWsMkkGIQJ5Gg3/ Pedro Uria Rodriguez, Amir Jafari, and Christopher M Ormerod. 2019. Language models and automated essay scoring. arXiv preprint arXiv:1909.09482. [↗](https://arxiv.org/abs/1909.09482)
46. Lawrence M. Rudner, Veronica Garcia, and Catherine Welch. 2006. An evaluation of IntelliMetric™ essay scoring system. The Journal of Technology, Learning and Assessment, 4(4). [find](https://scholar.google.com/scholar?q=Lawrence+M%2E+Rudner%2C+Veronica+Garcia%2C+and+Catherine+Welch%2E+2006%2E+An+evaluation+of+IntelliMetric%E2%84%A2+essay+scoring+system%2E+The+Journal+of+Technology%2C+Learning+and+Assessment%2C+4%284%29%2E)
47. Mark D. Shermis. 2015. Contrasting State-of-the-Art in the Machine Scoring of Short-Form Constructed Responses. Educational Assessment, 20(1):46–65. [↗](https://doi.org/10.1080/10627197.2015.997617)
48. Elana Shohamy, Smadar Donitsa-Schmidt, and Irit Ferman. 1996. Test impact revisited: Washback effect over time. Language testing, 13(3):298– 317. [↗](https://doi.org/10.1177/026553229601300305)
49. Steven E Stemler and Jessica Tsai. 2008. Best practices in interrater reliability three common approaches. Best practices in quantitative methods, pages 29–49. [↗](https://doi.org/10.4135/9781412995627.d5)
50. Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1882–1891. [find](https://scholar.google.com/scholar?q=Kaveh+Taghipour+and+Hwee+Tou+Ng%2E+2016%2E+A+neural+approach+to+automated+essay+scoring%2E+In+Proceedings+of+the+2016+conference+on+empirical+methods+in+natural+language+processing%2C+pages+1882%E2%80%931891%2E) [our card](https://md.research.elamur.ai/free/en/papers/taghipour-2016-neural-approach-automated-essay-scoring/)
51. Yi Tay, Minh Phan, Luu Anh Tuan, and Siu Cheung Hui. 2018. Skipflow: Incorporating neural coherence features for end-to-end automatic text scoring. In Proceedings of the AAAI conference on artificial intelligence, volume 32. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Minh+Phan%2C+Luu+Anh+Tuan%2C+and+Siu+Cheung+Hui%2E+2018%2E+Skipflow%3A+Incorporating+neural+coherence+features+for+end%2Dto%2Dend+automatic+text+scoring%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+)
52. Kari Tenfjord, Paul Meurer, and Knut Hofland. 2006. The ASK corpus - a language learner corpus of Norwegian as a second language. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Kari+Tenfjord%2C+Paul+Meurer%2C+and+Knut+Hofland%2E+2006%2E+The+ASK+corpus+%2D+a+language+learner+corpus+of+Norwegian+as+a+second+language%2E+In+Proceedings+of+the+Fifth+International+Conference+on+Language+Resou)
53. Rodrigo Wilkens, Patrick Watrin, Rémi Cardon, Alice Pintard, Isabelle Gribomont, and Thomas François. 2024. Exploring hybrid approaches to readability: experiments on the complementarity between linguistic features and transformers. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2316–2331. [find](https://scholar.google.com/scholar?q=Rodrigo+Wilkens%2C+Patrick+Watrin%2C+R%C3%A9mi+Cardon%2C+Alice+Pintard%2C+Isabelle+Gribomont%2C+and+Thomas+Fran%C3%A7ois%2E+2024%2E+Exploring+hybrid+approaches+to+readability%3A+experiments+on+the+complementarity+between+lingu)
54. Masaki Uto, Itsuki Aomi, Emiko Tsutsumi, and Maomi Ueno. 2023. Integration of prediction scores from various automated essay scoring models using item response theory. IEEE Transactions on Learning Technologies. [↗](https://doi.org/10.1109/tlt.2023.3253215)
55. David M Williamson, Xiaoming Xi, and F Jay Breyer. 2012. A framework for evaluation and use of automated scoring. Educational measurement: issues and practice, 31(1):2–13. [find](https://scholar.google.com/scholar?q=David+M+Williamson%2C+Xiaoming+Xi%2C+and+F+Jay+Breyer%2E+2012%2E+A+framework+for+evaluation+and+use+of+automated+scoring%2E+Educational+measurement%3A+issues+and+practice%2C+31%281%29%3A2%E2%80%9313%2E)
56. Masaki Uto, Yikuan Xie, and Maomi Ueno. 2020. Neural automated essay scoring incorporating handcrafted features. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6077–6088. [↗](https://doi.org/10.5715/jnlp.28.716)
57. Salvatore Valenti, Francesca Neri, and Alessandro Cucchiarelli. 2003. An overview of current research on automated essay grading. Journal of Information Technology Education: Research, 2(1):319–330. Publisher: Informing Science Institute. [↗](https://doi.org/10.28945/331)
58. Alexander Von Eye and Eun Young Mun. 2014. Analyzing rater agreement: Manifest variable methods. Psychology Press. [find](https://scholar.google.com/scholar?q=Alexander+Von+Eye+and+Eun+Young+Mun%2E+2014%2E+Analyzing+rater+agreement%3A+Manifest+variable+methods%2E+Psychology+Press%2E)
59. Sara Cushing Weigle. 2013. English language learners and automated scoring of essays: Critical considerations. Assessing Writing, 18(1):85– 99. Automated Assessment of Writing. [↗](https://doi.org/10.1016/j.asw.2012.10.006)
60. Rodrigo Wilkens, David Alfter, Xiaoou Wang, Alice Pintard, Anaïs Tack, Kevin P. Yancey, and Thomas François. 2022. FABRA: French aggregator-based readability assessment toolkit. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1217– 1233, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Rodrigo+Wilkens%2C+David+Alfter%2C+Xiaoou+Wang%2C+Alice+Pintard%2C+Ana%C3%AFs+Tack%2C+Kevin+P%2E+Yancey%2C+and+Thomas+Fran%C3%A7ois%2E+2022%2E+FABRA%3A+French+aggregator%2Dbased+readability+assessment+toolkit%2E+In+Proceedings+of+the+)
61. Rodrigo Wilkens, Alice Pintard, David Alfter, Vincent Folny, and Thomas François. 2023. TCFLE-8: a corpus of learner written productions for French as a foreign language and its application to automated essay scoring. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3447– 3465, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rodrigo+Wilkens%2C+Alice+Pintard%2C+David+Alfter%2C+Vincent+Folny%2C+and+Thomas+Fran%C3%A7ois%2E+2023%2E+TCFLE%2D8%3A+a+corpus+of+learner+written+productions+for+French+as+a+foreign+language+and+its+application+to+automat)
62. Xiaoming Xi. 2008. What and how much evidence do we need? critical considerations in validating an automated scoring system. In C.A. Chapelle, Y.R. Chung, and J. Xu, editors, Towards adaptive CALL: Natural language processing for diagnostic language assessment, pages 102–114. Iowa State University Ames, IA. [find](https://scholar.google.com/scholar?q=Xiaoming+Xi%2E+2008%2E+What+and+how+much+evidence+do+we+need%3F+critical+considerations+in+validating+an+automated+scoring+system%2E+In+C%2EA%2E+Chapelle%2C+Y%2ER%2E+Chung%2C+and+J%2E+Xu%2C+editors%2C+Towards+adaptive+CALL%3A+Na)
63. Jiayi Xie, Kaiwei Cai, Li Kong, Junsheng Zhou, and Weiguang Qu. 2022. Automated essay scoring via pairwise contrastive regression. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2724– 2733, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiayi+Xie%2C+Kaiwei+Cai%2C+Li+Kong%2C+Junsheng+Zhou%2C+and+Weiguang+Qu%2E+2022%2E+Automated+essay+scoring+via+pairwise+contrastive+regression%2E+In+Proceedings+of+the+29th+International+Conference+on+Computational+)
64. Duanli Yan and Brent Bridgeman. 2020. Validation of automated scoring systems. In Handbook of Automated Scoring, pages 297–318. Chapman and Hall/CRC. [↗](https://doi.org/10.1201/9781351264808-16)
65. Duanli Yan, André A Rupp, and Peter W Foltz. 2020. Handbook of automated scoring: Theory into practice. CRC Press. [find](https://scholar.google.com/scholar?q=Duanli+Yan%2C+Andr%C3%A9+A+Rupp%2C+and+Peter+W+Foltz%2E+2020%2E+Handbook+of+automated+scoring%3A+Theory+into+practice%2E+CRC+Press%2E)
66. Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560–1569. [find](https://scholar.google.com/scholar?q=Ruosong+Yang%2C+Jiannong+Cao%2C+Zhiyuan+Wen%2C+Youzheng+Wu%2C+and+Xiaodong+He%2E+2020%2E+Enhancing+automated+essay+scoring+performance+via+fine%2Dtuning+pre%2Dtrained+language+models+with+combination+of+regression+an)
67. Tal Yarkoni, David Balota, and Melvin Yap. 2008. Moving beyond coltheart’s n: A new measure of orthographic similarity. Psychonomic bulletin & review, 15(5):971–979. [find](https://scholar.google.com/scholar?q=Tal+Yarkoni%2C+David+Balota%2C+and+Melvin+Yap%2E+2008%2E+Moving+beyond+coltheart%E2%80%99s+n%3A+A+new+measure+of+orthographic+similarity%2E+Psychonomic+bulletin+%26+review%2C+15%285%29%3A971%E2%80%93979%2E)
68. Wajdi Zaghouani. 2002. AUTO-ÉVAL : vers un modèle d’évaluation automatique des textes. In Actes du colloque des étudiants en sciences du langage, page 16, Montréal, Canada. Université du Québec à Montréal. [find](https://scholar.google.com/scholar?q=Wajdi+Zaghouani%2E+2002%2E+AUTO%2D%C3%89VAL+%3A+vers+un+mod%C3%A8le+d%E2%80%99%C3%A9valuation+automatique+des+textes%2E+In+Actes+du+colloque+des+%C3%A9tudiants+en+sciences+du+langage%2C+page+16%2C+Montr%C3%A9al%2C+Canada%2E+Universit%C3%A9+du+Qu%C3%A9bec+%C3%A0+Mont)
69. Torsten Zesch, Michael Wojatzki, and Dirk Scholten-Akoun. 2015. Task-independent features for automated essay grading. In Proceedings of the tenth workshop on innovative use of NLP for building educational applications, pages 224–232. [find](https://scholar.google.com/scholar?q=Torsten+Zesch%2C+Michael+Wojatzki%2C+and+Dirk+Scholten%2DAkoun%2E+2015%2E+Task%2Dindependent+features+for+automated+essay+grading%2E+In+Proceedings+of+the+tenth+workshop+on+innovative+use+of+NLP+for+building+educat)