---
title: "KEYSCORE –- Keystroke-enhanced Automated Essay Scoring"
description: "The authors introduce the PISA 2025 FLA dataset. 23 references. Among them: 0."
authors: ["Nils-Jonathan Schaller", "Daniel Mora Melanchthon", "Thorben Jansen", "Olaf Köller", "Andrea Horbach"]
year: 2026
venue: "Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)"
source: "https://aclanthology.org/2026.bea-1.24/"
doi: "10.18653/v1/2026.bea-1.24"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schaller-2026-keyscore-keystroke-enhanced-automated-essay/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schaller-2026-keyscore-keystroke-enhanced-automated-essay/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/) / KEYSCORE –- Keystroke-enhanced Automated Essay Scoring

# KEYSCORE –- Keystroke-enhanced Automated Essay Scoring

**Authors:** [Nils-Jonathan Schaller](https://md.research.elamur.ai/free/en/authors/nils-jonathan-schaller/), [Daniel Mora Melanchthon](https://md.research.elamur.ai/free/en/authors/daniel-mora-melanchthon/), [Thorben Jansen](https://md.research.elamur.ai/free/en/authors/thorben-jansen/), Olaf Köller and [Andrea Horbach](https://md.research.elamur.ai/free/en/authors/andrea-horbach/) · **Year:** 2026 · **Venue:** [Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2026](https://md.research.elamur.ai/free/en/venues/bea/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.bea-1.24/) · [PDF](https://aclanthology.org/2026.bea-1.24.pdf) · [DOI 10.18653/v1/2026.bea-1.24](https://doi.org/10.18653/v1/2026.bea-1.24) · [OpenAlex](https://openalex.org/W7166818646)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Automated essay scoring usually works only on finished texts, which makes it hard to give learners feedback while they are still writing. This paper asks whether keystroke logs can fill that gap. The authors introduce the PISA 2025 FLA dataset. It contains roughly 3,900 argumentative EFL writing sessions by 15-year-old German students, collected during the PISA 2025 assessment. Half the students could use a gpt-4o-mini chatbot (Chat) and half could not (NoChat). From each session the authors derive 25 process features covering timing, pauses and bursts, deletions, production efficiency and navigation. They compare three kinds of scorers across several time points within the 30-minute session: XGBoost models on these features, DeBERTa-v3-base text models, and a combined model that concatenates text embeddings with keystroke features. Process features predict quality far better early on, when there is too little text to score. Text models win once essays are complete. The combined model gives the strongest results averaged over all time points. The study suggests that keystroke data could support real-time, formative feedback during writing.

## Key points
- **Dataset.** The PISA 2025 FLA set starts from 4,091 sessions and keeps 3,882 scored essays after cleaning (1,911 NoChat, 1,971 Chat). Scores are holistic on a 0–5 scale. Access is by request until September 2027, after which the data will be openly released.
- **Chat vs. NoChat.** Chat essays are longer (274 vs. 200 words) and score higher (mean 3.2 vs. 2.1). They were written with fewer keystrokes, which points to copy-pasting from the chatbot.
- **Early prediction (5 minutes).** XGBoost on process features reaches QWK .32 (NoChat) and .59 (Chat). DeBERTa trained on full texts reaches only .15 and .21.
- **Complete essays.** Text-only DeBERTa performs best, with QWK .86 / .95 against .76 / .91 for the best keystroke model.
- **Combined model.** Averaged over all thresholds, it achieves the highest QWK: .67 NoChat, .82 Chat, .79 overall. In Chat it passes QWK .70 by 10 minutes, whereas the text-only model needs 20 minutes.
- **Limitation.** Training used silver-standard labels, although evaluation was against human gold scores.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Christian-Albrechts-Universität zu Kiel](https://md.research.elamur.ai/free/en/universities/christian-albrechts-universitat-zu-kiel/) — University, Kiel, Germany · 6 works · 38,870 citations
- [Leibniz Institute for Science and Mathematics Education](https://md.research.elamur.ai/free/en/labs/leibniz-institute-for-science-and-mathematics-education/) — Research institute, Kiel, Germany · 4 works · 261 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🖍️ Grammatical error correction & learner language](https://md.research.elamur.ai/free/en/topics/grammatical-error-correction/) · 156 works

## Cite

Cite as: Nils-Jonathan Schaller, Daniel Mora Melanchthon, Thorben Jansen, Olaf Köller and Andrea Horbach. 2026. KEYSCORE –- Keystroke-enhanced Automated Essay Scoring. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schaller2026keyscore,
  title = {{KEYSCORE –- Keystroke-enhanced Automated Essay Scoring}},
  author = {Nils-Jonathan Schaller, Daniel Mora Melanchthon, Thorben Jansen, Olaf Köller and Andrea Horbach},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.24},
  url = {https://doi.org/10.18653/v1/2026.bea-1.24}
}
```

- [← previous Ketaba-OCR at AR-MS NakbaNLP 2026: Efficient Adaptation of Vision-Language Models for Handwritten Recognition](https://md.research.elamur.ai/free/en/papers/barmandah-2026-ketaba-ocr-ar-ms-nakbanlp/)
- [next → KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge](https://md.research.elamur.ai/free/en/papers/robertson-2026-kghalubench-knowledge-graph-based-hallucination/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/)

## References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Laura K. Allen, Matthew E. Jacovina, Mihai Dascalu, Rod D. Roscoe, Kevin M. Kent, Aaron D. Likens, and Danielle S. McNamara. 2016. {ENTER}ing the time series {space}: Uncovering the writing process through keystroke analyses. In Proceedings of the 9th International Conference on Educational Data Min… [find](https://scholar.google.com/scholar?q=Laura+K%2E+Allen%2C+Matthew+E%2E+Jacovina%2C+Mihai+Dascalu%2C+Rod+D%2E+Roscoe%2C+Kevin+M%2E+Kent%2C+Aaron+D%2E+Likens%2C+and+Danielle+S%2E+McNamara%2E+2016%2E+%7BENTER%7Ding+the+time+series+%7Bspace%7D%3A+Uncovering+the+writing+process+th)
2. Rianne Conijn, Christine Cook, Menno van Zaanen, and Luuk Van Waes. 2022. Early prediction of writing quality using keystroke logging. International Journal of Artificial Intelligence in Education, 32(4):835– 866. [find](https://scholar.google.com/scholar?q=Rianne+Conijn%2C+Christine+Cook%2C+Menno+van+Zaanen%2C+and+Luuk+Van+Waes%2E+2022%2E+Early+prediction+of+writing+quality+using+keystroke+logging%2E+International+Journal+of+Artificial+Intelligence+in+Education%2C+32)
3. Elena Cotos. 2015. Automated writing analysis for writing pedagogy: From healthy tension to tangible prospects. Writing and Pedagogy, 7(2-3):263–281. [find](https://scholar.google.com/scholar?q=Elena+Cotos%2E+2015%2E+Automated+writing+analysis+for+writing+pedagogy%3A+From+healthy+tension+to+tangible+prospects%2E+Writing+and+Pedagogy%2C+7%282%2D3%29%3A263%E2%80%93281%2E)
4. Linda Flower and John R. Hayes. 1981. A cognitive process theory of writing. College Composition and Communication, 32(4):365–387. [find](https://scholar.google.com/scholar?q=Linda+Flower+and+John+R%2E+Hayes%2E+1981%2E+A+cognitive+process+theory+of+writing%2E+College+Composition+and+Communication%2C+32%284%29%3A365%E2%80%93387%2E)
5. Steve Graham, Michael Hebert, and Karen R. Harris. 2015. Formative assessment and writing: A meta-analysis. The Elementary School Journal, 115(4):523–547. [find](https://scholar.google.com/scholar?q=Steve+Graham%2C+Michael+Hebert%2C+and+Karen+R%2E+Harris%2E+2015%2E+Formative+assessment+and+writing%3A+A+meta%2Danalysis%2E+The+Elementary+School+Journal%2C+115%284%29%3A523%E2%80%93547%2E)
6. John Hattie and Helen Timperley. 2007. The power of feedback. Review of Educational Research, 77(1):81– 112. [find](https://scholar.google.com/scholar?q=John+Hattie+and+Helen+Timperley%2E+2007%2E+The+power+of+feedback%2E+Review+of+Educational+Research%2C+77%281%29%3A81%E2%80%93+112%2E)
7. John R. Hayes. 2012. Modeling and remodeling writing. Written Communication, 29(3):369–388. [find](https://scholar.google.com/scholar?q=John+R%2E+Hayes%2E+2012%2E+Modeling+and+remodeling+writing%2E+Written+Communication%2C+29%283%29%3A369%E2%80%93388%2E)
8. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. [Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). CoRR, abs/2111.09543.
9. Stefan D. Keller, Johanna Fleckenstein, Maleika Krüger, Olaf Köller, and André A. Rupp. 2020. English writing skills of students in upper secondary education: Results from an empirical study in switzerland and germany. Journal of Second Language Writing, 48:100700. [find](https://scholar.google.com/scholar?q=Stefan+D%2E+Keller%2C+Johanna+Fleckenstein%2C+Maleika+Kr%C3%BCger%2C+Olaf+K%C3%B6ller%2C+and+Andr%C3%A9+A%2E+Rupp%2E+2020%2E+English+writing+skills+of+students+in+upper+secondary+education%3A+Results+from+an+empirical+study+in+switze)
10. Stefan D. Keller, Julian Lohmann, Ruth Trüb, Johanna Fleckenstein, Jennifer Meyer, Thorben Jansen, and Jens Möller. 2024. Language quality, content, structure: What analytic ratings tell us about efl writing skills at upper secondary school level in germany and switzerland. Journal of Second Languag… [find](https://scholar.google.com/scholar?q=Stefan+D%2E+Keller%2C+Julian+Lohmann%2C+Ruth+Tr%C3%BCb%2C+Johanna+Fleckenstein%2C+Jennifer+Meyer%2C+Thorben+Jansen%2C+and+Jens+M%C3%B6ller%2E+2024%2E+Language+quality%2C+content%2C+structure%3A+What+analytic+ratings+tell+us+about+efl+)
11. Valerie J. Shute. 2008. Focus on formative feedback. Review of Educational Research, 78(1):153–189. [find](https://scholar.google.com/scholar?q=Valerie+J%2E+Shute%2E+2008%2E+Focus+on+formative+feedback%2E+Review+of+Educational+Research%2C+78%281%29%3A153%E2%80%93189%2E)
12. Hansel Susanto, Alexander Agung Santoso Gunawan, and Muhammad Fikri Hasani. 2024. Development of automated essay scoring system using DeBERTa as a transformer-based language model. In Data Analytics in System Engineering. [find](https://scholar.google.com/scholar?q=Hansel+Susanto%2C+Alexander+Agung+Santoso+Gunawan%2C+and+Muhammad+Fikri+Hasani%2E+2024%2E+Development+of+automated+essay+scoring+system+using+DeBERTa+as+a+transformer%2Dbased+language+model%2E+In+Data+Analytics+i)
13. CoMeSySo 2023, volume 935 of Lecture Notes in Networks and Systems, pages 202–215, Cham. Springer. [find](https://scholar.google.com/scholar?q=CoMeSySo+2023%2C+volume+935+of+Lecture+Notes+in+Networks+and+Systems%2C+pages+202%E2%80%93215%2C+Cham%2E+Springer%2E)
14. Yu Tian, Scott Crossley, and Luuk Van Waes. 2025. The klicke corpus: Keystroke logging in compositions for knowledge evaluation. Journal of Writing Research, 17:23–60. [find](https://scholar.google.com/scholar?q=Yu+Tian%2C+Scott+Crossley%2C+and+Luuk+Van+Waes%2E+2025%2E+The+klicke+corpus%3A+Keystroke+logging+in+compositions+for+knowledge+evaluation%2E+Journal+of+Writing+Research%2C+17%3A23%E2%80%9360%2E)
15. Masaki Uto, Yikuan Xie, and Maomi Ueno. 2020. Neural automated essay scoring incorporating hand-crafted features. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6077–6088, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Masaki+Uto%2C+Yikuan+Xie%2C+and+Maomi+Ueno%2E+2020%2E+Neural+automated+essay+scoring+incorporating+hand%2Dcrafted+features%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+page)
16. Nina Vandermeulen, Mariëlle Leijten, and Luuk Van Waes. 2020. Reporting writing process feedback in the classroom. using keystroke logging data to reflect on writing processes. Journal of Writing Research, 12:109–140. [find](https://scholar.google.com/scholar?q=Nina+Vandermeulen%2C+Mari%C3%ABlle+Leijten%2C+and+Luuk+Van+Waes%2E+2020%2E+Reporting+writing+process+feedback+in+the+classroom%2E+using+keystroke+logging+data+to+reflect+on+writing+processes%2E+Journal+of+Writing+Rese)
17. Georgios Velentzas, Andrew Caines, Rita Borgo, Erin Pacquetet, Clive Hamilton, Taylor Arnold, Diane Nicholls, Paula Buttery, Thomas Gaillat, Nicolas Ballier, and Helen Yannakoudakis. 2024. [find](https://scholar.google.com/scholar?q=Georgios+Velentzas%2C+Andrew+Caines%2C+Rita+Borgo%2C+Erin+Pacquetet%2C+Clive+Hamilton%2C+Taylor+Arnold%2C+Diane+Nicholls%2C+Paula+Buttery%2C+Thomas+Gaillat%2C+Nicolas+Ballier%2C+and+Helen+Yannakoudakis%2E+2024%2E)
18. Logging Learning Agency Lab. 2024. Automated essay scoring keystrokes in writing by English learners. In Pro-2.0. Kaggle Competition. 1st-place writeup: ceedings of the 2024 Joint International Conference [https://www.kaggle.com/competitions/](https://www.kaggle.com/competitions/) learning-agency-lab-automated-essay-scoring-2/ on Computat… [↗](https://www.kaggle.com/competitions/)
19. ELRA and ICCL. Mariëlle Leijten and Luuk Van Waes. 2013. Keystroke A Appendix logging in writing research. Written Communication, 30:358–392. [find](https://scholar.google.com/scholar?q=ELRA+and+ICCL%2E+Mari%C3%ABlle+Leijten+and+Luuk+Van+Waes%2E+2013%2E+Keystroke+A+Appendix+logging+in+writing+research%2E+Written+Communication%2C+30%3A358%E2%80%93392%2E)
20. Shengjie Li and Vincent Ng. 2024. Automated essay scoring: A reflection on the state of the art. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17876– 17888, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shengjie+Li+and+Vincent+Ng%2E+2024%2E+Automated+essay+scoring%3A+A+reflection+on+the+state+of+the+art%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+17876%E2%80%93)
21. Julian F. Lohmann, Fynn Junge, Jens Möller, Johanna Fleckenstein, Ruth Trüb, Stefan Keller, Thorben Jansen, and Andrea Horbach. 2025. Neural networks or linguistic features? Comparing different machine-learning approaches for automated assessment of text quality traits among L1- and L2-Learners’ arg… [find](https://scholar.google.com/scholar?q=Julian+F%2E+Lohmann%2C+Fynn+Junge%2C+Jens+M%C3%B6ller%2C+Johanna+Fleckenstein%2C+Ruth+Tr%C3%BCb%2C+Stefan+Keller%2C+Thorben+Jansen%2C+and+Andrea+Horbach%2E+2025%2E+Neural+networks+or+linguistic+features%3F+Comparing+different+machin)
22. NoLen. All Len. NoLen. .25 .42 .55 .57 .70 .72 .05 .26 .42 .56 .66 .68 .25 .38 .48 .50 .65 .67 .52 .52 .64 .71 .77 .80 .25 .42 .57 .62 .74 .77 .52 .48 .55 .60 .58 .63 *Table 6: QWK for linear regression baselines at each time threshold. A LL = all keystroke features; L EN = text length only; N O L E… [find](https://scholar.google.com/scholar?q=NoLen%2E+All+Len%2E+NoLen%2E+%2E25+%2E42+%2E55+%2E57+%2E70+%2E72+%2E05+%2E26+%2E42+%2E56+%2E66+%2E68+%2E25+%2E38+%2E48+%2E50+%2E65+%2E67+%2E52+%2E52+%2E64+%2E71+%2E77+%2E80+%2E25+%2E42+%2E57+%2E62+%2E74+%2E77+%2E52+%2E48+%2E55+%2E60+%2E58+%2E63+%2ATable+6%3A+QWK+for+linear+regressi)
23. Category Feature Definition Temporal write_time init_pause First to last keystroke (s) Task start to first key (ms) brk_count brk_total brk_mean_dur brk_ratio Pauses ≥2 000 ms Sum of pause durations (s) Mean pause duration (ms) Break / total time (0–1) bst_count bst_mean_len bst_mean_dur Production… [find](https://scholar.google.com/scholar?q=Category+Feature+Definition+Temporal+write%5Ftime+init%5Fpause+First+to+last+keystroke+%28s%29+Task+start+to+first+key+%28ms%29+brk%5Fcount+brk%5Ftotal+brk%5Fmean%5Fdur+brk%5Fratio+Pauses+%E2%89%A52+000+ms+Sum+of+pause+durations+%28)

*Updated 1 November 2026*