--- title: "Using k-Shot Prompting with Large k for the Automated Scoring" description: "The authors examine whether a large language model can automatically score a German Written Elicited Imitation Test (WEIT). 2 references." authors: ["Malte Sternik", "Ronja Laarmann-Quante", "Anastasia Drackert"] year: 2026 venue: "Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)" source: "https://aclanthology.org/2026.bea-1.20/" doi: "10.18653/v1/2026.bea-1.20" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/sternik-2026-k-shot-prompting-large-k/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sternik-2026-k-shot-prompting-large-k/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/) / Using k-Shot Prompting with Large k for the Automated Scoring of a German Written Elicited Imitation Test # Using k-Shot Prompting with Large k for the Automated Scoring of a German Written Elicited Imitation Test **Authors:** Malte Sternik, [Ronja Laarmann-Quante](https://md.research.elamur.ai/free/en/authors/ronja-laarmann-quante/) and Anastasia Drackert · **Year:** 2026 · **Venue:** [Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2026](https://md.research.elamur.ai/free/en/venues/bea/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.bea-1.20/) · [PDF](https://aclanthology.org/2026.bea-1.20.pdf) · [DOI 10.18653/v1/2026.bea-1.20](https://doi.org/10.18653/v1/2026.bea-1.20) · [OpenAlex](https://openalex.org/W7166805334) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary The authors examine whether a large language model can automatically score a German Written Elicited Imitation Test (WEIT). In this test, learners of German as a foreign language briefly see a sentence and then type it from memory. Each response receives a score from 0 to 4 based on a rubric that separates typos, meaning-preserving changes, grammatical errors and loss of content. Using the WEIT dataset from Chifligarov et al. (2025), they prompted GPT-4o with up to 700 scored stimulus-response pairs but did not include the rubric. They studied three sources of variability: the number of shots k, which training examples were sampled, and repeated runs of the same prompt. Larger k raised accuracy significantly. Sample choice also mattered, though its effect faded at high k. Accuracy varied between identical runs, while QWK stayed more stable. The k-shot setup beat a rubric-only zero-shot GPT baseline but did not surpass earlier rule-based or BERT-based scorers. This suggests that LLM variability currently limits practical use, and that combining different model types is a promising direction. ## Key points - **Data:** The dataset has 3,900 stimulus-response pairs (195 participants × 20 sentences), split as in Chifligarov et al. (2025). The training pool contains 3,260 pairs, and the development set has 125 pairs balanced across scores. - **Experiment 1 design:** k ranged from 10 to 700 across 18 values. Each k was tested with 2 samples and 4 runs, giving 144 GPT-4o runs at temperature 0. Experiment 2 focused on small k (≤ 25). The evaluation metrics were accuracy and quadratically weighted kappa (QWK). - **Comparison with earlier models:** Differences in correctly scored items between GPT-k-shot and the rule-based or BERT-based (DistilBERT) models were not significant. Only the BERT-based model dropped significantly on unseen stimuli (χ²(1) = 11.31, p \< .001). - **Error patterns by score:** The rule-based model did best at the scale edges (scores 0, 1, 4), and BERT did best on score 2. GPT-k-shot had the highest F1 on score 3. GPT models tended to over-score grammatical errors (2 predicted as 3), whereas BERT and rule-based models tended to under-score (3 predicted as 2). - **Availability:** Code and data are released at gitlab.ruhr-uni-bochum.de/vamos-cl/german-weit-k-shot-prompting. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖍️ Grammatical error correction & learner language](https://md.research.elamur.ai/free/en/topics/grammatical-error-correction/) · 162 works ## Cite Cite as: Malte Sternik, Ronja Laarmann-Quante and Anastasia Drackert. 2026. Using k-Shot Prompting with Large k for the Automated Scoring of a German Written Elicited Imitation Test. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{sternik2026using, title = {{Using k-Shot Prompting with Large k for the Automated Scoring of a German Written Elicited Imitation Test}}, author = {Malte Sternik, Ronja Laarmann-Quante and Anastasia Drackert}, booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)}, year = {2026}, doi = {10.18653/v1/2026.bea-1.20}, url = {https://doi.org/10.18653/v1/2026.bea-1.20} } ``` - [← previous Using Interaction Log Data to Evaluate and Improve Feedback Accuracy in an Intelligent Language Tutoring System](https://md.research.elamur.ai/free/en/papers/soliar-2026-interaction-log-data-evaluate-improve/) - [next → Using LLMs for Automatic Discipline Annotation in a Diachronic Corpus of English Scientific Papers](https://md.research.elamur.ai/free/en/papers/bagdasarov-2026-llms-automatic-discipline-annotation-diachronic/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Berk Atıl, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. 2025. Non-determinism of “deterministic” LLM system settings in hosted environments. In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems, pages 135–148, Mumbai, India. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Berk+At%C4%B1l%2C+Sarp+Aykent%2C+Alexa+Chittams%2C+Lisheng+Fu%2C+Rebecca+J%2E+Passonneau%2C+Evan+Radcliffe%2C+Guru+Rajan+Rajagopal%2C+Adam+Sloan%2C+Tomasz+Tudrej%2C+Ferhan+Ture%2C+Zhe+Wu%2C+Lixinyu+Xu%2C+and+Breck+Baldwin%2E+2025%2E+No) 2. Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. 2025. [In-context learning with long-context models: An in-depth exploration](https://md.research.elamur.ai/free/en/papers/bertsch-2025-context-learning-long-context-models/). In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 12119–12149, Albuquerque, New Mexico. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2405.00200) *Updated 1 November 2026*