--- title: "Training data generation for context-dependent rubric-based" description: "Automatically grading short open-ended answers needs a lot of domain-specific training data with rubrics and both correct and… 6 references." authors: ["Pavel Šindelář", "Filip Prášil", "Dávid Slivka", "Christopher Bouma", "Ondrej Bojar"] year: 2026 venue: "Proceedings of Leveraging Derived Text Formats to Unlock Copyrighted Collections for Open Science (DTF) @ LREC 2026" source: "https://aclanthology.org/2026.dtf-1.6/" doi: "10.63317/3tt9vin8tdc4" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/sindelar-2026-training-data-generation-context-dependent/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sindelar-2026-training-data-generation-context-dependent/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/) / Training data generation for context-dependent rubric-based short answer grading # Training data generation for context-dependent rubric-based short answer grading **Authors:** Pavel Šindelář, Filip Prášil, Dávid Slivka, Christopher Bouma and [Ondrej Bojar](https://md.research.elamur.ai/free/en/authors/ondrej-bojar/) · **Year:** 2026 · **Venue:** [Proceedings of Leveraging Derived Text Formats to Unlock Copyrighted Collections for Open Science (DTF) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/dtf/) ([Leveraging Derived Text Formats to Unlock Copyrighted Collections for Open Science (DTF) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/dtf/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.dtf-1.6/) · [PDF](https://aclanthology.org/2026.dtf-1.6.pdf) · [DOI 10.63317/3tt9vin8tdc4](https://doi.org/10.63317/3tt9vin8tdc4) · [OpenAlex](https://openalex.org/W7166505809) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Automatically grading short open-ended answers needs a lot of domain-specific training data with rubrics and both correct and incorrect answers, but suitable data such as past OECD PISA items are confidential. The authors had access to the English portion of a PISA dataset under an NDA and wanted an open substitute for a subtask of the Sensemaking 2026 task at the ELOQUENT lab. They built a large synthetic candidate pool with gpt-oss-120b, using contexts drawn from openly available websites. The model generated questions, three-part grading rubrics and graded answers, and LLM judges filtered or iteratively refined each step. The confidential data entered the process only as simple derived text formats: relational features such as BAAI/bge-m3 similarities and n-gram overlaps between fields. These features were used to pick the synthetic samples closest to the reference. Three surrogate datasets resulted, and all of them resemble PISA more closely than plain prompted generation does. Only the most relaxed (third) selection method gave a statistically significant improvement when training graders. This matters because closed educational data can inform openly shareable resources without being disclosed. ## Key points - Each datapoint contains a context, a question, a rubric with full, partial and no credit sections, an answer, and a ternary label (FC=2, PC=1, NC=0). - **Candidate generation:** - Web texts were split into chunks of 150–800 words and cleaned by an LLM. - The LLM produced 1–5 questions per context. - Rubrics went through a judge–critique loop, capped at 10 iterations, a cap that was never reached. - Five candidate answers were generated for each credit level, imitating hurried high-school writing with occasional typos. - **Selection signal:** the features capture relations between fields, such as context–question cosine similarity via BAAI/bge-m3. Without parts of the original items, they reveal almost nothing about them. - **Third selection method:** in a Wilcoxon signed-rank test, models trained on the selected subset beat random subsets (p = 0.00091), with an average accuracy gain of 0.0206. - **Second selection method:** accuracy was slightly lower (0.58 vs 0.61), and its significance estimate was p = 0.715. - **Release:** the code and the generated dataset are published on GitHub (sindelarp/training-data-generation-for-context-dependent-rubric-based-short-answer-grading). - **Limitation:** most evaluations rely on a single training run. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Charles University](https://md.research.elamur.ai/free/en/universities/charles-university/) — University, Prague, Czechia · 133 works · 20,686 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 102 works ## Cite Cite as: Pavel Šindelář, Filip Prášil, Dávid Slivka, Christopher Bouma and Ondrej Bojar. 2026. Training data generation for context-dependent rubric-based short answer grading. Proceedings of Leveraging Derived Text Formats to Unlock Copyrighted Collections for Open Science (DTF) @ LREC 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{sindelar2026training, title = {{Training data generation for context-dependent rubric-based short answer grading}}, author = {Pavel Šindelář, Filip Prášil, Dávid Slivka, Christopher Bouma and Ondrej Bojar}, booktitle = {Proceedings of Leveraging Derived Text Formats to Unlock Copyrighted Collections for Open Science (DTF) @ LREC 2026}, year = {2026}, doi = {10.63317/3tt9vin8tdc4}, url = {https://doi.org/10.63317/3tt9vin8tdc4} } ``` - [← previous Training Biomedical Retrievers From Large-Scale Citation Contexts](https://md.research.elamur.ai/free/en/papers/wang-2026-training-biomedical-retrievers-large-scale/) - [next → Training Verifier to Assessing Complex Real-World Tool-Use Trajectories](https://md.research.elamur.ai/free/en/papers/sun-2026-training-verifier-assessing-complex-real/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/) ## References 6 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Chengcheng Guo, Bo Zhao, and Yanbing Bai. 2022. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications, pages 181–195. Springer. [find](https://scholar.google.com/scholar?q=Chengcheng+Guo%2C+Bo+Zhao%2C+and+Yanbing+Bai%2E+2022%2E+Deepcore%3A+A+comprehensive+library+for+coreset+selection+in+deep+learning%2E+In+International+Conference+on+Database+and+Expert+Systems+Applications%2C+pages) 2. Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. [Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP](https://md.research.elamur.ai/free/en/papers/khattab-2022-demonstrate-search-predict-composing-retrieval/). arXiv preprint arXiv:2212.14024. 3. Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. 2021. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 5464–5474. PMLR. [find](https://scholar.google.com/scholar?q=Krishnateja+Killamsetty%2C+Sivasubramanian+Durga%2C+Ganesh+Ramakrishnan%2C+Abir+De%2C+and+Rishabh+Iyer%2E+2021%2E+Grad%2Dmatch%3A+Gradient+matching+based+data+subset+selection+for+efficient+deep+model+training%2E+In+In) 4. Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920–19930. [find](https://scholar.google.com/scholar?q=Garima+Pruthi%2C+Frederick+Liu%2C+Satyen+Kale%2C+and+Mukund+Sundararajan%2E+2020%2E+Estimating+training+data+influence+by+tracing+gradient+descent%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A19920%E2%80%9319) 5. Frank. Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics, 1:196–202. [find](https://scholar.google.com/scholar?q=Frank%2E+Wilcoxon%2E+1945%2E+Individual+comparisons+by+ranking+methods%2E+Biometrics%2C+1%3A196%E2%80%93202%2E) 6. Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. [Less: Selecting influential data for targeted instruction tuning](https://md.research.elamur.ai/free/en/papers/xia-2024-less-selecting-influential-data-targeted/). arXiv preprint arXiv:2402.04333. *Updated 1 November 2026*