--- title: "AMATI at BEA 2026 Shared Task 2: Automatic Short Answer" description: "The paper presents AMATI's system for the BEA 2026 shared task, which targets rubric-based short… 22 references, cited by 1 paper here." authors: ["Alistair Willis", "Aisling Third"] year: 2026 venue: "Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)" source: "https://aclanthology.org/2026.bea-1.89/" doi: "10.18653/v1/2026.bea-1.89" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/willis-2026-amati-bea-2026-shared-task/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/willis-2026-amati-bea-2026-shared-task/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/) / AMATI at BEA 2026 Shared Task 2: Automatic Short Answer Grading with Inductive Logic Programming and a Large Language Model # AMATI at BEA 2026 Shared Task 2: Automatic Short Answer Grading with Inductive Logic Programming and a Large Language Model **Authors:** Alistair Willis and Aisling Third · **Year:** 2026 · **Venue:** [Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2026](https://md.research.elamur.ai/free/en/venues/bea/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.bea-1.89/) · [PDF](https://aclanthology.org/2026.bea-1.89.pdf) · [DOI 10.18653/v1/2026.bea-1.89](https://doi.org/10.18653/v1/2026.bea-1.89) · [OpenAlex](https://openalex.org/W7166798692) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/willis-2026-amati-bea-2026-shared-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper presents AMATI's system for the BEA 2026 shared task, which targets rubric-based short answer scoring in German under the unseen-answers setting. The authors build a neuro-symbolic grader. Its aim is to pair interpretable grading rules with the tolerance that LLMs show for messy student language. Questions, rubrics and responses are lemmatised with spaCy and encoded as logic facts. Grading rules are then induced per question in an Inductive Logic Programming style, with the clingo Answer Set Programming solver choosing among a small set of rule templates. A greedy covering loop then assembles the selected rules into an ordered theory. This theory is placed in the prompt for Mistral-large, alongside the training answers. Adding the rules improved the 3-way (correct / partially correct / incorrect) results, raising QWK from 0.721 to 0.749. The 2-way results did not change. The authors stress that their main goal is explainable, editable grading for educators, not leaderboard performance. ## Key points - **Encoding:** Lemmas come from spaCy `de_core_news_md` and are stored as `lemma_in_doc(Doc, Position, Lemma)` facts. Each response is linked to its gold grade by a label fact. - **Rule templates:** Six templates are used. They cover single lemmas, lemma pairs (optionally ordered) and "key information" (Pandey 2022), meaning terms found in the rubric but absent from the question. - **Rule induction:** clingo optimises compression to pick each rule, and covered responses are removed after every step. When applied, the first matching rule sets the grade, and the default is incorrect. - **Prompt variants and results:** Four prompt variants were tried on trial data, and the "Test ILP" variants performed worse. In the final evaluation, QWK was: | Task | Rules only | Mistral | Mistral + Theory | |---|---|---|---| | 2-way | 0.489 | 0.644 | 0.644 | | 3-way | 0.614 | 0.721 | 0.749 | - **Ranking, limitations and code:** The team placed 6th of 9 in the 2-way task and 5th of 8 in the 3-way task. The authors note that English prompts were used for German data, each experiment was run once, and the templates are lexically limited. Code is on GitHub (AMATI_BEA_submission). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [The Open University](https://md.research.elamur.ai/free/en/universities/the-open-university/) — University, Milton Keynes, United Kingdom · 10 works · 338 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works ## Cite Cite as: Alistair Willis and Aisling Third. 2026. AMATI at BEA 2026 Shared Task 2: Automatic Short Answer Grading with Inductive Logic Programming and a Large Language Model. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{willis2026amati, title = {{AMATI at BEA 2026 Shared Task 2: Automatic Short Answer Grading with Inductive Logic Programming and a Large Language Model}}, author = {Alistair Willis and Aisling Third}, booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)}, year = {2026}, doi = {10.18653/v1/2026.bea-1.89}, url = {https://doi.org/10.18653/v1/2026.bea-1.89} } ``` - [← previous AMALIA: A Fully Open Large Language Model for European Portuguese](https://md.research.elamur.ai/free/en/papers/simplicio-2026-amalia-fully-open-large-language/) - [next → Ambirig at SemEval-2026 Task 5: Distributional Ordinal Modelling for Ambiguous Word Senses in Narrative Contexts](https://md.research.elamur.ai/free/en/papers/roy-2026-ambirig-semeval-2026-task-5/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/) ## References 22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Dishank Aggarwal, Pritam Sil, Bhaskaran Raman, and Pushpak Bhattacharyya. 2025. “I Understand Why I Got This Grade”: Automatic Short Answer Grading (ASAG) with Feedback. In Artificial Intelligence in Education, pages 304–318, Cham. Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Dishank+Aggarwal%2C+Pritam+Sil%2C+Bhaskaran+Raman%2C+and+Pushpak+Bhattacharyya%2E+2025%2E+%E2%80%9CI+Understand+Why+I+Got+This+Grade%E2%80%9D%3A+Automatic+Short+Answer+Grading+%28ASAG%29+with+Feedback%2E+In+Artificial+Intelligence+in+) 2. Bikram Pratim Bhuyan, Amar Ramdane-Cherif, Ravi Tomar, and T. P. Singh. 2024. Neuro-symbolic artificial intelligence: a survey. Neural Computing and Applications, 36(21):12809–12844. [find](https://scholar.google.com/scholar?q=Bikram+Pratim+Bhuyan%2C+Amar+Ramdane%2DCherif%2C+Ravi+Tomar%2C+and+T%2E+P%2E+Singh%2E+2024%2E+Neuro%2Dsymbolic+artificial+intelligence%3A+a+survey%2E+Neural+Computing+and+Applications%2C+36%2821%29%3A12809%E2%80%9312844%2E) 3. Ian Clark. 2012. Formative Assessment: Assessment Is for Self-regulated Learning. Educational Psychology Review, 24(2):205–249. [find](https://scholar.google.com/scholar?q=Ian+Clark%2E+2012%2E+Formative+Assessment%3A+Assessment+Is+for+Self%2Dregulated+Learning%2E+Educational+Psychology+Review%2C+24%282%29%3A205%E2%80%93249%2E) 4. Andrew Cropper, Sebastijan Dumančić, Richard Evans, and Stephen H. Muggleton. 2022. Inductive logic programming at 30. Machine Learning, 111(1):147– 172. ArXiv: 2102.10556 ISBN: 0123456789. — [arXiv:2102.10556](https://arxiv.org/abs/2102.10556) [find](https://scholar.google.com/scholar?q=Andrew+Cropper%2C+Sebastijan+Dumanc%CC%8Cic%CC%81%2C+Richard+Evans%2C+and+Stephen+H%2E+Muggleton%2E+2022%2E+Inductive+logic+programming+at+30%2E+Machine+Learning%2C+111%281%29%3A147%E2%80%93+172%2E+ArXiv%3A+2102%2E10556+ISBN%3A+0123456789%2E+%E2%80%94+%5BarXiv) 5. Jiaxin Cui and Rohan Alexander. 2026. Same Prompt, Different Outcomes: Evaluating the Reproducibility of Data Analysis by LLMs. ArXiv: 2602.14349. — [arXiv:2602.14349](https://arxiv.org/abs/2602.14349) [find](https://scholar.google.com/scholar?q=Jiaxin+Cui+and+Rohan+Alexander%2E+2026%2E+Same+Prompt%2C+Different+Outcomes%3A+Evaluating+the+Reproducibility+of+Data+Analysis+by+LLMs%2E+ArXiv%3A+2602%2E14349%2E+%E2%80%94+%5BarXiv%3A2602%2E14349%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2602%2E14349) 6. Marius-Constantin Dinu, Claudiu Leoveanu-Condrei, Markus Holzleitner, Werner Zellinger, and Sepp Hochreiter. 2024. SymbolicAI: A framework for logic-based approaches combining generative models and solvers. _eprint: 2402.00854. [find](https://scholar.google.com/scholar?q=Marius%2DConstantin+Dinu%2C+Claudiu+Leoveanu%2DCondrei%2C+Markus+Holzleitner%2C+Werner+Zellinger%2C+and+Sepp+Hochreiter%2E+2024%2E+SymbolicAI%3A+A+framework+for+logic%2Dbased+approaches+combining+generative+models+and+so) 7. Myroslava O. Dzikovska, Rodney Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. 2013. [Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge](https://md.research.elamur.ai/free/en/papers/dzikovska-2013-semeval-2013-task-7-joint/). In Second Joint Conference on Lexica… 8. Martin Gebser, Roland Kaminski, Benjamin Kaufmann, and Torsten Schaub. 2019. Multi-shot ASP solving with clingo. Theory and Practice of Logic Programming, 19(1):27–82. [find](https://scholar.google.com/scholar?q=Martin+Gebser%2C+Roland+Kaminski%2C+Benjamin+Kaufmann%2C+and+Torsten+Schaub%2E+2019%2E+Multi%2Dshot+ASP+solving+with+clingo%2E+Theory+and+Practice+of+Logic+Programming%2C+19%281%29%3A27%E2%80%9382%2E) 9. Sebastian Gombert, Zhifan Sun, Fabian Zehner, Jannik Lossjew, Tobias Wyrwich, Berrit Katharina Czinczel, David Bednorz, Sascha Bernholt, Knut Neumann, Ute Harms, Aiso Heinze, and Hendrik Drachsler. 2026. [Report on the BEA 2026 Shared Task on Rubric-based Short Answer Scoring for German](https://md.research.elamur.ai/free/en/papers/gombert-2026-report-bea-2026-shared-task/). In Proceedin… 10. Carolin Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. 2024. Evaluating the Elementary Multilingual Capabilities of Large Language Models with \\MultiQ\\. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4476–4494. [find](https://scholar.google.com/scholar?q=Carolin+Holtermann%2C+Paul+R%C3%B6ttger%2C+Timm+Dill%2C+and+Anne+Lauscher%2E+2024%2E+Evaluating+the+Elementary+Multilingual+Capabilities+of+Large+Language+Models+with+%5C%3Cspan+style%3D%22font%2Dvariant%3Asmallcaps%3B%22%5C%3EMultiQ%5C%3C) 11. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal%2C+Ines+Montani%2C+Sofie+Van+Lan%2Ddeghem%2C+and+Adriane+Boyd%2E+2020%2E+spaCy%3A+Industrial%2Dstrength+Natural+Language+Processing+in+Python%2E) 12. Mark Law, Alessandra Russo, and Krysia Broda. 2014. Inductive Learning of Answer Set Programs. In European Conference on Logics in Artificial Intelligence (JELIA). ArXiv: 1608.01946. — [arXiv:1608.01946](https://arxiv.org/abs/1608.01946) [find](https://scholar.google.com/scholar?q=Mark+Law%2C+Alessandra+Russo%2C+and+Krysia+Broda%2E+2014%2E+Inductive+Learning+of+Answer+Set+Programs%2E+In+European+Conference+on+Logics+in+Artificial+Intelligence+%28JELIA%29%2E+ArXiv%3A+1608%2E01946%2E+%E2%80%94+%5BarXiv%3A1608%2E019) 13. Joan Lu, Bhavya Krishna Balasubramanian, Mike Joy, and Qiang Xu. 2025. Survey and Analysis for the Challenges in Computer Science to the Automation of Grading Systems. ACM Computing Surveys, 58(1). [find](https://scholar.google.com/scholar?q=Joan+Lu%2C+Bhavya+Krishna+Balasubramanian%2C+Mike+Joy%2C+and+Qiang+Xu%2E+2025%2E+Survey+and+Analysis+for+the+Challenges+in+Computer+Science+to+the+Automation+of+Grading+Systems%2E+ACM+Computing+Surveys%2C+58%281%29%2E) 14. Mistral AI. 2024. Mistral Large 3 Model Card. [find](https://scholar.google.com/scholar?q=Mistral+AI%2E+2024%2E+Mistral+Large+3+Model+Card%2E) 15. Stephen Muggleton. 1995. Inverse entailment and Progol. New Generation Computing, 13(3-4):245–286. [find](https://scholar.google.com/scholar?q=Stephen+Muggleton%2E+1995%2E+Inverse+entailment+and+Progol%2E+New+Generation+Computing%2C+13%283%2D4%29%3A245%E2%80%93286%2E) 16. Stephen Muggleton and Cao Feng. 1990. Efficient Induction of Logic Programs. New Generation Computing, pages 368–381. ISBN: 354063875X. [find](https://scholar.google.com/scholar?q=Stephen+Muggleton+and+Cao+Feng%2E+1990%2E+Efficient+Induction+of+Logic+Programs%2E+New+Generation+Computing%2C+pages+368%E2%80%93381%2E+ISBN%3A+354063875X%2E) 17. Suraj Jung Pandey. 2022. Modelling Alignment and Key Information for Automatic Grading. PhD Thesis, The Open University. [find](https://scholar.google.com/scholar?q=Suraj+Jung+Pandey%2E+2022%2E+Modelling+Alignment+and+Key+Information+for+Automatic+Grading%2E+PhD+Thesis%2C+The+Open+University%2E) 18. J R Quinlan. 1990. Learning logical definition from Relations. Machine Learning, 5(1990):239–266. [find](https://scholar.google.com/scholar?q=J+R+Quinlan%2E+1990%2E+Learning+logical+definition+from+Relations%2E+Machine+Learning%2C+5%281990%29%3A239%E2%80%93266%2E) 19. Ayush Thakur. 2025. Eisvogel: Evaluating German Language Proficiency. [find](https://scholar.google.com/scholar?q=Ayush+Thakur%2E+2025%2E+Eisvogel%3A+Evaluating+German+Language+Proficiency%2E) 20. Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. 2024. [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature, 625(7995):476–482. 21. Alistair Willis. 2015. Using NLP to Support Scalable Assessment of Short Free Text Responses. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 243–253, Denver, Colorado. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alistair+Willis%2E+2015%2E+Using+NLP+to+Support+Scalable+Assessment+of+Short+Free+Text+Responses%2E+In+Proceedings+of+the+Tenth+Workshop+on+Innovative+Use+of+NLP+for+Building+Educational+Applications%2C+pages) 22. Xiao-Wen Yang, Jie-Jing Shao, Lan-Zhe Guo, Bo-Wen Zhang, Zhi Zhou, Lin-Han Jia, Wang-Zhou Dai, and Yu-Feng Li. 2025. Neuro-Symbolic Artificial Intelligence: Towards Improving the Reasoning Abilities of Large Language Models. In Proceedings of the Thirty-Fourth International Joint Conference on Artif… [find](https://scholar.google.com/scholar?q=Xiao%2DWen+Yang%2C+Jie%2DJing+Shao%2C+Lan%2DZhe+Guo%2C+Bo%2DWen+Zhang%2C+Zhi+Zhou%2C+Lin%2DHan+Jia%2C+Wang%2DZhou+Dai%2C+and+Yu%2DFeng+Li%2E+2025%2E+Neuro%2DSymbolic+Artificial+Intelligence%3A+Towards+Improving+the+Reasoning+Abilities+o) *Updated 31 October 2026*