---
title: "GradeLegal: Automated Grading for German Legal Cases (2026)"
description: "German law students sit long written case exams scored on an 18-point scale (0–18). 64 references."
authors: ["Abdullah Al Zubaer", "Lorenz Wendlinger", "Simon Alexander Nonn", "Michael Granitzer", "Jelena Mitrovic"]
year: 2026
venue: "arXiv:2605.21076"
source: "https://arxiv.org/abs/2605.21076"
doi: "10.48550/arxiv.2605.21076"
arxiv: "2605.21076"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zubaer-2026-gradelegal-automated-grading-german-legal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zubaer-2026-gradelegal-automated-grading-german-legal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/31/) / GradeLegal: Automated Grading for German Legal Cases

# GradeLegal: Automated Grading for German Legal Cases

**Authors:** Abdullah Al Zubaer, Lorenz Wendlinger, Simon Alexander Nonn, [Michael Granitzer](https://md.research.elamur.ai/free/en/authors/michael-granitzer/) and [Jelena Mitrovic](https://md.research.elamur.ai/free/en/authors/jelena-mitrovic/) · **Year:** 2026 · **Venue:** arXiv:2605.21076 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2605.21076) · [PDF](https://arxiv.org/pdf/2605.21076) · [DOI 10.48550/arxiv.2605.21076](https://doi.org/10.48550/arxiv.2605.21076) · [arXiv 2605.21076](https://arxiv.org/abs/2605.21076) · [OpenAlex](https://openalex.org/W7161956756)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
German law students sit long written case exams scored on an 18-point scale (0–18). Expert graders are scarce, so results can take weeks or months. The authors ask whether LLMs can approximate expert grades on such long-form solutions. They use an existing non-public set of 87 expert-graded student solutions, 71 in criminal law and 16 in public law. On this set they benchmark 27 proprietary and open models, both reasoning and non-reasoning. Each prompt variant adds more task material in steps, such as a sample solution, a grading rubric, or both. Agreement with experts is measured by quadratic weighted kappa (QWK).

With the rubric and the sample solution together, reasoning models such as GPT-5 reach about 0.91 QWK in public law, roughly matching human agreement. Criminal law tops out near 0.60. Both figures are well above earlier attempts. Small ensembles of open models beat their best member and narrow the gap to closed models. This matters because exam data is privacy-sensitive and benefits from local deployment. The authors see the approach as suitable for low-stakes formative feedback only, not for final exam decisions.

## Key points
- The evaluation covers 27 LLMs on 87 solutions (71 criminal law, 16 public law), scored on the German 0–18 ordinal scale. The data comes from earlier work by Strecker et al. and Wendlinger et al. and is not public, which reduces contamination risk.
- The best results come from GPT-5 given a rubric plus a sample solution: QWK 0.911 in public law and 0.599 in criminal law. A previous criminal-law study had reported a best of about 0.058.
- Rubric-only prompts outperform solution-only prompts, and combining the two gives peak agreement. Under identical full-context prompts, reasoning models beat non-reasoning ones.
- Ensembles of three fixed open models (Open-L, Open-S), combined by min or median, gain up to 0.15 QWK over their best member. Median aggregation is the more reliable rule for stronger ensembles, but gains are not universal.
- Several open and legal-domain models failed outright because of context length limits or output format problems. The code, the public law prompt and the instructor materials are released on GitHub; the student answers are not.

## Interesting ideas
- **Step-by-step prompt ablation.** Adding one kind of task information at a time (nothing, then exemplar, then rubric, then both) shows which inputs actually drive agreement, instead of testing a single all-in prompt.
- **Point-referenced rubrics over holistic exemplars.** Explicit point allocation tied to checkable evidence beats a single model answer, while the exemplar adds cues for holistic qualities such as structure and coherence.
- **Median aggregation and comparison against the best member.** Taking the median of several graders tolerates members whose scores are calibrated differently, and an ensemble only counts as a win if it beats its strongest member.
- **Error-bucket analysis of combined systems.** Ensemble gains came from shifting particular kinds of mistakes rather than shrinking all errors evenly, which shows up only when errors are binned by type.
- **Private evaluation data against contamination.** Using an unpublished corpus makes it less likely that the models saw the answers during pretraining.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Evaluate readers with knowledge sources switched on incrementally (e.g., VerbNet roles, WordNet ontology) to measure what each layer really contributes.
- **I2** (answer): Grade school-test answers by matching rubric items to frames in the student's text, so each awarded point traces to explicit evidence, with a reference answer only as a secondary signal.
- **I3** (learn): When witness parsers or LLM teacher tiers vote on silver labels, use median-style aggregation and always report gain over the best single witness as a control.
- **I4** (tree): When combining the perceptron parser and gparse, bucket disagreements by relation type to see which error modes the combination actually fixes.
- **I5** (learn): Keep a held-out private test set of fables and word problems for judging LLM teacher tiers, so their scores reflect reasoning rather than memorization.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Passau](https://md.research.elamur.ai/free/en/universities/university-of-passau/) — University, Passau, Germany · 4 works · 656 citations

Also on the byline: [Deggendorf Institute of Technology](https://ror.org/02kw5st29).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 182 works · [⚖️ Law](https://md.research.elamur.ai/free/en/shelves/social-sciences/law/) · 77 works

🧭 Topics (1): [⚖️ Legal AI](https://md.research.elamur.ai/free/en/topics/legal-ai/) · 134 works

## Cite

Cite as: Abdullah Al Zubaer, Lorenz Wendlinger, Simon Alexander Nonn, Michael Granitzer and Jelena Mitrovic. 2026. GradeLegal: Automated Grading for German Legal Cases. arXiv:2605.21076.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zubaer2026gradelegal,
  title = {{GradeLegal: Automated Grading for German Legal Cases}},
  author = {Abdullah {Al Zubaer}, Lorenz Wendlinger, Simon Alexander Nonn, Michael Granitzer and Jelena Mitrovic},
  year = {2026},
  doi = {10.48550/arxiv.2605.21076},
  eprint = {2605.21076},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.21076}
}
```

- [← previous GQLBench: A Large-Scale Cross-Domain, Cross-Dialect Benchmark for NL2GQL](https://md.research.elamur.ai/free/en/papers/su-2026-gqlbench-large-scale-cross-domain/)
- [next → Gradient Descenders at SemEval-2026 Task 9: Data-Centric Counterfactual Augmentation for Multi-Label Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/nhan-2026-gradient-descenders-semeval-2026-task/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/31/)

## References

64 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdullah Al Zubaer, Stephan Geschwind, Deborah Voss, Lorenz Wendlinger, Johann Graf Lambsdorff, Michael Granitzer, and Jelena Mitrović. 2025. Comparative Study of Language Models and Prompt Paradigms in Short Answer Grading. In 2025 IEEE 37th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, Athens, Greece, 321–329. doi:10.1109/ICTAI66417.2025.00049 [↗](https://doi.org/10.1109/ictai66417.2025.00049)
2. Abdullah Al Zubaer, Michael Granitzer, and Jelena Mitrović. 2023. Performance analysis of large language models in the domain of legal argument mining. Frontiers in artificial intelligence 6 (2023), 1278796. [↗](https://doi.org/10.3389/frai.2023.1278796)
3. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. [Longformer: The Long-Document Transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv:2004.05150 [cs.CL] https://arxiv.org/abs/2004. 05150 [↗](https://arxiv.org/abs/2004.05150)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Kaplan, et al. 2020. [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 1877–1901. [https://proceedings.neurips.cc/paper_files/paper/2020/file/](https://proceedings.neurips.cc/paper_files/paper/2020/file/) 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf [↗](https://proceedings.neurips.cc/paper_files/paper/2020/file/)
5. Marius Büttner and Ivan Habernal. 2024. Answering legal questions from laymen in German civil law system. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, St. Julian’s, Malta, 2015–2027. doi:10. 18653/v1/2024.eacl-long.122 [↗](https://aclanthology.org/2024.eacl-long.122/)
6. Odysseas S. Chlapanis, Dimitris Galanis, Nikolaos Aletras, and Ion Androutsopoulos. 2025. GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations. In Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, Suzhou, China, 25099–25119. doi:10.18653/v1/2025.findings-emnlp.1368 [↗](https://doi.org/10.18653/v1/2025.findings-emnlp.1368)
7. Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, et al. 2024. SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 129672–129695. doi:10.52202/079017-4120 [↗](https://doi.org/10.52202/079017-4120)
8. Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, and Michael Desa. 2024. SaulLM-7B: A pioneering Large Language Model for Law. arXiv:2403.03883 [cs.CL] https://arxiv.org/abs/2403.03883 [↗](https://arxiv.org/abs/2403.03883)
9. Kevin L. Cope, Jens Frankenreiter, Scott Hirst, Eric A. Posner, Daniel Schwarcz, and Dane Thorley. 2025. Grading Machines: Can AI Exam-Grading Replace Law Professors? doi:10.2139/ssrn.5851362 [↗](https://doi.org/10.2139/ssrn.5851362)
10. Scott A. Crossley, Perpetual Baffour, L. Burleigh, and Jules King. 2025. A large-scale corpus for assessing source-based writing quality: ASAP 2.0. Assessing Writing 65 (2025), 100954. doi:10.1016/j.asw.2025.100954 [↗](https://doi.org/10.1016/j.asw.2025.100954)
11. DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] https://arxiv.org/abs/2412.19437 [↗](https://arxiv.org/abs/2412.19437)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. doi:10.18653/v1/N19-1423 [↗](https://doi.org/10.18653/v1/n19-1423)
13. Ahmed M. Elmassry, Nazar Zaki, Negmeldin Alsheikh, and Mohammed Mediani. 2025. A Systematic Review of Pretrained Models in Automated Essay Scoring. IEEE Access 13 (2025), 121902–121917. doi:10.1109/ACCESS.2025.3584784 [↗](https://doi.org/10.1109/access.2025.3584784)
14. Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, et al. 2025. LEXam: Benchmarking Legal Reasoning on 340 Law Exams. arXiv:2505.12864 [cs.CL] https://arxiv.org/abs/2505.12864 [↗](https://arxiv.org/abs/2505.12864)
15. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. 2024. [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://md.research.elamur.ai/free/en/papers/fei-2023-lawbench-benchmarking-legal-knowledge-large/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Miami, Florida, USA, 7933– 7962. doi:10.18653/v1/2024.emnlp-main.452 [↗](https://doi.org/10.18653/v1/2024.emnlp-main.452)
16. Freie Universität Berlin, Department of Law. 2026. Grading Scales for Incoming Exchange Students. [https://www.jura.fu-berlin.de/en/international/](https://www.jura.fu-berlin.de/en/international/) studierendenaustausch/incomings/noten/ [↗](https://www.jura.fu-berlin.de/en/international/)
17. Shahriar Golchin, Nikhil Garuda, Christopher Impey, and Matthew Wenger. 2025. Grading Massive Open Online Courses Using Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics. Association for Computational Linguistics, Abu Dhabi, UAE, 3899–3912. [↗](https://doi.org/10.48550/arxiv.2406.11102)
18. Google DeepMind. 2025. Gemini 2.5. [https://blog.google/technology/googledeepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-thinking](https://blog.google/technology/googledeepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-thinking) Accessed: 2026-01-03. [↗](https://blog.google/technology/googledeepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-thinking)
19. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, et al. 2023. [LEGALBENCH: a collaboratively built benchmark for measuring legal reasoning in large language models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/). In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 1915, 157 pages. [↗](https://doi.org/10.2139/ssrn.4583531)
20. Ali Hakimi Parizi, Yuyang Liu, Prudhvi Nokku, Sina Gholamian, and David Emerson. 2023. A Comparative Study of Prompting Strategies for Legal Text Classification. In Proceedings of the Natural Legal Language Processing Workshop 2023. Association for Computational Linguistics, Singapore, 258–265. doi:10. 18653/v1/2023.nllp-1.25 [↗](https://aclanthology.org/2023.nllp-1.25/)
21. Ben Hamner, Jaison Morgan, lynnvandev, Mark Shermis, and Tom Vander Ark. 2012. The Hewlett Foundation: Automated Essay Scoring. [https://kaggle.com/](https://kaggle.com/) competitions/asap-aes. Kaggle. [↗](https://kaggle.com/)
22. Emaan Hariri and Daniel E Ho. 2026. AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark. In Proceedings of the Twentieth International Conference on Artificial Intelligence and Law (ICAIL ’25). Association for Computing Machinery, New York, NY, USA, 177–187. doi:10.1145/3769126.3769256 [↗](https://doi.org/10.1145/3769126.3769256)
23. Yue Huang and Joshua Wilson. 2025. Evaluating LLM-Based Automated Essay Scoring: Accuracy, Fairness, and Validity. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Works in Progress. National Council on Measurement in Education (NCME), Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States, 71–83. [find](https://scholar.google.com/scholar?q=Yue+Huang+and+Joshua+Wilson%2E+2025%2E+Evaluating+LLM%2DBased+Automated+Essay+Scoring%3A+Accuracy%2C+Fairness%2C+and+Validity%2E+In+Proceedings+of+the+Artificial+Intelligence+in+Measurement+and+Education+Conference)
24. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165 [↗](https://doi.org/10.1145/3600006.3613165)
25. LAION and LeoLM. 2025. LeoLM/leo-hessianai-70b-chat. Hugging Face. [https://huggingface.co/LeoLM/leo-hessianai-70b-chat](https://huggingface.co/LeoLM/leo-hessianai-70b-chat) Accessed: 2026-01-25. [↗](https://huggingface.co/LeoLM/leo-hessianai-70b-chat)
26. Shengjie Li and Vincent Ng. 2024. Automated Essay Scoring: A Reflection on the State of the Art. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Miami, Florida, USA, 17876–17888. doi:10.18653/v1/2024.emnlp-main.991 [↗](https://doi.org/10.18653/v1/2024.emnlp-main.991)
27. Pei Yee Liew and Ian K. T. Tan. 2025. On Automated Essay Grading using Large Language Models. In Proceedings of the 2024 8th International Conference on Computer Science and Artificial Intelligence (CSAI ’24). Association for Computing Machinery, New York, NY, USA, 204–211. doi:10.1145/3709026.3709030 [↗](https://doi.org/10.1145/3709026.3709030)
28. LiteLLM Contributors. 2024. Completion API Documentation. [https://docs.litellm.ai/docs/completion](https://docs.litellm.ai/docs/completion) Accessed January 9, 2026. [↗](https://docs.litellm.ai/docs/completion)
29. Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, et al. 2026. Ministral 3. arXiv:2601.08584 [cs.CL] https://arxiv.org/abs/2601.08584 [↗](https://arxiv.org/abs/2601.08584)
30. Watheq Ahmad Mansour, Salam Albatarni, Sohaila Eltanbouly, and Tamer Elsayed. 2024. Can Large Language Models Automatically Score Proficiency of Written Essays?. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). ELRA and ICCL, Torino, Italia, 2777–2786. https://aclanthology.org/2024.lrec-main.247/ [↗](https://aclanthology.org/2024.lrec-main.247/)
31. Meta AI. 2024. LLaMA 3.3 Model Card. Meta. [https://github.com/meta-llama/](https://github.com/meta-llama/) llama-models/blob/main/models/llama3_3/MODEL_CARD.md Accessed: 2026-01-26. [↗](https://github.com/meta-llama/)
32. Mistral AI. 2025. Mistral 3. Mistral AI. [https://mistral.ai/news/mistral-3](https://mistral.ai/news/mistral-3) Accessed: 2026-01-26. [↗](https://mistral.ai/news/mistral-3)
33. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, et al. 2025. [gpt-oss-120b & gpt-oss-20b Model Card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/). arXiv:2508.10925 [cs.CL] https://arxiv.org/abs/2508.10925 [↗](https://arxiv.org/abs/2508.10925)
34. OpenAI. 2024. GPT-4o mini: Advancing Cost-Efficient Intelligence. OpenAI. [https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/) Accessed: 2026-01-26. [↗](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)
35. OpenAI. 2024. Hello GPT-4o. OpenAI. [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/) Accessed: 2026-01-03. [↗](https://openai.com/index/hello-gpt-4o/)
36. OpenAI. 2024. OpenAI API Reference: Chat. OpenAI. [https://platform.openai.com/docs/api-reference/chat](https://platform.openai.com/docs/api-reference/chat) Accessed: 2026-01-26. [↗](https://platform.openai.com/docs/api-reference/chat)
37. OpenAI. 2025. GPT-5 System Card. [https://cdn.openai.com/gpt-5-system-card](https://cdn.openai.com/gpt-5-system-card). pdf. Accessed: 2026-01-03. [↗](https://cdn.openai.com/gpt-5-system-card)
38. OpenAI. 2025. GPT-5.1 Instant and GPT-5.1 Thinking System Card Addendum. System card addendum. OpenAI. [https://cdn.openai.com/pdf/4173ec8d-1229-47db-96de-06d87147e07e/5_1_system_card.pdf](https://cdn.openai.com/pdf/4173ec8d-1229-47db-96de-06d87147e07e/5_1_system_card.pdf) Accessed: 2026-01-03. [↗](https://cdn.openai.com/pdf/4173ec8d-1229-47db-96de-06d87147e07e/5_1_system_card.pdf)
39. OpenAI. 2025. Introducing GPT-4.1 in the API. OpenAI. [https://openai.com/](https://openai.com/) index/gpt-4-1/ Accessed: 2026-01-03. [↗](https://openai.com/)
40. OpenAI. 2025. Update to GPT-5 System Card: GPT-5.2. System card update. OpenAI. [https://cdn.openai.com/pdf/3a4153c8-c748-4b71-8e31-aecbde944f8d/](https://cdn.openai.com/pdf/3a4153c8-c748-4b71-8e31-aecbde944f8d/) oai_5_2_system-card.pdf Accessed: 2026-01-03. [↗](https://cdn.openai.com/pdf/3a4153c8-c748-4b71-8e31-aecbde944f8d/)
41. OpenRouter, Inc. 2026. OpenRouter. [https://openrouter.ai/](https://openrouter.ai/) Accessed Jan 9, 2026. [↗](https://openrouter.ai/)
42. Christopher Ormerod and Gitit Kehat. 2025. Long context Automated Essay Scoring with Language Models. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers. National Council on Measurement in Education (NCME), Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States, 35–42. [↗](https://doi.org/10.48550/arxiv.2509.10417)
43. Ramon Pires, Roseval Malaquias Junior, and Rodrigo Nogueira. 2026. Automatic Legal Writing Evaluation of LLMs. In Proceedings of the Twentieth International Conference on Artificial Intelligence and Law (ICAIL ’25). Association for Computing Machinery, New York, NY, USA, 420–424. doi:10.1145/3769126.3769227 [↗](https://doi.org/10.1145/3769126.3769227)
44. Qwen Team. 2025. Qwen Blog: Research and Model Advancements. Qwen. [https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd) Accessed: 2026-01-25. [↗](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd)
45. Dadi Ramesh and Suresh Kumar Sanampudi. 2022. An Automated Essay Scoring Systems: A Systematic Literature Review. Artificial Intelligence Review 55, 3 (2022), 2495–2527. doi:10.1007/s10462-021-10068-2 [↗](https://doi.org/10.1007/s10462-021-10068-2)
46. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/). In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, Singapore, 10776–10787.
47. Sergio Servantez, Joe Barrow, Kristian Hammond, and Rajiv Jain. 2024. Chain of Logic: Rule-Based Reasoning with Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, Bangkok, Thailand, 2721–2733. doi:10.18653/v1/2024.findings-acl.159 [↗](https://doi.org/10.18653/v1/2024.findings-acl.159)
48. Kathrin Seßler, Maurice Fürstenberg, Babette Bühler, and Enkelejda Kasneci. 2025. Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring. In Proceedings of the 15th International Learning Analytics and Knowledge Conference (LAK ’25). Association for Computing Machinery, New York, NY, USA, 462–472. doi:10.1145/3706468. 3706527 [↗](https://doi.org/10.1145/3706468)
49. Yashothara Shanmugarasa, Ming Ding, Chamikara Mahawaga Arachchige, and Thierry Rakotoarivelo. 2025. SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation. In Proceedings of the 20th ACM Asia Conference on Computer and Communications Security (ASIA CCS ’25). Association for Computing Machinery, New York, NY, USA, 425–441. [find](https://scholar.google.com/scholar?q=Yashothara+Shanmugarasa%2C+Ming+Ding%2C+Chamikara+Mahawaga+Arachchige%2C+and+Thierry+Rakotoarivelo%2E+2025%2E+SoK%3A+The+Privacy+Paradox+of+Large+Language+Models%3A+Advancements%2C+Privacy+Risks%2C+and+Mitigation%2E+In+P)
50. Michael B. Strecker, Susanne Hähnchen, Martin Heidebach, Marie Herberger, Simon Alexander Nonn, Sarah Großkopf, Gökhan Erol, Menaf Erol, Ilja Garber, Constantin Höhmann, Enci Huang, Clemens Hufeld, and Louisa Zachmann. 2026. KI-Unterstützung und Rohpunkteschemata: Die Zukunft der juristischen Klausurkorrektur? [https://ordnungderwissenschaft.de/wp-content/uploads/](https://ordnungderwissenschaft.de/wp-content/uploads/) 2025/12/Herberger.pdf [↗](https://ordnungderwissenschaft.de/wp-content/uploads/)
51. Swiss AI Initiative. 2025. Apertus-70B-Instruct-2509. Hugging Face. [https://huggingface.co/swiss-ai/Apertus-70B-Instruct-2509](https://huggingface.co/swiss-ai/Apertus-70B-Instruct-2509) Accessed: 2026-01-25. [↗](https://huggingface.co/swiss-ai/Apertus-70B-Instruct-2509)
52. Gemma Team. 2025. Gemma 3 Technical Report. arXiv:2503.19786 [cs.CL] https://arxiv.org/abs/2503.19786 [↗](https://arxiv.org/abs/2503.19786)
53. Qwen Team. 2025. [Qwen3 Technical Report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv:2505.09388 [cs.CL] https://arxiv.org/abs/2505.09388 [↗](https://arxiv.org/abs/2505.09388)
54. Qwen Team. 2025. QwQ-32B: Embracing the Power of Reinforcement Learning. [https://qwenlm.github.io/blog/qwq-32b/](https://qwenlm.github.io/blog/qwq-32b/) [↗](https://qwenlm.github.io/blog/qwq-32b/)
55. Masaki Uto. 2021. A review of deep-neural automated essay scoring models. Behaviormetrika 48, 2 (July 2021), 459–484. doi:10.1007/s41237-021-00142-y [↗](https://doi.org/10.1007/s41237-021-00142-y)
56. UTTER Project. 2025. EuroLLM-22B-Instruct-2512. Hugging Face. [https://huggingface.co/utter-project/EuroLLM-22B-Instruct-2512](https://huggingface.co/utter-project/EuroLLM-22B-Instruct-2512) Accessed: 2026-01-25. [↗](https://huggingface.co/utter-project/EuroLLM-22B-Instruct-2512)
57. Yongjie Wang, Chuang Wang, Ruobing Li, and Hui Lin. 2022. On the Use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Seattle, United States, 3416–3425. doi:10.18653/v1/2022.naacl-main.249 [↗](https://doi.org/10.18653/v1/2022.naacl-main.249)
58. Oliver Wardas and Florian Matthes. 2026. LLMs for Legal Subsumption in German Employment Contracts. In Proceedings of the Twentieth International Conference on Artificial Intelligence and Law (ICAIL ’25). Association for Computing Machinery, New York, NY, USA, 169–176. doi:10.1145/3769126.3769209 [↗](https://doi.org/10.1145/3769126.3769209)
59. Lorenz Wendlinger, Christian Braun, Abdullah Al Zubaer, Simon Alexander Nonn, Sarah Großkopf, Christofer Fellicious, and Michael Granitzer. 2024. On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education. arXiv:2412.15902 [cs.CL] https://arxiv.org/abs/2412.15902 LLMs Task Agnostic Reas oning. GPT-5.1 GPT-5 GPT-5.2 0.497 ±0.045 0.446 ±0.054 0.454 ±0.040 0.454 ±0.031 0.598 ±0.048 0.559 ±0.023 0.453 ±0.069 0.481 ±0.033 0.587 ±0.042 0.305 ±0.026 0.599 ±0.005 0.589 ±0.023 Non-Reas. *Table 7: Agreement across prompt variants for criminal law. Bold/underline = best/runner-up per column. Values are Pearson Corr. ±SD over three runs. Top-3 reasoning and non-reasoning models from Table 4. Shaded : open-source LLMs; Non-shaded: closed-source LLMs. Baseline: Corr.≈0.058 [59].* Instr. +Rubric Instr. Instr.+Rubric +Solution +Solution Mistral-L a GPT-4o GPT-4.1 0.429 ±0.019 0.376 ±0.044 0.552 ±0.011 0.228 ±0.014 0.269 ±0.044 0.400 ±0.031 0.369 ±0.050 0.581 ±0.030 0.492 ±0.008 0.471 ±0.049 0.325 ±0.051 0.541 ±0.016 a Mistral-Large-3-675B [↗](https://arxiv.org/abs/2412.15902)
60. Yiquan Wu, Siying Zhou, Yifei Liu, Weiming Lu, Xiaozhong Liu, Yating Zhang, Changlong Sun, Fei Wu, and Kun Kuang. 2023. Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Singapore, 12060–12075. doi:10.18653/ v1/2023.emnlp-main.740 [↗](https://aclanthology.org/2023.emnlp-main.740/)
61. Jiayi Xie, Kaiwei Cai, Li Kong, Junsheng Zhou, and Weiguang Qu. 2022. Automated Essay Scoring via Pairwise Contrastive Regression. In Proceedings of the 29th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 2724–2733. https://aclanthology.org/2022.coling-1.240/ [↗](https://aclanthology.org/2022.coling-1.240/)
62. Fangyi Yu, Lee Quartey, and Frank Schilder. 2023. Exploring the Effectiveness of Prompt Engineering for Legal Reasoning Tasks. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, Toronto, Canada, 13582–13596. doi:10.18653/v1/2023.findings-acl.858 [↗](https://doi.org/10.18653/v1/2023.findings-acl.858)
63. Weikang Yuan, Junjie Cao, Zhuoren Jiang, Yangyang Kang, Jun Lin, et al. 2024. Can Large Language Models Grasp Legal Theories? Enhance Legal Reasoning with Insights from Multi-Agent Collaboration. In Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, Miami, Florida, USA, 7577–7597. doi:10.18653/v1/2024.findings-emnlp.445 [↗](https://doi.org/10.18653/v1/2024.findings-emnlp.445)
64. Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, and Daniel E. Ho. 2025. [A Reasoning-Focused Legal Retrieval Benchmark](https://md.research.elamur.ai/free/en/papers/zheng-2025-reasoning-focused-legal-retrieval-benchmark/). In Proceedings of the 2025 Symposium on Computer Science and Law (Munich, Germany) (CSLAW ’25). Association for Computing Machinery, New York, NY, USA, 169–193. doi:10.1145/3709025.3712219 *Figure 1: Performance improvement (Δ) over the Task Agnostic baseline for each model, averaged across the criminal and public dataset, comparing Instr.+Rubric, Instr.+Solution, and Instr.+Rubric+Sol.* [↗](https://doi.org/10.1145/3709025.3712219)

*Updated 1 November 2026*