--- title: "BenGER Platform — paper by Nagl and Grabmair (2026)" description: "The authors present BenGER (Benchmark for German Law), an open-source browser platform that… 6 references, cited by 1 paper here." authors: ["Sebastian Nagl", "Matthias Grabmair"] year: 2026 venue: "arXiv:2604.13583" source: "https://arxiv.org/abs/2604.13583" doi: "10.48550/arxiv.2604.13583" arxiv: "2604.13583" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/nagl-2026-benger-platform-collaborative-web-platform/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nagl-2026-benger-platform-collaborative-web-platform/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/) / BenGER Platform: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks # BenGER Platform: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks **Authors:** [Sebastian Nagl](https://md.research.elamur.ai/free/en/authors/sebastian-nagl/) and [Matthias Grabmair](https://md.research.elamur.ai/free/en/authors/matthias-grabmair/) · **Year:** 2026 · **Venue:** arXiv:2604.13583 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2604.13583) · [PDF](https://arxiv.org/pdf/2604.13583) · [DOI 10.48550/arxiv.2604.13583](https://doi.org/10.48550/arxiv.2604.13583) · [arXiv 2604.13583](https://arxiv.org/abs/2604.13583) · [OpenAlex](https://openalex.org/W7154569069) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/nagl-2026-benger-platform-collaborative-web-platform/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Benchmarking LLMs on legal reasoning usually involves several disconnected tools. Experts supply material, engineers turn it into benchmark items, annotations are collected in a separate tool, models are run through one-off scripts, and evaluation code is rewritten for each study. This fragmentation hides decisions from legal experts and makes results hard to reproduce. The authors present BenGER (Benchmark for German Law), an open-source browser platform that handles the whole cycle in one system: task and reference-solution authoring, collaborative human annotation, batch runs of arbitrary LLMs, and scoring with a mix of metric families. It supports several organizations working together on one deployment, with each organization's data kept separate and access controlled by role. It can also give annotators LLM-generated feedback anchored in the reference answers. This is a short demo paper, so it reports no experiments. Its contribution is the system design and a live public instance. The aim is to let non-technical legal experts, public bodies and NGOs build and audit benchmarks without relying on engineers. ## Key points - The platform follows a six-stage workflow: - task creation with reference solutions - annotation - optional formative feedback - model execution - evaluation - analysis and export - Supported task formats are free-text reasoning, multiple choice and span annotation. Metric families are lexical, semantic, factual, classification and LLM-as-a-judge. - The stack is a Next.js (TypeScript) frontend and a FastAPI (Python) backend on PostgreSQL. Redis and Celery workers run model jobs and evaluations in the background. Deployment uses Docker Compose or Kubernetes. - Each organization's data is kept separate in storage, and permissions depend on roles such as administrator, contributor and annotator. API keys can be set per user or per project, so institutions can use their own credentials. - The authors position BenGER against LabelStudio, Doccano, Lawnotation and DeepWrite, which each cover only part of the pipeline. A public instance runs at what-a-benger.net, and the code is promised as free software on acceptance (ICAIL 2026 demo). ## Interesting ideas - BenGER treats an evaluation run as a saved, shareable object made of four parts: task definition, reference solution, model configuration and chosen metrics. Such a run can be audited and repeated later instead of living inside a notebook. - The same task is scored with several kinds of metrics side by side, from surface overlap to factual checks to a judge model. No single score can then dominate the conclusion. - Annotator feedback is grounded in the reference answer and modeled on the German "Repetitor" tutor. It names missing reasoning steps and common mistakes rather than just marking an answer wrong. - Human baselines are collected inside the same system as model runs, and agreement and consistency signals are tracked. Noisy or inconsistently interpreted annotations then do not silently skew conclusions. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): every evaluation, including its negative controls, could be saved as a versioned bundle of data slice, model build and metric set, so regressions are rerun against exactly the same setup. - **I2** (answer): answers on fable QA and school tests could be scored both by string match and by a fact-level check against the world graph, revealing answers that are right for the wrong reasons. - **I3** (answer): the maths word-problem solver could compare its derivation with a reference solution and report which step is missing, keeping the explanation chain-based rather than giving a bare verdict. - **I4** (learn): per-item agreement indicators among witness parsers could be stored next to the silver data, so low-agreement items are flagged before they enter training. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Technical University of Munich](https://md.research.elamur.ai/free/en/universities/technical-university-of-munich/) — University, Munich, Germany · 55 works · 48,474 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 182 works 🧭 Topics (1): [⚖️ Legal AI](https://md.research.elamur.ai/free/en/topics/legal-ai/) · 134 works ## Cite Cite as: Sebastian Nagl and Matthias Grabmair. 2026. BenGER Platform: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks. arXiv:2604.13583. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{nagl2026benger, title = {{BenGER Platform: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks}}, author = {Sebastian Nagl and Matthias Grabmair}, year = {2026}, doi = {10.48550/arxiv.2604.13583}, eprint = {2604.13583}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2604.13583} } ``` - [← previous Benchmarking Multilingual Temporal Reasoning in LLMs: The Temporal Reasoning Dataset](https://md.research.elamur.ai/free/en/papers/mazzia-2026-benchmarking-multilingual-temporal-reasoning-llms/) - [next → BertKittens at SemEval-2026 Task 3: Multi-Domain Aspect Sentiment with BERT/DeBERTa Ensembles for VA Regression and Aspect–Opinion–VA Triplets](https://md.research.elamur.ai/free/en/papers/sukhodolsky-2026-bertkittens-semeval-2026-task-3/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/) ## References 6 references, 0 of them with a card of ours. ↗ leads to the work. 1. Yu Fan, Jingwei Ni, Jakob Merane, Etienne Salimbeni, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. 2025. LEXam: Benchmarking Legal Reasoning on 340 Law Exams. arXiv:2505.12864 [cs] doi:10.48550/arXiv.2505.12864 [↗](https://doi.org/10.48550/arxiv.2505.12864) 2. Neel Guha, Daniel E. Ho, Julian Nyarko, and Christopher Ré. 2022. LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoning. arXiv:2209.06120 [cs] [↗](https://arxiv.org/abs/2209.06120) 3. Urs Kramer, Michael Granitzer, and Johann Graf Lambsdorff. 2024. DeepWrite: Annotation and Extraction of Legal Texts. [https://extract-annotations.deepwrite.pads.fim.uni-passau.de/](https://extract-annotations.deepwrite.pads.fim.uni-passau.de/). [↗](https://extract-annotations.deepwrite.pads.fim.uni-passau.de/) 4. Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text Annotation Tool for Human. [https://github.com/](https://github.com/) doccano/doccano Software available from https://github.com/doccano/doccano. [↗](https://github.com/) 5. Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2025. Label Studio: Data labeling software. [https://github.com/HumanSignal/label-studio](https://github.com/HumanSignal/label-studio) Open source software available from https://github.com/HumanSignal/label-studio. [↗](https://github.com/HumanSignal/label-studio) 6. Gijs van Dijck, Carlos Aguilera, Chris van der Lans, Shashank Chakravarthy, and Sander van Essel. 2022. Lawnotation: A Formal Language for Legal Rules. [https://www.lawnotation.org/](https://www.lawnotation.org/). [↗](https://www.lawnotation.org/) *Updated 31 October 2026*