--- title: "HHU-SyLo at SemEval-2026 Task 11: Logic in the Loop –" description: "The authors compare three approaches. 95.29% accuracy. 9 references, cited by 1 paper here." authors: ["Wiebke Petersen", "Cherine Jaziri", "Diem Tran"] year: 2026 venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)" source: "https://aclanthology.org/2026.semeval-1.400/" doi: "10.18653/v1/2026.semeval-1.400" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/petersen-2026-hhu-sylo-semeval-2026-task/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/petersen-2026-hhu-sylo-semeval-2026-task/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/13/) / HHU-SyLo at SemEval-2026 Task 11: Logic in the Loop – Hybridizing LLMs and Theorem Provers for Robust Formal Reasoning # HHU-SyLo at SemEval-2026 Task 11: Logic in the Loop – Hybridizing LLMs and Theorem Provers for Robust Formal Reasoning **Authors:** [Wiebke Petersen](https://md.research.elamur.ai/free/en/authors/wiebke-petersen/), Cherine Jaziri and Diem Tran · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.400/) · [PDF](https://aclanthology.org/2026.semeval-1.400.pdf) · [DOI 10.18653/v1/2026.semeval-1.400](https://doi.org/10.18653/v1/2026.semeval-1.400) · [OpenAlex](https://openalex.org/W7166869954) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/petersen-2026-hhu-sylo-semeval-2026-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This system paper addresses SemEval-2026 Task 11, which tests whether models can judge syllogistic validity without being swayed by how believable a conclusion sounds (the content effect, or belief bias). The task has four subtasks: English validity checking, validity checking with irrelevant premises to identify, and multilingual versions of both covering 11 languages. The authors compare three approaches. The first is direct LLM judgement (DNI). The second translates text into first-order logic, either with an LLM or with a regex rule engine, and checks it with the Otter theorem prover (TBI-FOL). The third maps text to Aristotelian triples checked against a Prolog knowledge base (TBI-SYL). All experiments use compact models with 135M–4B parameters. The FOL pipeline worked best, and Otter's refutation proofs also identify which premises were relevant. The best English system reached 95.29% accuracy. Results were sensitive to prompt and few-shot choices. The work shows that small LLMs can serve as reliable parsers when a symbolic solver handles the actual inference. ## Key points - **Architectures:** DNI, Neur-/Rule-TBI-FOL (Otter) and TBI-SYL (Prolog KB encoding all 256 classical moods). Existential import is added automatically as existence clauses. - **Models:** SmolLM, Qwen, Llama and Gemma families, run with greedy decoding. The top configuration, Eng2FOL5, used SmolLM3-3B for Subtask 1 and Qwen3-4B-Instruct for Subtask 2. - **Rankings:** Subtask 1 reached 95.29% (19th of 45). Subtask 2 reached 95.79% (9th of 16), with premise F1 of 92.37 and TCE of 3.26. Subtask 3 reached 88.02% (12th of 15) using native multilingual input. Subtask 4 reached 77.08% (11th of 15) after first translating inputs into English with Google Translate. - **Simplification:** Adding an LLM step that simplifies text before FOL translation hurt performance. TCE rose from 4.17 to 12.5, and accuracy on plausible-valid items fell from 89.58 to 70.83. - **Language variation:** Accuracy differed widely across languages, with Dutch scoring a perfect 100% while Swahili managed only 52%. Code is available on GitHub (WiebkePetersen/HHU-SyLo-SemEval2026-Task11). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Heinrich Heine University Düsseldorf](https://md.research.elamur.ai/free/en/universities/heinrich-heine-university-dusseldorf/) — University, Düsseldorf, Germany · 34 works · 1,487 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works ## Cite Cite as: Wiebke Petersen, Cherine Jaziri and Diem Tran. 2026. HHU-SyLo at SemEval-2026 Task 11: Logic in the Loop – Hybridizing LLMs and Theorem Provers for Robust Formal Reasoning. Proceedings of the 20th International Workshop on Semantic Evaluation (2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{petersen2026hhu, title = {{HHU-SyLo at SemEval-2026 Task 11: Logic in the Loop – Hybridizing LLMs and Theorem Provers for Robust Formal Reasoning}}, author = {Wiebke Petersen, Cherine Jaziri and Diem Tran}, booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)}, year = {2026}, doi = {10.18653/v1/2026.semeval-1.400}, url = {https://doi.org/10.18653/v1/2026.semeval-1.400} } ``` - [← previous HFT at BEA 2026 Shared Task 2: Blunt-Edge Models for Hybrid Grading](https://md.research.elamur.ai/free/en/papers/pado-2026-hft-bea-2026-shared-task/) - [next → Hi-SEMFLOW: Lie Algebra–Based Semantic Flow for Span-Level Informal Language Identification in Hindi](https://md.research.elamur.ai/free/en/papers/ravikiran-2026-hi-semflow-lie-algebra-based/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/13/) ## References 9 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. 2023. [Evaluating large language models with NeuBAROCO: Syllogistic reasoning ability and human-like biases](https://md.research.elamur.ai/free/en/papers/ando-2023-evaluating-large-language-models-neubaroco/). In Proceedings of the 4th Natural Logic Meets Machine Learning Workshop, pages 1–11, Nancy, France. Associat… 2. Jonathan St BT Evans, Julie L Barston, and Paul Pollard. 1983. On the conflict between logic and belief in syllogistic reasoning. Memory & Cognition, 11(3):295–306. [find](https://scholar.google.com/scholar?q=Jonathan+St+BT+Evans%2C+Julie+L+Barston%2C+and+Paul+Pollard%2E+1983%2E+On+the+conflict+between+logic+and+belief+in+syllogistic+reasoning%2E+Memory+%26+Cognition%2C+11%283%29%3A295%E2%80%93306%2E) 3. Wenyue Hua, Kaijie Zhu, Lingyao Li, Lizhou Fan, Mingyu Jin, Shuhang Lin, Haochen Xue, Zelong Li, Jindong Wang, and Yongfeng Zhang. 2025. Disentangling logic: The role of context in large language model reasoning capabilities. In Findings of the Association for Computational Linguistics: ACL 2025, pa… [find](https://scholar.google.com/scholar?q=Wenyue+Hua%2C+Kaijie+Zhu%2C+Lingyao+Li%2C+Lizhou+Fan%2C+Mingyu+Jin%2C+Shuhang+Lin%2C+Haochen+Xue%2C+Zelong+Li%2C+Jindong+Wang%2C+and+Yongfeng+Zhang%2E+2025%2E+Disentangling+logic%3A+The+role+of+context+in+large+language+mode) 4. Geonhee Kim, Marco Valentino, and Andre Freitas. 2025. Reasoning circuits in language models: A mechanistic interpretation of syllogistic inference. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10074–10095, Vienna, Austria. Association for Computational Linguistics.… [find](https://scholar.google.com/scholar?q=Geonhee+Kim%2C+Marco+Valentino%2C+and+Andre+Freitas%2E+2025%2E+Reasoning+circuits+in+language+models%3A+A+mechanistic+interpretation+of+syllogistic+inference%2E+In+Findings+of+the+Association+for+Computational+Li) 5. Aheli Poddar, Saptarshi Sahoo, and Sujata Ghosh. 2025. Understanding syllogistic reasoning in llms from formal and natural language perspectives. CoRR, abs/2512.12620. [find](https://scholar.google.com/scholar?q=Aheli+Poddar%2C+Saptarshi+Sahoo%2C+and+Sujata+Ghosh%2E+2025%2E+Understanding+syllogistic+reasoning+in+llms+from+formal+and+natural+language+perspectives%2E+CoRR%2C+abs%2F2512%2E12620%2E) 6. Marco Valentino, Geonhee Kim, Dhairya Dalal, Zhixue Zhao, and André Freitas. 2025. Mitigating content effects on reasoning in language models through fine-grained activation steering. arXiv preprint arXiv:2505.12189. — [arXiv:2505.12189](https://arxiv.org/abs/2505.12189) [find](https://scholar.google.com/scholar?q=Marco+Valentino%2C+Geonhee+Kim%2C+Dhairya+Dalal%2C+Zhixue+Zhao%2C+and+Andr%C3%A9+Freitas%2E+2025%2E+Mitigating+content+effects+on+reasoning+in+language+models+through+fine%2Dgrained+activation+steering%2E+arXiv+preprint+a) 7. Marco Valentino, Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, and André Freitas. 2026. [Semeval-2026 task 11: Disentangling content and formal reasoning in large language models](https://md.research.elamur.ai/free/en/papers/valentino-2026-semeval-2026-task-11-disentangling/). In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). 8. Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. 2024. [Faithful logical reasoning via symbolic chain-of-thought](https://md.research.elamur.ai/free/en/papers/xu-2024-faithful-logical-reasoning-symbolic-chain/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13326–13365, Bangkok, Thailand. Associat… 9. Shi Zong and Jimmy Lin. 2024. Categorical syllogisms revisited: A review of the logical reasoning abilities of LLMs for analyzing categorical syllogisms. In Proceedings of the 1st Workshop on NLP for Science (NLP4Science), pages 230–239, Miami, FL, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shi+Zong+and+Jimmy+Lin%2E+2024%2E+Categorical+syllogisms+revisited%3A+A+review+of+the+logical+reasoning+abilities+of+LLMs+for+analyzing+categorical+syllogisms%2E+In+Proceedings+of+the+1st+Workshop+on+NLP+for+) *Updated 31 October 2026*