---
title: "GIL-Zaragoza at SemEval 2026 Task 11 — Lopez-Ponce 2026"
description: "The paper describes a team's entry to SemEval-2026 Task 11. 48.69 accuracy. 19 references, cited by 1 paper here."
authors: ["Francisco Lopez-Ponce", "Lucia Pitarch", "Iván Saavedra Martínez", "Ignacio Huitzil", "Sergio Ojeda Trueba", "Fernando Bobillo", "Gemma Bel-Enguix"]
year: 2026
venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)"
source: "https://aclanthology.org/2026.semeval-1.308/"
doi: "10.18653/v1/2026.semeval-1.308"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ponce-2026-gil-zaragoza-semeval-2026-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ponce-2026-gil-zaragoza-semeval-2026-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / GIL-Zaragoza at SemEval 2026 Task 11: Comparing Classification, Autoformalization, and Ontologies for Formal Reasoning Capabilities

# GIL-Zaragoza at SemEval 2026 Task 11: Comparing Classification, Autoformalization, and Ontologies for Formal Reasoning Capabilities

**Authors:** Francisco Lopez-Ponce, Lucia Pitarch, Iván Saavedra Martínez, Ignacio Huitzil, [Sergio Ojeda Trueba](https://md.research.elamur.ai/free/en/authors/sergio-luis-ojeda-trueba/), Fernando Bobillo and [Gemma Bel-Enguix](https://md.research.elamur.ai/free/en/authors/gemma-bel-enguix/) · **Year:** 2026 · **Venue:** Proceedings of the 20th International Workshop on Semantic Evaluation (2026) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.308/) · [PDF](https://aclanthology.org/2026.semeval-1.308.pdf) · [DOI 10.18653/v1/2026.semeval-1.308](https://doi.org/10.18653/v1/2026.semeval-1.308) · [OpenAlex](https://openalex.org/W7166872991)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper describes a team's entry to SemEval-2026 Task 11. The task checks whether models can judge if a syllogism is logically valid without being swayed by whether its content matches real-world knowledge. The team worked only on Subtask 1, which uses English syllogisms with no distractor premises, and compared three strategies:

- **Supervised classification:** TF-IDF features with classical classifiers, plus a fine-tuned BERT debiased with GroupDRO.
- **Autoformalization:** DPO-aligned Llama3 models translate each syllogism into first-order logic, and Prover9 checks the inference.
- **Neuro-symbolic:** GPT writes OWL 2 ontologies, which the HermiT reasoner evaluates.

The simplest method won. A Logistic Regression submission placed 40th out of 45 teams, scoring 72.25% on accuracy and 20.37 on the combined metric. The classical models were less sensitive to content than BERT. The symbolic pipelines were held back by poor translations, mismatches between generated logic and solver syntax, and error-prone serialization. The authors conclude that autoformalization is still a weak point for LLMs and that external solvers need robust syntax parsing. Lightweight classifiers remain useful because they show little content bias.

## Key points
- **Evaluation metric:** the official ranking metric (PRM) combines Accuracy with Total Content Effect (TCE), which measures sensitivity to plausible versus implausible content. The best official run had a TCE of 11.77.
- **Development results (80/20 split):** Naive Bayes reached the highest PRM (32.61), ahead of Logistic Regression (31.96) and SVM (29.71). BERT with GroupDRO scored 21.92 because of its higher TCE (8.09).
- **Autoformalization models:** four DPO checkpoints trained on FOLIO were used (Kurosawama/Llama-3.1-8B/3.2-3B, vanilla and Instruct "LogLim"). Instruct variants made fewer errors.
- **Autoformalization errors:** the models produced XOR operators that Prover9 rejects, which excluded roughly 27% of answers for one checkpoint. They also sometimes dropped the conclusion entirely.
- **Ontology approach:** using gpt-4o-mini, it scored 48.69 accuracy, 27.08 TCE and 11.23 PRM. The authors blame limited prompt exploration and the verbose RDF/XML format required by OwlReady2.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/)

## Institutions

- [Universidad Nacional Autónoma de México](https://md.research.elamur.ai/free/en/universities/universidad-nacional-autonoma-de-mexico/) — University, Mexico City, Mexico · 5 works · 283 citations

Also on the byline: [Universidad de Zaragoza](https://ror.org/012a91z28) · [Instituto de Investigación en Ingeniería de Aragón](https://ror.org/04eqgej32).

## Cite

Cite as: Francisco Lopez-Ponce, Lucia Pitarch, Iván Saavedra Martínez, Ignacio Huitzil, Sergio Ojeda Trueba, Fernando Bobillo and Gemma Bel-Enguix. 2026. GIL-Zaragoza at SemEval 2026 Task 11: Comparing Classification, Autoformalization, and Ontologies for Formal Reasoning Capabilities. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lopezponce2026gil,
  title = {{GIL-Zaragoza at SemEval 2026 Task 11: Comparing Classification, Autoformalization, and Ontologies for Formal Reasoning Capabilities}},
  author = {Francisco Lopez-Ponce, Lucia Pitarch, Iván Saavedra Martínez, Ignacio Huitzil, Sergio Ojeda Trueba, Fernando Bobillo and Gemma Bel-Enguix},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.308},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.308}
}
```

- [← previous German General Social Survey Personas: A Survey-Derived Persona Prompt Collection for Population-Aligned LLM Studies](https://md.research.elamur.ai/free/en/papers/rupprecht-2026-german-general-social-survey-personas/)
- [next → Gretino: A Greek and Latin Dataset to Benchmark Retrieval Systems in Classical Languages](https://md.research.elamur.ai/free/en/papers/toyin-2026-gretino-greek-latin-dataset-benchmark/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 oth… [our card](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/)
2. Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. 2025. Matharena: Evaluating llms on uncontaminated math competitions. [find](https://scholar.google.com/scholar?q=Mislav+Balunovic%CC%81%2C+Jasper+Dekoninck%2C+Ivo+Petrov%2C+Nikola+Jovanovic%CC%81%2C+and+Martin+Vechev%2E+2025%2E+Matharena%3A+Evaluating+llms+on+uncontaminated+math+competitions%2E)
3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
4. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2024. Language models show human-like content effects on reasoning tasks. Preprint, arXiv:2207.07051. [find](https://scholar.google.com/scholar?q=Ishita+Dasgupta%2C+Andrew+K%2E+Lampinen%2C+Stephanie+C%2E+Y%2E+Chan%2C+Hannah+R%2E+Sheahan%2C+Antonia+Creswell%2C+Dharshan+Kumaran%2C+James+L%2E+McClelland%2C+and+Felix+Hill%2E+2024%2E+Language+models+show+human%2Dlike+content+eff)
5. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. [our card](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
6. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… [find](https://scholar.google.com/scholar?q=Aaron+Grattafiori%2C+Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Alex+Vaughan%2C+Amy+Yang%2C+Angela+Fan%2C+Anirudh+Goyal%2C+Anth)
8. Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, and 16 others. 2024. FOLIO:… [our card](https://md.research.elamur.ai/free/en/papers/han-2022-folio-natural-language-reasoning-first/)
9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
10. Francisco F. López-Ponce and Gemma Bel-Enguix. 2025. Into the limits of logic: Alignment methods for formal logical reasoning. In Proceedings of The 3rd Workshop on Mathematical Natural Language Processing (MathNLP 2025), pages 112–123, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Francisco+F%2E+L%C3%B3pez%2DPonce+and+Gemma+Bel%2DEnguix%2E+2025%2E+Into+the+limits+of+logic%3A+Alignment+methods+for+formal+logical+reasoning%2E+In+Proceedings+of+The+3rd+Workshop+on+Mathematical+Natural+Language+Proce)
11. W. McCune. 2005–2010. Prover9 and mace4. [http://www.cs.unm.edu/~mccune/prover9/](http://www.cs.unm.edu/~mccune/prover9/). [↗](http://www.cs.unm.edu/~mccune/prover9/)
12. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925. [our card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/)
13. Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3806–3824, Singapore. Association for Computational Lingui… [our card](https://md.research.elamur.ai/free/en/papers/pan-2023-logic-lm-empowering-large-language/)
14. Auguste Poiroux, Gail Weiss, Viktor Kunčak, and Antoine Bosselut. 2025. Reliable evaluation and benchmarks for statement autoformalization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17947–17969, Suzhou, China. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Auguste+Poiroux%2C+Gail+Weiss%2C+Viktor+Kunc%CC%8Cak%2C+and+Antoine+Bosselut%2E+2025%2E+Reliable+evaluation+and+benchmarks+for+statement+autoformalization%2E+In+Proceedings+of+the+2025+Conference+on+Empirical+Methods+)
15. Xin Quan, Marco Valentino, Louise A. Dennis, and Andre Freitas. 2024. Verification and refinement of natural language explanations through LLM-symbolic theorem proving. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2933–2958, Miami, Florida, USA. As… [find](https://scholar.google.com/scholar?q=Xin+Quan%2C+Marco+Valentino%2C+Louise+A%2E+Dennis%2C+and+Andre+Freitas%2E+2024%2E+Verification+and+refinement+of+natural+language+explanations+through+LLM%2Dsymbolic+theorem+proving%2E+In+Proceedings+of+the+2024+Conf)
16. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290. [our card](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/)
17. Szegedy. 2022. Autoformalization with large language models. Preprint, arXiv:2205.12615. [our card](https://md.research.elamur.ai/free/en/papers/wu-2022-autoformalization-large-language-models/)
18. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Preprint, arXiv:2305.10601. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Preprint%2C+arXiv%3A)
19. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837. [our card](https://md.research.elamur.ai/free/en/papers/yue-2025-reinforcement-learning-really-incentivize-reasoning/)