---
title: "HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal"
description: "The paper addresses SemEval-2026 Task 11. 100% accuracy on ST1. 24 references, cited by 1 paper here."
authors: ["Abdullah Shaikh", "Zain Naqi", "Taha Zahid", "Sandesh Kumar", "Abdul Samad"]
year: 2026
venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)"
source: "https://aclanthology.org/2026.semeval-1.139/"
doi: "10.18653/v1/2026.semeval-1.139"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shaikh-2026-habib-taz-semeval-2026-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shaikh-2026-habib-taz-semeval-2026-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/13/) / HABIB\_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization

# HABIB\_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization

**Authors:** Abdullah Shaikh, Zain Naqi, Taha Zahid, [Sandesh Kumar](https://md.research.elamur.ai/free/en/authors/sandesh-kumar/) and [Abdul Samad](https://md.research.elamur.ai/free/en/authors/abdul-samad/) · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.139/) · [PDF](https://aclanthology.org/2026.semeval-1.139.pdf) · [DOI 10.18653/v1/2026.semeval-1.139](https://doi.org/10.18653/v1/2026.semeval-1.139) · [OpenAlex](https://openalex.org/W7166888476)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/shaikh-2026-habib-taz-semeval-2026-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses SemEval-2026 Task 11. Systems must decide whether a syllogism's conclusion follows formally from its premises, regardless of how believable it is. The task covers 12 languages, and some tracks add distractor premises that the system must filter out. The authors target the "content effect", where models tend to accept plausible conclusions. They fine-tune NLI-pretrained mDeBERTa-v3/DeBERTa-v3 encoders, adding a span-pooling premise-selection head for the noisy tracks.

Only 960 English training items were provided, so they built a rule-based generator over the 256 Aristotelian moods and figures. It draws nouns from WordNet hierarchies and adds symbolic-variable and rephrased variants, HANS samples, and template-based translations. Training combines three objectives:
- adaptive Group DRO over validity/plausibility (and language) subgroups,
- a scheduled differentiable bias penalty,
- KL consistency between simple and rephrased syllogisms.

The system ranked first with perfect scores on three subtasks and sixth on the hardest one. This suggests that controlled synthetic data plus bias-aware objectives can largely separate logic from plausibility.

## Key points
- Results: ranking score 100.0, 0.00% bias and 100% accuracy on ST1 (English), ST2 (noisy English) and ST3 (multilingual). On ST4 (noisy multilingual), the system placed 6th with 89.06% accuracy/F1, 2.89% bias and a 37.78 ranking score.
- Synthetic corpus sizes per subtask: 23,524 (ST1), 27,988 (ST2), 388,992 (ST3) and 648,632 (ST4) samples. The generator uses WordNet via NLTK, DeepL for noun translation, and HANS for syntactic robustness.
- Backbones:
  - mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 for ST1/ST3;
  - DeBERTa-v3-large-mnli-fever-anli-ling-wanli for ST2, where scaling up was key to the perfect score;
  - microsoft/mdeberta-v3-base for ST4.
- Ablations:
  - On ST3, adding the bias penalty alone cut the content effect from 8.29% to 1.04%. HANS data then lifted the system to a perfect score.
  - Symbolic training data consistently improved generalization.
- On ST4, KL regularization gave the most consistent cross-lingual behavior. It needs many forward passes, though, so it was trained on only about 4% of the data. Code and generation scripts are released on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Habib University](https://md.research.elamur.ai/free/en/universities/habib-university/) — University, Karachi, Pakistan · 5 works · 4 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Abdullah Shaikh, Zain Naqi, Taha Zahid, Sandesh Kumar and Abdul Samad. 2026. HABIB\_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shaikh2026habib,
  title = {{HABIB\_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization}},
  author = {Abdullah Shaikh, Zain Naqi, Taha Zahid, Sandesh Kumar and Abdul Samad},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.139},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.139}
}
```

- [← previous H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs](https://md.research.elamur.ai/free/en/papers/tekin-2026-h3fusion-helpful-harmless-honest-fusion/)
- [next → HAD: HAllucination Detection Language Models Based on a Comprehensive Hallucination Taxonomy](https://md.research.elamur.ai/free/en/papers/xu-2026-had-hallucination-detection-language-models/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/13/)

## References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. [Translation artifacts in cross-lingual transfer learning](https://md.research.elamur.ai/free/en/papers/artetxe-2020-translation-artifacts-cross-lingual-transfer/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674–7684, Online. Association for Computational Linguistics.
2. L. Bertolazzi, A. Gatt, A. M. Gliozzo, G. Greco, A. Madotto, and M. Valentino. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. arXiv preprint, arXiv:2406.11341. [find](https://scholar.google.com/scholar?q=L%2E+Bertolazzi%2C+A%2E+Gatt%2C+A%2E+M%2E+Gliozzo%2C+G%2E+Greco%2C+A%2E+Madotto%2C+and+M%2E+Valentino%2E+2024%2E+A+systematic+analysis+of+large+language+models+as+soft+reasoners%3A+The+case+of+syllogistic+inferences%2E+arXiv+preprin)
3. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly Media, Inc. [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Ewan+Klein%2C+and+Edward+Loper%2E+2009%2E+Natural+Language+Processing+with+Python%3A+Analyzing+Text+with+the+Natural+Language+Toolkit%2E+O%E2%80%99Reilly+Media%2C+Inc%2E)
4. F. Cheng, H. Li, F. Liu, R. van Rooij, K. Zhang, and Z. Lin. 2025. Empowering llms with logical reasoning: A comprehensive survey. arXiv preprint, arXiv:2502.15652. [find](https://scholar.google.com/scholar?q=F%2E+Cheng%2C+H%2E+Li%2C+F%2E+Liu%2C+R%2E+van+Rooij%2C+K%2E+Zhang%2C+and+Z%2E+Lin%2E+2025%2E+Empowering+llms+with+logical+reasoning%3A+A+comprehensive+survey%2E+arXiv+preprint%2C+arXiv%3A2502%2E15652%2E)
5. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2024. Language models show human-like content effects on reasoning tasks. Preprint, arXiv:2207.07051. [find](https://scholar.google.com/scholar?q=Ishita+Dasgupta%2C+Andrew+K%2E+Lampinen%2C+Stephanie+C%2E+Y%2E+Chan%2C+Hannah+R%2E+Sheahan%2C+Antonia+Creswell%2C+Dharshan+Kumaran%2C+James+L%2E+McClelland%2C+and+Felix+Hill%2E+2024%2E+Language+models+show+human%2Dlike+content+eff)
6. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. [Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). In International Conference on Learning Representations.
7. Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, and 51 others. 2024. [Trustllm: Trustworthiness in large…](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/)
8. G. Kim, M. Valentino, D. Dalal, Z. Zhao, and A. Freitas. 2024. Reasoning circuits in language models: A mechanistic interpretation of syllogistic inference. arXiv preprint, arXiv:2408.08590. [find](https://scholar.google.com/scholar?q=G%2E+Kim%2C+M%2E+Valentino%2C+D%2E+Dalal%2C+Z%2E+Zhao%2C+and+A%2E+Freitas%2E+2024%2E+Reasoning+circuits+in+language+models%3A+A+mechanistic+interpretation+of+syllogistic+inference%2E+arXiv+preprint%2C+arXiv%3A2408%2E08590%2E)
9. Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86. [find](https://scholar.google.com/scholar?q=Solomon+Kullback+and+Richard+A+Leibler%2E+1951%2E+On+information+and+sufficiency%2E+The+Annals+of+Mathematical+Statistics%2C+22%281%29%3A79%E2%80%9386%2E)
10. Moritz Laurer, Wouter van Atteveldt, Andreu Salleras Casas, and Kasper Welbers. 2022. Less annotating, more classifying – addressing the data scarcity issue of supervised machine learning with deep transfer learning and BERT-NLI. Preprint, Open Science Framework. [find](https://scholar.google.com/scholar?q=Moritz+Laurer%2C+Wouter+van+Atteveldt%2C+Andreu+Salleras+Casas%2C+and+Kasper+Welbers%2E+2022%2E+Less+annotating%2C+more+classifying+%E2%80%93+addressing+the+data+scarcity+issue+of+supervised+machine+learning+with+deep+tr)
11. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
12. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). Preprint, arXiv:1902.01007.
13. George A. Miller. 1995. [Wordnet: a lexical database for english](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Commun. ACM, 38(11):39–41.
14. OpenAI. 2026. ChatGPT (gpt-5.2 version). [https://chat.openai.com](https://chat.openai.com). Large language model accessed via web interface. [↗](https://chat.openai.com)
15. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 20… — [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
16. N. Patel, M. Kulkarni, M. Parmar, A. Budhiraja, M. Nakamura, N. Varshney, and C. Baral. 2024. [Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models](https://md.research.elamur.ai/free/en/papers/patel-2024-multi-logieval-evaluating-multi-step/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). Available…
17. X. Quan, M. Valentino, L. A. Dennis, and A. Freitas. 2024. [Verification and refinement of natural language explanations through llm-Symbolic theorem proving](https://md.research.elamur.ai/free/en/papers/quan-2024-verification-refinement-natural-language-explanations/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2933–2958. Available: arXiv:2405.0137…
18. Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. Preprint, arXiv:1911.08731. [find](https://scholar.google.com/scholar?q=Shiori+Sagawa%2C+Pang+Wei+Koh%2C+Tatsunori+B%2E+Hashimoto%2C+and+Percy+Liang%2E+2020%2E+Distributionally+robust+neural+networks+for+group+shifts%3A+On+the+importance+of+regularization+for+worst%2Dcase+generalization%2E)
19. M. Valentino, G. Kim, D. Dalal, Z. Zhao, and A. Freitas. 2025. Mitigating content effects on reasoning in language models through fine-grained activation steering. arXiv preprint, arXiv:2505.12189. — [arXiv:2505.12189](https://arxiv.org/abs/2505.12189) [find](https://scholar.google.com/scholar?q=M%2E+Valentino%2C+G%2E+Kim%2C+D%2E+Dalal%2C+Z%2E+Zhao%2C+and+A%2E+Freitas%2E+2025%2E+Mitigating+content+effects+on+reasoning+in+language+models+through+fine%2Dgrained+activation+steering%2E+arXiv+preprint%2C+arXiv%3A2505%2E12189%2E+%E2%80%94+)
20. Marco Valentino, Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, and André Freitas. 2026. Semeval-2026 task 11: Disentangling content and formal reasoning in language models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). [find](https://scholar.google.com/scholar?q=Marco+Valentino%2C+Leonardo+Ranaldi%2C+Giulia+Pucci%2C+Federico+Ranaldi%2C+and+Andr%C3%A9+Freitas%2E+2026%2E+Semeval%2D2026+task+11%3A+Disentangling+content+and+formal+reasoning+in+language+models%2E+In+Proceedings+of+the+2)
21. W. Wan, Z. Yang, Y. Chen, C. Luo, R. Wang, K. Cai, N. Kang, L. Lin, and K. Wang. 2025. Sr-fot: A syllogistic-reasoning framework of thought for large language models tackling knowledge-based reasoning tasks. arXiv preprint, arXiv:2501.11599. — [arXiv:2501.11599](https://arxiv.org/abs/2501.11599) [find](https://scholar.google.com/scholar?q=W%2E+Wan%2C+Z%2E+Yang%2C+Y%2E+Chen%2C+C%2E+Luo%2C+R%2E+Wang%2C+K%2E+Cai%2C+N%2E+Kang%2C+L%2E+Lin%2C+and+K%2E+Wang%2E+2025%2E+Sr%2Dfot%3A+A+syllogistic%2Dreasoning+framework+of+thought+for+large+language+models+tackling+knowledge%2Dbased+reasoning)
22. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. [Transf…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
23. X. Wu, Y. Bu, and 1 others. 2024. Content-free logical modification of large language models by disentangling and modifying logic representation. In Proceedings of the AAAI Conference on Artificial Intelligence. DOI pending. [find](https://scholar.google.com/scholar?q=X%2E+Wu%2C+Y%2E+Bu%2C+and+1+others%2E+2024%2E+Content%2Dfree+logical+modification+of+large+language+models+by+disentangling+and+modifying+logic+representation%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+In)
24. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2019. [Freelb: Enhanced adversarial training for language understanding](https://md.research.elamur.ai/free/en/papers/zhu-2019-freelb-enhanced-adversarial-training-natural/). CoRR, abs/1909.11764.

*Updated 31 October 2026*