--- title: "ModusPonens at SemEval-2026 Task 11: Breaking the" description: "The paper addresses belief bias in syllogistic reasoning for SemEval-2026 Task 11. 87.23% accuracy. 3 references, cited by 1 paper here." authors: ["Soumyajit Roy", "Manav Malhotra"] year: 2026 venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)" source: "https://aclanthology.org/2026.semeval-1.10/" doi: "10.18653/v1/2026.semeval-1.10" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/roy-2026-modusponens-semeval-2026-task-11/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/roy-2026-modusponens-semeval-2026-task-11/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/15/) / ModusPonens at SemEval-2026 Task 11: Breaking the Plausibility Trap in LLMs via Conflict-Aware Ensembling # ModusPonens at SemEval-2026 Task 11: Breaking the Plausibility Trap in LLMs via Conflict-Aware Ensembling **Authors:** [Soumyajit Roy](https://md.research.elamur.ai/free/en/authors/soumyajit-roy/) and Manav Malhotra · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.10/) · [PDF](https://aclanthology.org/2026.semeval-1.10.pdf) · [DOI 10.18653/v1/2026.semeval-1.10](https://doi.org/10.18653/v1/2026.semeval-1.10) · [OpenAlex](https://openalex.org/W7166837715) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/roy-2026-modusponens-semeval-2026-task-11/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper addresses belief bias in syllogistic reasoning for SemEval-2026 Task 11. LLMs often confuse whether a conclusion follows logically with whether it sounds true. The hardest case is invalid arguments with believable conclusions. Using roughly 768 training syllogisms split into four validity/plausibility subgroups, the authors fine-tune Qwen-2.5-14B-Instruct with LoRA in two ways. The "Believer" is trained on the original data distribution. The "Skeptic" is trained on a set where samples whose validity and plausibility disagree are duplicated five times. At inference, each model's probabilities over the "true" and "false" tokens are averaged with equal weight (α = 0.5). The authors also report negative results. Symbolic chain-of-thought, nonsense-word substitution, ORPO, and a consistency loss on abstracted inputs all hurt performance. They attribute this to an "abstraction penalty": with so little data, removing familiar semantic content makes reasoning harder. The ensemble lowers the Total Content Effect to 3.21 and reaches a Combined Score of 39.09. This shows that simple data reweighting plus probability fusion can reduce content bias without losing accuracy. ## Key points - **Setup:** unsloth/Qwen2.5-14B-Instruct-bnb-4bit with LoRA (r = 16, all attention and MLP projections), trained on a single NVIDIA A40. The 80/20 train/validation split is stratified by the four subgroups (VP, VI, IP, II). - **Believer vs. Skeptic on invalid-but-plausible items:** - Believer: 87.23% accuracy on these items; TCE 4.36; Combined Score 34.60. - Skeptic: 95.74% on these items, but it over-rejects valid arguments with implausible conclusions; TCE 5.10; Combined Score 33.94. - **Ensemble results:** TCE 3.21 and Combined Score 39.09, where the score is accuracy divided by (1 + log(1 + TCE)). Reported overall accuracy differs within the paper: 94.27% in the abstract versus 95.31% in the validation table. - **Failed approaches:** - Structural CoT collapsed to 58.85% accuracy. - Nonsense augmentation reached 88.02% accuracy with TCE 14.92. - ORPO reached 77.60% accuracy with TCE 30.56. - CLCT reached a Combined Score of 23.02. - **Decoding choice:** predictions come from a softmax over only the logits of the "true" and "false" tokens, which avoids malformed generated outputs. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works ## Cite Cite as: Soumyajit Roy and Manav Malhotra. 2026. ModusPonens at SemEval-2026 Task 11: Breaking the Plausibility Trap in LLMs via Conflict-Aware Ensembling. Proceedings of the 20th International Workshop on Semantic Evaluation (2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{roy2026modusponens, title = {{ModusPonens at SemEval-2026 Task 11: Breaking the Plausibility Trap in LLMs via Conflict-Aware Ensembling}}, author = {Soumyajit Roy and Manav Malhotra}, booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)}, year = {2026}, doi = {10.18653/v1/2026.semeval-1.10}, url = {https://doi.org/10.18653/v1/2026.semeval-1.10} } ``` - [← previous Modelling Legal Compliance in a Consent Wizard Application as Part of a Research-Centered and User-Oriented Data Infrastructure](https://md.research.elamur.ai/free/en/papers/strathmann-2026-modelling-legal-compliance-consent-wizard/) - [next → MoLAN: A Unified Modality-Aware Noise Dynamic Editing Framework for Multimodal Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/xu-2026-molan-unified-modality-aware-noise/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/15/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. our system defaults to strict thresholding (\> 0.5 → True). Consequently, the model leans slightly towards ‘False’ (Invalid) when entirely uncertain, demanding a higher burden of proof to validate an argument. [find](https://scholar.google.com/scholar?q=our+system+defaults+to+strict+thresholding+%28%5C%3E+0%2E5+%E2%86%92+True%29%2E+Consequently%2C+the+model+leans+slightly+towards+%E2%80%98False%E2%80%99+%28Invalid%29+when+entirely+uncertain%2C+demanding+a+higher+burden+of+proof+to+validate+an+) 2. Leonardo Bertolazzi, Albert Gatt, and Raffaella Bernardi. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. Preprint, arXiv:2406.11341. [find](https://scholar.google.com/scholar?q=Leonardo+Bertolazzi%2C+Albert+Gatt%2C+and+Raffaella+Bernardi%2E+2024%2E+A+systematic+analysis+of+large+language+models+as+soft+reasoners%3A+The+case+of+syllogistic+inferences%2E+Preprint%2C+arXiv%3A2406%2E11341%2E) 3. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2024. Language models show human-like content effects on reasoning tasks. Preprint, arXiv:2207.07051. \#\# Conclusion [find](https://scholar.google.com/scholar?q=Ishita+Dasgupta%2C+Andrew+K%2E+Lampinen%2C+Stephanie+C%2E+Y%2E+Chan%2C+Hannah+R%2E+Sheahan%2C+Antonia+Creswell%2C+Dharshan+Kumaran%2C+James+L%2E+McClelland%2C+and+Felix+Hill%2E+2024%2E+Language+models+show+human%2Dlike+content+eff) *Updated 31 October 2026*