--- title: "Diagnosing the First-Order Logical Reasoning Ability Through" description: "Pretrained Transformers score well on many NLU benchmarks, but it is unclear whether they actually… 31 references, cited by 64 papers here." authors: ["Jidong Tian", "Yitian Li", "Wenqing Chen", "Liqiang Xiao", "Hao He", "Yaohui Jin"] year: 2021 venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/2021.emnlp-main.303/" doi: "10.18653/v1/2021.emnlp-main.303" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/tian-2021-diagnosing-first-order-logical-reasoning/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tian-2021-diagnosing-first-order-logical-reasoning/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI # Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI **Authors:** Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, [Hao He](https://md.research.elamur.ai/free/en/authors/hao-he/) and Yaohui Jin · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.303/) · [PDF](https://aclanthology.org/2021.emnlp-main.303.pdf) · [DOI 10.18653/v1/2021.emnlp-main.303](https://doi.org/10.18653/v1/2021.emnlp-main.303) · [OpenAlex](https://openalex.org/W3212781905) **Citations:** 22 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/tian-2021-diagnosing-first-order-logical-reasoning/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary Pretrained Transformers score well on many NLU benchmarks, but it is unclear whether they actually perform first-order logic (FOL) reasoning or exploit dataset artifacts and commonsense cues. The authors address this with LogicNLI, a semi-automatically generated NLI-style benchmark. Each instance pairs a premise of facts and rules with a statement, and the facts and rules are built from logic templates over names and adjectives. The predicates are deliberately meaningless, so logic is separated from world knowledge. The label set adds a fourth class, Paradox, for cases where both a statement and its negation can be derived. Every instance also includes its proofs and reasoning path. Using this resource, they evaluate BERT, RoBERTa, and XLNet on four aspects: - accuracy - robustness to irrelevant sentences - generalization to more reasoning hops - proof-based traceability RoBERTa performs best, yet none of the models reasons reliably or generalizes as humans do. This matters because it shows that strong NLU scores can mask shallow logic. It also gives a reusable recipe for building disentangled reasoning diagnostics. ## Key points - LogicNLI has over 30K instances and covers all seven FOL operators: ∧, ∨, ¬, →, ≡, ∀, and ∃. It is compared with LogiQA, ReClor, CLUTRR, LTL, and SoftReasoner, each of which covers fewer operators or lacks natural language or proofs. - The task assumes an open world and uses four labels: Entailment, Contradiction, Neutral, and Paradox. Because of Paradox, a model must explore a minimum of two reasoning paths, which lessens its dependence on bias. - Separate test sets target each diagnostic aspect. Traceability is measured with P-EM, an exact-match score on proofs. - In a variant without Paradox (d-LogicNLI), models become almost untraceable, with a maximum P-EM of 4.1% (XLNet). Their accuracy drops sharply on harder multi-hop tests, with δA→G of 44.0% for BERT, 40.0% for RoBERTa, and 36.8% for XLNet. - RoBERTa leads on LogicNLI and XLNet leads on d-LogicNLI. The authors attribute RoBERTa's lead to its larger pretraining corpus, which helps on more complex reasoning. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 63 works · 15,047 citations Also on the byline: [State Key Laboratory of Advanced Optical Communication Systems and Networks](https://ror.org/03dz8mb53). 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works ## Cite Cite as: Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He and Yaohui Jin. 2021. Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{tian2021diagnosing, title = {{Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI}}, author = {Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He and Yaohui Jin}, booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}, year = {2021}, doi = {10.18653/v1/2021.emnlp-main.303}, url = {https://doi.org/10.18653/v1/2021.emnlp-main.303} } ``` - [← previous AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models](https://md.research.elamur.ai/free/en/papers/madabushi-2021-astitchinlanguagemodels-dataset-methods-exploration-idiomaticity/) - [next → How to Adapt Your Pretrained Multilingual Model to 1600 Languages](https://md.research.elamur.ai/free/en/papers/ebrahimi-2021-how-adapt-your-pretrained-multilingual/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) ## References 31 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Md Mosharaf Hossain, Kathleen E. Hamilton, Alexis Palmer, and Eduardo Blanco. 2020b. Predicting the focus of negation: Model and error analysis. In ACL. [find](https://scholar.google.com/scholar?q=Md+Mosharaf+Hossain%2C+Kathleen+E%2E+Hamilton%2C+Alexis+Palmer%2C+and+Eduardo+Blanco%2E+2020b%2E+Predicting+the+focus+of+negation%3A+Model+and+error+analysis%2E+In+ACL%2E) 2. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. [Mathqa: Towards interpretable math word problem solving with operation-based formalisms](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/). In NAACL-HLT. 3. Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020c. [An analysis of natural language inference benchmarks through the lens of negation](https://md.research.elamur.ai/free/en/papers/hossain-2020-analysis-natural-language-inference-benchmarks/). In EMNLP. 4. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. [Cosmos QA: machine reading comprehension with contextual commonsense reasoning](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/). In EMNLP-IJCNLP, pages 2391–2401. 5. Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In ACL. [find](https://scholar.google.com/scholar?q=Koustuv+Sinha%2C+Shagun+Sodhani%2C+Jin+Dong%2C+Joelle+Pineau%2C+and+William+L%2E+Hamilton%2E+2019%2E+CLUTRR%3A+A+diagnostic+benchmark+for+inductive+reasoning+from+text%2E+In+ACL%2E) 6. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. [Are natural language inference models imppressive? learning implicature and presupposition](https://md.research.elamur.ai/free/en/papers/jeretic-2020-are-natural-language-inference-models/). In ACL. 7. Shane Storks, Qiaozi Gao, and Joyce Y. Chai. 2019. [Recent advances in natural language inference: A survey of benchmarks, resources, and approaches](https://md.research.elamur.ai/free/en/papers/storks-2019-recent-advances-natural-language-inference/). CoRR. 8. Yichen Jiang and Mohit Bansal. 2019. [Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA](https://md.research.elamur.ai/free/en/papers/jiang-2019-avoiding-reasoning-shortcuts-adversarial-evaluation/). In ACL. 9. Saku Sugawara, Pontus Stenetorp, Kentaro Inui, and Akiko Aizawa. 2020. Assessing the benchmarking capacity of machine reading comprehension datasets. In AAAI. [find](https://scholar.google.com/scholar?q=Saku+Sugawara%2C+Pontus+Stenetorp%2C+Kentaro+Inui%2C+and+Akiko+Aizawa%2E+2020%2E+Assessing+the+benchmarking+capacity+of+machine+reading+comprehension+datasets%2E+In+AAAI%2E) 10. Aikaterini-Lida Kalouli, Richard S. Crouch, and Valeria de Paiva. 2020. Hy-nli: a hybrid system for natural language inference. In COLING. [find](https://scholar.google.com/scholar?q=Aikaterini%2DLida+Kalouli%2C+Richard+S%2E+Crouch%2C+and+Valeria+de+Paiva%2E+2020%2E+Hy%2Dnli%3A+a+hybrid+system+for+natural+language+inference%2E+In+COLING%2E) 11. Divyansh Kaushik and Zachary C. Lipton. 2018. [How much reading does reading comprehension require? A critical investigation of popular benchmarks](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/). In ACL. 12. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. [RACE: large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In EMNLP. 13. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. [Logiqa: A challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/). In IJCAI. 14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+C) 15. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In ACL. 16. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). In ACL. 17. Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. Quartz: An open-domain dataset of qualitative relationship questions. In EMNLP-IJCNLP. [find](https://scholar.google.com/scholar?q=Oyvind+Tafjord%2C+Matt+Gardner%2C+Kevin+Lin%2C+and+Peter+Clark%2E+2019%2E+Quartz%3A+An+open%2Ddomain+dataset+of+qualitative+relationship+questions%2E+In+EMNLP%2DIJCNLP%2E) 18. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. [olmpics - on what language model pre-training captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/). TACL. 19. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In NAACL-HLT. 20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NeurIPS. 21. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In NeurIPS. 22. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In ICLR. 23. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100, 000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In EMNLP. 24. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2DHLT%2E) 25. Abhilasha Ravichander, Aakanksha Naik, Carolyn Penstein Rosé, and Eduard H. Hovy. 2019. [EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference](https://md.research.elamur.ai/free/en/papers/ravichander-2019-equate-benchmark-evaluation-framework-quantitative/). In CoNLL. 26. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In NeurIPS. 27. Marco Túlio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? evaluating consistency of question-answering models. In ACL. [find](https://scholar.google.com/scholar?q=Marco+T%C3%BAlio+Ribeiro%2C+Carlos+Guestrin%2C+and+Sameer+Singh%2E+2019%2E+Are+red+roses+red%3F+evaluating+consistency+of+question%2Danswering+models%2E+In+ACL%2E) 28. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In EMNLP. 29. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+In+AAAI%2E) 30. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le-Bras, and Yejin Choi. 2019. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). CoRR. 31. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. [Reclor: A reading comprehension dataset requiring logical reasoning](https://md.research.elamur.ai/free/en/papers/yu-2020-reclor-reading-comprehension-dataset-requiring/). In ICLR. *Updated 24 October 2026*