---
title: "Towards LogiGLUE — paper by Luo et al. (2023)"
description: "They then release LogiGLUE, a benchmark that puts 24 datasets into a unified format for training and… 91 references, cited by 22 papers here."
authors: ["Man Luo", "Shrinidhi Kumbhar", "Ming shen", "Mihir Parmar", "Neeraj Varshney", "Pratyay Banerjee", "Somak Aditya", "Chitta Baral"]
year: 2023
venue: "arXiv:2310.00836"
source: "https://arxiv.org/abs/2310.00836"
doi: "10.48550/arxiv.2310.00836"
arxiv: "2310.00836"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/luo-2023-towards-logiglue-brief-survey-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/luo-2023-towards-logiglue-brief-survey-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models

# Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models

**Authors:** [Man Luo](https://md.research.elamur.ai/free/en/authors/man-luo/), [Shrinidhi Kumbhar](https://md.research.elamur.ai/free/en/authors/shrinidhi-kumbhar/), Ming shen, [Mihir Parmar](https://md.research.elamur.ai/free/en/authors/mihir-parmar/), [Neeraj Varshney](https://md.research.elamur.ai/free/en/authors/neeraj-varshney/), Pratyay Banerjee, Somak Aditya and [Chitta Baral](https://md.research.elamur.ai/free/en/authors/chitta-baral/) · **Year:** 2023 · **Venue:** arXiv:2310.00836 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.00836) · [PDF](https://arxiv.org/pdf/2310.00836) · [DOI 10.48550/arxiv.2310.00836](https://doi.org/10.48550/arxiv.2310.00836) · [arXiv 2310.00836](https://arxiv.org/abs/2310.00836) · [OpenAlex](https://openalex.org/W4387323979)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [22](https://md.research.elamur.ai/free/en/papers/luo-2023-towards-logiglue-brief-survey-benchmark/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks how well language models handle logical reasoning expressed in natural language. Formal knowledge-representation systems needed heavy expert effort and did not scale. The authors first survey the field. They cover three reasoning types (deductive, inductive, abductive), the usual task formats, how datasets are built, and how LLMs are applied. They then release LogiGLUE, a benchmark that puts 24 datasets into a unified format for training and testing.

Using LogiGLUE, they fine-tune Flan-T5 with single-task training, multi-task training and chain-of-thought distillation. The resulting model is called LogiT5. Multi-task training and CoT distillation gave the best results. They also test GPT-4, GPT-3.5-turbo and LLaMA-2-chat with zero-shot CoT prompting on held-out test sets.

All three LLMs show the same ranking: abductive reasoning is easiest, then deductive, with inductive weakest. GPT-4 leads overall. The small fine-tuned LogiT5 matches or beats larger LLMs on some inductive and abductive datasets. The work offers a shared, extensible testbed for measuring and improving reasoning in LMs.

## Key points
- LogiGLUE has 24 datasets grouped as deductive, inductive, abductive and mixed. Examples include bAbi, RuleBERT, PrOntoQA, CLUTTR, AbductionRule, ReClor and LogiQA 2.0.
- The survey sorts logical-reasoning datasets into four task formats: MCQA, FFQA, fact verification and NLI. It names three ways datasets are built: crowdsourcing, exam extraction and synthetic generation.
- GPT-4 average scores by reasoning type:
  - abductive: about 98.8
  - mixed: 83.0
  - deductive: 73.6
  - inductive: 31.6
- LogiT5 has under 1B parameters and clearly improves on Flan-T5. Its deductive average rises from 41.5 to 56.7, and its AbductionRule score jumps from 3.8 to 96.1. However, LLMs beat LogiT5 on ReClor and LogiQA, possibly because these public exam datasets appeared in their training data.
- On LogiQA, adding 15K CoT examples raised Flan-T5 accuracy from 0.37 to 0.41. This suggests CoT distillation helps more as the amount of data grows.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 128 works · 41,700 citations
- [Arizona State University](https://md.research.elamur.ai/free/en/universities/arizona-state-university/) — University, Tempe, United States · 54 works · 2,356 citations
- [Indian Institute of Technology Kharagpur](https://md.research.elamur.ai/free/en/universities/indian-institute-of-technology-kharagpur/) — University, Kharagpur, India · 25 works · 714 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Man Luo, Shrinidhi Kumbhar, Ming shen, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya and Chitta Baral. 2023. Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models. arXiv:2310.00836.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{luo2023logiglue,
  title = {{Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models}},
  author = {Man Luo, Shrinidhi Kumbhar, Ming shen, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya and Chitta Baral},
  year = {2023},
  doi = {10.48550/arxiv.2310.00836},
  eprint = {2310.00836},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.00836}
}
```

- [← previous Prediction of Arabic Legal Rulings using Large Language Models](https://md.research.elamur.ai/free/en/papers/ammar-2023-prediction-arabic-legal-rulings-using/)
- [next → “Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text](https://md.research.elamur.ai/free/en/papers/hada-2023-fifty-shades-bias-normative-ratings/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

91 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anonymous. [Logicbench: Towards systematic evaluation of logical reasoning ability of large language models](https://md.research.elamur.ai/free/en/papers/parmar-2024-logicbench-systematic-evaluation-logical-reasoning/). In Submitted to The Twelfth International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=71kocBuhNO](https://openreview.net/forum?id=71kocBuhNO). under review. [↗](https://openreview.net/forum?id=71kocBuhNO)
2. Stephen H. Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. Hinge-loss markov random fields and probabilistic soft logic. J. Mach. Learn. Res., 18(1):3846–3912, jan 2017. ISSN 1532-4435. [find](https://scholar.google.com/scholar?q=Stephen+H%2E+Bach%2C+Matthias+Broecheler%2C+Bert+Huang%2C+and+Lise+Getoor%2E+Hinge%2Dloss+markov+random+fields+and+probabilistic+soft+logic%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+18%281%29%3A3846%E2%80%933912%2C+jan+2017%2E+ISSN+1532%2D4435%2E)
3. Pratyay Banerjee, Chitta Baral, Man Luo, Arindam Mitra, Kuntal Pal, Tran C Son, and Neeraj Varshney. Can transformers reason about effects of actions? arXiv:2012.09938, 2020. — [arXiv:2012.09938](https://arxiv.org/abs/2012.09938) [find](https://scholar.google.com/scholar?q=Pratyay+Banerjee%2C+Chitta+Baral%2C+Man+Luo%2C+Arindam+Mitra%2C+Kuntal+Pal%2C+Tran+C+Son%2C+and+Neeraj+Varshney%2E+Can+transformers+reason+about+effects+of+actions%3F+arXiv%3A2012%2E09938%2C+2020%2E+%E2%80%94+%5BarXiv%3A2012%2E09938%5D%28http)
4. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. [Abductive commonsense reasoning](https://md.research.elamur.ai/free/en/papers/bhagavatula-2019-abductive-commonsense-reasoning/). In International Conference on Learning Representations, 2019.
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
6. Ruth MJ Byrne, Jonathan St BT Evans, and Stephen E Newstead. Human reasoning: The psychology of deduction. Psychology Press, 2019. [find](https://scholar.google.com/scholar?q=Ruth+MJ+Byrne%2C+Jonathan+St+BT+Evans%2C+and+Stephen+E+Newstead%2E+Human+reasoning%3A+The+psychology+of+deduction%2E+Psychology+Press%2C+2019%2E)
7. Chunkit Chan, Xin Liu, Tsz Ho Chan, Jiayang Cheng, Yangqiu Song, Ginny Wong, and Simon See. Self-consistent narrative prompts on abductive natural language inference. arXiv preprint arXiv:2309.08303, 2023. [find](https://scholar.google.com/scholar?q=Chunkit+Chan%2C+Xin+Liu%2C+Tsz+Ho+Chan%2C+Jiayang+Cheng%2C+Yangqiu+Song%2C+Ginny+Wong%2C+and+Simon+See%2E+Self%2Dconsistent+narrative+prompts+on+abductive+natural+language+inference%2E+arXiv+preprint+arXiv%3A2309%2E08303%2C+)
8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
9. Peter Clark, Oyvind Tafjord, and Kyle Richardson. [Transformers as soft reasoners over language](https://md.research.elamur.ai/free/en/papers/clark-2020-transformers-soft-reasoners-over-language/). In Christian Bessiere (ed.), IJCAI, 7 2020. doi: 10.24963/ijcai.2020/537. URL https://doi.org/10.24963/ijcai.2020/537. [↗](https://doi.org/10.24963/ijcai.2020/537)
10. Alain Colmerauer and Philippe Roussel. The birth of prolog. In History of programming languages— II. 1996. [find](https://scholar.google.com/scholar?q=Alain+Colmerauer+and+Philippe+Roussel%2E+The+birth+of+prolog%2E+In+History+of+programming+languages%E2%80%94+II%2E+1996%2E)
11. Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Antonia+Creswell%2C+Murray+Shanahan%2C+and+Irina+Higgins%2E+Selection%2Dinference%3A+Exploiting+large+language+models+for+interpretable+logical+reasoning%2E+In+The+Eleventh+International+Conference+on+Learning+Re)
12. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.[07051, 2022](https://md.research.elamur.ai/free/en/papers/dasgupta-2022-language-models-show-human-like/).
13. Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. Problog: A probabilistic prolog and its application in link discovery. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI’07, pp. 2468–2473, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=Luc+De+Raedt%2C+Angelika+Kimmig%2C+and+Hannu+Toivonen%2E+Problog%3A+A+probabilistic+prolog+and+its+application+in+link+discovery%2E+In+Proceedings+of+the+20th+International+Joint+Conference+on+Artifical+Intelli)
14. J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL-HLT, 2019a.
15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL, Minneapolis, Minnesota, June 2019b. ACL. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/ N19-1423. [↗](https://aclanthology.org/)
16. Xuemeng Song Liqiang Nie Fangkai Jiao, Yangyang Guo. Merit: Meta-path guided contrastive learning for logical reasoning. arXiv preprint arXiv:2203.00357, 2022. [find](https://scholar.google.com/scholar?q=Xuemeng+Song+Liqiang+Nie+Fangkai+Jiao%2C+Yangyang+Guo%2E+Merit%3A+Meta%2Dpath+guided+contrastive+learning+for+logical+reasoning%2E+arXiv+preprint+arXiv%3A2203%2E00357%2C+2022%2E)
17. Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. Language models can be logical solvers. arXiv preprint arXiv:2311.06158, 2023. — [arXiv:2311.06158](https://arxiv.org/abs/2311.06158) [find](https://scholar.google.com/scholar?q=Jiazhan+Feng%2C+Ruochen+Xu%2C+Junheng+Hao%2C+Hiteshi+Sharma%2C+Yelong+Shen%2C+Dongyan+Zhao%2C+and+Weizhu+Chen%2E+Language+models+can+be+logical+solvers%2E+arXiv+preprint+arXiv%3A2311%2E06158%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E06158%5D%28h)
18. Jörg Frohberg and Frank Binder. Crass: A novel data set and benchmark to test counterfactual reasoning of large language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2126–2140, 2022. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Frohberg+and+Frank+Binder%2E+Crass%3A+A+novel+data+set+and+benchmark+to+test+counterfactual+reasoning+of+large+language+models%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+and+Evaluation+Conf)
19. Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. [Leveraging pre-trained large language models to construct and utilize world models for model-based task planning](https://md.research.elamur.ai/free/en/papers/guan-2023-leveraging-pre-trained-large-language/). arXiv preprint arXiv:2305.14909, 2023.
20. Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, et al. [Folio: Natural language reasoning with first-order logic](https://md.research.elamur.ai/free/en/papers/han-2022-folio-natural-language-reasoning-first/). arXiv preprint arXiv:2209.00840, 2022.
21. Simon Jerome Han, Keith J Ransom, Andrew Perfors, and Charles Kemp. Inductive reasoning in humans and large language models. Cognitive Systems Research, pp. 101155, 2023. [find](https://scholar.google.com/scholar?q=Simon+Jerome+Han%2C+Keith+J+Ransom%2C+Andrew+Perfors%2C+and+Charles+Kemp%2E+Inductive+reasoning+in+humans+and+large+language+models%2E+Cognitive+Systems+Research%2C+pp%2E+101155%2C+2023%2E)
22. Li Erran Li3 Eric Xing Hanlin Zhang1, Yi-Fan Zhang2. The impact of symbolic representations on in-context learning for few-shot reasoning. arXiv preprint arXiv:2212.08686, 2022. — [arXiv:2212.08686](https://arxiv.org/abs/2212.08686) [find](https://scholar.google.com/scholar?q=Li+Erran+Li3+Eric+Xing+Hanlin+Zhang1%2C+Yi%2DFan+Zhang2%2E+The+impact+of+symbolic+representations+on+in%2Dcontext+learning+for+few%2Dshot+reasoning%2E+arXiv+preprint+arXiv%3A2212%2E08686%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E08686%5D%28h)
23. Weinan He, Canming Huang, Yongmei Liu, and Xiaodan Zhu. WinoLogic: A zero-shot logic-based diagnostic dataset for Winograd Schema Challenge. In EMNLP, November 2021. URL https://aclanthology.org/2021.emnlp-main.307. [↗](https://aclanthology.org/2021.emnlp-main.307)
24. Evan Heit. What Is Induction and Why Study It?, pp. 1–24. Cambridge University Press, 2007. doi: 10.1017/CBO9780511619304.002. — [doi:10.1017/cbo9780511619304.002](https://doi.org/10.1017/cbo9780511619304.002) [find](https://scholar.google.com/scholar?q=Evan+Heit%2E+What+Is+Induction+and+Why+Study+It%3F%2C+pp%2E+1%E2%80%9324%2E+Cambridge+University+Press%2C+2007%2E+doi%3A+10%2E1017%2FCBO9780511619304%2E002%2E+%E2%80%94+%5Bdoi%3A10%2E1017%2Fcbo9780511619304%2E002%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1017%2Fcbo978051161)
25. Chadi Helwe, Chloé Clavel, and Fabian Suchanek. Logitorch: A pytorch-based library for logical reasoning on natural language. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2022. [find](https://scholar.google.com/scholar?q=Chadi+Helwe%2C+Chlo%C3%A9+Clavel%2C+and+Fabian+Suchanek%2E+Logitorch%3A+A+pytorch%2Dbased+library+for+logical+reasoning+on+natural+language%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Lang)
26. Jerry R Hobbs, Mark E Stickel, Douglas E Appelt, and Paul Martin. Interpretation as abduction. Artificial intelligence, 63(1-2):69–142, 1993. [find](https://scholar.google.com/scholar?q=Jerry+R+Hobbs%2C+Mark+E+Stickel%2C+Douglas+E+Appelt%2C+and+Paul+Martin%2E+Interpretation+as+abduction%2E+Artificial+intelligence%2C+63%281%2D2%29%3A69%E2%80%93142%2C+1993%2E)
27. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. [Cosmos QA: Machine reading comprehension with contextual commonsense reasoning](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/). In EMNLP-IJCNLP, November 2019. doi: 10.18653/v1/D19-1243. URL https://aclanthology.org/D19-1243. [↗](https://aclanthology.org/D19-1243)
28. Alexey Ignatiev, Nina Narodytska, and Joao Marques-Silva. Abduction-based explanations for machine learning models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1511–1519, 2019. [find](https://scholar.google.com/scholar?q=Alexey+Ignatiev%2C+Nina+Narodytska%2C+and+Joao+Marques%2DSilva%2E+Abduction%2Dbased+explanations+for+machine+learning+models%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+33%2C+pp%2E+151)
29. Mete Ismayilzada, Debjit Paul, Syrielle Montariol, Mor Geva, and Antoine Bosselut. CRoW: Benchmarking commonsense reasoning in real-world tasks. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9785–9821,… [find](https://scholar.google.com/scholar?q=Mete+Ismayilzada%2C+Debjit+Paul%2C+Syrielle+Montariol%2C+Mor+Geva%2C+and+Antoine+Bosselut%2E+CRoW%3A+Benchmarking+commonsense+reasoning+in+real%2Dworld+tasks%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+Bali+%28eds%2E%29%2C+Pr)
30. Fangkai Jiao, Zhiyang Teng, Shafiq Joty, Bosheng Ding, Aixin Sun, Zhengyuan Liu, and Nancy F Chen. Logicllm: Exploring self-supervised logic-enhanced training for large language models. arXiv preprint arXiv:2305.13718, 2023. [find](https://scholar.google.com/scholar?q=Fangkai+Jiao%2C+Zhiyang+Teng%2C+Shafiq+Joty%2C+Bosheng+Ding%2C+Aixin+Sun%2C+Zhengyuan+Liu%2C+and+Nancy+F+Chen%2E+Logicllm%3A+Exploring+self%2Dsupervised+logic%2Denhanced+training+for+large+language+models%2E+arXiv+preprint)
31. Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. TaxiNLI: Taking a ride up the NLU hill. In CoNLL, November 2020. doi: 10.18653/v1/2020.conll-1.4. URL https://aclanthology.org/2020.conll-1.4. [↗](https://aclanthology.org/2020.conll-1.4)
32. Jia Deng Kaiyu Yang and Danqi Chen. Generating natural language proofs with verifier-guided search. arXiv preprint arXiv:2205.12443, 2022. [find](https://scholar.google.com/scholar?q=Jia+Deng+Kaiyu+Yang+and+Danqi+Chen%2E+Generating+natural+language+proofs+with+verifier%2Dguided+search%2E+arXiv+preprint+arXiv%3A2205%2E12443%2C+2022%2E)
33. Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. [Lambada: Backward chaining for automated reasoning in natural language](https://md.research.elamur.ai/free/en/papers/kazemi-2023-lambada-backward-chaining-automated-reasoning/). arXiv preprint arXiv:2212.13894, 2022.
34. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamile Lukosiute, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saura… — [Measuring Faithfulness in Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/lanham-2023-measuring-faithfulness-chain-thought-reasoning/)
35. Joohyung Lee and Man Luo. Strong equivalence for lpmln programs. In 35th International Conference on Logic Programming (ICLP 2019), 2018. [find](https://scholar.google.com/scholar?q=Joohyung+Lee+and+Man+Luo%2E+Strong+equivalence+for+lpmln+programs%2E+In+35th+International+Conference+on+Logic+Programming+%28ICLP+2019%29%2C+2018%2E)
36. Joohyung Lee and Yi Wang. Weighted rules under the stable model semantics. In KRR, 2016. [find](https://scholar.google.com/scholar?q=Joohyung+Lee+and+Yi+Wang%2E+Weighted+rules+under+the+stable+model+semantics%2E+In+KRR%2C+2016%2E)
37. Vladimir Lifschitz. Answer set programming. Springer Berlin, 2019. [find](https://scholar.google.com/scholar?q=Vladimir+Lifschitz%2E+Answer+set+programming%2E+Springer+Berlin%2C+2019%2E)
38. Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. [Wanli: Worker and ai collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/). arXiv preprint arXiv:2201.05955, 2022.
39. Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2947–2962, 2023a. doi: 10.1109/ TASLP.2023.3293046. [find](https://scholar.google.com/scholar?q=Hanmeng+Liu%2C+Jian+Liu%2C+Leyang+Cui%2C+Zhiyang+Teng%2C+Nan+Duan%2C+Ming+Zhou%2C+and+Yue+Zhang%2E+Logiqa+2%2E0%E2%80%94an+improved+dataset+for+logical+reasoning+in+natural+language+understanding%2E+IEEE%2FACM+Transactions+on+Au)
40. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. [Evaluating the logical reasoning ability of chatgpt and gpt-4](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/). arXiv preprint arXiv:2304.03439, 2023b.
41. Hanmeng Liu, Zhiyang Teng, Leyang Cui, Chaoli Zhang, Qiji Zhou, and Yue Zhang. Logicot: Logical chain-of-thought instruction-tuning data collection with gpt-4. arXiv preprint arXiv:2305.12147, 2023c. [find](https://scholar.google.com/scholar?q=Hanmeng+Liu%2C+Zhiyang+Teng%2C+Leyang+Cui%2C+Chaoli+Zhang%2C+Qiji+Zhou%2C+and+Yue+Zhang%2E+Logicot%3A+Logical+chain%2Dof%2Dthought+instruction%2Dtuning+data+collection+with+gpt%2D4%2E+arXiv+preprint+arXiv%3A2305%2E12147%2C+2023c%2E)
42. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. [Logiqa: A challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/). In IJCAI, 2020. doi: 10.24963/ijcai.2020/501. URL https://doi.org/10.24963/ijcai.2020/501. [↗](https://doi.org/10.24963/ijcai.2020/501)
43. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. [Logiqa: a challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/). In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3622–3628, 2…
44. Yinhan Liu, Myle Ott, Naman Goyal, and et al. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv:1907.11692, 2019.
45. Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark. arXiv preprint arXiv:2103.13009, 2021. [find](https://scholar.google.com/scholar?q=Nicholas+Lourie%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Unicorn+on+rainbow%3A+A+universal+commonsense+reasoning+model+on+a+new+multitask+benchmark%2E+arXiv+preprint+arXiv%3A2103%2E13009%2C+2021%2E)
46. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. [Chameleon: Plug-and-play compositional reasoning with large language models](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/). arXiv preprint arXiv:2304.09842, 2023.
47. Man Luo, Kazuma Hashimoto, Semih Yavuz, Zhiwei Liu, Chitta Baral, and Yingbo Zhou. Choose your qa model wisely: A systematic study of generative and extractive readers for question answering. Spa-NLP 2022, pp. 7, 2022. [find](https://scholar.google.com/scholar?q=Man+Luo%2C+Kazuma+Hashimoto%2C+Semih+Yavuz%2C+Zhiwei+Liu%2C+Chitta+Baral%2C+and+Yingbo+Zhou%2E+Choose+your+qa+model+wisely%3A+A+systematic+study+of+generative+and+extractive+readers+for+question+answering%2E+Spa%2DNLP+)
48. John McCarthy. Artificial intelligence, logic and formalizing common sense. In Philosophical logic and artificial intelligence. Springer, 1989. [find](https://scholar.google.com/scholar?q=John+McCarthy%2E+Artificial+intelligence%2C+logic+and+formalizing+common+sense%2E+In+Philosophical+logic+and+artificial+intelligence%2E+Springer%2C+1989%2E)
49. Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. [SPARTQA: A textual question answering benchmark for spatial reasoning](https://md.research.elamur.ai/free/en/papers/mirzaee-2021-spartqa-textual-question-answering-benchmark/). In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao…
50. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470–3487, 2022a. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Asso)
51. Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. [Numglue: A suite of fundamental yet challenging mathematical reasoning tasks](https://md.research.elamur.ai/free/en/papers/mishra-2022-numglue-suite-fundamental-yet-challenging/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa…
52. Se-Young Yun Namgyu Ho, Laura Schmid. [Large language models are reasoning teachers](https://md.research.elamur.ai/free/en/papers/ho-2023-large-language-models-are-reasoning/). arXiv preprint arXiv:2212.10071, 2022.
53. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. [Adversarial nli: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885–4901, 2020.
54. Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments. In ACL, 2019. [find](https://scholar.google.com/scholar?q=Timothy+Niven+and+Hung%2DYu+Kao%2E+Probing+neural+network+comprehension+of+natural+language+arguments%2E+In+ACL%2C+2019%2E)
55. Theo X. Olausson*, Alex Gu*, Ben Lipkin*, Cedegao E. Zhang*, Armando Solar-Lezama, Joshua B. Tenenbaum, and Roger P. Levy. Linc: A neuro-symbolic approach for logical reasoning by combining language models with first-order logic provers. 2023. [find](https://scholar.google.com/scholar?q=Theo+X%2E+Olausson%2A%2C+Alex+Gu%2A%2C+Ben+Lipkin%2A%2C+Cedegao+E%2E+Zhang%2A%2C+Armando+Solar%2DLezama%2C+Joshua+B%2E+Tenenbaum%2C+and+Roger+P%2E+Levy%2E+Linc%3A+A+neuro%2Dsymbolic+approach+for+logical+reasoning+by+combining+language+m)
56. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. [Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning](https://md.research.elamur.ai/free/en/papers/pan-2023-logic-lm-empowering-large-language/). arXiv preprint arXiv:2305.12295, 2023.
57. Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. [Art: Automatic multi-step reasoning and tool-use for large language models](https://md.research.elamur.ai/free/en/papers/paranjape-2023-art-automatic-multi-step-reasoning/). arXiv preprint arXiv:2303.09014, 2023.
58. Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral. In-boxbart: Get instructions into biomedical multi-task learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 112–128, 2022. [find](https://scholar.google.com/scholar?q=Mihir+Parmar%2C+Swaroop+Mishra%2C+Mirali+Purohit%2C+Man+Luo%2C+Murad+Mohammad%2C+and+Chitta+Baral%2E+In%2Dboxbart%3A+Get+instructions+into+biomedical+multi%2Dtask+learning%2E+In+Findings+of+the+Association+for+Computatio)
59. Gabriele Paul. Approaches to abductive reasoning: an overview. Artificial intelligence review, 7(2): 109–152, 1993. [find](https://scholar.google.com/scholar?q=Gabriele+Paul%2E+Approaches+to+abductive+reasoning%3A+an+overview%2E+Artificial+intelligence+review%2C+7%282%29%3A+109%E2%80%93152%2C+1993%2E)
60. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. Kyle Richardson and Ashish Sabharwal. Pushing the limits of rule reasoning in transformers through natural language satisfiability. arXiv:2112.09054, 2021. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+Kyle+Richardson+and+Ashish+Sabharwal%2E+Pushing+the+limits+of)
61. Matthew Richardson and Pedro Domingos. Markov logic networks. Mach. Learn., 62(1–2):107–136, feb 2006. ISSN 0885-6125. doi: 10.1007/s10994-006-5833-1. URL https://doi.org/10. 1007/s10994-006-5833-1. [find](https://scholar.google.com/scholar?q=Matthew+Richardson+and+Pedro+Domingos%2E+Markov+logic+networks%2E+Mach%2E+Learn%2E%2C+62%281%E2%80%932%29%3A107%E2%80%93136%2C+feb+2006%2E+ISSN+0885%2D6125%2E+doi%3A+10%2E1007%2Fs10994%2D006%2D5833%2D1%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F10%2E+1007%2Fs10994%2D006%2D5833%2D1%2E)
62. Mohammed Saeed, Naser Ahmadi, Preslav Nakov, and Paolo Papotti. RuleBERT: Teaching soft rules to pre-trained lms. In EMNLP, November 2021. URL https://aclanthology.org/ 2021.emnlp-main.110. [↗](https://aclanthology.org/)
63. Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal. [Prover: Proof generation for interpretable reasoning over rules](https://md.research.elamur.ai/free/en/papers/saha-2020-prover-proof-generation-interpretable-reasoning/). In EMNLP, 2020.
64. Soumya Sanyal, Yichong Xu, Shuohang Wang, Ziyi Yang, Reid Pryzant, Wenhao Yu, Chenguang Zhu, and Xiang Ren. Apollo: A simple approach for adaptive pretraining of language models for logical reasoning. arXiv preprint arXiv:2212.09282, 2022. [find](https://scholar.google.com/scholar?q=Soumya+Sanyal%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ziyi+Yang%2C+Reid+Pryzant%2C+Wenhao+Yu%2C+Chenguang+Zhu%2C+and+Xiang+Ren%2E+Apollo%3A+A+simple+approach+for+adaptive+pretraining+of+language+models+for+logical+reasoning%2E)
65. Abulhair Saparov and He He. [Language models are greedy reasoners: A systematic formal analysis of chain-of-thought](https://md.research.elamur.ai/free/en/papers/saparov-2022-language-models-greedy-reasoners-systematic/). In The Eleventh International Conference on Learning Representations, 2023a. URL [https://openreview.net/forum?id=qFVVBzXxR2V](https://openreview.net/forum?id=qFVVBzXxR2V). [↗](https://openreview.net/forum?id=qFVVBzXxR2V)
66. Abulhair Saparov and He He. [Language models are greedy reasoners: A systematic formal analysis of chain-of-thought](https://md.research.elamur.ai/free/en/papers/saparov-2022-language-models-greedy-reasoners-systematic/). In The Eleventh International Conference on Learning Representations, 2023b. URL [https://openreview.net/forum?id=qFVVBzXxR2V](https://openreview.net/forum?id=qFVVBzXxR2V). [↗](https://openreview.net/forum?id=qFVVBzXxR2V)
67. Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim, and He He. [Testing the general deductive reasoning capacity of large language models using ood examples](https://md.research.elamur.ai/free/en/papers/saparov-2023-testing-general-deductive-reasoning-capacity/). arXiv preprint arXiv:2305.15269, 2023.
68. Bruno Sauce and Louis D Matzel. Inductive reasoning. Encyclopedia of animal cognition and behavior, 6:1–8, 2017. [find](https://scholar.google.com/scholar?q=Bruno+Sauce+and+Louis+D+Matzel%2E+Inductive+reasoning%2E+Encyclopedia+of+animal+cognition+and+behavior%2C+6%3A1%E2%80%938%2C+2017%2E)
69. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). arXiv preprint arXiv:2302.04761, 2023.
70. Koustuv Sinha, Shagun Sodhani, Jin Dong, and et al. Clutrr: A diagnostic benchmark for inductive reasoning from text. In EMNLP, 2019. [find](https://scholar.google.com/scholar?q=Koustuv+Sinha%2C+Shagun+Sodhani%2C+Jin+Dong%2C+and+et+al%2E+Clutrr%3A+A+diagnostic+benchmark+for+inductive+reasoning+from+text%2E+In+EMNLP%2C+2019%2E)
71. Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017. [find](https://scholar.google.com/scholar?q=Robyn+Speer%2C+Joshua+Chin%2C+and+Catherine+Havasi%2E+Conceptnet+5%2E5%3A+An+open+multilingual+graph+of+general+knowledge%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+volume+31%2C+2017%2E)
72. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615, 2022.
73. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Transactions on Machine Learning Resear…
74. Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427, 2023. [find](https://scholar.google.com/scholar?q=Theodore+Sumers%2C+Shunyu+Yao%2C+Karthik+Narasimhan%2C+and+Thomas+L+Griffiths%2E+Cognitive+architectures+for+language+agents%2E+arXiv+preprint+arXiv%3A2309%2E02427%2C+2023%2E)
75. Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. [ProofWriter: Generating implications, proofs, and abductive statements over natural language](https://md.research.elamur.ai/free/en/papers/tafjord-2021-proofwriter-generating-implications-proofs-abductive/). In Findings-ACL-IJCNLP, August 2021. doi: 10.18653/v1/2021.findings-acl.317. URL https://aclanthology.org/2021. findings-acl.317. [↗](https://aclanthology.org/2021)
76. Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. Wiqa: A dataset for “what if...” reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language… [find](https://scholar.google.com/scholar?q=Niket+Tandon%2C+Bhavana+Dalvi%2C+Keisuke+Sakaguchi%2C+Peter+Clark%2C+and+Antoine+Bosselut%2E+Wiqa%3A+A+dataset+for+%E2%80%9Cwhat+if%2E%2E%2E%E2%80%9D+reasoning+over+procedural+text%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+M)
77. Zhiyang Teng, Ruoxi Ning, Jian Liu, Qiji Zhou, Yue Zhang, et al. Glore: Evaluating logical reasoning of large language models. arXiv preprint arXiv:2310.09107, 2023. — [arXiv:2310.09107](https://arxiv.org/abs/2310.09107) [find](https://scholar.google.com/scholar?q=Zhiyang+Teng%2C+Ruoxi+Ning%2C+Jian+Liu%2C+Qiji+Zhou%2C+Yue+Zhang%2C+et+al%2E+Glore%3A+Evaluating+logical+reasoning+of+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E09107%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E09107%5D%28https%3A%2F%2Farxiv)
78. Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. [Diagnosing the first-order logical reasoning ability through logicnli](https://md.research.elamur.ai/free/en/papers/tian-2021-diagnosing-first-order-logical-reasoning/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3738–3747, 2021a.
79. Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. [Diagnosing the first-order logical reasoning ability through LogicNLI](https://md.research.elamur.ai/free/en/papers/tian-2021-diagnosing-first-order-logical-reasoning/). In EMNLP, November 2021b. URL https://aclanthology.org/2021.emnlp-main.303. [↗](https://aclanthology.org/2021.emnlp-main.303)
80. Neeraj Varshney, Mihir Parmar, Nisarg Patel, Divij Handa, Sayantan Sarkar, Man Luo, and Chitta Baral. Can nlp models correctly reason over contexts that break the common assumptions? arXiv preprint arXiv:2305.12096, 2023. — [arXiv:2305.12096](https://arxiv.org/abs/2305.12096) [find](https://scholar.google.com/scholar?q=Neeraj+Varshney%2C+Mihir+Parmar%2C+Nisarg+Patel%2C+Divij+Handa%2C+Sayantan+Sarkar%2C+Man+Luo%2C+and+Chitta+Baral%2E+Can+nlp+models+correctly+reason+over+contexts+that+break+the+common+assumptions%3F+arXiv+preprint+ar)
81. Ashish Vaswani, Noam Shazeer, Niki Parmar, and et al. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NeurIPS, volume 30. Curran Associates, Inc., 2017. URL [https://proceedings.neurips.cc/](https://proceedings.neurips.cc/) paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. [↗](https://proceedings.neurips.cc/)
82. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Advances in neural information processing systems, 32, 2019.
83. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
84. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. [Towards ai-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). arXiv preprint arXiv:1502.05698, 2015a.
85. Jason Weston, Antoine Bordes, and et al. [Towards ai-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). arXiv:1502.05698, 2015b.
86. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. CoRR, abs/2307.02477, 2023. doi: 10.48550/arXiv. 2307.02477. URL h… [find](https://scholar.google.com/scholar?q=Zhaofeng+Wu%2C+Linlu+Qiu%2C+Alexis+Ross%2C+Ekin+Aky%C3%BCrek%2C+Boyuan+Chen%2C+Bailin+Wang%2C+Najoung+Kim%2C+Jacob+Andreas%2C+and+Yoon+Kim%2E+Reasoning+or+reciting%3F+exploring+the+capabilities+and+limitations+of+language+mod)
87. Zhilin Yang, Zihang Dai, Yiming Yang, and et al. Carbonell. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In NeurIPS, volume 32. Curran Associates, Inc., 2019. URL [https://proceedings.neurips.cc/paper/2019/file/](https://proceedings.neurips.cc/paper/2019/file/) dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf. [↗](https://proceedings.neurips.cc/paper/2019/file/)
88. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. [Reclor: A reading comprehension dataset requiring logical reasoning](https://md.research.elamur.ai/free/en/papers/yu-2020-reclor-reading-comprehension-dataset-requiring/). In ICLR, 2019.
89. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. [Reclor: A reading comprehension dataset requiring logical reasoning](https://md.research.elamur.ai/free/en/papers/yu-2020-reclor-reading-comprehension-dataset-requiring/). In International Conference on Learning Representations, 2020.
90. Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502, 2022. — [arXiv:2205.11502](https://arxiv.org/abs/2205.11502) [find](https://scholar.google.com/scholar?q=Honghua+Zhang%2C+Liunian+Harold+Li%2C+Tao+Meng%2C+Kai%2DWei+Chang%2C+and+Guy+Van+den+Broeck%2E+On+the+paradox+of+learning+to+reason+from+data%2E+arXiv+preprint+arXiv%3A2205%2E11502%2C+2022%2E+%E2%80%94+%5BarXiv%3A2205%2E11502%5D%28https%3A%2F%2Fa)
91. Dataset Abductive NLI CLUTTR Abduction Rules Adversarial NLI LogiNLI proofWriter RuleBERT Folio LogiQA baBi Electric-bird Winologic Wanli BigBench ProntoQA License Apache-2.0 license Attribution-NonCommercial 4.0 International MIT License Attribution-NonCommercial 4.0 International Creative Commons… [find](https://scholar.google.com/scholar?q=Dataset+Abductive+NLI+CLUTTR+Abduction+Rules+Adversarial+NLI+LogiNLI+proofWriter+RuleBERT+Folio+LogiQA+baBi+Electric%2Dbird+Winologic+Wanli+BigBench+ProntoQA+License+Apache%2D2%2E0+license+Attribution%2DNonCo)

*Updated 2 November 2026*