---
title: "Language Models as Inductive Reasoners — Yang 2024"
description: "The authors also propose automatic metrics that agree well with human judgments. 45 references, cited by 61 papers here."
authors: ["Zonglin Yang", "Li Dong", "Xinya Du", "Hao Cheng", "Erik Cambria", "Xiaodong Liu", "Jianfeng Gao", "Furu Wei"]
year: 2024
venue: "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.eacl-long.13/"
doi: "10.18653/v1/2024.eacl-long.13"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2024-language-models-as-inductive-reasoners/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2024-language-models-as-inductive-reasoners/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Language Models as Inductive Reasoners

# Language Models as Inductive Reasoners

**Authors:** [Zonglin Yang](https://md.research.elamur.ai/free/en/authors/zonglin-yang/), [Li Dong](https://md.research.elamur.ai/free/en/authors/li-dong/), [Xinya Du](https://md.research.elamur.ai/free/en/authors/xinya-du/), [Hao Cheng](https://md.research.elamur.ai/free/en/authors/hao-cheng/), [Erik Cambria](https://md.research.elamur.ai/free/en/authors/erik-cambria/), [Xiaodong Liu](https://md.research.elamur.ai/free/en/authors/xiaodong-liu/), [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) and [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) · **Year:** 2024 · **Venue:** [Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2024](https://md.research.elamur.ai/free/en/venues/eacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.eacl-long.13/) · [PDF](https://aclanthology.org/2024.eacl-long.13.pdf) · [DOI 10.18653/v1/2024.eacl-long.13](https://doi.org/10.18653/v1/2024.eacl-long.13) · [OpenAlex](https://openalex.org/W4411630358)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [61](https://md.research.elamur.ai/free/en/papers/yang-2024-language-models-as-inductive-reasoners/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Classical computational induction, such as Inductive Logic Programming, works over symbolic first-order logic. This makes it dependent on hand-preprocessed input, fragile when labels are wrong, and unable to treat synonymous expressions as equivalent. The authors reframe induction as a fully natural-language task: given facts written as ordinary text, a pretrained language model must generate a general rule that explains them.

To support this task they built DEER, a set of 1.2k rule–fact pairs. Each rule was written by an expert, and its supporting facts were gathered from web text. A companion resource, DEERLET, contains rules produced by GPT-J and annotated with labels for several quality aspects. The authors also propose automatic metrics that agree well with human judgments.

They then analyse how well PLMs induce rules across rule types, topics, numbers of input facts and model sizes. Finally, they introduce CoLM (chain-of-language-models), a framework grounded in philosophical criteria for valid induction, which beats baselines in both automatic and human evaluation. The work connects ILP and NLP and points toward LLM-assisted scientific hypothesis generation.

## Key points
- DEER pairs each rule with six supporting facts, three long paragraphs and three short core sentences, taken from search-engine results and Wikipedia.
- DEER spans six topics: zoology, botany, geology, astronomy, history and physics. It also covers four FOL rule types: universal, existential, conjunctive and disjunctive implications.
- In a manual check by four graduate students, 95.5% of facts contained enough information to induce their rule.
- DEERLET stores (fact, rule, label0–label3) tuples, with rules produced by GPT-J through in-context learning. It is used to evaluate classifiers of specific rule qualities.
- CoLM chains five PLM-based modules. M1 proposes candidate rules, and M2–M5 each test whether a rule satisfies one requirement of induction. Data and code are released at github.com/ZonglinY/Inductive_Reasoning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Texas at Dallas](https://md.research.elamur.ai/free/en/universities/the-university-of-texas-at-dallas/) — University, Richardson, United States · 47 works · 5,363 citations
- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 73 works · 24,390 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao and Furu Wei. 2024. Language Models as Inductive Reasoners. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2024language,
  title = {{Language Models as Inductive Reasoners}},
  author = {Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao and Furu Wei},
  booktitle = {Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.eacl-long.13},
  url = {https://doi.org/10.18653/v1/2024.eacl-long.13}
}
```

- [← previous Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://md.research.elamur.ai/free/en/papers/beck-2024-sensitivity-performance-robustness-deconstructing-effect/)
- [next → Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards](https://md.research.elamur.ai/free/en/papers/wang-2024-arithmetic-control-llms-diverse-user/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacopo Amidei, Paul Piwek, and Alistair Willis. 2020. Identifying annotator bias: A new irt-based method for bias identification. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4787–4797. Internatio… [find](https://scholar.google.com/scholar?q=Jacopo+Amidei%2C+Paul+Piwek%2C+and+Alistair+Willis%2E+2020%2E+Identifying+annotator+bias%3A+A+new+irt%2Dbased+method+for+bias+identification%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+L)
2. Aristotle Aristotle. 1994. Posterior analytics, volume 1. Clarendon Press Oxford, UK. [find](https://scholar.google.com/scholar?q=Aristotle+Aristotle%2E+1994%2E+Posterior+analytics%2C+volume+1%2E+Clarendon+Press+Oxford%2C+UK%2E)
3. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: an automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 2…
4. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. [Abductive commonsense reasoning](https://md.research.elamur.ai/free/en/papers/bhagavatula-2019-abductive-commonsense-reasoning/). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.…
5. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. [Transformers as soft reasoners over language](https://md.research.elamur.ai/free/en/papers/clark-2020-transformers-soft-reasoners-over-language/). In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3882–3890. ijcai.org.
7. Andrew Cropper, Sebastijan Dumancic, Richard Evans, and Stephen H. Muggleton. 2022. Inductive logic programming at 30. Mach. Learn., 111(1):147–172. [find](https://scholar.google.com/scholar?q=Andrew+Cropper%2C+Sebastijan+Dumancic%2C+Richard+Evans%2C+and+Stephen+H%2E+Muggleton%2E+2022%2E+Inductive+logic+programming+at+30%2E+Mach%2E+Learn%2E%2C+111%281%29%3A147%E2%80%93172%2E)
8. Wang-Zhou Dai and Stephen H. Muggleton. 2021. Abductive knowledge induction from raw data. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 1845–1851. ijcai.org. [find](https://scholar.google.com/scholar?q=Wang%2DZhou+Dai+and+Stephen+H%2E+Muggleton%2E+2021%2E+Abductive+knowledge+induction+from+raw+data%2E+In+Proceedings+of+the+Thirtieth+International+Joint+Conference+on+Artificial+Intelligence%2C+IJCAI+2021%2C+Virtua)
9. Rajarshi Das, Ameya Godbole, Ankita Naik, Elliot Tower, Manzil Zaheer, Hannaneh Hajishirzi, Robin Jia, and Andrew McCallum. 2022. Knowledge base question answering by case-based reasoning over subgraphs. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland… [find](https://scholar.google.com/scholar?q=Rajarshi+Das%2C+Ameya+Godbole%2C+Ankita+Naik%2C+Elliot+Tower%2C+Manzil+Zaheer%2C+Hannaneh+Hajishirzi%2C+Robin+Jia%2C+and+Andrew+McCallum%2E+2022%2E+Knowledge+base+question+answering+by+case%2Dbased+reasoning+over+subgrap)
10. Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. Case-based reasoning for natural language queries over knowledge bases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process… [find](https://scholar.google.com/scholar?q=Rajarshi+Das%2C+Manzil+Zaheer%2C+Dung+Thai%2C+Ameya+Godbole%2C+Ethan+Perez%2C+Jay+Yoon+Lee%2C+Lizhen+Tan%2C+Lazaros+Polymenakos%2C+and+Andrew+McCallum%2E+2021%2E+Case%2Dbased+reasoning+for+natural+language+queries+over+kno)
11. Joe Davison, Joshua Feldman, and Alexander M. Rush. 2019. [Commonsense knowledge mining from pre-trained models](https://md.research.elamur.ai/free/en/papers/davison-2019-commonsense-knowledge-mining-pretrained-models/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong…
12. Luc De Raedt. 2010. Inductive logic programming. [find](https://scholar.google.com/scholar?q=Luc+De+Raedt%2E+2010%2E+Inductive+logic+programming%2E)
13. Peter A Flach and Antonis C Kakas. 2000. Abductive and inductive reasoning: background and issues. In Abduction and induction, pages 1–27. Springer. [find](https://scholar.google.com/scholar?q=Peter+A+Flach+and+Antonis+C+Kakas%2E+2000%2E+Abductive+and+inductive+reasoning%3A+background+and+issues%2E+In+Abduction+and+induction%2C+pages+1%E2%80%9327%2E+Springer%2E)
14. Evan Heit. 2000. Properties of inductive reasoning. Psychonomic Bulletin & Review, 7(4):569–592. [find](https://scholar.google.com/scholar?q=Evan+Heit%2E+2000%2E+Properties+of+inductive+reasoning%2E+Psychonomic+Bulletin+%26+Review%2C+7%284%29%3A569%E2%80%93592%2E)
15. Karen Spärck Jones. 2004. A statistical interpretation of term specificity and its application in retrieval. J. Documentation, 60(5):493–502. [find](https://scholar.google.com/scholar?q=Karen+Sp%C3%A4rck+Jones%2E+2004%2E+A+statistical+interpretation+of+term+specificity+and+its+application+in+retrieval%2E+J%2E+Documentation%2C+60%285%29%3A493%E2%80%93502%2E)
16. Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out-of-distribution. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ananya+Kumar%2C+Aditi+Raghunathan%2C+Robbie+Matthew+Jones%2C+Tengyu+Ma%2C+and+Percy+Liang%2E+2022%2E+Fine%2Dtuning+can+distort+pretrained+features+and+underperform+out%2Dof%2Ddistribution%2E+In+The+Tenth+International+Co)
17. Yiyang Li and Hai Zhao. 2023. EM pre-training for multi-party dialogue response generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 92–103. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Yiyang+Li+and+Hai+Zhao%2E+2023%2E+EM+pre%2Dtraining+for+multi%2Dparty+dialogue+response+generation%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+P)
18. Yufei Li, Zexin Li, Yingfan Gao, and Cong Liu. 2023. White-box multi-objective adversarial attack on dialogue generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 1778–1792. A… [find](https://scholar.google.com/scholar?q=Yufei+Li%2C+Zexin+Li%2C+Yingfan+Gao%2C+and+Cong+Liu%2E+2023%2E+White%2Dbox+multi%2Dobjective+adversarial+attack+on+dialogue+generation%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational)
19. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael D. Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. [How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/). In Proceedings of the 2016 Conference on Empirical Methods in Nat…
20. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). CoRR, abs/2107.13586.
21. Yu Meng, Yunyi Zhang, Jiaxin Huang, Xuan Wang, Yu Zhang, Heng Ji, and Jiawei Han. 2021. Distantly-supervised named entity recognition with noise-robust learning and language model augmented self-training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNL… [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Yunyi+Zhang%2C+Jiaxin+Huang%2C+Xuan+Wang%2C+Yu+Zhang%2C+Heng+Ji%2C+and+Jiawei+Han%2E+2021%2E+Distantly%2Dsupervised+named+entity+recognition+with+noise%2Drobust+learning+and+language+model+augmented+self%2Dtrain)
22. Hugo Mercier. 2018. The enigma of reason. In The enigma of reason. Harvard University Press. [find](https://scholar.google.com/scholar?q=Hugo+Mercier%2E+2018%2E+The+enigma+of+reason%2E+In+The+enigma+of+reason%2E+Harvard+University+Press%2E)
23. Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings…
24. Kanishka Misra, Julia Taylor Rayz, and Allyson Ettinger. 2022. A property induction framework for neural language models. CoRR, abs/2205.06910. — [arXiv:2205.06910](https://arxiv.org/abs/2205.06910) [find](https://scholar.google.com/scholar?q=Kanishka+Misra%2C+Julia+Taylor+Rayz%2C+and+Allyson+Ettinger%2E+2022%2E+A+property+induction+framework+for+neural+language+models%2E+CoRR%2C+abs%2F2205%2E06910%2E+%E2%80%94+%5BarXiv%3A2205%2E06910%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2205%2E06910%29)
25. Stephen H. Muggleton and Luc De Raedt. 1994. Inductive logic programming: Theory and methods. J. Log. Program., 19/20:629–679. [find](https://scholar.google.com/scholar?q=Stephen+H%2E+Muggleton+and+Luc+De+Raedt%2E+1994%2E+Inductive+logic+programming%3A+Theory+and+methods%2E+J%2E+Log%2E+Program%2E%2C+19%2F20%3A629%E2%80%93679%2E)
26. Stephen H. Muggleton, Luc De Raedt, David Poole, Ivan Bratko, Peter A. Flach, Katsumi Inoue, and Ashwin Srinivasan. 2012. ILP turns 20 - biography and future challenges. Mach. Learn., 86(1):3–23. [find](https://scholar.google.com/scholar?q=Stephen+H%2E+Muggleton%2C+Luc+De+Raedt%2C+David+Poole%2C+Ivan+Bratko%2C+Peter+A%2E+Flach%2C+Katsumi+Inoue%2C+and+Ashwin+Srinivasan%2E+2012%2E+ILP+turns+20+%2D+biography+and+future+challenges%2E+Mach%2E+Learn%2E%2C+86%281%29%3A3%E2%80%9323%2E)
27. John D Norton. 2005. A little survey of induction. [find](https://scholar.google.com/scholar?q=John+D+Norton%2E+2005%2E+A+little+survey+of+induction%2E)
28. Jason Obeid and Enamul Hoque. 2020. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. In Proceedings of the 13th International Conference on Natural Language Generation, INLG 2020, Dublin, Ireland, December 15-18, 2020, pages 138–147. Association f… [find](https://scholar.google.com/scholar?q=Jason+Obeid+and+Enamul+Hoque%2E+2020%2E+Chart%2Dto%2Dtext%3A+Generating+natural+language+descriptions+for+charts+by+adapting+the+transformer+model%2E+In+Proceedings+of+the+13th+International+Conference+on+Natural)
29. Samir Okasha. 2002. Philosophy of science: A very short introduction, volume 67. Oxford Paperbacks. [find](https://scholar.google.com/scholar?q=Samir+Okasha%2E+2002%2E+Philosophy+of+science%3A+A+very+short+introduction%2C+volume+67%2E+Oxford+Paperbacks%2E)
30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.
31. Linlu Qiu and Liwei Jiang. 2023. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. [find](https://scholar.google.com/scholar?q=Linlu+Qiu+and+Liwei+Jiang%2E+2023%2E+Phenomenal+yet+puzzling%3A+Testing+inductive+reasoning+capabilities+of+language+models+with+hypothesis+refinement%2E)
32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67.
33. Stuart Russell and Peter Norvig. 2020. Artificial Intelligence: A Modern Approach (4th Edition). Pearson. [find](https://scholar.google.com/scholar?q=Stuart+Russell+and+Peter+Norvig%2E+2020%2E+Artificial+Intelligence%3A+A+Modern+Approach+%284th+Edition%29%2E+Pearson%2E)
34. Merrilee H Salmon. 1989. Introduction to logic and critical thinking. [find](https://scholar.google.com/scholar?q=Merrilee+H+Salmon%2E+1989%2E+Introduction+to+logic+and+critical+thinking%2E)
35. Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan… [find](https://scholar.google.com/scholar?q=Koustuv+Sinha%2C+Shagun+Sodhani%2C+Jin+Dong%2C+Joelle+Pineau%2C+and+William+L%2E+Hamilton%2E+2019%2E+CLUTRR%3A+A+diagnostic+benchmark+for+inductive+reasoning+from+text%2E+In+Proceedings+of+the+2019+Conference+on+Empiri)
36. Raymond M Smullyan. 1995. First-order logic. Courier Corporation. [find](https://scholar.google.com/scholar?q=Raymond+M+Smullyan%2E+1995%2E+First%2Dorder+logic%2E+Courier+Corporation%2E)
37. Komal K. Teru, Etienne G. Denis, and William L. Hamilton. 2020. Inductive relation prediction by subgraph reasoning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9448–… [find](https://scholar.google.com/scholar?q=Komal+K%2E+Teru%2C+Etienne+G%2E+Denis%2C+and+William+L%2E+Hamilton%2E+2020%2E+Inductive+relation+prediction+by+subgraph+reasoning%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+Learning%2C+ICML+2020%2C)
38. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
39. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [find](https://scholar.google.com/scholar?q=Ben+Wang+and+Aran+Komatsuzaki%2E+2021%2E+GPT%2DJ%2D6B%3A+A+6+Billion+Parameter+Autoregressive+Language+Model%2E)
40. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2023a. Learning to generate novel scientific directions with contextualized literature-based discovery. arXiv preprint arXiv:2305.14259. [find](https://scholar.google.com/scholar?q=Qingyun+Wang%2C+Doug+Downey%2C+Heng+Ji%2C+and+Tom+Hope%2E+2023a%2E+Learning+to+generate+novel+scientific+directions+with+contextualized+literature%2Dbased+discovery%2E+arXiv+preprint+arXiv%3A2305%2E14259%2E)
41. Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. 2023b. Hypothesis search: Inductive reasoning with language models. CoRR, abs/2309.05660. [find](https://scholar.google.com/scholar?q=Ruocheng+Wang%2C+Eric+Zelikman%2C+Gabriel+Poesia%2C+Yewen+Pu%2C+Nick+Haber%2C+and+Noah+D%2E+Goodman%2E+2023b%2E+Hypothesis+search%3A+Inductive+reasoning+with+language+models%2E+CoRR%2C+abs%2F2309%2E05660%2E)
42. Kaiyu Yang and Jia Deng. 2021. Learning symbolic rules for reasoning in quasi-natural language. CoRR, abs/2111.12038. — [arXiv:2111.12038](https://arxiv.org/abs/2111.12038) [find](https://scholar.google.com/scholar?q=Kaiyu+Yang+and+Jia+Deng%2E+2021%2E+Learning+symbolic+rules+for+reasoning+in+quasi%2Dnatural+language%2E+CoRR%2C+abs%2F2111%2E12038%2E+%E2%80%94+%5BarXiv%3A2111%2E12038%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2111%2E12038%29)
43. Kun Zhao, Bohao Yang, Chenghua Lin, Wenge Rong, Aline Villavicencio, and Xiaohui Cui. 2023. Evaluating open-domain dialogues in latent space with next sentence prediction and mutual information. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Kun+Zhao%2C+Bohao+Yang%2C+Chenghua+Lin%2C+Wenge+Rong%2C+Aline+Villavicencio%2C+and+Xiaohui+Cui%2E+2023%2E+Evaluating+open%2Ddomain+dialogues+in+latent+space+with+next+sentence+prediction+and+mutual+information%2E+In+Pr)
44. Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. 2023. Goal driven discovery of distributional differences via language descriptions. arXiv preprint arXiv:2302.14233. [find](https://scholar.google.com/scholar?q=Ruiqi+Zhong%2C+Peter+Zhang%2C+Steve+Li%2C+Jinwoo+Ahn%2C+Dan+Klein%2C+and+Jacob+Steinhardt%2E+2023%2E+Goal+driven+discovery+of+distributional+differences+via+language+descriptions%2E+arXiv+preprint+arXiv%3A2302%2E14233%2E)
45. Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. 2023. Large language models can learn rules. arXiv preprint arXiv:2310.07064. — [arXiv:2310.07064](https://arxiv.org/abs/2310.07064) [find](https://scholar.google.com/scholar?q=Zhaocheng+Zhu%2C+Yuan+Xue%2C+Xinyun+Chen%2C+Denny+Zhou%2C+Jian+Tang%2C+Dale+Schuurmans%2C+and+Hanjun+Dai%2E+2023%2E+Large+language+models+can+learn+rules%2E+arXiv+preprint+arXiv%3A2310%2E07064%2E+%E2%80%94+%5BarXiv%3A2310%2E07064%5D%28https%3A%2F)

*Updated 24 October 2026*