---
title: "JustLogic — paper by Chen et al. (2025) — summary and ideas"
description: "The authors argue that current deductive reasoning benchmarks for LLMs… 0.4 points on JustLogic. 36 references, cited by 10 papers here."
authors: ["Michael K. Chen", "Xikun Zhang", "Dacheng Tao"]
year: 2025
venue: "arXiv:2501.14851"
source: "https://arxiv.org/abs/2501.14851"
doi: "10.48550/arxiv.2501.14851"
arxiv: "2501.14851"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-justlogic-comprehensive-benchmark-evaluating-deductive/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-justlogic-comprehensive-benchmark-evaluating-deductive/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models

# JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models

**Authors:** Michael K. Chen, [Xikun Zhang](https://md.research.elamur.ai/free/en/authors/xikun-zhang/) and [Dacheng Tao](https://md.research.elamur.ai/free/en/authors/dacheng-tao/) · **Year:** 2025 · **Venue:** arXiv:2501.14851 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2501.14851) · [PDF](https://arxiv.org/pdf/2501.14851) · [DOI 10.48550/arxiv.2501.14851](https://doi.org/10.48550/arxiv.2501.14851) · [arXiv 2501.14851](https://arxiv.org/abs/2501.14851) · [OpenAlex](https://openalex.org/W4406879608)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/chen-2025-justlogic-comprehensive-benchmark-evaluating-deductive/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors argue that current deductive reasoning benchmarks for LLMs have three weaknesses. They are simple in either their language or their argument structure. They let models score well by using world knowledge instead of logic. Their error analysis is shallow. To fix this, they built JustLogic, a code-generated dataset. Argument structures are assembled programmatically from seven propositional-logic argument forms, and real sentences sampled from GenericsKB-Best fill them in. Each item gives a paragraph of premises and a statement, and the model must judge the statement as true, false, or uncertain. Because the sentences are combined at random, the resulting arguments rarely match reality. A new test confirmed that prior knowledge barely helps on JustLogic, while it inflates scores on manually curated sets. Most SOTA models fell below the average human score. Only DeepSeek R1 beat it, and no model reached the human ceiling. The benchmark therefore exposes real gaps in LLM logic and supports detailed diagnosis by argument form and reasoning depth.

## Key points
- **Prior knowledge test:** the accuracy gap |∆| when the premises are removed is only 0.4 points on JustLogic. It is 27.1 on LogiQA 2.0, 6.7 on FOLIO, 3.7 on ProofWriter and 2.0 on CLUTRR.
- **Human vs. model scores:** humans averaged 73.0% and the human ceiling is 100%. DeepSeek R1 scored 80.9% and OpenAI o1 72.9%. GPT-4o reached 65.6% with CoT, and random guessing gives 33.3%.
- **Prompting vs. scale:** prompting matters more than model size. Llama3-8B with chain-of-thought (57.8%) beat zero-shot Llama3-70B (53.1%).
- **Effect of depth:** accuracy generally falls as reasoning depth grows (depths 1–7, grouped into low, medium and high). OpenAI o1-mini matches DeepSeek R1 at low depth but collapses at medium and high depth.
- **Effect of argument form:** hypothetical syllogism and constructive dilemma are solved more reliably. Models find reductio ad absurdum, disjunctive syllogism and modus tollens more difficult. Code and data are available at github.com/michaelchen-lab/JustLogic.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Michael K. Chen, Xikun Zhang and Dacheng Tao. 2025. JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models. arXiv:2501.14851.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2025justlogic,
  title = {{JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models}},
  author = {Michael K. Chen, Xikun Zhang and Dacheng Tao},
  year = {2025},
  doi = {10.48550/arxiv.2501.14851},
  eprint = {2501.14851},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.14851}
}
```

- [← previous Investigating Idiomaticity in Word Representations](https://md.research.elamur.ai/free/en/papers/he-2025-investigating-idiomaticity-word-representations/)
- [next → Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model](https://md.research.elamur.ai/free/en/papers/xu-2025-let-s-focus-neuron-neuron/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/)

## References

36 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nan, L., Kasai, J., Yu, T., Zhang, R., Joty, S., Fabbri, A. R., Kryscinski, W., Lin, X. V., Xiong, C., and Radev, D. [Folio: Natural language reasoning with first-order logic](https://md.research.elamur.ai/free/en/papers/han-2022-folio-natural-language-reasoning-first/). arXiv preprint arXiv:2209.00840, 2022. URL https://arxiv.org/abs/2209.00840. [↗](https://arxiv.org/abs/2209.00840)
2. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
3. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
4. Amazon. Amazon Mechanical Turk, 2005. URL [https://www.mturk.com/](https://www.mturk.com/). [↗](https://www.mturk.com/)
5. Beurer-Kellner, L., Fischer, M., and Vechev, M. Guiding LLMs the right way: Fast, non-invasive constrained generation. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning,… [find](https://scholar.google.com/scholar?q=Beurer%2DKellner%2C+L%2E%2C+Fischer%2C+M%2E%2C+and+Vechev%2C+M%2E+Guiding+LLMs+the+right+way%3A+Fast%2C+non%2Dinvasive+constrained+generation%2E+In+Salakhutdinov%2C+R%2E%2C+Kolter%2C+Z%2E%2C+Heller%2C+K%2E%2C+Weller%2C+A%2E%2C+Oliver%2C+N%2E%2C+Scarlett%2C+J)
6. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). arXiv preprint arXiv:2412.16720, 2024.
7. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E)
8. Bhakthavatsalam, S., Anastasiades, C., and Clark, P. Genericskb: A knowledge base of generic statements. arXiv preprint arXiv:2005.00660, 2020. [find](https://scholar.google.com/scholar?q=Bhakthavatsalam%2C+S%2E%2C+Anastasiades%2C+C%2E%2C+and+Clark%2C+P%2E+Genericskb%3A+A+knowledge+base+of+generic+statements%2E+arXiv+preprint+arXiv%3A2005%2E00660%2C+2020%2E)
9. Johnson, R. A Logic Book: Fundamentals of Reasoning. Cengage Learning, 5 edition, 2006. [find](https://scholar.google.com/scholar?q=Johnson%2C+R%2E+A+Logic+Book%3A+Fundamentals+of+Reasoning%2E+Cengage+Learning%2C+5+edition%2C+2006%2E)
10. Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., and Hajishirzi, H. [Unifiedqa: Crossing format boundaries with a single qa system](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/). arXiv preprint arXiv:2005.00700, 2020.
11. Brown, T. B. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 2020.
12. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457, 2018.
13. Kincaid, J. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Chief of Naval Technical Training, 1975. [find](https://scholar.google.com/scholar?q=Kincaid%2C+J%2E+Derivation+of+new+readability+formulas+%28automated+readability+index%2C+fog+count+and+flesch+reading+ease+formula%29+for+navy+enlisted+personnel%2E+Chief+of+Naval+Technical+Training%2C+1975%2E)
14. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
15. Liu, H., Liu, J., Cui, L., Teng, Z., Duan, N., Zhou, M., and Zhang, Y. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023a. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Liu%2C+J%2E%2C+Cui%2C+L%2E%2C+Teng%2C+Z%2E%2C+Duan%2C+N%2E%2C+Zhou%2C+M%2E%2C+and+Zhang%2C+Y%2E+Logiqa+2%2E0%E2%80%94an+improved+dataset+for+logical+reasoning+in+natural+language+understanding%2E+IEEE%2FACM+Transactions+on+Audio%2C+Speech%2C+a)
16. Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. [A survey on in-context learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/). arXiv preprint arXiv:2301.00234, 2022.
17. Liu, H., Teng, Z., Cui, L., Zhang, C., Zhou, Q., and Zhang, Y. Logicot: Logical chain-of-thought instruction tuning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2908–2921, 2023b. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Teng%2C+Z%2E%2C+Cui%2C+L%2E%2C+Zhang%2C+C%2E%2C+Zhou%2C+Q%2E%2C+and+Zhang%2C+Y%2E+Logicot%3A+Logical+chain%2Dof%2Dthought+instruction+tuning%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2023%2C+pp%2E+2908%E2%80%93)
18. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. [Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). arXiv preprint arXiv:1903.00161, 2019.
19. Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. [Logiqa: A challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/). arXiv preprint arXiv:2007.08124, 2020.
20. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
21. OpenAI. Learning to Reason with LLMs, 9 2024. URL [https://openai.com/index/](https://openai.com/index/) learning-to-reason-with-llms/. [↗](https://openai.com/index/)
22. Fahlman, S. E. and Hinton, G. E. Connectionist architectures for artificial intelligence. Computer, 20(01):100– 109, 1987. [find](https://scholar.google.com/scholar?q=Fahlman%2C+S%2E+E%2E+and+Hinton%2C+G%2E+E%2E+Connectionist+architectures+for+artificial+intelligence%2E+Computer%2C+20%2801%29%3A100%E2%80%93+109%2C+1987%2E)
23. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [find](https://scholar.google.com/scholar?q=Sakaguchi%2C+K%2E%2C+Bras%2C+R%2E+L%2E%2C+Bhagavatula%2C+C%2E%2C+and+Choi%2C+Y%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2C+2021%2E)
24. Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Benson, L., Sun, L., Zubova, E., Qiao, Y., Burtell, M., Peng, D., Fan, J., Liu, Y., Wong, B., Sailor, M., Ni, A., Saparov, A., Pang, R. Y., Padmakumar, V., Joshi, N., Kazemi, M., Kim, N., and He, H. Testing the general deductive reasoning capac… — [Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples](https://md.research.elamur.ai/free/en/papers/saparov-2023-testing-general-deductive-reasoning-capacity/)
25. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625, 2022.
26. Sinha, K., Sodhani, S., Dong, J., Pineau, J., and Hamilton, W. L. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint arXiv:1908.06177, 2019. [find](https://scholar.google.com/scholar?q=Sinha%2C+K%2E%2C+Sodhani%2C+S%2E%2C+Dong%2C+J%2E%2C+Pineau%2C+J%2E%2C+and+Hamilton%2C+W%2E+L%2E+Clutrr%3A+A+diagnostic+benchmark+for+inductive+reasoning+from+text%2E+arXiv+preprint+arXiv%3A1908%2E06177%2C+2019%2E)
27. Sprague, Z., Ye, X., Bostrom, K., Chaudhuri, S., and Durrett, G. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. arXiv preprint arXiv:2310.16049, 2023. [find](https://scholar.google.com/scholar?q=Sprague%2C+Z%2E%2C+Ye%2C+X%2E%2C+Bostrom%2C+K%2E%2C+Chaudhuri%2C+S%2E%2C+and+Durrett%2C+G%2E+Musr%3A+Testing+the+limits+of+chain%2Dof%2Dthought+with+multistep+soft+reasoning%2E+arXiv+preprint+arXiv%3A2310%2E16049%2C+2023%2E)
28. Tafjord, O., Mishra, B. D., and Clark, P. [Proofwriter: Generating implications, proofs, and abductive statements over natural language](https://md.research.elamur.ai/free/en/papers/tafjord-2021-proofwriter-generating-implications-proofs-abductive/). arXiv preprint arXiv:2012.13048, 2020.
29. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
30. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171, 2022.
31. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022.
32. Xu, R., Wang, Z., Fan, R.-Z., and Liu, P. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824, 2024. [find](https://scholar.google.com/scholar?q=Xu%2C+R%2E%2C+Wang%2C+Z%2E%2C+Fan%2C+R%2E%2DZ%2E%2C+and+Liu%2C+P%2E+Benchmarking+benchmark+leakage+in+large+language+models%2E+arXiv+preprint+arXiv%3A2404%2E18824%2C+2024%2E)
33. Yadav, V., Bethard, S., and Surdeanu, M. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. arXiv preprint arXiv:1911.07176, 2019. [find](https://scholar.google.com/scholar?q=Yadav%2C+V%2E%2C+Bethard%2C+S%2E%2C+and+Surdeanu%2C+M%2E+Quick+and+%28not+so%29+dirty%3A+Unsupervised+selection+of+justification+sentences+for+multi%2Dhop+question+answering%2E+arXiv+preprint+arXiv%3A1911%2E07176%2C+2019%2E)
34. Yu, W., Jiang, Z., Dong, Y., and Feng, J. [Reclor: A reading comprehension dataset requiring logical reasoning](https://md.research.elamur.ai/free/en/papers/yu-2020-reclor-reading-comprehension-dataset-requiring/). arXiv preprint arXiv:2002.04326, 2020.
35. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Holtzman%2C+A%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
36. Zhang, H., Li, L. H., Meng, T., Chang, K.-W., and Broeck, G. V. d. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502, 2022. — [arXiv:2205.11502](https://arxiv.org/abs/2205.11502) [find](https://scholar.google.com/scholar?q=Zhang%2C+H%2E%2C+Li%2C+L%2E+H%2E%2C+Meng%2C+T%2E%2C+Chang%2C+K%2E%2DW%2E%2C+and+Broeck%2C+G%2E+V%2E+d%2E+On+the+paradox+of+learning+to+reason+from+data%2E+arXiv+preprint+arXiv%3A2205%2E11502%2C+2022%2E+%E2%80%94+%5BarXiv%3A2205%2E11502%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F220)

*Updated 31 October 2026*