---
title: "On Memorization of Large Language Models in Logical Reasoning"
description: "They define LiMem, a memorization score that multiplies a model's accuracy on a puzzle set by the… 80 references, cited by 42 papers here."
authors: ["Chulin Xie", "Yangsibo Huang", "Chiyuan Zhang", "Da Yu", "Xinyun Chen", "Bill Yuchen Lin", "Bo Li", "Badih Ghazi", "Ravi Kumar"]
year: 2025
venue: "Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics"
source: "https://aclanthology.org/2025.ijcnlp-long.148/"
doi: "10.18653/v1/2025.ijcnlp-long.148"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xie-2025-memorization-large-language-models-logical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xie-2025-memorization-large-language-models-logical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / On Memorization of Large Language Models in Logical Reasoning

# On Memorization of Large Language Models in Logical Reasoning

**Authors:** [Chulin Xie](https://md.research.elamur.ai/free/en/authors/chulin-xie/), [Yangsibo Huang](https://md.research.elamur.ai/free/en/authors/yangsibo-huang/), [Chiyuan Zhang](https://md.research.elamur.ai/free/en/authors/chiyuan-zhang/), Da Yu, [Xinyun Chen](https://md.research.elamur.ai/free/en/authors/xinyun-chen/), [Bill Yuchen Lin](https://md.research.elamur.ai/free/en/authors/bill-yuchen-lin/), [Bo Li](https://md.research.elamur.ai/free/en/authors/bo-li/), Badih Ghazi and Ravi Kumar · **Year:** 2025 · **Venue:** [Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/ijcnlp/) ([IJCNLP 2025](https://md.research.elamur.ai/free/en/venues/ijcnlp/2025/)) · **Publisher:** [The Asian Federation of Natural Language Processing and The Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/the-asian-federation-of-natural-language-processing-and-the-association-for/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.ijcnlp-long.148/) · [PDF](https://aclanthology.org/2025.ijcnlp-long.148.pdf) · [DOI 10.18653/v1/2025.ijcnlp-long.148](https://doi.org/10.18653/v1/2025.ijcnlp-long.148) · [OpenAlex](https://openalex.org/W7137858089)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [42](https://md.research.elamur.ai/free/en/papers/xie-2025-memorization-large-language-models-logical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs score well on hard reasoning benchmarks yet still fail on trivial questions. This raises the suspicion that some benchmark success comes from recalling familiar problems. The authors test this idea in a controlled setting. They define LiMem, a memorization score that multiplies a model's accuracy on a puzzle set by the fraction of solved puzzles it fails after small perturbations that keep the solving principle and difficulty the same. To support the metric, they build two dynamic logic-puzzle generators. One is based on Knights and Knaves (K&K) and grounded in Boolean satisfiability. The other is DynamicZebra. Both can create puzzles at a chosen difficulty, perturb them at the language or problem level, recompute answers and produce step-by-step solutions. After evaluating 17 LLMs, they fine-tune Llama3-8B and GPT4o-mini on K&K. The fine-tuned models fit the training puzzles almost perfectly but break under perturbation, which points to memorization. Their accuracy on unseen puzzles also keeps improving. Memorization and real reasoning skill therefore grow together, which matters for interpreting benchmark scores and contamination.

## Key points
- **Metric:** LiMem = Acc · (1 − CR). CR is the share of originally solved puzzles that the model still solves after local perturbation. The score ranges from 0 to 1.
- **K&K benchmark design:**
  - It has an abstract module with a Generator, Solver, Reasoner (CoT synthesis) and Perturber.
  - Puzzle difficulty is controlled by the number of people and by logical-tree width and depth.
- **Fine-tuning findings:**
  - Models trained only on question–answer pairs still gain generalization and transfer across difficulty levels.
  - CoT fine-tuning yields higher test accuracy than answer-only fine-tuning.
  - Even fully corrupted CoTs help somewhat. Shuffled steps hurt more than a single wrong step.
- **Per-sample analysis:** Embeddings from the fine-tuned model predict whether a given puzzle is solved by memorization or by reasoning better than embeddings from the base model do.
- **Generalization of findings:** DynamicZebra results with Qwen2.5-7B mirror the K&K findings. Code and data are available at memkklogic.github.io.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 353 works · 200,043 citations
- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 184 works · 69,372 citations
- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 90 works · 39,109 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi and Ravi Kumar. 2025. On Memorization of Large Language Models in Logical Reasoning. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xie2025memorization,
  title = {{On Memorization of Large Language Models in Logical Reasoning}},
  author = {Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi and Ravi Kumar},
  booktitle = {Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.ijcnlp-long.148},
  url = {https://doi.org/10.18653/v1/2025.ijcnlp-long.148}
}
```

- [← previous Gemini Embedding: Generalizable Embeddings from Gemini](https://md.research.elamur.ai/free/en/papers/lee-2025-gemini-embedding-generalizable-embeddings-gemini/)
- [next → AdaptThink: Reasoning Models Can Learn When to Think](https://md.research.elamur.ai/free/en/papers/zhang-2025-adaptthink-reasoning-models-can-learn/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

80 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. [Fine-grained analysis of sentence embeddings using auxiliary prediction tasks](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/). In ICLR.
2. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. 2024. [Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs](https://md.research.elamur.ai/free/en/papers/balloccu-2024-leak-cheat-repeat-data-contamination/). arXiv:2402.03927.
3. Mikhail Belkin. 2021. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica. [find](https://scholar.google.com/scholar?q=Mikhail+Belkin%2E+2021%2E+Fit+without+fear%3A+remarkable+mathematical+phenomena+of+deep+learning+through+the+prism+of+interpolation%2E+Acta+Numerica%2E)
4. Mikhail Belkin, Daniel J Hsu, and Partha Mitra. 2018. Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate. In NeurIPS. [find](https://scholar.google.com/scholar?q=Mikhail+Belkin%2C+Daniel+J+Hsu%2C+and+Partha+Mitra%2E+2018%2E+Overfitting+or+perfect+fitting%3F+Risk+bounds+for+classification+and+regression+rules+that+interpolate%2E+In+NeurIPS%2E)
5. Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. 2024. Emergent and predictable memorization in large language models. NeurIPS, 36. [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Usvsn+Prashanth%2C+Lintang+Sutawika%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Shivanshu+Purohit%2C+and+Edward+Raff%2E+2024%2E+Emergent+and+predictable+memorization+in+large+language+models%2E+NeurIP)
6. Boolean satisfiability problem. Boolean satisfiability problem — Wikipedia, the free encyclopedia. [Online; accessed 20-Nov-2024]. [find](https://scholar.google.com/scholar?q=Boolean+satisfiability+problem%2E+Boolean+satisfiability+problem+%E2%80%94+Wikipedia%2C+the+free+encyclopedia%2E+%5BOnline%3B+accessed+20%2DNov%2D2024%5D%2E)
7. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In ICLR. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Daphne+Ippolito%2C+Matthew+Jagielski%2C+Katherine+Lee%2C+Florian+Tramer%2C+and+Chiyuan+Zhang%2E+2023%2E+Quantifying+memorization+across+neural+language+models%2E+In+ICLR%2E)
8. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. [Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/). In USENIX Security.
9. Xinyun Chen, Ryan Andrew Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. In ICML. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Ryan+Andrew+Chi%2C+Xuezhi+Wang%2C+and+Denny+Zhou%2E+2024%2E+Premise+order+matters+in+reasoning+with+large+language+models%2E+In+ICML%2E)
10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2024. Scaling instruction-finetuned language models. JMLR. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+and+1+others%2E+2024%2E+Scaling+instruction%2Dfinetuned+language+mo)
11. Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. [Transformers as soft reasoners over language](https://md.research.elamur.ai/free/en/papers/clark-2020-transformers-soft-reasoners-over-language/). In IJCAI.
12. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/). In ACL.
13. Yuntian Deng, Yejin Choi, and Stuart Shieber. 2024. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv:2405.14838. [find](https://scholar.google.com/scholar?q=Yuntian+Deng%2C+Yejin+Choi%2C+and+Stuart+Shieber%2E+2024%2E+From+explicit+cot+to+implicit+cot%3A+Learning+to+internalize+cot+step+by+step%2E+arXiv%3A2405%2E14838%2E)
14. Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. 2023. Implicit chain of thought reasoning via knowledge distillation. arXiv:2311.01460. [find](https://scholar.google.com/scholar?q=Yuntian+Deng%2C+Kiran+Prasad%2C+Roland+Fernandez%2C+Paul+Smolensky%2C+Vishrav+Chaudhary%2C+and+Stuart+Shieber%2E+2023%2E+Implicit+chain+of+thought+reasoning+via+knowledge+distillation%2E+arXiv%3A2311%2E01460%2E)
15. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaïd Harchaoui, and Yejin Choi. 2024. [Faith and fate: Limits of transformers on compositiona…](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/)
16. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+and+1+others%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+)
17. Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, and Giorgos Stamou. 2024. [Puzzle solving using reasoning of large language models: A survey](https://md.research.elamur.ai/free/en/papers/giadikiaroglou-2024-puzzle-solving-reasoning-large-language/). In IJCAI.
18. Shahriar Golchin and Mihai Surdeanu. 2023. Data contamination quiz: A tool to detect and estimate contamination in large language models. arXiv:2311.06233. [find](https://scholar.google.com/scholar?q=Shahriar+Golchin+and+Mihai+Surdeanu%2E+2023%2E+Data+contamination+quiz%3A+A+tool+to+detect+and+estimate+contamination+in+large+language+models%2E+arXiv%3A2311%2E06233%2E)
19. Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. 2024. Changing answer order can decrease MMLU accuracy. arXiv:2406.19470. — [arXiv:2406.19470](https://arxiv.org/abs/2406.19470) [find](https://scholar.google.com/scholar?q=Vipul+Gupta%2C+David+Pantoja%2C+Candace+Ross%2C+Adina+Williams%2C+and+Megan+Ung%2E+2024%2E+Changing+answer+order+can+decrease+MMLU+accuracy%2E+arXiv%3A2406%2E19470%2E+%E2%80%94+%5BarXiv%3A2406%2E19470%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2406%2E19470)
20. Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. 2023. SoK: memorization in general-purpose large language models. arXiv:2310.18362. — [arXiv:2310.18362](https://arxiv.org/abs/2310.18362) [find](https://scholar.google.com/scholar?q=Valentin+Hartmann%2C+Anshuman+Suri%2C+Vincent+Bindschaedler%2C+David+Evans%2C+Shruti+Tople%2C+and+Robert+West%2E+2023%2E+SoK%3A+memorization+in+general%2Dpurpose+large+language+models%2E+arXiv%3A2310%2E18362%2E+%E2%80%94+%5BarXiv%3A2310%2E1)
21. Luxi He, Yangsibo Huang, Weijia Shi, Tinghao Xie, Haotian Liu, Yue Wang, Luke Zettlemoyer, Chiyuan Zhang, Danqi Chen, and Peter Henderson. 2024. Fantastic copyrighted beasts and how (not) to generate them. arXiv:2406.14526. [find](https://scholar.google.com/scholar?q=Luxi+He%2C+Yangsibo+Huang%2C+Weijia+Shi%2C+Tinghao+Xie%2C+Haotian+Liu%2C+Yue+Wang%2C+Luke+Zettlemoyer%2C+Chiyuan+Zhang%2C+Danqi+Chen%2C+and+Peter+Henderson%2E+2024%2E+Fantastic+copyrighted+beasts+and+how+%28not%29+to+generate+)
22. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In EMNLP. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Percy+Liang%2E+2019%2E+Designing+and+interpreting+probes+with+control+tasks%2E+In+EMNLP%2E)
23. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. [Large language models are reasoning teachers](https://md.research.elamur.ai/free/en/papers/ho-2023-large-language-models-are-reasoning/). In ACL.
24. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). In ACL.
25. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. [LiveCodeBench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv:2403.07974.
26. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. 2024. A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity. In ICML. [find](https://scholar.google.com/scholar?q=Andrew+Lee%2C+Xiaoyan+Bai%2C+Itamar+Pres%2C+Martin+Wattenberg%2C+Jonathan+K+Kummerfeld%2C+and+Rada+Mihalcea%2E+2024%2E+A+mechanistic+understanding+of+alignment+algorithms%3A+A+case+study+on+DPO+and+toxicity%2E+In+ICML%2E)
27. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In ACL. [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Daphne+Ippolito%2C+Andrew+Nystrom%2C+Chiyuan+Zhang%2C+Douglas+Eck%2C+Chris+Callison%2DBurch%2C+and+Nicholas+Carlini%2E+2022%2E+Deduplicating+training+data+makes+language+models+better%2E+In+ACL%2E)
28. Bill Yuchen Lin. 2024. Math Olympiad becomes easier for AI; Common sense is still hard. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2E+2024%2E+Math+Olympiad+becomes+easier+for+AI%3B+Common+sense+is+still+hard%2E)
29. Bill Yuchen Lin, Ronan Le Bras, and Yejin Choi. 2024. ZebraLogic: benchmarking the logical reasoning ability of language models. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Ronan+Le+Bras%2C+and+Yejin+Choi%2E+2024%2E+ZebraLogic%3A+benchmarking+the+logical+reasoning+ability+of+language+models%2E)
30. Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. 2025. Zebralogic: On the scaling limits of llms for logical reasoning. arXiv:2502.01100. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Ronan+Le+Bras%2C+Kyle+Richardson%2C+Ashish+Sabharwal%2C+Radha+Poovendran%2C+Peter+Clark%2C+and+Yejin+Choi%2E+2025%2E+Zebralogic%3A+On+the+scaling+limits+of+llms+for+logical+reasoning%2E+arXiv%3A2502%2E0110)
31. Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. 2022a. Towards understanding grokking: An effective theory of representation learning. NeurIPS. [find](https://scholar.google.com/scholar?q=Ziming+Liu%2C+Ouail+Kitouni%2C+Niklas+S+Nolte%2C+Eric+Michaud%2C+Max+Tegmark%2C+and+Mike+Williams%2E+2022a%2E+Towards+understanding+grokking%3A+An+effective+theory+of+representation+learning%2E+NeurIPS%2E)
32. Ziming Liu, Eric J Michaud, and Max Tegmark. 2022b. Omnigrok: Grokking beyond algorithmic data. In ICLR. [find](https://scholar.google.com/scholar?q=Ziming+Liu%2C+Eric+J+Michaud%2C+and+Max+Tegmark%2E+2022b%2E+Omnigrok%3A+Grokking+beyond+algorithmic+data%2E+In+ICLR%2E)
33. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. Analyzing leakage of personally identifiable information in language models. In S & P. [find](https://scholar.google.com/scholar?q=Nils+Lukas%2C+Ahmed+Salem%2C+Robert+Sim%2C+Shruti+Tople%2C+Lukas+Wutschitz%2C+and+Santiago+Zanella%2DB%C3%A9guelin%2E+2023%2E+Analyzing+leakage+of+personally+identifiable+information+in+language+models%2E+In+S+%26+P%2E)
34. Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo J Taylor, and Dan Roth. 2024. [A peek into token bias: Large language models are not yet genuine reasoners](https://md.research.elamur.ai/free/en/papers/jiang-2024-peek-into-token-bias-large/). EMNLP.
35. R Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D Hardy, and Thomas L Griffiths. 2024. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. PNAS, 121(41):e2322420121. [find](https://scholar.google.com/scholar?q=R+Thomas+McCoy%2C+Shunyu+Yao%2C+Dan+Friedman%2C+Mathew+D+Hardy%2C+and+Thomas+L+Griffiths%2E+2024%2E+Embers+of+autoregression+show+how+large+language+models+are+shaped+by+the+problem+they+are+trained+to+solve%2E+PNA)
36. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? natural language attack on text classification and entailment. AAAI. [find](https://scholar.google.com/scholar?q=Di+Jin%2C+Zhijing+Jin%2C+Joey+Tianyi+Zhou%2C+and+Peter+Szolovits%2E+2020%2E+Is+bert+really+robust%3F+natural+language+attack+on+text+classification+and+entailment%2E+AAAI%2E)
37. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. [Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models](https://md.research.elamur.ai/free/en/papers/mirzadeh-2024-gsm-symbolic-understanding-limitations-mathematical/). arXiv:2410.05229.
38. Philip N Johnson-Laird and Ruth MJ Byrne. 1990. Meta-logical problems: Knights, knaves, and rips. Cognition. [find](https://scholar.google.com/scholar?q=Philip+N+Johnson%2DLaird+and+Ruth+MJ+Byrne%2E+1990%2E+Meta%2Dlogical+problems%3A+Knights%2C+knaves%2C+and+rips%2E+Cognition%2E)
39. Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. [Copyright violations and large language models](https://md.research.elamur.ai/free/en/papers/karamolegkou-2023-copyright-violations-large-language-models/). In EMNLP.
40. Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. 2024. BoardgameQA: A dataset for natural language reasoning with contradictory information. In NeurIPS. [find](https://scholar.google.com/scholar?q=Mehran+Kazemi%2C+Quan+Yuan%2C+Deepti+Bhatia%2C+Najoung+Kim%2C+Xin+Xu%2C+Vaiva+Imbrasaite%2C+and+Deepak+Ramachandran%2E+2024%2E+BoardgameQA%3A+A+dataset+for+natural+language+reasoning+with+contradictory+information%2E+In+)
41. Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. 2023. [The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning](https://md.research.elamur.ai/free/en/papers/kim-2023-cot-collection-improving-zero-shot/). In EMNLP.
42. Philipp Mondorf and Barbara Plank. 2024. Liar, liar, logical mire: A benchmark for suppositional reasoning in large language models. arXiv:2406.12546. [find](https://scholar.google.com/scholar?q=Philipp+Mondorf+and+Barbara+Plank%2E+2024%2E+Liar%2C+liar%2C+logical+mire%3A+A+benchmark+for+suppositional+reasoning+in+large+language+models%2E+arXiv%3A2406%2E12546%2E)
43. Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. 2023. Grokking of hierarchical structure in vanilla transformers. In ACL. [find](https://scholar.google.com/scholar?q=Shikhar+Murty%2C+Pratyusha+Sharma%2C+Jacob+Andreas%2C+and+Christopher+D+Manning%2E+2023%2E+Grokking+of+hierarchical+structure+in+vanilla+transformers%2E+In+ACL%2E)
44. Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, and Jenia Jitsev. 2024. Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models. arXiv:2406.02061. [find](https://scholar.google.com/scholar?q=Marianna+Nezhurina%2C+Lucia+Cipolina%2DKun%2C+Mehdi+Cherti%2C+and+Jenia+Jitsev%2E+2024%2E+Alice+in+wonderland%3A+Simple+tasks+showing+complete+reasoning+breakdown+in+state%2Dof%2Dthe%2Dart+large+language+models%2E+arXiv%3A24)
45. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. [Adversarial NLI: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). In ACL, pages 4885–4901.
46. Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. 2024. Proving test set contamination in black box language models. ICLR. [find](https://scholar.google.com/scholar?q=Yonatan+Oren%2C+Nicole+Meister%2C+Niladri+Chatterji%2C+Faisal+Ladhak%2C+and+Tatsunori+B+Hashimoto%2E+2024%2E+Proving+test+set+contamination+in+black+box+language+models%2E+ICLR%2E)
47. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. [Memorization without overfitting: Analyzing the training dynamics of large language models](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/). NeurIPS.
48. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. [Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning](https://md.research.elamur.ai/free/en/papers/pan-2023-logic-lm-empowering-large-language/). In EMNLP.
49. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. 2024. The instruction hierarchy: Training LLMs to prioritize privileged instructions. arXiv:2404.13208. [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Kai+Xiao%2C+Reimar+Leike%2C+Lilian+Weng%2C+Johannes+Heidecke%2C+and+Alex+Beutel%2E+2024%2E+The+instruction+hierarchy%3A+Training+LLMs+to+prioritize+privileged+instructions%2E+arXiv%3A2404%2E13208%2E)
50. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. [LogicBench: towards systematic evaluation of logical reasoning ability of large language models](https://md.research.elamur.ai/free/en/papers/parmar-2024-logicbench-systematic-evaluation-logical-reasoning/). In ACL.
51. Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. 2024a. Grokking of implicit reasoning in transformers: A mechanistic journey to the edge of generalization. In NeurIPS. [find](https://scholar.google.com/scholar?q=Boshi+Wang%2C+Xiang+Yue%2C+Yu+Su%2C+and+Huan+Sun%2E+2024a%2E+Grokking+of+implicit+reasoning+in+transformers%3A+A+mechanistic+journey+to+the+edge+of+generalization%2E+In+NeurIPS%2E)
52. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. 2022. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177. [find](https://scholar.google.com/scholar?q=Alethea+Power%2C+Yuri+Burda%2C+Harri+Edwards%2C+Igor+Babuschkin%2C+and+Vedant+Misra%2E+2022%2E+Grokking%3A+Generalization+beyond+overfitting+on+small+algorithmic+datasets%2E+arXiv%3A2201%2E02177%2E)
53. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, and 1 others. 2023a. [DecodingTrust: a comprehensive assessment of trustworthiness in GPT models](https://md.research.elamur.ai/free/en/papers/wang-2023-decodingtrust-comprehensive-assessment-trustworthiness-gpt/). In NeurIPS.
54. Akshara Prabhakar, Thomas L Griffiths, and R Thomas McCoy. 2024. Deciphering the factors influencing the efficacy of chain-of-thought: Probability, memorization, and noisy reasoning. arXiv:2407.01687. [find](https://scholar.google.com/scholar?q=Akshara+Prabhakar%2C+Thomas+L+Griffiths%2C+and+R+Thomas+McCoy%2E+2024%2E+Deciphering+the+factors+influencing+the+efficacy+of+chain%2Dof%2Dthought%3A+Probability%2C+memorization%2C+and+noisy+reasoning%2E+arXiv%3A2407%2E01687%2E)
55. USVSN Sai Prashanth, Alvin Deng, Kyle O’Brien, Jyothir SV, Mohammad Aflah Khan, Jaydeep Borkar, Christopher A Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, and 1 others. 2024. Recite, reconstruct, recollect: Memorization in LMs as a multifaceted phenomenon. arXiv:2406.17746. — [arXiv:2406.17746](https://arxiv.org/abs/2406.17746) [find](https://scholar.google.com/scholar?q=USVSN+Sai+Prashanth%2C+Alvin+Deng%2C+Kyle+O%E2%80%99Brien%2C+Jyothir+SV%2C+Mohammad+Aflah+Khan%2C+Jaydeep+Borkar%2C+Christopher+A+Choquette%2DChoo%2C+Jacob+Ray+Fuehne%2C+Stella+Biderman%2C+Tracy+Ke%2C+and+1+others%2E+2024%2E+Recite%2C+r)
56. Haritz Puerto, Tilek Chubakov, Xiaodan Zhu, Harish Tayyar Madabushi, and Iryna Gurevych. 2024. Fine-tuning with divergent chains of thought boosts reasoning through self-correction in language models. arXiv:2407.03181. — [arXiv:2407.03181](https://arxiv.org/abs/2407.03181) [find](https://scholar.google.com/scholar?q=Haritz+Puerto%2C+Tilek+Chubakov%2C+Xiaodan+Zhu%2C+Harish+Tayyar+Madabushi%2C+and+Iryna+Gurevych%2E+2024%2E+Fine%2Dtuning+with+divergent+chains+of+thought+boosts+reasoning+through+self%2Dcorrection+in+language+models%2E)
57. Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. [Impact of pretraining term frequencies on few-shot numerical reasoning](https://md.research.elamur.ai/free/en/papers/razeghi-2022-impact-pretraining-term-frequencies-few/). In Findings of EMNLP 2022, pages 840–854.
58. Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. 2023. To the cutoff... and beyond? A longitudinal perspective on LLM data contamination. In ICLR. [find](https://scholar.google.com/scholar?q=Manley+Roberts%2C+Himanshu+Thakur%2C+Christine+Herlihy%2C+Colin+White%2C+and+Samuel+Dooley%2E+2023%2E+To+the+cutoff%2E%2E%2E+and+beyond%3F+A+longitudinal+perspective+on+LLM+data+contamination%2E+In+ICLR%2E)
59. Joshua Robinson and David Wingate. 2023. [Leveraging large language models for multiple choice question answering](https://md.research.elamur.ai/free/en/papers/robinson-2022-leveraging-large-language-models-multiple/). In ICLR.
60. Abulhair Saparov and He He. 2023. [Language models are greedy reasoners: A systematic formal analysis of chain-of-thought](https://md.research.elamur.ai/free/en/papers/saparov-2022-language-models-greedy-reasoners-systematic/). In ICLR.
61. SAT solver. 2024. Sat solver — Wikipedia, the free encyclopedia. [Online; accessed 20-Nov-2024]. [find](https://scholar.google.com/scholar?q=SAT+solver%2E+2024%2E+Sat+solver+%E2%80%94+Wikipedia%2C+the+free+encyclopedia%2E+%5BOnline%3B+accessed+20%2DNov%2D2024%5D%2E)
62. Raymond Smullyan. 1978. What is the Name of this Book? Prentice-Hall. [find](https://scholar.google.com/scholar?q=Raymond+Smullyan%2E+1978%2E+What+is+the+Name+of+this+Book%3F+Prentice%2DHall%2E)
63. Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, and 1 others. 2024. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv:2402.19450. [find](https://scholar.google.com/scholar?q=Saurabh+Srivastava%2C+Anto+PV%2C+Shashank+Menon%2C+Ajay+Sukumar%2C+Alan+Philipose%2C+Stevin+Prince%2C+Sooraj+Thomas%2C+and+1+others%2E+2024%2E+Functional+benchmarks+for+robust+evaluation+of+reasoning+performance%2C+and+t)
64. Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021. Adversarial GLUE: A multi-task benchmark for robustness evaluation of language models. In NeurIPS Datasets and Benchmarks. [find](https://scholar.google.com/scholar?q=Boxin+Wang%2C+Chejian+Xu%2C+Shuohang+Wang%2C+Zhe+Gan%2C+Yu+Cheng%2C+Jianfeng+Gao%2C+Ahmed+Hassan+Awadallah%2C+and+Bo+Li%2E+2021%2E+Adversarial+GLUE%3A+A+multi%2Dtask+benchmark+for+robustness+evaluation+of+language+models%2E+)
65. Haochun Wang, Sendong Zhao, Zewen Qiang, Bing Qin, and Ting Liu. 2024b. Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models. arXiv:2402.01349. — [arXiv:2402.01349](https://arxiv.org/abs/2402.01349) [find](https://scholar.google.com/scholar?q=Haochun+Wang%2C+Sendong+Zhao%2C+Zewen+Qiang%2C+Bing+Qin%2C+and+Ting+Liu%2E+2024b%2E+Beyond+the+answers%3A+Reviewing+the+rationality+of+multiple+choice+question+answering+for+the+evaluation+of+large+language+models%2E)
66. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In ICLR.
67. Boyi Wei, Weijia Shi, Yangsibo Huang, Noah A Smith, Chiyuan Zhang, Luke Zettlemoyer, Kai Li, and Peter Henderson. 2024. Evaluating copyright takedown methods for language models. In NeurIPS Datasets and Benchmark. [find](https://scholar.google.com/scholar?q=Boyi+Wei%2C+Weijia+Shi%2C+Yangsibo+Huang%2C+Noah+A+Smith%2C+Chiyuan+Zhang%2C+Luke+Zettlemoyer%2C+Kai+Li%2C+and+Peter+Henderson%2E+2024%2E+Evaluating+copyright+takedown+methods+for+language+models%2E+In+NeurIPS+Datasets+a)
68. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. [Large language models are better reasoners with self-verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/). In EMNLP.
69. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2024. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In MAACL, pages 1819–1862. [find](https://scholar.google.com/scholar?q=Zhaofeng+Wu%2C+Linlu+Qiu%2C+Alexis+Ross%2C+Ekin+Aky%C3%BCrek%2C+Boyuan+Chen%2C+Bailin+Wang%2C+Najoung+Kim%2C+Jacob+Andreas%2C+and+Yoon+Kim%2E+2024%2E+Reasoning+or+reciting%3F+exploring+the+capabilities+and+limitations+of+langua)
70. Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. Benchmarking benchmark leakage in large language models. arXiv:2404.18824. [find](https://scholar.google.com/scholar?q=Ruijie+Xu%2C+Zengzhi+Wang%2C+Run%2DZe+Fan%2C+and+Pengfei+Liu%2E+2024%2E+Benchmarking+benchmark+leakage+in+large+language+models%2E+arXiv%3A2404%2E18824%2E)
71. Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. 2023. [Rethinking benchmark and contamination for language models with rephrased samples](https://md.research.elamur.ai/free/en/papers/yang-2023-rethinking-benchmark-contamination-language-models/). arXiv:2311.04850.
72. Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, and Jingbo Shang. 2024. Data contamination can cross language barriers. arXiv:2406.13236. [find](https://scholar.google.com/scholar?q=Feng+Yao%2C+Yufan+Zhuang%2C+Zihao+Sun%2C+Sunan+Xu%2C+Animesh+Kumar%2C+and+Jingbo+Shang%2E+2024%2E+Data+contamination+can+cross+language+barriers%2E+arXiv%3A2406%2E13236%2E)
73. Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. 2024. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv:2407.20311. [find](https://scholar.google.com/scholar?q=Tian+Ye%2C+Zicheng+Xu%2C+Yuanzhi+Li%2C+and+Zeyuan+Allen%2DZhu%2E+2024%2E+Physics+of+language+models%3A+Part+2%2E1%2C+grade%2Dschool+math+and+the+hidden+reasoning+process%2E+arXiv%3A2407%2E20311%2E)
74. Zebra Puzzle. Zebra puzzle — Wikipedia, the free encyclopedia. [Online; accessed 18-May-2025]. [find](https://scholar.google.com/scholar?q=Zebra+Puzzle%2E+Zebra+puzzle+%E2%80%94+Wikipedia%2C+the+free+encyclopedia%2E+%5BOnline%3B+accessed+18%2DMay%2D2025%5D%2E)
75. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). NeurIPS, 35:15476–15488.
76. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, and 1 others. 2024. A careful examination of large language model performance on grade school arithmetic. arXiv:2405.00332. [find](https://scholar.google.com/scholar?q=Hugh+Zhang%2C+Jeff+Da%2C+Dean+Lee%2C+Vaughn+Robinson%2C+Catherine+Wu%2C+Will+Song%2C+Tiffany+Zhao%2C+Pranav+Raja%2C+Dylan+Slack%2C+Qin+Lyu%2C+and+1+others%2E+2024%2E+A+careful+examination+of+large+language+model+performance+)
77. Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In ICML.
78. Wenjie Zhou, Qiang Wang, Mingzhou Xu, Ming Chen, and Xiangyu Duan. 2024. Revisiting the self-consistency challenges in multi-choice question formats for large language model evaluation. In LREC-COLING. [find](https://scholar.google.com/scholar?q=Wenjie+Zhou%2C+Qiang+Wang%2C+Mingzhou+Xu%2C+Ming+Chen%2C+and+Xiangyu+Duan%2E+2024%2E+Revisiting+the+self%2Dconsistency+challenges+in+multi%2Dchoice+question+formats+for+large+language+model+evaluation%2E+In+LREC%2DCOLING)
79. Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2024. Dyval: Graph-informed dynamic evaluation of large language models. In ICLR. [find](https://scholar.google.com/scholar?q=Kaijie+Zhu%2C+Jiaao+Chen%2C+Jindong+Wang%2C+Neil+Zhenqiang+Gong%2C+Diyi+Yang%2C+and+Xing+Xie%2E+2024%2E+Dyval%3A+Graph%2Dinformed+dynamic+evaluation+of+large+language+models%2E+In+ICLR%2E)
80. Yongshuo Zong, Tingyang Yu, Bingchen Zhao, Ruchika Chavhan, and Timothy Hospedales. 2024. Fool your (vision and) language model with embarrassingly simple permutations. ICML. [find](https://scholar.google.com/scholar?q=Yongshuo+Zong%2C+Tingyang+Yu%2C+Bingchen+Zhao%2C+Ruchika+Chavhan%2C+and+Timothy+Hospedales%2E+2024%2E+Fool+your+%28vision+and%29+language+model+with+embarrassingly+simple+permutations%2E+ICML%2E)

*Updated 30 October 2026*