---
title: "Alice in Wonderland — paper by Nezhurina et al. (2024)"
description: "They propose template-based, falsification-oriented benchmarks that generate fresh problem… 94 references, cited by 28 papers here."
authors: ["Marianna Nezhurina", "Lucia Cipolina-Kun", "Mehdi Cherti", "Jenia Jitsev"]
year: 2024
venue: "arXiv:2406.02061"
source: "https://arxiv.org/abs/2406.02061"
doi: "10.48550/arxiv.2406.02061"
arxiv: "2406.02061"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nezhurina-2024-alice-wonderland-simple-tasks-showing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nezhurina-2024-alice-wonderland-simple-tasks-showing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models

# Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models

**Authors:** [Marianna Nezhurina](https://md.research.elamur.ai/free/en/authors/marianna-nezhurina/), Lucia Cipolina-Kun, [Mehdi Cherti](https://md.research.elamur.ai/free/en/authors/mehdi-cherti/) and [Jenia Jitsev](https://md.research.elamur.ai/free/en/authors/jenia-jitsev/) · **Year:** 2024 · **Venue:** arXiv:2406.02061 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.02061) · [PDF](https://arxiv.org/pdf/2406.02061) · [DOI 10.48550/arxiv.2406.02061](https://doi.org/10.48550/arxiv.2406.02061) · [arXiv 2406.02061](https://arxiv.org/abs/2406.02061) · [OpenAlex](https://openalex.org/W4399401573)

**Citations:** 20 citations (OpenAlex, 2026-10-14) · [28](https://md.research.elamur.ai/free/en/papers/nezhurina-2024-alice-wonderland-simple-tasks-showing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors ask whether high scores on standardized benchmarks (MMLU, GSM8k, MATH, etc.) really show that LLMs generalize and reason robustly. To test this, they use a tiny family puzzle, the AIW problem: Alice has N brothers and M sisters; how many sisters does her brother have? The answer is M + 1. By plugging different small numbers into this template, they create versions that are equally easy and differ only in surface detail. They then measure how often each model answers correctly over many repeated trials and prompt styles.

Every tested model, including GPT-4 and Claude 3 Opus, does poorly on average. Accuracy also swings sharply between versions that should be equally hard. Control tasks (AIW Light) show the models can parse the text, handle family relations and do the arithmetic, so the failures are not due to these basic skills. Models often defend wrong answers with confident, plausible-sounding explanations, and they rarely fix them when asked to recheck. The authors argue that current benchmarks miss such weaknesses. They propose template-based, falsification-oriented benchmarks that generate fresh problem instances and are harder to leak into training data.

## Key points
- **AIW template:** N and M are natural numbers ≤ 7, giving variations 1–4. For example, 3 brothers and 6 sisters gives the answer 7.
- **GPT-4 example:** gpt-4-0613 was run 60 times per variation with STANDARD, THINKING and RESTRICTED prompts. Accuracy was near 0 on variation 3 and near 1 on variation 4, regardless of prompt type.
- **Model scale:** Larger models (GPT-4/4o, Claude 3 Opus/3.5 Sonnet, Qwen 2.5 72B, Llama 3.1 405B) score clearly above zero, while smaller models stay close to zero. Reasoning models such as DeepSeek-R1 and o1-mini also show unstable accuracy across variations.
- **Failed fixes:** Chain-of-thought prompting and multi-turn self-verification do not repair the failures. Majority voting cannot help either, because the most frequent answers on hard variations are wrong.
- **Authors and resources:** The authors are from LAION and the Juelich Supercomputing Center, among others. Code and raw experiment data are published in the AIW repo.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti and Jenia Jitsev. 2024. Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models. arXiv:2406.02061.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nezhurina2024alice,
  title = {{Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models}},
  author = {Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti and Jenia Jitsev},
  year = {2024},
  doi = {10.48550/arxiv.2406.02061},
  eprint = {2406.02061},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.02061}
}
```

- [← previous Word Embeddings Are Steers for Language Models](https://md.research.elamur.ai/free/en/papers/han-2024-word-embeddings-are-steers-language/)
- [next → Attribute First, then Generate: Locally-attributable Grounded Text Generation](https://md.research.elamur.ai/free/en/papers/slobodkin-2024-attribute-first-then-generate-locally/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

94 references, 42 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International Conference on Machine Learning, pages 8748–8763. PMLR, 20…
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
3. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67, 2020. [https://discord.gg/BZqhreFazY](https://discord.gg/BZqhreFazY) http… [↗](https://discord.gg/BZqhreFazY)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
5. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
6. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonya… [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann%2C+Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Elena+Buchatskaya%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Diego+de+las+Casas%2C+Lisa+Anne+Hendricks%2C+Johannes+Welbl%2C+Aidan+Clark%2C+Tom+Hennigan%2C+Eric+Noland%2C+Ka)
7. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199–22213, 2022.
8. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
9. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
10. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
11. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
12. Melanie Mitchell. How do we know how smart ai systems are? Science, 381(6654):eadj5957, 2023. [find](https://scholar.google.com/scholar?q=Melanie+Mitchell%2E+How+do+we+know+how+smart+ai+systems+are%3F+Science%2C+381%286654%29%3Aeadj5957%2C+2023%2E)
13. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv e-prints, pages arXiv–2307, 2023. [find](https://scholar.google.com/scholar?q=Zhaofeng+Wu%2C+Linlu+Qiu%2C+Alexis+Ross%2C+Ekin+Aky%C3%BCrek%2C+Boyuan+Chen%2C+Bailin+Wang%2C+Najoung+Kim%2C+Jacob+Andreas%2C+and+Yoon+Kim%2E+Reasoning+or+reciting%3F+exploring+the+capabilities+and+limitations+of+language+mod)
14. Shahriar Golchin and Mihai Surdeanu. [Time travel in llms: Tracing data contamination in large language models](https://md.research.elamur.ai/free/en/papers/golchin-2023-time-travel-llms-tracing-data/). arXiv preprint arXiv:2308.08493, 2023.
15. Changmao Li and Jeffrey Flanigan. [Task contamination: Language models may not be few-shot anymore](https://md.research.elamur.ai/free/en/papers/li-2023-task-contamination-language-models-may/). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18471–18480, 2024.
16. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. [Mathematical capabilities of chatgpt](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/). Advances in Neural Information Processing Systems, 36, 2024.
17. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+et+al%2E+Language+models+%28mostly%29+know+what+th)
18. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, 2023.
19. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). In The Eleventh International Conference on Learning Representations, 202…
20. Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. [Self-contrast: Better reflection through inconsistent solving perspectives](https://md.research.elamur.ai/free/en/papers/zhang-2024-self-contrast-better-reflection-through/). arXiv preprint arXiv:2401.02009, 2024.
21. Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. [Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies](https://md.research.elamur.ai/free/en/papers/pan-2024-automatically-correcting-large-language-models/). Transactions of the Association for Computational Linguistics, 12:484–506, 2024.
22. OpenAI. Gpt-4 turbo and gpt-4 model docs. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+turbo+and+gpt%2D4+model+docs%2E)
23. Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2C+2024%2E)
24. Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. [Ask me anything: A simple strategy for prompting language models](https://md.research.elamur.ai/free/en/papers/arora-2022-ask-me-anything-simple-strategy/). In The Eleventh International Conference on Learning Representations, 2022.
25. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022.
26. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. [A prompt pattern catalog to enhance prompt engineering with chatgpt](https://md.research.elamur.ai/free/en/papers/white-2023-prompt-pattern-catalog-enhance-prompt/). arXiv preprint arXiv:2302.11382, 2023.
27. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. vllm, 2024. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+E%2E+Gonzalez%2C+Hao+Zhang%2C+and+Ion+Stoica%2E+vllm%2C+2024%2E)
28. Lewis Carroll. Through the Looking Glass. Penguin Books, 1871. [find](https://scholar.google.com/scholar?q=Lewis+Carroll%2E+Through+the+Looking+Glass%2E+Penguin+Books%2C+1871%2E)
29. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025.
30. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). arXiv preprint arXiv:2501.19393, 2025.
31. OpenThoughts Team. OpenThinker 32B. [https://huggingface.co/open-thoughts/OpenThinker-32B](https://huggingface.co/open-thoughts/OpenThinker-32B), January 2025. [↗](https://huggingface.co/open-thoughts/OpenThinker-32B)
32. OpenThoughts Team. Open Thoughts. [https://open-thoughts.ai](https://open-thoughts.ai), January 2025. [↗](https://open-thoughts.ai)
33. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. [Limo: Less is more for reasoning](https://md.research.elamur.ai/free/en/papers/ye-2025-limo-less-more-reasoning/). arXiv preprint arXiv:2502.03387, 2025.
34. Art of Problem Solving. American invitational mathematics examination (AIME). [find](https://scholar.google.com/scholar?q=Art+of+Problem+Solving%2E+American+invitational+mathematics+examination+%28AIME%29%2E)
35. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
36. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling, 2024.
37. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457, 2018.
38. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.
39. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
40. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computati)
41. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations, 2020.
42. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WINOGRANDE: an adversarial winograd schema challenge at scale, 2019. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+WINOGRANDE%3A+an+adversarial+winograd+schema+challenge+at+scale%2C+2019%2E)
43. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Advances+in+Neural+Informati)
44. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625, 2022.
45. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, 2022.
46. Jacob Pfau, William Merrill, and Samuel R Bowman. Let’s think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404.15758, 2024. [find](https://scholar.google.com/scholar?q=Jacob+Pfau%2C+William+Merrill%2C+and+Samuel+R+Bowman%2E+Let%E2%80%99s+think+dot+by+dot%3A+Hidden+computation+in+transformer+language+models%2E+arXiv+preprint+arXiv%3A2404%2E15758%2C+2024%2E)
47. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023. — [Let's Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
48. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hu… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
49. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Al… — [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
50. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). In The Thirteenth International Conference on Learning Representations, 20…
51. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. [Faith and fate: Limits of transformers on compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/). Advances in Neural Information Processing Systems, 36, 2024.
52. Martha Lewis and Melanie Mitchell. Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXiv preprint arXiv:2402.08955, 2024. — [arXiv:2402.08955](https://arxiv.org/abs/2402.08955) [find](https://scholar.google.com/scholar?q=Martha+Lewis+and+Melanie+Mitchell%2E+Using+counterfactual+tasks+to+evaluate+the+generality+of+analogical+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E08955%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E08955%5D%28h)
53. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv preprint arXiv:2309.12288, 2023. [find](https://scholar.google.com/scholar?q=Lukas+Berglund%2C+Meg+Tong%2C+Max+Kaufmann%2C+Mikita+Balesni%2C+Asa+Cooper+Stickland%2C+Tomasz+Korbak%2C+and+Owain+Evans%2E+The+reversal+curse%3A+Llms+trained+on%22+a+is+b%22+fail+to+learn%22+b+is+a%22%2E+arXiv+preprint+arXiv%3A)
54. Arsenii Kirillovich Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain. Transactions on machine learning research, 2023. [find](https://scholar.google.com/scholar?q=Arsenii+Kirillovich+Moskvichev%2C+Victor+Vikram+Odouard%2C+and+Melanie+Mitchell%2E+The+conceptarc+benchmark%3A+Evaluating+understanding+and+generalization+in+the+arc+domain%2E+Transactions+on+machine+learning+r)
55. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
56. Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao, and Timothy Hospedales. [Fool your (vision and) language model with embarrassingly simple permutations](https://md.research.elamur.ai/free/en/papers/zong-2023-fool-vision-language-model-embarrassingly/). In Forty-first International Conference on Machine Learning, 2024.
57. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2)
58. Zhijing Jin, Jiarui Liu, LYU Zhiheng, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona T Diab, and Bernhard Schölkopf. Can large language models infer causation from correlation? In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Jiarui+Liu%2C+LYU+Zhiheng%2C+Spencer+Poff%2C+Mrinmaya+Sachan%2C+Rada+Mihalcea%2C+Mona+T+Diab%2C+and+Bernhard+Sch%C3%B6lkopf%2E+Can+large+language+models+infer+causation+from+correlation%3F+In+The+Twelfth+Inte)
59. Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng LYU, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, and Bernhard Schölkopf. CLadder: A benchmark to assess causal reasoning capabilities of language models. In Thirty-seventh Conference on Neural Info… [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Yuen+Chen%2C+Felix+Leeb%2C+Luigi+Gresele%2C+Ojasv+Kamal%2C+Zhiheng+LYU%2C+Kevin+Blin%2C+Fernando+Gonzalez+Adauto%2C+Max+Kleiman%2DWeiner%2C+Mrinmaya+Sachan%2C+and+Bernhard+Sch%C3%B6lkopf%2E+CLadder%3A+A+benchmark+to+)
60. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. [Are nlp models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, 2021.
61. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. [Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models](https://md.research.elamur.ai/free/en/papers/mirzadeh-2024-gsm-symbolic-understanding-limitations-mathematical/). arXiv preprint arXiv:2410.05229, 2024.
62. Ricardo Dominguez-Olmedo, Florian E Dorner, and Moritz Hardt. Training on the test task confounds evaluation and emergence. arXiv preprint arXiv:2407.07890, 2024. — [arXiv:2407.07890](https://arxiv.org/abs/2407.07890) [find](https://scholar.google.com/scholar?q=Ricardo+Dominguez%2DOlmedo%2C+Florian+E+Dorner%2C+and+Moritz+Hardt%2E+Training+on+the+test+task+confounds+evaluation+and+emergence%2E+arXiv+preprint+arXiv%3A2407%2E07890%2C+2024%2E+%E2%80%94+%5BarXiv%3A2407%2E07890%5D%28https%3A%2F%2Farxiv%2Eor)
63. HuggingFace. Fineweb: decanting the web for the finest text data at scale. [find](https://scholar.google.com/scholar?q=HuggingFace%2E+Fineweb%3A+decanting+the+web+for+the+finest+text+data+at+scale%2E)
64. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. [Datacomp-lm: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/). Advances in Neural Information Processing Systems, 37:14200–142…
65. Cohere. Introducing command r+: A scalable llm built for business. [find](https://scholar.google.com/scholar?q=Cohere%2E+Introducing+command+r%2B%3A+A+scalable+llm+built+for+business%2E)
66. Cohere. Command r+ documentation, 2024. [find](https://scholar.google.com/scholar?q=Cohere%2E+Command+r%2B+documentation%2C+2024%2E)
67. Mosaic. Introducing dbrx: A new state-of-the-art open llm. [find](https://scholar.google.com/scholar?q=Mosaic%2E+Introducing+dbrx%3A+A+new+state%2Dof%2Dthe%2Dart+open+llm%2E)
68. Karl Raimund Popper. The logic of scientific discovery. 1934. [find](https://scholar.google.com/scholar?q=Karl+Raimund+Popper%2E+The+logic+of+scientific+discovery%2E+1934%2E)
69. OpenAI. Gpt-4o model docs., 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4o+model+docs%2E%2C+2024%2E)
70. OpenAI. Announcement: Hello gpt-4o, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Announcement%3A+Hello+gpt%2D4o%2C+2024%2E)
71. OpenAI. Introducing chatgpt, 11 2022. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Introducing+chatgpt%2C+11+2022%2E)
72. OpenAI. Models - openai gpt 3.5 turbo docs. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Models+%2D+openai+gpt+3%2E5+turbo+docs%2E)
73. OpenAI. Models - openai gpt 3.5 turbo update. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Models+%2D+openai+gpt+3%2E5+turbo+update%2E)
74. Anthropic. Claude 3.5 sonnet, 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Claude+3%2E5+sonnet%2C+2024%2E)
75. Anthropic. Introducing the next generation of claude, 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Introducing+the+next+generation+of+claude%2C+2024%2E)
76. Sundar Pichai and Demis Hassabis. Introducing gemini: Google’s most capable ai model yet, 2023. [find](https://scholar.google.com/scholar?q=Sundar+Pichai+and+Demis+Hassabis%2E+Introducing+gemini%3A+Google%E2%80%99s+most+capable+ai+model+yet%2C+2023%2E)
77. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
78. Sundar Pichai and Demis Hassabis. Our next-generation model: Gemini 1.5, 2024. [find](https://scholar.google.com/scholar?q=Sundar+Pichai+and+Demis+Hassabis%2E+Our+next%2Dgeneration+model%3A+Gemini+1%2E5%2C+2024%2E)
79. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530,…
80. Google. Gemma: Introducing new state-of-the-art open models, 2024. [find](https://scholar.google.com/scholar?q=Google%2E+Gemma%3A+Introducing+new+state%2Dof%2Dthe%2Dart+open+models%2C+2024%2E)
81. Google. Gemma model card, 2024. [find](https://scholar.google.com/scholar?q=Google%2E+Gemma+model+card%2C+2024%2E)
82. Mistral-AI-Team. Mistral ai models api versioning, 2024. [find](https://scholar.google.com/scholar?q=Mistral%2DAI%2DTeam%2E+Mistral+ai+models+api+versioning%2C+2024%2E)
83. Mistral-AI-Team. Au large \| mistral ai \| frontier ai in your hands, 2024. [find](https://scholar.google.com/scholar?q=Mistral%2DAI%2DTeam%2E+Au+large+%5C%7C+mistral+ai+%5C%7C+frontier+ai+in+your+hands%2C+2024%2E)
84. Mistral-AI-Team. Announcement mixtral ai 7x22b, 2024. [find](https://scholar.google.com/scholar?q=Mistral%2DAI%2DTeam%2E+Announcement+mixtral+ai+7x22b%2C+2024%2E)
85. Mistral-AI-Team. Announcement mixtral ai 7x8b, 2024. [find](https://scholar.google.com/scholar?q=Mistral%2DAI%2DTeam%2E+Announcement+mixtral+ai+7x8b%2C+2024%2E)
86. Mistral-AI-Team. Announcement mistral ai 7b, 2023. [find](https://scholar.google.com/scholar?q=Mistral%2DAI%2DTeam%2E+Announcement+mistral+ai+7b%2C+2023%2E)
87. Cohere. Model card for c4ai command r+, 2024. [find](https://scholar.google.com/scholar?q=Cohere%2E+Model+card+for+c4ai+command+r%2B%2C+2024%2E)
88. Meta. Meta and microsoft introduce the next generation of llama \| meta, 2023. [find](https://scholar.google.com/scholar?q=Meta%2E+Meta+and+microsoft+introduce+the+next+generation+of+llama+%5C%7C+meta%2C+2023%2E)
89. Meta. Introducing meta llama 3: The most capable openly available llm to date, 2024. [find](https://scholar.google.com/scholar?q=Meta%2E+Introducing+meta+llama+3%3A+The+most+capable+openly+available+llm+to+date%2C+2024%2E)
90. Meta. Meta llama 3 model card., 2024. [find](https://scholar.google.com/scholar?q=Meta%2E+Meta+llama+3+model+card%2E%2C+2024%2E)
91. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609, 2023.
92. Alibaba. Introducing qwen1.5, 2024. [find](https://scholar.google.com/scholar?q=Alibaba%2E+Introducing+qwen1%2E5%2C+2024%2E)
93. Alibaba. Hello qwen2, 2024. [find](https://scholar.google.com/scholar?q=Alibaba%2E+Hello+qwen2%2C+2024%2E)
94. TogetherAI. Togetherai api, 2024. [find](https://scholar.google.com/scholar?q=TogetherAI%2E+Togetherai+api%2C+2024%2E)

*Updated 3 November 2026*