---
title: "Auxiliary task demands mask the capabilities of smaller language"
description: "The authors argue that scores, and claims about when abilities emerge, should be read as products… 53 references, cited by 34 papers here."
authors: ["Jennifer Hu", "Michael C. Frank"]
year: 2024
venue: "arXiv:2404.02418"
source: "https://arxiv.org/abs/2404.02418"
doi: "10.48550/arxiv.2404.02418"
arxiv: "2404.02418"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hu-2024-auxiliary-task-demands-mask-capabilities/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hu-2024-auxiliary-task-demands-mask-capabilities/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Auxiliary task demands mask the capabilities of smaller language models

# Auxiliary task demands mask the capabilities of smaller language models

**Authors:** [Jennifer Hu](https://md.research.elamur.ai/free/en/authors/jennifer-hu/) and [Michael C. Frank](https://md.research.elamur.ai/free/en/authors/michael-c-frank/) · **Year:** 2024 · **Venue:** arXiv:2404.02418 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.02418) · [PDF](https://arxiv.org/pdf/2404.02418) · [DOI 10.48550/arxiv.2404.02418](https://doi.org/10.48550/arxiv.2404.02418) · [arXiv 2404.02418](https://arxiv.org/abs/2404.02418) · [OpenAlex](https://openalex.org/W4393968405)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/hu-2024-auxiliary-task-demands-mask-capabilities/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Developmental psychologists know that children can fail a test because of its extra requirements, such as understanding the question, even when they have the targeted skill. Hu and Frank ask whether these auxiliary "task demands" also hide what language models know, and whether weaker models are hit harder. They compare paired evaluations of the same ability. One pair sets free generation of an answer against selection from fixed options. The other sets an explicit metalinguistic prompt against a direct readout of string probabilities. These pairs span four domains, namely judging grammaticality, predicting words, and reasoning both by analogy and by reflection. The models are open-source families released in several sizes or with intermediate training checkpoints. In every domain, the more demanding format produces lower scores. This "demand gap" narrows as models gain parameters and training, so small models look worse than their underlying knowledge warrants. The authors argue that scores, and claims about when abilities emerge, should be read as products of evaluation design rather than direct measures of a model's intelligence.

## Key points
- The production vs. forced choice contrast is applied to analogical reasoning (digit-sequence puzzles from Webb et al., 2023) and reflective reasoning (intuition-trap problems from Hagendorff et al., 2023).
- The metalinguistic vs. probability contrast is applied to word prediction (LAMBADA) and grammaticality judgment (minimal pairs from BLiMP, Dentella et al., 2023a and Hu et al., 2024).
- Checkpoint analyses of OLMo 7B on analogical reasoning and word prediction show the demand gap shrinking over training. The other two domains were not analyzed this way because the fully trained OLMo 7B still performed poorly on their high-demand versions.
- All model families except Pythia showed qualitatively similar patterns. The authors speculate that Pythia differs because it was trained on roughly an order of magnitude less data.
- An evaluation counts as low-demand when it matches the training objective (token probabilities) or restricts the answer space (forced choice). All models are available via Huggingface Transformers, and the code and data are at github.com/jennhu/lm-task-demands.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 499 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Jennifer Hu and Michael C. Frank. 2024. Auxiliary task demands mask the capabilities of smaller language models. arXiv:2404.02418.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hu2024auxiliary,
  title = {{Auxiliary task demands mask the capabilities of smaller language models}},
  author = {Jennifer Hu and Michael C. Frank},
  year = {2024},
  doi = {10.48550/arxiv.2404.02418},
  eprint = {2404.02418},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.02418}
}
```

- [← previous Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models](https://md.research.elamur.ai/free/en/papers/ormazabal-2024-reka-core-flash-edge-series/)
- [next → Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation](https://md.research.elamur.ai/free/en/papers/chhun-2024-do-language-models-enjoy-their/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

53 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Elika Bergelson and Daniel Swingley. At 6–9 months, human infants know the meanings of many common nouns. Proceedings of the National Academy of Sciences, 109(9):3253–3258, February 2012. doi: 10.1073/pnas.1113380109. URL https://doi.org/10.1073/pnas. 1113380109. — [doi:10.1073/pnas.1113380109](https://doi.org/10.1073/pnas.1113380109) [↗](https://doi.org/10.1073/pnas)
2. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A Suite for Analyzing Large Language Models Across Training and… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+Aviya+Skowron%2C+Lintang+Sutaw)
3. Samuel Bowman. The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7484–7499, Dublin, Ireland, May 2022. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Samuel+Bowman%2E+The+Dangers+of+Underclaiming%3A+Reasons+for+Caution+When+Reporting+How+NLP+Systems+Fail%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume)
4. Samuel R. Bowman and George Dahl. What Will it Take to Fix Benchmarking in Natural Language Understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4843–4855, Online, June 2021. Association… [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman+and+George+Dahl%2E+What+Will+it+Take+to+Fix+Benchmarking+in+Natural+Language+Understanding%3F+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Co)
5. Peter Carruthers. Mindreading in infancy. Mind & Language, 28(2):141–172, 2013. [find](https://scholar.google.com/scholar?q=Peter+Carruthers%2E+Mindreading+in+infancy%2E+Mind+%26+Language%2C+28%282%29%3A141%E2%80%93172%2C+2013%2E)
6. Tyler A. Chang and Benjamin K. Bergen. [Language Model Behavior: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/chang-2024-language-model-behavior-comprehensive-survey/). Computational Linguistics, 50(1):293–350, March 2024. ISSN 0891-2017. doi: 10.1162/coli_a_00492. URL https://doi.org/10.1162/coli_a_00492. [↗](https://doi.org/10.1162/coli_a_00492)
7. Lee J Cronbach and Paul E Meehl. Construct validity in psychological tests. Psychological Bulletin, 52(4):281, 1955. [find](https://scholar.google.com/scholar?q=Lee+J+Cronbach+and+Paul+E+Meehl%2E+Construct+validity+in+psychological+tests%2E+Psychological+Bulletin%2C+52%284%29%3A281%2C+1955%2E)
8. Vittoria Dentella, Fritz Günther, and Evelina Leivada. Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias. Proceedings of the National Academy of Sciences, 120(51):e2309583120, December 2023a. doi: 10.1073/pnas.2309583120… — [doi:10.1073/pnas.2309583120](https://doi.org/10.1073/pnas.2309583120) [find](https://scholar.google.com/scholar?q=Vittoria+Dentella%2C+Fritz+G%C3%BCnther%2C+and+Evelina+Leivada%2E+Systematic+testing+of+three+Language+Models+reveals+low+language+accuracy%2C+absence+of+response+stability%2C+and+a+yes%2Dresponse+bias%2E+Proceedings+of)
9. Vittoria Dentella, Elliot Murphy, Gary Marcus, and Evelina Leivada. Testing AI performance on less frequent aspects of language reveals insensitivity to underlying meaning, 2023b. URL https://arxiv.org/abs/2302.12313. — [arXiv:2302.12313](https://arxiv.org/abs/2302.12313) [↗](https://arxiv.org/abs/2302.12313)
10. Chaz Firestone. Performance vs. competence in human–machine comparisons. Proceedings of the National Academy of Sciences, 117(43):26562–26571, October 2020. URL https://doi.org/10.1073/pnas.1905334117. Publisher: Proceedings of the National Academy of Sciences. — [doi:10.1073/pnas.1905334117](https://doi.org/10.1073/pnas.1905334117) [↗](https://doi.org/10.1073/pnas.1905334117)
11. Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451–452, 2023a. [find](https://scholar.google.com/scholar?q=Michael+C+Frank%2E+Baby+steps+in+evaluating+the+capacities+of+large+language+models%2E+Nature+Reviews+Psychology%2C+2%288%29%3A451%E2%80%93452%2C+2023a%2E)
12. Michael C Frank. Openly accessible llms can help us to understand human cognition. Nature Human Behaviour, 7(11):1825–1827, 2023b. [find](https://scholar.google.com/scholar?q=Michael+C+Frank%2E+Openly+accessible+llms+can+help+us+to+understand+human+cognition%2E+Nature+Human+Behaviour%2C+7%2811%29%3A1825%E2%80%931827%2C+2023b%2E)
13. Shane Frederick. Cognitive Reflection and Decision Making. Journal of Economic Perspectives, 19(4):25–42, 2005. doi: 10.1257/089533005775196732. URL [https://www.aeaweb.org/](https://www.aeaweb.org/) articles?id=10.1257/089533005775196732. — [doi:10.1257/089533005775196732](https://doi.org/10.1257/089533005775196732) [↗](https://www.aeaweb.org/)
14. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, A… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+Pouya+Tafti%2C+L%C3%A9onard+Hussenot%2C+Aakanksha)
15. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, W… [find](https://scholar.google.com/scholar?q=Dirk+Groeneveld%2C+Iz+Beltagy%2C+Pete+Walsh%2C+Akshita+Bhagia%2C+Rodney+Kinney%2C+Oyvind+Tafjord%2C+Ananya+Harsh+Jha%2C+Hamish+Ivison%2C+Ian+Magnusson%2C+Yizhong+Wang%2C+Shane+Arora%2C+David+Atkinson%2C+Russell+Authur%2C+Khyat)
16. Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in Chat-GPT. Nature Computational Science, 3(10):833–838, October 2023. ISSN 2662-8457. doi: 10.1038/s43588-023-00527-x. URL https://doi.org/10.1038/… — [doi:10.1038/s43588-023-00527-x](https://doi.org/10.1038/s43588-023-00527-x) [find](https://scholar.google.com/scholar?q=Thilo+Hagendorff%2C+Sarah+Fabi%2C+and+Michal+Kosinski%2E+Human%2Dlike+intuitive+behavior+and+reasoning+biases+emerged+in+large+language+models+but+disappeared+in+Chat%2DGPT%2E+Nature+Computational+Science%2C+3%2810%29%3A)
17. Petra Hendriks and Charlotte Koster. Production/comprehension asymmetries in language acquisition. Asymmetries in Language Acquisition, 120(8):1887–1897, August 2010. ISSN 0024-3841. doi: 10.1016/j.lingua.2010.02.002. URL [https://www.sciencedirect.com/](https://www.sciencedirect.com/) science/article/pii/S0024384110000215. — [doi:10.1016/j.lingua.2010.02.002](https://doi.org/10.1016/j.lingua.2010.02.002) [↗](https://www.sciencedirect.com/)
18. Jesse Hoogland, Alexander Gietelink Oldenziel, Daniel Murfet, and Stan van Wingerden. Towards Developmental Interpretability, 2023. URL [https://www.lesswrong.com/posts/](https://www.lesswrong.com/posts/) TjaeCWvLZtEDAS5Ex/towards-developmental-interpretability. [↗](https://www.lesswrong.com/posts/)
19. Jennifer Hu and Roger Levy. [Prompting is not a substitute for probability measurements in large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/). In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5040–5060, Singapore, December 2023. Assoc…
20. Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. Language models align with human judgments on key grammatical constructions, 2024. URL https://arxiv.org/abs/2402.01676. — [arXiv:2402.01676](https://arxiv.org/abs/2402.01676) [↗](https://arxiv.org/abs/2402.01676)
21. Anna A. Ivanova. Running cognitive evaluations on large language models: The do’s and the don’ts, 2023. URL https://arxiv.org/abs/2312.01276. — [arXiv:2312.01276](https://arxiv.org/abs/2312.01276) [↗](https://arxiv.org/abs/2312.01276)
22. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
23. Roni Katzir. Why Large Language Models Are Poor Theories of Human Linguistic Cognition: A Reply to Piantadosi. Biolinguistics, 17, 2023. URL https://doi.org/10.5964/bioling. 13153. — [doi:10.5964/bioling](https://doi.org/10.5964/bioling) [↗](https://doi.org/10.5964/bioling)
24. Carina Kauf, Emmanuele Chersoni, Alessandro Lenci, Evelina Fedorenko, and Anna A. Ivanova. Comparing Plausibility Estimates in Base and Instruction-Tuned Large Language Models, 2024. [URL https://arxiv.org/abs/2403.14859](https://md.research.elamur.ai/free/en/papers/kauf-2024-log-probabilities-reliable-estimate-semantic/). [↗](https://arxiv.org/abs/2403.14859)
25. Rachel Keen. Representation of objects and events: Why do infants look so smart and toddlers look so dumb? Current Directions in Psychological Science, 12(3):79–83, 2003. [find](https://scholar.google.com/scholar?q=Rachel+Keen%2E+Representation+of+objects+and+events%3A+Why+do+infants+look+so+smart+and+toddlers+look+so+dumb%3F+Current+Directions+in+Psychological+Science%2C+12%283%29%3A79%E2%80%9383%2C+2003%2E)
26. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench:… [find](https://scholar.google.com/scholar?q=Douwe+Kiela%2C+Max+Bartolo%2C+Yixin+Nie%2C+Divyansh+Kaushik%2C+Atticus+Geiger%2C+Zhengxuan+Wu%2C+Bertie+Vidgen%2C+Grusha+Prasad%2C+Amanpreet+Singh%2C+Pratik+Ringshia%2C+Zhiyi+Ma%2C+Tristan+Thrush%2C+Sebastian+Riedel%2C+Zeerak+)
27. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large Language Models are Zero-Shot Reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 22199–22213. Curran Assoc…
28. Andrew K Lampinen, Ishita Dasgupta, Stephanie C Y Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models, like humans, show content effects on reasoning tasks. PNAS Nexus, 3(7):pgae233, July 2024. ISSN 2752-6542. doi: 10.1093/pnasnexus/pgae233… — [doi:10.1093/pnasnexus/pgae233](https://doi.org/10.1093/pnasnexus/pgae233) [find](https://scholar.google.com/scholar?q=Andrew+K+Lampinen%2C+Ishita+Dasgupta%2C+Stephanie+C+Y+Chan%2C+Hannah+R+Sheahan%2C+Antonia+Creswell%2C+Dharshan+Kumaran%2C+James+L+McClelland%2C+and+Felix+Hill%2E+Language+models%2C+like+humans%2C+show+content+effects+on+)
29. Andrew Kyle Lampinen. [Can language models handle recursively nested grammatical structures? A case study on comparing models and humans](https://md.research.elamur.ai/free/en/papers/lampinen-2024-can-language-models-handle-recursively/), 2023. URL https://arxiv.org/abs/2210.15303. [↗](https://arxiv.org/abs/2210.15303)
30. Tal Linzen. [How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://md.research.elamur.ai/free/en/papers/linzen-2020-how-can-we-accelerate-progress/) In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5210–5217, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.465. URL ht…
31. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). Transactions of the Association for Computational Linguistics, 4:521–535, 2016. doi: 10.1162/tacl_a_00115. URL https://aclanthology.org/Q16-1037. Place: Cambridge, MA Publisher: MIT… [↗](https://aclanthology.org/Q16-1037)
32. Chenyang Lyu, Minghao Wu, and Alham Fikri Aji. Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2402.13887. [↗](https://arxiv.org/)
33. Rebecca Marvin and Tal Linzen. Targeted Syntactic Evaluation of Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1192–1202, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1151. URL https:… [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+Targeted+Syntactic+Evaluation+of+Language+Models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pp%2E+1192%E2%80%931202%2C+Brussels%2C+Bel)
34. Kanishka Misra. minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models, 2022. URL https://arxiv.org/abs/2203.13112. [↗](https://arxiv.org/abs/2203.13112)
35. Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating Cognitive Maps and Planning in Large Language Models with CogEval. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL http… [find](https://scholar.google.com/scholar?q=Ida+Momennejad%2C+Hosein+Hasanbeig%2C+Felipe+Vieira+Frujeri%2C+Hiteshi+Sharma%2C+Nebojsa+Jojic%2C+Hamid+Palangi%2C+Robert+Ness%2C+and+Jonathan+Larson%2E+Evaluating+Cognitive+Maps+and+Planning+in+Large+Language+Models)
36. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885–4901, Online, July 2020. Association for Comp…
37. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annua…
38. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, 2022. URL https://arxiv.org/abs/2201.02177. [↗](https://arxiv.org/abs/2201.02177)
39. Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. AI and the Everything in the Whole Wide World Benchmark. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021. URL htt… [find](https://scholar.google.com/scholar?q=Deborah+Raji%2C+Emily+Denton%2C+Emily+M%2E+Bender%2C+Alex+Hanna%2C+and+Amandalynne+Paullada%2E+AI+and+the+Everything+in+the+Whole+Wide+World+Benchmark%2E+In+J%2E+Vanschoren+and+S%2E+Yeung+%28eds%2E%29%2C+Proceedings+of+the+Neu)
40. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, and Dirk Hovy. [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://md.research.elamur.ai/free/en/papers/rottger-2024-political-compass-or-spinning-arrow/), 2024. URL https://arxiv.org/abs/2402.16786. [↗](https://arxiv.org/abs/2402.16786)
41. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Commun. ACM, 64(9):99–106, August 2021. ISSN 0001-0782. doi: 10.1145/3474381. URL https://doi.org/10.1145/3474381. Place: New York, NY, USA Publisher: Association for… [↗](https://doi.org/10.1145/3474381)
42. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. [Are Emergent Abilities of Large Language Models a Mirage?](https://md.research.elamur.ai/free/en/papers/schaeffer-2023-emergent-abilities-large-language-models/) In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=ITw9edRDlD](https://openreview.net/forum?id=ITw9edRDlD). [↗](https://openreview.net/forum?id=ITw9edRDlD)
43. David Schlangen. Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research, 2019. URL https://arxiv.org/abs/1908.10747. — [arXiv:1908.10747](https://arxiv.org/abs/1908.10747) [↗](https://arxiv.org/abs/1908.10747)
44. James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht. On the Stepwise Nature of Self-Supervised Learning. In Proceedings of the 40th International Conference on Machine Learning, 2023. URL [https://proceedings.mlr.press/](https://proceedings.mlr.press/) v202/simon23a/simon23a.pdf. [↗](https://proceedings.mlr.press/)
45. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Al… — [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
46. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
47. Polina Tsvilodub, Hening Wang, Sharon Grosch, and Michael Franke. Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods, 2024. URL https://arxiv.org/abs/2403.00998. — [arXiv:2403.00998](https://arxiv.org/abs/2403.00998) [↗](https://arxiv.org/abs/2403.00998)
48. Esra Nur Turan-Küçük and Melissa M Kibbe. Three-year-olds’ ability to plan for mutually exclusive future possibilities is limited primarily by their representations of possible plans, not possible events. Cognition, 244:105712, 2024. [find](https://scholar.google.com/scholar?q=Esra+Nur+Turan%2DK%C3%BC%C3%A7%C3%BCk+and+Melissa+M+Kibbe%2E+Three%2Dyear%2Dolds%E2%80%99+ability+to+plan+for+mutually+exclusive+future+possibilities+is+limited+primarily+by+their+representations+of+possible+plans%2C+not+possible+eve)
49. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8, 2020. URL https://doi.org/ 10.1162/tacl_a_00321. Publisher: MIT P…
50. Taylor Webb, Keith J. Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526–1541, September 2023. ISSN 2397-3374. doi: 10.1038/s41562-023-01659-w. URL https://doi.org/10.1038/s41562-023-01659-w. [↗](https://doi.org/10.1038/s41562-023-01659-w)
51. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. [Emergent Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Transactions on Machine…
52. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Proc… — [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
53. Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. The Generative AI Paradox: “What It Can Create, It May Not Understand”. In The Twelfth Int… [find](https://scholar.google.com/scholar?q=Peter+West%2C+Ximing+Lu%2C+Nouha+Dziri%2C+Faeze+Brahman%2C+Linjie+Li%2C+Jena+D%2E+Hwang%2C+Liwei+Jiang%2C+Jillian+Fisher%2C+Abhilasha+Ravichander%2C+Khyathi+Chandu%2C+Benjamin+Newman%2C+Pang+Wei+Koh%2C+Allyson+Ettinger%2C+and+Ye)

*Updated 30 October 2026*