---
title: "Language Models Fail to Introspect About Their Knowledge of"
description: "This work tests whether large language models can introspect, meaning whether they can accurately… 48 references, cited by 7 papers here."
authors: ["Siyuan Song", "Jennifer Hu", "Kyle Mahowald"]
year: 2025
venue: "arXiv:2503.07513"
source: "https://arxiv.org/abs/2503.07513"
doi: "10.48550/arxiv.2503.07513"
arxiv: "2503.07513"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/song-2025-language-models-fail-introspect-about/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/song-2025-language-models-fail-introspect-about/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) / Language Models Fail to Introspect About Their Knowledge of Language

# Language Models Fail to Introspect About Their Knowledge of Language

**Authors:** [Siyuan Song](https://md.research.elamur.ai/free/en/authors/siyuan-song/), [Jennifer Hu](https://md.research.elamur.ai/free/en/authors/jennifer-hu/) and [Kyle Mahowald](https://md.research.elamur.ai/free/en/authors/kyle-mahowald/) · **Year:** 2025 · **Venue:** arXiv:2503.07513 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2503.07513) · [PDF](https://arxiv.org/pdf/2503.07513) · [DOI 10.48550/arxiv.2503.07513](https://doi.org/10.48550/arxiv.2503.07513) · [arXiv 2503.07513](https://arxiv.org/abs/2503.07513) · [OpenAlex](https://openalex.org/W4415100619)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/song-2025-language-models-fail-introspect-about/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This work tests whether large language models can introspect, meaning whether they can accurately report on their own linguistic knowledge without fine-tuning. The question matters for interpretability. It also matters for linguists who evaluate models with human-style questions, such as asking whether a sentence is grammatical.

The authors examined 21 open-source LLMs on two tasks: judging grammaticality in minimal sentence pairs (Exp. 1) and choosing the better next word after a prefix (Exp. 2). For each item, they computed two scores:
- a **Direct** score, from string log probabilities, treated as the ground truth of what the model knows;
- a **Meta** score, from the probabilities the model assigns to answering "1" or "2" to an explicit prompt.

Introspection was defined as a model's Meta scores tracking its own Direct scores better than those of a highly similar model, with similarity controlled in a regression.

Meta–Direct alignment grew as models became more similar, but there was no extra "same model" advantage. The authors conclude that prompted judgments carry real information but are separate from the knowledge that drives string probabilities, so they should not be used as a substitute for it.

## Key points
- Model similarity was defined in two ways. The feature-based version ranks pairs as self > seed variant > base/instruct > same family > other. The empirical version is the correlation between two models' Direct scores.
- Both methods reached high task accuracy. Direct beat Meta for smaller models, while larger models were near ceiling on both. Larger models' Meta scores also predicted probabilities better. Consistency between methods was measured with Cohen's κ.
- Exp. 2 used four datasets: wikipedia, news, nonsense and randomseq. Models were near ceiling on the first two and responded at chance on the last two, as expected.
- In the regression, Direct–Direct similarity had a strong effect (p \< .0001), but the self pairing showed no advantage. The same family and other pairings actually scored higher relative to self than expected. These results held when only large models were analysed.
- Code and data are released at github.com/SiyuanSong2004/language-introspection, and all models are available on Huggingface.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Siyuan Song, Jennifer Hu and Kyle Mahowald. 2025. Language Models Fail to Introspect About Their Knowledge of Language. arXiv:2503.07513.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{song2025language,
  title = {{Language Models Fail to Introspect About Their Knowledge of Language}},
  author = {Siyuan Song, Jennifer Hu and Kyle Mahowald},
  year = {2025},
  doi = {10.48550/arxiv.2503.07513},
  eprint = {2503.07513},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.07513}
}
```

- [← previous HunyuanOCR Technical Report](https://md.research.elamur.ai/free/en/papers/team-2025-vision-hunyuanocr-technical-report/)
- [next → Liger: Linearizing Large Language Models to Gated Recurrent Structures](https://md.research.elamur.ai/free/en/papers/lan-2025-liger-linearizing-large-language-models/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/)

## References

48 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI@Meta. Llama 3 Model Card. 2024. URL [https://github.com/meta-llama/llama3/blob/](https://github.com/meta-llama/llama3/blob/) main/MODEL_CARD.md. [↗](https://github.com/meta-llama/llama3/blob/)
2. Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. [Which of these best describes multiple choice evaluation with LLMs? a) forced B) flawed C) fixable D) all of the above](https://md.research.elamur.ai/free/en/papers/balepur-2025-which-these-best-describes-multiple/). In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual…
3. Christopher C. Berger, Tara C. Dennehy, John A. Bargh, and Ezequiel Morsella. Nisbett and Wilson (1977) Revisited: The Little That We Can Know and Can Tell. Social Cognition, 34(3):167–195, June 2016. ISSN 0278-016X. doi: 10.1521/soco.2016.34.3.167. URL https://doi.org/10.1521/soco.2016.34.3.167. P… — [doi:10.1521/soco.2016.34.3.167](https://doi.org/10.1521/soco.2016.34.3.167) [↗](https://doi.org/10.1521/soco.2016.34.3.167)
4. Jan Betley, Xuchan Bao, Martín Soto, Anna Sztyber-Betley, James Chua, and Owain Evans. Tell me about yourself: LLMs are aware of their learned behaviors. In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/](https://openreview.net/) forum?id=IjQ2Jtemzy. [↗](https://openreview.net/)
5. Felix Jedidja Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, and Owain Evans. Looking inward: Language models can learn about themselves by introspection. In The Thirteenth International Conference on Learning Representations, 2025. URL https://… [find](https://scholar.google.com/scholar?q=Felix+Jedidja+Binder%2C+James+Chua%2C+Tomek+Korbak%2C+Henry+Sleight%2C+John+Hughes%2C+Robert+Long%2C+Ethan+Perez%2C+Miles+Turpin%2C+and+Owain+Evans%2E+Looking+inward%3A+Language+models+can+learn+about+themselves+by+intro)
6. Edwin G Boring. A history of introspection. Psychological Bulletin, 50(3):169, 1953. [find](https://scholar.google.com/scholar?q=Edwin+G+Boring%2E+A+history+of+introspection%2E+Psychological+Bulletin%2C+50%283%29%3A169%2C+1953%2E)
7. Alex Byrne. Introspection. Philosophical Topics, 33(1):79–104, 2005. [find](https://scholar.google.com/scholar?q=Alex+Byrne%2E+Introspection%2E+Philosophical+Topics%2C+33%281%29%3A79%E2%80%93104%2C+2005%2E)
8. N. Chomsky. Syntactic Structures. The Hague: Mouton, 1957. [find](https://scholar.google.com/scholar?q=N%2E+Chomsky%2E+Syntactic+Structures%2E+The+Hague%3A+Mouton%2C+1957%2E)
9. Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. [A survey of the state of explainable ai for natural language processing](https://md.research.elamur.ai/free/en/papers/danilevsky-2020-survey-state-explainable-ai-natural/). In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th Internati…
10. Vittoria Dentella, Fritz Günther, and Evelina Leivada. Systematic testing of three language models reveals low language accuracy, absence of response stability, and a yes-response bias. Proceedings of the National Academy of Sciences, 120(51):e2309583120, 2023. [find](https://scholar.google.com/scholar?q=Vittoria+Dentella%2C+Fritz+G%C3%BCnther%2C+and+Evelina+Leivada%2E+Systematic+testing+of+three+language+models+reveals+low+language+accuracy%2C+absence+of+response+stability%2C+and+a+yes%2Dresponse+bias%2E+Proceedings+of)
11. Rudresh Dwivedi, Devam Dave, Het Naik, Smiti Singhal, Rana Omer, Pankesh Patel, Bin Qian, Zhenyu Wen, Tejal Shah, Graham Morgan, et al. Explainable ai (xai): Core ideas, techniques, and solutions. ACM computing surveys, 55(9):1–33, 2023. [find](https://scholar.google.com/scholar?q=Rudresh+Dwivedi%2C+Devam+Dave%2C+Het+Naik%2C+Smiti+Singhal%2C+Rana+Omer%2C+Pankesh+Patel%2C+Bin+Qian%2C+Zhenyu+Wen%2C+Tejal+Shah%2C+Graham+Morgan%2C+et+al%2E+Explainable+ai+%28xai%29%3A+Core+ideas%2C+techniques%2C+and+solutions%2E+ACM)
12. W Nelson Francis and Henry Kucera. Brown corpus manual. Letters to the Editor, 5(2):7, 1979. [find](https://scholar.google.com/scholar?q=W+Nelson+Francis+and+Henry+Kucera%2E+Brown+corpus+manual%2E+Letters+to+the+Editor%2C+5%282%29%3A7%2C+1979%2E)
13. Richard Futrell and Kyle Mahowald. How linguistics learned to stop worrying and love the language models. Behavioral and Brain Sciences, pp. 1–98, 2025. doi: 10.1017/ S0140525X2510112X. [find](https://scholar.google.com/scholar?q=Richard+Futrell+and+Kyle+Mahowald%2E+How+linguistics+learned+to+stop+worrying+and+love+the+language+models%2E+Behavioral+and+Brain+Sciences%2C+pp%2E+1%E2%80%9398%2C+2025%2E+doi%3A+10%2E1017%2F+S0140525X2510112X%2E)
14. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. [Neural language models as psycholinguistic subjects: Representations of syntactic state](https://md.research.elamur.ai/free/en/papers/futrell-2019-neural-language-models-as-psycholinguistic/). In Proceedings of NAACL-HLT 2019, pp. 32–42, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
15. E. Gibson and E. Fedorenko. The need for quantitative methods in syntax and semantics research. Language and Cognitive Processes, 2010. [find](https://scholar.google.com/scholar?q=E%2E+Gibson+and+E%2E+Fedorenko%2E+The+need+for+quantitative+methods+in+syntax+and+semantics+research%2E+Language+and+Cognitive+Processes%2C+2010%2E)
16. Jennifer Hu and Michael Frank. [Auxiliary task demands mask the capabilities of smaller language models](https://md.research.elamur.ai/free/en/papers/hu-2024-auxiliary-task-demands-mask-capabilities/). In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum?id=U5BUzSn4tD](https://openreview.net/forum?id=U5BUzSn4tD). [↗](https://openreview.net/forum?id=U5BUzSn4tD)
17. Jennifer Hu and Roger Levy. [Prompting is not a substitute for probability measurements in large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/). In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5040–5060, Singapore, December 2023. Assoc…
18. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. A Systematic Assessment of Syntactic Generalization in Neural Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1725–1744, Online, July 2020. Association for Computation… [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+A+Systematic+Assessment+of+Syntactic+Generalization+in+Neural+Language+Models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Asso)
19. Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. Language models align with human judgments on key grammatical constructions. Proceedings of the National Academy of Sciences, 121(36):e2400917121, 2024. [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Kyle+Mahowald%2C+Gary+Lupyan%2C+Anna+Ivanova%2C+and+Roger+Levy%2E+Language+models+align+with+human+judgments+on+key+grammatical+constructions%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+121)
20. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
21. Roni Katzir. Why Large Language Models Are Poor Theories of Human Linguistic Cognition: A Reply to Piantadosi. Biolinguistics, 17, 2023. URL https://doi.org/10.5964/bioling. 13153. — [doi:10.5964/bioling](https://doi.org/10.5964/bioling) [↗](https://doi.org/10.5964/bioling)
22. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking Cognitive Biases in Large Language Models as Evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 517–545, Bangko… [find](https://scholar.google.com/scholar?q=Ryan+Koo%2C+Minhwa+Lee%2C+Vipul+Raheja%2C+Jong+Inn+Park%2C+Zae+Myung+Kim%2C+and+Dongyeop+Kang%2E+Benchmarking+Cognitive+Biases+in+Large+Language+Models+as+Evaluators%2E+In+Lun%2DWei+Ku%2C+Andre+Martins%2C+and+Vivek+Sriku)
23. Evelina Leivada, Fritz Günther, and Vittoria Dentella. Reply to Hu et al.: Applying different evaluation standards to humans vs. Large Language Models overestimates AI performance. Proceedings of the National Academy of Sciences, 121(36):e2406752121, September 2024. doi: 10.1073/pnas.2406752121. URL… — [doi:10.1073/pnas.2406752121](https://doi.org/10.1073/pnas.2406752121) [find](https://scholar.google.com/scholar?q=Evelina+Leivada%2C+Fritz+G%C3%BCnther%2C+and+Vittoria+Dentella%2E+Reply+to+Hu+et+al%2E%3A+Applying+different+evaluation+standards+to+humans+vs%2E+Large+Language+Models+overestimates+AI+performance%2E+Proceedings+of+the+)
24. Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. Trustworthy ai: From principles to practices. ACM Computing Surveys, 55(9):1–46, 2023. [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Peng+Qi%2C+Bo+Liu%2C+Shuai+Di%2C+Jingen+Liu%2C+Jiquan+Pei%2C+Jinfeng+Yi%2C+and+Bowen+Zhou%2E+Trustworthy+ai%3A+From+principles+to+practices%2E+ACM+Computing+Surveys%2C+55%289%29%3A1%E2%80%9346%2C+2023%2E)
25. David A Lieberman. Behaviorism and the mind: A (limited) call for a return to introspection. American Psychologist, 34(4):319, 1979. [find](https://scholar.google.com/scholar?q=David+A+Lieberman%2E+Behaviorism+and+the+mind%3A+A+%28limited%29+call+for+a+return+to+introspection%2E+American+Psychologist%2C+34%284%29%3A319%2C+1979%2E)
26. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. [Assessing the ability of lstms to learn syntax-sensitive dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). Transactions of the Association for Computational Linguistics, 4:521–535, 2016.
27. Kyle Mahowald. A discerning several thousand judgments: GPT-3 rates the article + adjective + numeral + noun construction. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 265–273, Dubr… [find](https://scholar.google.com/scholar?q=Kyle+Mahowald%2E+A+discerning+several+thousand+judgments%3A+GPT%2D3+rates+the+article+%2B+adjective+%2B+numeral+%2B+noun+construction%2E+In+Andreas+Vlachos+and+Isabelle+Augenstein+%28eds%2E%29%2C+Proceedings+of+the+17th+Co)
28. Kyle Mahowald, Peter Graff, Jeremy Hartman, and Edward Gibson. Snap judgments: A small n acceptability paradigm (snap) for linguistic acceptability judgments. Language, 92 (3):619–635, 2016. [find](https://scholar.google.com/scholar?q=Kyle+Mahowald%2C+Peter+Graff%2C+Jeremy+Hartman%2C+and+Edward+Gibson%2E+Snap+judgments%3A+A+small+n+acceptability+paradigm+%28snap%29+for+linguistic+acceptability+judgments%2E+Language%2C+92+%283%29%3A619%E2%80%93635%2C+2016%2E)
29. Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1192–1202, 2018. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pp%2E+1192%E2%80%931202%2C+2018%2E)
30. Kanishka Misra. minicons: Enabling flexible behavioral and representational analyses of transformer language models. arXiv:2203.13112, 2022. URL https://arxiv.org/abs/ 2203.13112. [↗](https://arxiv.org/abs/)
31. Richard E. Nisbett and Timothy D. Wilson. Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84:231–259, 1977. doi: 10.1037/0033-295X.84.3.231. — [doi:10.1037/0033-295x.84.3.231](https://doi.org/10.1037/0033-295x.84.3.231) [find](https://scholar.google.com/scholar?q=Richard+E%2E+Nisbett+and+Timothy+D%2E+Wilson%2E+Telling+more+than+we+can+know%3A+Verbal+reports+on+mental+processes%2E+Psychological+Review%2C+84%3A231%E2%80%93259%2C+1977%2E+doi%3A+10%2E1037%2F0033%2D295X%2E84%2E3%2E231%2E+%E2%80%94+%5Bdoi%3A10%2E1037%2F003)
32. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024. [find](https://scholar.google.com/scholar?q=Team+OLMo%2C+Pete+Walsh%2C+Luca+Soldaini%2C+Dirk+Groeneveld%2C+Kyle+Lo%2C+Shane+Arora%2C+Akshita+Bhagia%2C+Yuling+Gu%2C+Shengyi+Huang%2C+Matt+Jordan%2C+et+al%2E+2+olmo+2+furious%2E+arXiv+preprint+arXiv%3A2501%2E00656%2C+2024%2E)
33. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL [https://openreview.net/forum?id=4NJBV6Wp0h](https://openreview.net/forum?id=4NJBV6Wp0h). [↗](https://openreview.net/forum?id=4NJBV6Wp0h)
34. Steven Piantadosi. Modern language models refute chomsky’s approach to language. Lingbuzz Preprint, lingbuzz, 7180, 2023. [find](https://scholar.google.com/scholar?q=Steven+Piantadosi%2E+Modern+language+models+refute+chomsky%E2%80%99s+approach+to+language%2E+Lingbuzz+Preprint%2C+lingbuzz%2C+7180%2C+2023%2E)
35. Eric Schwitzgebel. The unreliability of naive introspection. Philosophical Review, 117(2): 245–273, 2008. [find](https://scholar.google.com/scholar?q=Eric+Schwitzgebel%2E+The+unreliability+of+naive+introspection%2E+Philosophical+Review%2C+117%282%29%3A+245%E2%80%93273%2C+2008%2E)
36. Eric Schwitzgebel. Introspection. In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2024 edition, 2024. [find](https://scholar.google.com/scholar?q=Eric+Schwitzgebel%2E+Introspection%2E+In+Edward+N%2E+Zalta+and+Uri+Nodelman+%28eds%2E%29%2C+The+Stanford+Encyclopedia+of+Philosophy%2E+Metaphysics+Research+Lab%2C+Stanford+University%2C+Fall+2024+edition%2C+2024%2E)
37. BF Skinner. Representations and misrepresentations. Behavioral and Brain Sciences, 7(4): 655–667, 1984. [find](https://scholar.google.com/scholar?q=BF+Skinner%2E+Representations+and+misrepresentations%2E+Behavioral+and+Brain+Sciences%2C+7%284%29%3A+655%E2%80%93667%2C+1984%2E)
38. Jon Sprouse. A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43(1):155–167, March 2011. ISSN 1554-3528. doi: 10.3758/s13428-010-0039-7. URL [http://www.springerlink.com/index/](http://www.springerlink.com/index/) 10.3758/s13428-010-0039-7. — [doi:10.3758/s13428-010-0039-7](https://doi.org/10.3758/s13428-010-0039-7) [↗](http://www.springerlink.com/index/)
39. Jon Sprouse, Carson T Schütze, and Diogo Almeida. A comparison of informal and formal acceptability judgments using a random sample from linguistic inquiry 2001–2010. Lingua, 134:219–248, 2013. [find](https://scholar.google.com/scholar?q=Jon+Sprouse%2C+Carson+T+Sch%C3%BCtze%2C+and+Diogo+Almeida%2E+A+comparison+of+informal+and+formal+acceptability+judgments+using+a+random+sample+from+linguistic+inquiry+2001%E2%80%932010%2E+Lingua%2C+134%3A219%E2%80%93248%2C+2013%2E)
40. Leonard Talmy. Introspection as a methodology in linguistics. In Ten lectures on cognitive semantics, pp. 218–262. Brill, 2018. [find](https://scholar.google.com/scholar?q=Leonard+Talmy%2E+Introspection+as+a+methodology+in+linguistics%2E+In+Ten+lectures+on+cognitive+semantics%2C+pp%2E+218%E2%80%93262%2E+Brill%2C+2018%2E)
41. Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL [https://qwenlm.github.io/blog/qwen2.5/](https://qwenlm.github.io/blog/qwen2.5/). [↗](https://qwenlm.github.io/blog/qwen2.5/)
42. Tristan Thrush, Jared Moore, Miguel Monares, Christopher Potts, and Douwe Kiela. I am a Strange Dataset: Metalinguistic Tests for Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume… [find](https://scholar.google.com/scholar?q=Tristan+Thrush%2C+Jared+Moore%2C+Miguel+Monares%2C+Christopher+Potts%2C+and+Douwe+Kiela%2E+I+am+a+Strange+Dataset%3A+Metalinguistic+Tests+for+Language+Models%2E+In+Lun%2DWei+Ku%2C+Andre+Martins%2C+and+Vivek+Srikumar+%28eds)
43. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. “my answer is C”: First-token probabilities do not match text answers in instruction-tuned language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Ass… — [``My Answer is C'': First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-my-answer-is-c-first/)
44. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8, 2020. URL https://doi.org/ 10.1162/tacl_a_00321. Publisher: MIT P…
45. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Alex Warstadt, Aaron Mue… [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Aaron+Mueller%2C+Leshem+Choshen%2C+Ethan+Wilcox%2C+Chengxu+Zhuang%2C+Juan+Ciro%2C+Rafael+Mosquera%2C+Bhargavi+Paranjabe%2C+Adina+Williams%2C+Tal+Linzen%2C+and+Ryan+Cotterell%2E+Findings+of+the+BabyLM+chall)
46. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. What do rnn language models learn about filler–gap dependencies? In Proceedings of the 2018 [EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp](https://md.research.elamur.ai/free/en/papers/wilcox-2018-rnn-language-models-learn-about/). 211–221, 2018.
47. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
48. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai D… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)

*Updated 30 October 2026*